Skip to content
← Publications
KCC 2025

Dynamic Replica Replacement Technique Based on Deep Reinforcement Learning for Cost-Effective Cluster Utilization in Heterogeneous Kubernetes Environment

Seoeun Kwon, Minjae Kang, Donggyun Kim, Heonchang Yu

Korea Computer Congress (KCC 2025)

Summary

The study redistributes a fixed replica total across nodes with different capabilities and prices. DQN uses response latency, CPU utilization, and node cost to adjust placement.

Role
Third authorundergraduate research mentor
Outcome
Comparable mean and P95 latency to static placement on three workers, with about 20% lower cost in the active-time model.
DQN-based scheduling system architecture across a heterogeneous Kubernetes cluster
The DQN-based scheduling system

Different Performance and Cost with the Same Number of Replicas

With different node capabilities and hourly prices, the same replica total can produce different latency and cost depending on placement. The study considered response latency and node cost alongside CPU utilization.

Dynamic replica replacement keeps the total fixed while changing the distribution across nodes in real time. The goal was to reduce cost while maintaining performance.

DQN Placement with a Fixed Replica Total

A Deep Q-Network (DQN) agent observes the cluster and chooses the next placement. Its state contains CPU utilization, mean response latency, and replica count for each worker. These three measurements across three workers form a 9-dimensional state.

An action is a vector that changes each node’s replica count by −1, 0, or +1. Its sum is constrained to zero, so replicas removed from one worker are added elsewhere. The learning problem is where to place the same number of replicas, rather than increasing the total count with load.

After a placement change, new latency, CPU utilization, and replica counts form the next state and reward. The reward combines mean HTTP latency, CPU usage, and a cost term based on replica counts multiplied by node rates. A negative sign on their weighted sum gives lower rewards to placements with higher latency or resource burden.

DQN selects among these discrete placement actions. Training used the MLP policy in stable-baselines3, with ε-greedy exploration, a target network, and experience replay.

ε-greedy combines choosing the currently preferred action with exploring alternatives. Experience replay reuses earlier state, action, reward, and next-state transitions, while a target network reduces the instability of continuously changing learning targets. Training can therefore reuse past experience rather than only follow consecutive observations.

Dynamic and static placement with a fixed replica count

The Tencent Cloud cluster consisted of one master and three workers with different clock speeds and hourly rates of $0.15, $0.31, and $0.12. Vegeta generated load at 15-second intervals for a CPU-intensive PHP matrix multiplication service, with metrics collected through Prometheus and Istio. The static baseline kept one replica on each worker. The proposed method also retained three replicas in total, isolating the effect of redistribution from an increase in total capacity.

The cost model sums each worker’s active time multiplied by its hourly rate. Mean and P95 latency are compared alongside cost to examine whether lower-cost placement also preserves performance for slower requests.

Latency Comparable to Static Placement with About 20% Lower Modeled Cost

DQN-based redistribution maintained mean and P95 latency similar to the static placement while reducing modeled cost by about 20%. Changing the distribution across nodes altered the performance–cost balance without changing the total replica count.

The paper’s cost tables show a much shorter modeled active time for worker 3, with placement concentrated on workers 1 and 2. Worker 3 had the lowest hourly rate but was used less, distinguishing the result from placing everything on the cheapest node. The redistribution reflects processing performance together with cost.

Redistributing the fixed replica total maintained performance and reduced modeled cost in this comparison. I participated as a co-author and undergraduate research mentor.