Dynamic Replica Replacement Technique Based on Deep Reinforcement Learning for Cost-Effective Cluster Utilization in Heterogeneous Kubernetes Environment
Korea Computer Congress (KCC 2025)
Summary
The study redistributes a fixed replica total across nodes with different capabilities and prices. DQN uses response latency, CPU utilization, and node cost to adjust placement.
- Role
- Third authorundergraduate research mentor
- Outcome
- Comparable mean and P95 latency to static placement on three workers, with about 20% lower cost in the active-time model.
Different Performance and Cost with the Same Number of Replicas
With different node capabilities and hourly prices, the same replica total can produce different latency and cost depending on placement. The study considered response latency and node cost alongside CPU utilization.
Dynamic replica replacement keeps the total fixed while changing the distribution across nodes in real time. The goal was to reduce cost while maintaining performance.
DQN Placement with a Fixed Replica Total
A Deep Q-Network (DQN) agent observes the cluster and chooses the next placement. Its state contains CPU utilization, mean response latency, and replica count for each worker. These three measurements across three workers form a 9-dimensional state.
An action is a vector that changes each node’s replica count by −1, 0, or +1. Its sum is constrained to zero, so replicas removed from one worker are added elsewhere. The learning problem is where to place the same number of replicas, rather than increasing the total count with load.
After a placement change, new latency, CPU utilization, and replica counts form the next state and reward. The reward combines mean HTTP latency, CPU usage, and a cost term based on replica counts multiplied by node rates. A negative sign on their weighted sum gives lower rewards to placements with higher latency or resource burden.
DQN selects among these discrete placement actions. Training used the MLP policy in stable-baselines3, with ε-greedy exploration, a target network, and experience replay.
ε-greedy combines choosing the currently preferred action with exploring alternatives. Experience replay reuses earlier state, action, reward, and next-state transitions, while a target network reduces the instability of continuously changing learning targets. Training can therefore reuse past experience rather than only follow consecutive observations.
Dynamic and static placement with a fixed replica count
The Tencent Cloud cluster consisted of one master and three workers with different clock speeds and hourly rates of $0.15, $0.31, and $0.12. Vegeta generated load at 15-second intervals for a CPU-intensive PHP matrix multiplication service, with metrics collected through Prometheus and Istio. The static baseline kept one replica on each worker. The proposed method also retained three replicas in total, isolating the effect of redistribution from an increase in total capacity.
The cost model sums each worker’s active time multiplied by its hourly rate. Mean and P95 latency are compared alongside cost to examine whether lower-cost placement also preserves performance for slower requests.
Latency Comparable to Static Placement with About 20% Lower Modeled Cost
DQN-based redistribution maintained mean and P95 latency similar to the static placement while reducing modeled cost by about 20%. Changing the distribution across nodes altered the performance–cost balance without changing the total replica count.
The paper’s cost tables show a much shorter modeled active time for worker 3, with placement concentrated on workers 1 and 2. Worker 3 had the lowest hourly rate but was used less, distinguishing the result from placing everything on the cheapest node. The redistribution reflects processing performance together with cost.
Redistributing the fixed replica total maintained performance and reduced modeled cost in this comparison. I participated as a co-author and undergraduate research mentor.