Dynamic Replica Replacement Technique Based on Deep Reinforcement Learning for Cost-Effective Cluster Utilization in Heterogeneous Kubernetes Environment
Korea Computer Congress (KCC 2025)
Rather than adding replicas, this study redistributes a fixed total across nodes with different performance and prices. A DQN policy weighs both latency and cost.
- Role
- Co-author · undergraduate research mentor
- Outcome
- Comparable mean and P95 latency to static placement on three workers, with about 20% lower cost in the active-time model.
Background
Nodes with different performance and hourly prices can produce different latency and cost for the same replica total. This study considered response latency and node cost alongside CPU utilization when adjusting placement.
This paper keeps the total replica count fixed and re-places only the distribution across nodes in real time, aiming to cut cost while holding performance.
Approach
I helped design a Deep Q-Network (DQN) reinforcement-learning agent. The state is each worker’s CPU utilization, average response latency, and current replica count (9-dimensional for three nodes); the action is a discrete choice of −1 / 0 / +1 replicas per node while keeping the total fixed. The reward jointly weighs average HTTP response time, per-node CPU usage, and the replicas placed on each node times that VM’s hourly cost — so the agent learns the balance between performance and cost.
DQN selects among discrete −1, 0, and +1 replica adjustments while preserving the total. Training used the MLP policy in stable-baselines3, with ε-greedy exploration, a target network, and experience replay.
Results
On Tencent Cloud, a heterogeneous cluster of one master and three differently-clocked workers ($0.15 / $0.31 / $0.12 per hour) ran a CPU-intensive PHP matrix-multiply service, load-tested with vegeta every 15s and measured via Prometheus and Istio. The baseline was a static placement of one replica per worker (default scheduling).
DQN redistribution maintained mean and P95 latency comparable to static placement while reducing modeled cost by about 20%. Cost was calculated from each worker’s active time and hourly price.
In this comparison, redistributing a fixed replica total reduced modeled cost without adding capacity or materially increasing latency. I participated as a co-author and undergraduate research mentor.