LARE-HPA: Co-optimizing Latency and Resource Efficiency for Horizontal Pod Autoscaling in Kubernetes
22nd International Conference on Service-Oriented Computing
- Distinguished Paper (Top 6)
- SCIE-compatible top-tier
Summary
A lower Threshold reduces latency but uses more resources. LARE-HPA adjusts the Threshold from workload volatility and Cool Down Time from request trends to balance latency and resource efficiency.
- Role
- First authorLARE-HPA research
- Outcome
- 50.34% lower mean latency on NASA-HTTP; SLO attainment rose from 93.86% to 97.24%, using about 9.5% more resources than HPA.
Fixed Threshold and Cool Down Time Under Changing Load
CPU-based HPA calculates the desired replica count from average CPU utilization relative to the configured Threshold. During scale-in, stabilization considers recommendations over a time window to avoid releasing replicas after a brief dip. This study examines the latency and resource trade-offs of a fixed Threshold and Cool Down Time.
A lower Threshold triggers earlier scale-out and reduces latency, but increases CPU allocation. In the paper’s fixed-Threshold comparison, mean allocation was 1,280.0 millicores at 60% and 1,046.15 millicores at 90%, a difference of about 22%. Cool Down Time creates another trade-off: a long interval retains unnecessary replicas, while a short one can cause flapping as removed replicas are needed again.
Steady growth and short bursts need different settings. New applications also have little history for choosing suitable values in advance. I designed the controller to adjust both the Threshold and Cool Down Time using requests observed during execution.
Two control paths built from observed metrics
System metrics from kube-state-metrics and request metrics from Istio are collected in Prometheus. Mean CPU utilization determines the replica count needed now, while request history adjusts the Threshold and Cool Down Time used in subsequent decisions.
The Forecaster and Threshold Coordinator derive the CPU Threshold from forecasts and recent request changes. The CDT Decider uses the regression trend and residuals of observed requests to determine the waiting count. The AutoScaler combines both outputs to decide expansion and reduction.
The controller runs as one pod separate from the application, with its components processed in parallel. It updates Deployment replicas through the Kubernetes API without modifying the target service’s code.
Online ARIMA updated from observations without pretraining
New applications need to respond to load changes before a large training dataset is available. Rather than an offline predictor such as Bi-LSTM, I considered a model that updates as requests arrive.
ARIMA combines autoregressive terms from past observations with moving-average terms from past errors, using differencing to handle changes in the series level. The online approach approximates error terms that cannot be observed directly with a longer autoregressive model, allowing coefficients to be updated from observations arriving during execution.
Online Newton-Step updates the model coefficients. The model predicts request volume, adjusts its coefficients using the prediction loss when an observation arrives, and then forecasts the next interval. Its output feeds the Threshold Coordinator, allowing history collected during service operation to be used without completing an offline training phase.
Lower CPU Thresholds under high volatility
The same increase has different implications for services with different traffic volumes. The Threshold Coordinator combines recent absolute request changes with the absolute difference between the next-interval forecast and the current observation. The experiment used the 10 most recent intervals, keeping observed changes in the decision even when the forecast misses.
The z-score subtracts the historical mean from the recent volatility value and divides by the historical standard deviation. Positive values indicate more variation than usual; negative values indicate greater stability. Services with different traffic volumes can therefore be assessed relative to their own histories rather than by the same absolute increase.
High volatility lowers the Threshold to trigger earlier scale-out, while stable demand raises it to make fuller use of existing replicas. An inverse sigmoid maps the z-score between the minimum and maximum Thresholds, keeping even extreme inputs within the configured range.
Within the experimental 50–95% range, a z-score of zero gives a calculated midpoint of 72.5%. Increasing positive values move toward 50%, while increasingly negative values move toward 95%. Services with different request volumes therefore use the same adjustment rule relative to their own histories.
Cool Down Time adjusted to request trends
A linear regression over recent requests indicates whether demand is rising or falling. Rising demand lengthens Cool Down Time to avoid releasing replicas during a brief dip. Falling demand shortens it so unused resources can be returned sooner.
Trend analysis uses the 60 most recent observations. The regression slope over time is classified as rising, falling, or unchanged and represented by +1, −1, or 0. This direction changes the CDT Counter one step at a time rather than adjusting the waiting period in proportion to the slope magnitude.
The controller checks residuals as well as the fitted line before changing Cool Down Time. Residuals are the differences between observations and the fitted line. The Durbin–Watson statistic ranges from 0 to 4, with values closer to 2 indicating weaker first-order residual autocorrelation.
The configured acceptance interval of 1.616–2.384 corresponds to the 5% critical-value table for 60 observations and one predictor, time, excluding the intercept. The table gives dU = 1.616; checking negative autocorrelation as well gives the opposite boundary, 4−dU = 2.384.
The CDT Counter changes by one step with the request trend only when the statistic falls within this interval. It increases for rising demand and decreases for falling demand; adjustments are withheld outside the interval.
Immediate scale-out, conditional scale-in
The desired replica count is the ceiling of the current count multiplied by the ratio of mean CPU utilization to the adjusted Threshold, then bounded by the configured minimum and maximum. Forecast requests do not directly determine the count; the forecast-adjusted Threshold is applied to the current CPU metric.
The CDT Counter counts metric collection periods rather than seconds. CDTdesired is the target waiting count, and CDTcurrent is the remaining count. Each cycle decrements the remaining count; an executed scale-in resets it to the target so the next reduction must wait again.
When the desired replica count is higher than the current count, the controller requests scale-out without waiting for Cool Down Time. A scale-in requires the remaining CDT Counter to be zero and the trend direction returned by the CDT Decider not to be rising (+1). During intervals classified as rising, replicas are retained through a brief CPU dip to avoid releasing capacity that may soon be needed again.
-
Map request volatility to a Threshold
Combine load forecasts with recent request changes. Compare volatility with its history and map the relative value into the configured CPU Threshold range.
-
Adjust Cool Down Time from the trend
Check the request trend and residual autocorrelation. When the acceptance condition is met, increase the CDT Counter for rising demand and decrease it for falling demand.
-
Calculate the desired replica count
Round up current replicas × average CPU utilization ÷ adjusted Threshold, then limit the result to the configured minimum and maximum.
-
Check scale-out and scale-in conditions
Request scale-out immediately. Execute scale-in only when the CDT Counter has expired and demand is not rising, then reset the counter.
NASA-HTTP evaluation against three baselines
A PHP trigonometric service ran on Intel Core i9-10900K hardware with 10 CPUs and 32GiB of memory. Requests from outside the cluster replayed the real NASA-HTTP pattern to compare responses during volatile periods. The response-time SLO was one second.
The baselines were default HPA, pretrained Bi-LSTM, and Microscaler’s online Bayesian optimization (Online-BO). Bi-LSTM was trained on 50% of the workload, with a per-pod request handling limit of 50. LARE-HPA used 10 Threshold observation intervals, 60 CDT observations, and a 50–95% Threshold range.
The evaluation measured mean and percentile latency, SLO attainment, and average CPU allocation together. Since preparing replicas earlier can reduce latency, the same comparison examined the performance gain alongside additional resource allocation.
50.34% Lower Mean Latency with SLO Compliance Reaching 97.24%
Mean latency fell by 50.34%, from 562.8ms with HPA to 279.5ms, and was 39.52% and 46.18% lower than Bi-LSTM and Online-BO respectively. SLO attainment rose from 93.86% to 97.24%, an increase of 3.38 percentage points. Latency and SLO were compared on the same request history.
The response distribution improved as well as the mean. The 50th percentile fell from 528.6ms to 255.5ms and the 95th from 1,189.0ms to 600.1ms. The difference appeared in both the central and slower portions of the distribution.
| Method | Mean latency (ms) | SLO attainment | Resources (millicores) |
|---|---|---|---|
| HPA | 562.8 | 93.86% | 582.76 |
| LARE-HPA | 279.5 | 97.24% | 638.13 |
| Bi-LSTM | 462.1 | 93.06% | 875.85 |
| Online-BO | 519.3 | 95.15% | 1,274.75 |
Resource trade-offs and research outcomes
Mean CPU allocation rose by about 9.5%, from 582.76 millicores with default HPA to 638.13 with LARE-HPA. Bi-LSTM allocated 875.85 millicores and Online-BO 1,274.75 in the same experiment. LARE-HPA reduced latency with more resources than default HPA, while allocating less than the two prediction and optimization baselines.
The paper attributes Bi-LSTM’s weaker response to bursts to the limits of its learned patterns, and Online-BO’s over-provisioning and delayed response to its scaling trigger. LARE-HPA instead updates forecasts without pretraining and combines current CPU metrics with request variation to adjust both expansion and resource release.
I conducted this research as first author. It received a Distinguished Paper award (Top 6) at ICSOC 2024, an SCIE-compatible top-tier venue. A patent was filed, the software copyright registered, and the code released as open source.
Patent filed (10-2024-0128875) and software copyright registered (C-2024-043831).