LARE-HPA: Co-optimizing Latency and Resource Efficiency for Horizontal Pod Autoscaling in Kubernetes

Fixed settings could not follow changing demand

CPU-based HPA calculates desired replicas from average CPU utilization relative to the configured target. Downscale stabilization considers recommendations over a time window to avoid shrinking on a brief dip. This study examines the latency and resource trade-offs of fixing the CPU target and retention period.

A lower CPU target triggers earlier expansion but uses more resources: the tested low-threshold configuration used about 22% more CPU. A higher target retains less capacity, but load can outgrow it between measurements. A long cool-down retains excess pods; a short one can lead to repeated contraction and expansion.

Steady growth and short bursts call for different settings. A fixed threshold and retention period cannot adapt to both, and a new application has little history to profile. I designed the controller to adjust both values using requests observed during execution.

Forecasting without a training dataset

New applications need forecasts before a large training dataset is available. Rather than an offline predictor such as Bi-LSTM, I considered a model that updates as requests arrive.

I used online ARIMA in a single controller Pod. The controller reads system and Istio request metrics from Prometheus, then adjusts the CPU target and downscale retention period while keeping the replica-count formula unchanged.

Updating forecasts as requests arrive

Online ARIMA must handle unobserved noise terms. Following prior Online ARIMA work, the implementation approximates the moving-average component with a higher-order autoregressive model, allowing forecasts to be updated from observed request history.

I adopted Online ARIMA’s Online Newton-Step update, incorporating each new observation’s prediction error without retraining over the full history. The resulting forecast feeds the CPU-target and retention adjustments.

Mapping volatility to a CPU target

Request volumes differ by service, so raw counts are a poor shared scaling criterion. The controller measures volatility relative to each application’s history and maps it to a CPU target. The experiment used a target range of 50–95%.

Volatility combines recent average request changes with forecast error, so prediction misses also affect scaling decisions. A z-score compares this value with the application’s historical variation.

An inverse sigmoid maps higher volatility to a lower CPU target. The controller expands earlier during rapid changes and uses existing pods more fully when demand is stable. The bounded mapping also prevents extreme inputs from pushing the target outside its configured range.

Adjusting retention from request trends

A linear regression over recent requests determines the trend. Rising demand lengthens the cool-down to avoid releasing pods during a brief dip. Falling demand shortens it so unused resources are released sooner.

The regression slope alone does not change retention. A Durbin–Watson residual check must fall within the study’s acceptance range before the counter moves one step up or down with the trend. Otherwise trend-based adjustment is withheld.

Expanding promptly and checking before contraction

Restoring released capacity requires waiting for containers to start and for new measurements. Scale-up is therefore requested immediately, while scale-down requires both an expired counter and a trend that is not rising.

The controller updates replica counts through the Kubernetes API without changing application code. It requires metrics from Prometheus, Istio, and kube-state-metrics; threshold ranges and observation windows are configurable.

Results

Using the NASA-HTTP request trace and a PHP trigonometric-computation service, the study compared HPA, offline Bi-LSTM, and Online Bayesian Optimization (Microscaler) under a 1-second SLO. Mean latency fell from 562.8ms with HPA to 279.5ms, a 50.34% reduction; reductions against Bi-LSTM and Online-BO were 39.52% and 46.18%. SLO attainment rose from 93.86% to 97.24%, an increase of 3.38 percentage points.

LARE-HPA used about 9.5% more resources than HPA, compared with roughly twice HPA’s resource use for Online-BO. Bi-LSTM performed worse on bursts outside its training patterns, while Online-BO’s later scaling trigger missed sudden increases. I led this work as first author. It received a Distinguished Paper award (Top 6) at ICSOC 2024, a SCIE-compatible top-tier venue. A patent was filed, the software copyright registered, and the code released as open source.