Proactive Auto Scaling Based on Marginal Request Change Analysis for Reducing Tail Latency in Kubernetes Cluster

Bursts could outpace CPU-based scaling

CPU-based HPA adjusts replica count by comparing current utilization with a configured target. A lower target expands capacity earlier but uses more resources; a higher target waits for more load to accumulate.

The experimental environment collected CPU metrics every 15 seconds. A request burst between measurements could build load before scaling began. The study therefore considered request growth alongside CPU utilization to prepare replicas earlier.

Detecting request growth before CPU overload

p-HPA monitors how quickly request volume is growing. Request changes provide an additional signal for bursts that current CPU utilization alone may miss, allowing replicas to be added before CPU reaches its target.

Istio collects per-service request counts for Prometheus. The p-Analyzer uses them to calculate additional replicas, and the p-Caller requests expansion through the Kubernetes API. Both modules run alongside HPA.

Normalizing request changes

Raw request deltas can make decisions sensitive to large values. An arctangent maps each increase into a bounded angle, providing a measure of how steeply demand is rising.

Shortening the observation window during bursts

A long observation window reacts slowly to bursts, while a short one is sensitive to brief fluctuations. The window expands backward from recent requests until cumulative change reaches the criterion. It therefore uses fewer intervals during a burst and more during gradual change.

Combining request growth with current CPU load

The same growth rate can require different replica counts at different CPU utilization levels. Multiplying the average slope by current CPU utilization accounts for both growth and existing load.

p-HPA adds replicas early only when it detects a burst. Otherwise, HPA continues managing the replica count, retaining the existing controller alongside proactive expansion.

Responding to bursts without prior training

LSTM and Bi-LSTM forecasting requires collecting data and training a model. This study instead focused on detecting short bursts from live requests without prior training. Request growth and current CPU utilization determine the additional replica count. Latency improved, at the cost of about 13% more resources on average.

Latency and resource use

On a Kubernetes cluster (Intel Xeon Cascade Lake, 4 vCPUs, 16 GB) running a CPU-intensive PHP matrix-product app, linear, sinusoidal, and exponential patterns of 0–2,000 requests per minute were driven and compared against the default HPA at thresholds 70, 80, and 90.

Tail and P95 latency decreased across the tested workload–threshold combinations. Under exponential load at the 90% target, tail latency fell from 1,106.6ms to 879.8ms. The table compares measurements from the supplied manuscript at that fixed target.

Average resource use was about 13% higher because replicas were prepared earlier. Evaluation covered synthetic linear, sinusoidal, and exponential CPU-bound workloads; services with other bottlenecks would require further testing.

I led this work as first author. The follow-ups I flagged — time-series (ARIMA) forecasting and automatically determining the optimal threshold — became the online-ARIMA, adaptive-threshold design of LARE-HPA.