Skip to content
← Publications
IC 2024 2024

Proactive Auto Scaling Based on Marginal Request Change Analysis for Reducing Tail Latency in Kubernetes Cluster

Donggyun Kim, Heonchang Yu

International Conference on Innovative Computing

Summary

Requests may already be queued when CPU reaches the Threshold. p-HPA uses request growth and current CPU utilization to calculate additional replicas and respond to bursts earlier.

Role
First authorproactive autoscaling research
Outcome
Lower latency across linear, sinusoidal, and exponential workloads at 70%, 80%, and 90% thresholds, with about 13% more resources on average.
p-HPA framework architecture: p-Analyzer and p-Caller across master and worker nodes
The p-HPA framework architecture (p-Analyzer · p-Caller)

Request Bursts That Can Outpace CPU-Based Scaling

CPU-based HPA adjusts replica count by comparing current utilization with a configured Threshold. A lower Threshold triggers earlier scale-up but uses more resources; a higher one waits for more load to accumulate.

The experimental environment collected CPU metrics every 15 seconds. A burst between measurements could build up load before scaling began. The study therefore considered request growth alongside CPU utilization to prepare replicas earlier.

LSTM and Bi-LSTM require a dataset and a training phase. p-HPA skips that preparation by focusing on sharp increases in incoming requests during operation rather than predicting long-term patterns.

Request Growth Ahead of the CPU Threshold

p-HPA monitors how quickly request volume is growing. Request changes provide an early signal for bursts that current CPU utilization alone may miss, allowing replicas to be added before CPU reaches the Threshold.

Istio collects per-service request counts for Prometheus. The p-Analyzer uses them to calculate additional replicas, and the p-Caller requests scale-up through the Kubernetes API. Both modules run alongside HPA.

Request Changes Normalized with arctan

Request increases can become very large, making raw deltas an unstable comparison scale. arctan maps them into a bounded angle. Faster growth produces a larger angle without an unbounded value, allowing growth rates to be compared within the same range.

The p-Analyzer divides the difference in request counts over a 60-second interval by time, applies arctan, and converts it to −90 to 90 degrees. Positive values indicate growth and negative values decline; steeper growth approaches 90 degrees. This normalized change determines the observation window instead of raw request counts.

Shorter Observation Windows During Request Bursts

The controller also adjusts how many recent intervals to examine. A long window can react too slowly to a burst, while a short one can overreact to a brief fluctuation.

Starting with the latest interval, the algorithm moves backward and accumulates normalized changes. When the total reaches the paper’s reference value of 90, it divides by the number of selected intervals to obtain the average growth rate. A sharp increase reaches the reference in fewer intervals, while gradual growth draws on a longer history.

The resulting observation window reduces the dilution of bursts by a fixed-length average. It also passes an average over the selected intervals to the replica calculation rather than deciding expansion from a single request count.

Additional Replicas from Request Growth and CPU Utilization

The same request growth rate can require different replica counts depending on current CPU utilization. The method uses average growth and CPU utilization together to reflect both the trend and the current load.

Current CPU utilization and average growth indicate the remaining headroom before utilization reaches 100%. The paper evaluates this within a ten-minute horizon, twice the default stabilization period. At the same growth rate, an already higher CPU load needs additional replicas sooner, motivating the use of both request trends and current load.

p-HPA adds replicas early only when it detects a burst. Otherwise, HPA continues managing the replica count, retaining the existing controller alongside proactive expansion.

Three workload patterns across three Threshold settings

The evaluation ran a CPU-intensive PHP matrix-product service on an Intel Xeon Cascade Lake cluster with 4 vCPUs and 16GB of memory. Linear, sinusoidal, and exponential workloads ranged from 0 to 2,000 requests per minute. p-HPA was compared with default HPA at CPU Thresholds of 70%, 80%, and 90%.

Linear load tests sustained growth, sinusoidal load repeated increases and decreases, and exponential load accelerating growth. Separate 70%, 80%, and 90% Threshold settings test whether the benefit of proactive expansion changes with the baseline HPA configuration.

The evaluation compares P95 and tail latency alongside the average resources needed for earlier replica creation. Matching the workload pattern and Threshold between methods allows latency gains and increased allocation to be interpreted together.

Lower Latency with About 13% More Resources

Tail latency and P95 decreased across all tested workload and Threshold combinations. Under exponential load at the 90% Threshold, tail latency fell from 1,106.6ms with HPA to 879.8ms with p-HPA. The table compares measurements from the supplied manuscript at that fixed Threshold.

Average resource use was about 13% higher than HPA because replicas were prepared earlier. Evaluation covered synthetic linear, sinusoidal, and exponential CPU-bound workloads. Services with other bottlenecks would require further testing.

I conducted this research as first author. The proposed follow-ups, ARIMA-based time-series forecasting and automatic Threshold adjustment, later became part of LARE-HPA’s Online ARIMA and adaptive Threshold design.

Tail latency in ms from Table 1 of the supplied manuscript, with the CPU Threshold fixed at 90%.
WorkloadHPA (ms)p-HPA (ms)
Linear2,504.22,332.6
Sinusoidal2,497.02,360.8
Exponential1,106.6879.8