Skip to content
← Back to Work
Kubernetes · Reliability

LARE-HPA

  • Autoscaling
  • Forecasting
  • Self-tuning

Summary

A Kubernetes autoscaler that retunes its threshold and cool-down time on its own as the workload changes. It runs online with no pre-training, co-optimizing the trade-off between latency and resource efficiency. It lowers the threshold as request volatility grows and adjusts the cool-down time to the request trend. It can scale any application independently.

Role
First AuthorDeveloper
Outcome
Mean latency −50.34% vs HPAICSOC Distinguished
LARE-HPA architecture
LARE-HPA Architecture

Before we start

"What percentage should the scaling threshold be?" In autoscaling, there is no single right threshold, so operators have relied on their experience to set it. I started by asking whether the system could adapt to changes in load and adjust the threshold automatically.

With a static threshold, response latency and resource use vary with load. I implemented a controller that adjusts the scaling threshold based on request volatility and the cool-down time before scaling in based on request trends.

The derivation and experiments are in the LARE-HPA paper under Publications.

Fixed settings could not adapt to demand

A lower HPA threshold scales out earlier but also uses more resources. In the paper’s experiment, CPU allocation at a 60% threshold was about 22% higher than at 90%. A higher threshold can delay scaling and increase response latency. Balancing response latency and resource use therefore required adjusting the threshold as load changed.

A schematic comparison of 60% and 90% CPU thresholds; it does not show experimental measurements. The left trace holds replicas constant to illustrate an increasing load crossing the 60% threshold before the 90% threshold. The right side takes the same load from incoming requests (3.6 CPU cores needed, with each Pod requesting 1 core), branches it into the two CPU thresholds, and applies the HPA replica formula. A 60% threshold creates 6 Pods, each used up to 60%, leaving 2.4 cores spare; a 90% threshold creates 4 Pods, leaving 0.4 cores spare.
Resource Usage by Threshold

If requests increase again immediately after Pods are scaled in, flapping occurs as scale-in and scale-out repeat. Simply lengthening the cool-down time to address this leaves more Pods than needed after load decreases. I therefore needed to adjust the cool-down time based on request trends, in addition to adjusting the threshold.

AI models can forecast load to help the system respond, but new applications lacked the data needed for offline training. Retraining the model whenever request patterns changed also added work, so I chose to keep updating the model with data collected during execution.

LARE-HPA forecasts load online, adjusts the threshold based on request volatility, and adjusts the cool-down time based on request trends. It uses data collected during execution rather than a separately prepared training dataset.

One Pod, three core processes

As load changed, the controller needed to keep updating the threshold and cool-down time and use those results to adjust the actual number of Pods. I used multiprocessing to run three independent processes in parallel within a single Pod. The processes handle load forecasting and threshold updates, request-trend analysis and cool-down adjustment, and actual scaling, respectively. They share the threshold and cool-down counters through shared memory, with locks around reads and updates to prevent concurrent changes to the same value. The scaling process calculates the required Pod count from current CPU utilization and the updated threshold, then checks the cool-down conditions before applying scale-in through the Kubernetes API.

The Metrics Collector queries system metrics and Istio Envoy request metrics stored in Prometheus. The Forecaster and Threshold Coordinator run in the same process to forecast load and update the threshold, while the CDT Decider runs in a separate process to adjust the cool-down time based on request trends. The AutoScaler uses these results to adjust Deployment replicas through the Kubernetes API.

Updating forecasts, thresholds, and cool-down times

The Forecaster uses online-learning ARIMA to predict load for the next interval. While conventional ARIMA requires offline training with sufficient data, online-learning ARIMA updates its autoregressive coefficients with Online Newton-Step as request data arrives, allowing it to adapt to changes in request patterns without offline retraining. Forecasts use request metric values from the 10 most recent time points, and negative predicted request values are set to zero.

The Threshold Coordinator adjusts the threshold using forecast load and recent request metrics. Request counts have no fixed upper limit, while the threshold must stay within a percentage range, so changes in request counts needed to be normalized. I calculated volatility as a z-score by comparing request changes over the last 10 intervals with the full history, then mapped it to a threshold between 50% and 95% using an inverse sigmoid function. When volatility is low, a higher threshold reduces unnecessary Pod scale-out; when volatility is high, a lower threshold enables an earlier response to rising demand.

The CDT Decider adjusts the cool-down time based on request trends. It fits a linear regression to request metric values from the 60 most recent time points to identify whether requests are increasing or decreasing, and checks residual autocorrelation with the Durbin-Watson statistic. The trend affects the cool-down time only when the statistic falls within the configured range of 1.616–2.384. During an upward trend, it increases the cool-down counter up to 60 intervals to avoid prematurely reducing Pods in response to a temporary drop in requests. During a downward trend, it shortens the counter to a minimum of one interval to avoid retaining unnecessary Pods for too long.

Deployment without application code changes

LARE-HPA is built on the python:3.11-slim image and deployed as a container. To adjust the replica count through the Kubernetes API, I used a dedicated ServiceAccount for in-cluster authentication and granted the ClusterRole only the permissions needed for scaling and checking state. I configured patch, get, list, and watch permissions on deployments/scale and statefulsets/scale, and read permissions on pods and services.

Request history from Prometheus feeds the Forecaster and Threshold Coordinator to adjust the CPU threshold, and the CDT Decider to adjust the cool-down time. Parallel processes in one Pod share values through multiprocessing.Value. The AutoScaler combines these with mean CPU utilization, patches Deployment replicas through the Kubernetes API, and evaluates metrics again in the next cycle. The Pod layout below illustrates a replica-count change; it is not an experimental measurement.
LARE-HPA Online Self-Tuning Loop

I made the scaling target configurable without changes to application code so the controller could be applied independently rather than tied to a particular application. A ConfigMap defines the application to be scaled, Prometheus address, threshold (75% by default), replica range (1–16), and metric collection interval (60 seconds), and a script deploys the controller. Separate logs record forecast load, threshold, cool-down time, and errors so the metrics and settings used to adjust the Pod count can be checked.

Results

I replayed NASA-HTTP request patterns against a PHP application that computes trigonometric functions, set the SLO to a response time of one second, and compared LARE-HPA with HPA, Bi-LSTM, and Online-BO. Mean latency fell by 50.34%, from 562.8ms with HPA to 279.5ms with LARE-HPA, and 95th-percentile latency fell from 1189.0ms to 600.1ms. SLO satisfaction increased from 93.86% to 97.24%, the highest among the four methods. CPU allocation was 638.13 millicores, about 9.5% more than HPA (582.76), but less than Bi-LSTM (875.85) and Online-BO (1274.75).

MethodLatency (ms)Resource Usage (millicore)SLO Satisfaction (%)
mean50th75th90th95th
HPA562.8528.6770.41001.11189.0582.7693.86
LARE279.5255.5368.1494.0600.1638.1397.24
Bi-LSTM462.1404.1664.3903.71062.6875.8593.06
Online-BO519.3519.9722.2872.1966.31274.7595.15

Wrapping up

The work was selected as a Distinguished Paper (Top 6) at ICSOC 2024, an SCIE-compatible top-tier venue. A patent application was filed, the software copyright was registered, and the implementation was released as open source.

I updated the prediction model from incoming request data without prior training and adjusted the threshold and cool-down time as load changed. In the experiments, I compared response latency and resource use together. The controller adapts its scaling criteria to metrics collected during execution rather than relying only on fixed values set by an operator.