ARTT-HPA: Heterogeneity-Aware Autoscaling with Adaptive Resilient Thresholding and Traffic Allocation for Multi-Replica Microservices on Kubernetes (tentative title)

Scaling across nodes with different performance

ARTT-HPA extends LARE-HPA to clusters whose nodes have different processing capabilities. Increasing replica count did not resolve the imbalance: traffic allocation ignored node performance, while HPA scaled from average CPU utilization. Near 300 RPS, a slow replica reached 100% CPU but continued receiving requests, sharply increasing P99 latency.

LARE-HPA also needed a better threshold signal. Volatility became small under gradually rising demand, delaying adaptation; on the non-seasonal NASDAQ workload, cumulative SLO attainment fell below default HPA. The follow-up separates the direction of demand from short-term variation and adjusts traffic allocation alongside scaling.

Separating trend from volatility

The threshold uses separate, dynamically weighted signals for trend and volatility. This distinguishes steady growth from short bursts instead of treating both as the same change in demand.

RANSAC estimates the trend with less sensitivity to outliers than ordinary least squares. Its inlier criterion uses Median Absolute Deviation (MAD), reducing the influence of extreme observations on the cutoff. The fitted slope represents trend, while the mean absolute residual represents volatility.

Accounting for per-request CPU cost

CPU consumption for the same request differs between nodes. Multiplying volatility by per-request CPU utilization makes the threshold reflect the additional processing cost on slower nodes.

Welford’s online algorithm updates the mean and variance without recalculating the full history. A standard-normal-quantile interval filters outliers before they affect the normalization bounds. Volatility has a fixed lower bound of zero, so only its upper bound is updated.

Prediction-based traffic allocation — Holt with a Kalman filter

Current active-request counts or equal traffic shares were not enough to estimate each replica’s processing capacity. The traffic scheduler predicts next-interval CPU utilization with Holt double-exponential smoothing and adjusts traffic shares from that forecast.

Large Holt smoothing factors respond quickly but are sensitive to noise; small factors respond slowly. A Kalman filter adjusts them from prediction and observation uncertainty. Measurement noise is also updated from an EWMA of recent prediction residuals so the filter can respond to changes after a long history has accumulated.

Bounded adjustments and new-replica initialization

Applying the full load difference to traffic shares can shift overload to the opposite side in the next interval. A sigmoid bounds each adjustment to reduce oscillation.

New replicas have no observation history. Warm-start initialization uses a reference replica in the same component with sufficient history, providing a starting estimate and avoiding too little traffic simply because measurements are missing. The module communicates through the Kubernetes API without changing application code.

Status and results

A heterogeneous cluster of one master and five differently-performing workers (Kubernetes 1.27.5, Istio 1.23.2, Prometheus 3.2.1) ran five microservices that chain matrix multiplications of different sizes. Across a strongly seasonal workload (NASA-HTTP) and a non-seasonal one (NASDAQ), under a 5-second SLO, it was compared with the default HPA, LARE-HPA, and Istio Least Request.

On NASDAQ, mean P99 fell from 4,180.0ms with default HPA, 3,864.0ms with Least Request, and 4,476.2ms with LARE-HPA to 1,018.3ms with ARTT-HPA: reductions of 75.64%, 73.65%, and 77.25%. ARTT-HPA had the fewest 5-second timeouts on both workloads.

On NASA-HTTP, Least Request had a lower mean P99: 1,139.8ms versus 1,763.5ms. However, it had 5,364,327 timeouts versus 4,391,250 for ARTT-HPA. Failed requests can be absent from latency calculations, so latency and timeout counts must be considered together.

Average component CPU utilization was 50.22%, compared with 32.50% for default HPA, 46.43% for Least Request, and 36.80% for LARE-HPA. The evaluation considered this resource utilization alongside latency and timeout counts. The study is being prepared for submission; its title is tentative.