Skip to content
← Publications
KIPS ACK 2024

An Analysis of Service Latency Based on Replica Resource Granularity in Kubernetes Environment

Geonwoo Kim, Donggyun Kim, Heonchang Yu

Annual Conference of KIPS (ACK 2024)

Summary

With the same total CPU, latency can still differ by per-pod allocation and replica count. The study divides one CPU core among 1–16 replicas to examine resource granularity.

Role
Second authorundergraduate research mentor
Outcome
Splitting one replica into two raised P95 by 220.58% and P99 by 660.56% in the CPU-bound experiment.
Line chart of service latency (mean/90th/95th/99th percentile) versus replica count
Service latency vs replica resource granularity

Different Latency with the Same Total CPU

The replica count needed by HPA depends on the base resources assigned to each pod. Even with the same total CPU, latency can differ according to how many replicas share that budget: its resource granularity.

Many small pods can reduce unused capacity at low load, but may add application overhead per pod, interference between processes sharing physical resources, and host OS kernel load. The study held total CPU allocation constant while varying per-pod resources and replica count.

Separating total resources from per-pod allocation

Resource granularity also changes the unit in which HPA adds or releases capacity. For example, two CPU cores require two replicas at one core per pod, or four replicas at 0.5 cores per pod. The total is the same, but the resource increment per replica and the number of running processes differ.

Smaller pods allow finer allocation at low load, but each still requires an application process. Partitioned resources do not simply translate into additional request-processing capacity, motivating measurements of latency and CPU utilization under load.

One CPU Core Divided Across 1 to 16 Replicas

Instances with two 3.1GHz AMD EPYC cores and 4GB of memory formed one master and one worker. One CPU core was divided evenly among 1, 2, 4, 8, 10, or 16 replicas. Per-pod allocations were 1, 0.5, 0.25, 0.125, and 0.1 cores; the paper’s table rounds the 16-replica allocation to approximately 0.063 cores.

One pod with one core and four pods with 0.25 cores each have the same total CPU allocation. What changes is the resource available to each pod processing a request and the number of application instances running concurrently. This compares partitioning without increasing the total resource budget.

An Apache web server ran a CPU-intensive trigonometric computation. The load generator sent 300 HTTP requests per minute for 300 seconds while response latency and CPU utilization were measured.

Every configuration received the same request rate for the same duration. Larger replica counts were not given more incoming work. Mean latency, upper latency percentiles, and CPU utilization were compared together.

660.56% Higher P99 Latency After Splitting One Replica into Two

With total CPU held constant, adding replicas could still increase latency. Splitting one replica into two raised P95 and P99 by 220.58% and 660.56%. Mean latency generally increased, but some metrics were similar or lower between eight and ten replicas.

The upper tail can grow more sharply than the mean, so an unchanged total CPU budget does not establish equivalent request performance. The finer allocation units must be considered together with the latency of multiple processes sharing limited CPU capacity.

Container overhead and inter-process contention were proposed explanations for the higher latency. Their individual contributions were not isolated and would need separate measurement.

CPU utilization also increased with finer allocation. Partitioning changed latency and utilization even with the same total CPU allocation. Choosing per-pod resources therefore requires evaluating both response time and utilization under actual request load alongside the total resource budget. I participated as a co-author and undergraduate research mentor.