Resource metrics pipeline

Resource usage metrics, such as container CPU and memory usage, are available in Kubernetes through the Metrics API. These metrics can be accessed either directly by the user with the kubectl top command, or by a controller in the cluster, for example Horizontal Pod Autoscaler, to make decisions.

The Metrics API

Through the Metrics API, you can get the amount of resource currently used by a given node or a given pod. This API doesn't store the metric values, so it's not possible, for example, to get the amount of resources used by a given node 10 minutes ago.

The API is no different from any other API:

  • it is discoverable through the same endpoint as the other Kubernetes APIs under the path: /apis/metrics.k8s.io/
  • it offers the same security, scalability, and reliability guarantees

The API is defined in k8s.io/metrics repository. You can find more information about the API there.

Measuring Resource Usage

CPU

CPU is reported as the average usage, in CPU cores, over a period of time. This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in both Linux and Windows kernels). The kubelet chooses the window for the rate calculation.

Memory

Memory is reported as the working set, in bytes, at the instant the metric was collected. In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap. The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages.

Metrics Server

Metrics Server is a cluster-wide aggregator of resource usage data. By default, it is deployed in clusters created by kube-up.sh script as a Deployment object. If you use a different Kubernetes setup mechanism, you can deploy it using the provided deployment components.yaml file.

Metrics Server collects metrics from the Summary API, exposed by Kubelet on each node, and is registered with the main API server via Kubernetes aggregator.

Learn more about the metrics server in the design doc.

Summary API Source

The Kubelet gathers stats at node, volume, pod and container level, and emits their statistics in the Summary API for consumers to read.

Pre-1.23, these resources have been primarily gathered from cAdvisor. However, in 1.23 with the introduction of the PodAndContainerStatsFromCRI FeatureGate, container and pod level stats can be gathered by the CRI implementation. Note: this also requires support from the CRI implementations (containerd >= 1.6.0, CRI-O >= 1.23.0).

Last modified December 23, 2021 at 4:04 AM PST : Fix the description of the Summary API Source section (9aa7f71e4)