Garbage collector on pod termination is not active

kube_controller_manager_terminated_pod

Severity

low

Vendor

Kubernetes

Resource

kube-controller-manager

Description

--terminated-pod-gc-threshold determine number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods

Current setting for garbage collection is 12,500 terminated pods which might be too high for your system to sustain. Ensure you choose an appropriate threshold value based on your system resources.

Examples

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: weak
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager (1)
    - --controllers=*,bootstrapsigner,tokencleaner
    image: gcr.io/google_containers/kube-apiserver-amd64:v1.6.0

1	When `--terminated-pod-gc-threshold` is not set, garbage collector occurs when 12500 terminated pod is reached, which could be too high for your system.

Mitigation / Fix

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: good
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
    - --controllers=*,bootstrapsigner,tokencleaner
    - --terminated-pod-gc-threshold=500 (1)
    image: gcr.io/<image>

1	Verify `--terminated-pod-gc-threshold` is set appropriately.