Resource Limits

When scheduling, K8s takes into consideration available resources (CPU and Memory) at the nodes. However, by default, there is no upper limit, so eventually pods could “fill up” nodes. To prevent this, and to ensure certain resources at the container, pod or namespace level, the concept of request and limits is introduced.

Pod (Container) level

In each container in a pod, two thing can be defined to specify resource requirements and limits:

requests:
- These act as requirements
- They define the guaranteed amount of CPU and memory that a container will get
limits
- Define the maximum amount of CPU and memory that the container will get
- In case of going over the CPU limit, the CPU available for the container will be throttled
- If going over the memory limit, the container will be killed (OOM)

Units

CPU is specified in cores, which i typically equivalent to one vCPU in cloud environments like AWS, GCP, or Azure. In the definitions, fractional values can be be used with prefixes (e.g 100m is equal to 0.1). For memory, either bytes are used, either in base 1000 (K,M,G) or 1024 (Ki, Mi, Gi).

Below is an example of a pod definition with resource allocation:

apiVersion: v1
kind: Pod
metadata:
  name: simple-webapp-color
spec:
  containers:
  - name: simple-webapp-color
    image: simple-webapp-color
    resources:
      requests:
        memory: "1Gi"
        cpu: 1
      limits:
        memory: "2Gi"
        cpu: 2

Combination Behavior

-	No Limit	Limit Set
No Request	A container can use as much as it wants, potentially starving others in the same node	K8s assumes an implicit `request` equal to the limits
Request Set	Guarantees that containers will get their minumum resources needed, but they are free to use more. This is the recommended configuration for CPU, NOT MEMORY. More info	The container is guaranteed a certain amount but can peak and use up to the limit, however idle resources might be generated in the case of CPU (reserved but not utilized). Recommended for memory

Namespace level

At the namespace level, two things can be done to limit/handle resources:

LimitRanges

These are k8s object that represent default and max/min resource values for all pods in a namespace. In other words, it can be used to guarantee that even pods that don’t declare anything will have a request and/or limit.

For example for CPU:

apiVersion: v1
kind: LimitRange
metadata:
  name: cpu-resource-constraint # Can be anything
spec:
  limits:
    - default:
        cpu: 500m  #Default limit set on containers that dont declare anything
      defaultRequest:
        cpu: 500m # Default request set
      max:
        cpu: 1 # Max amount of cpu any container can ask for
      min:
        cpu: 100m # Min amount of cpu any container needs to ask for
      type: Container

ResourceQuotas

This is used to limit or control resource usage added up for the whole namespace (e.g all containers in the namespace). For example

apiVersion: v1
kind: ResourceQuota
metadata:
  name: my-resource-quota # Can be anything
spec:
  hard:
    requests.cpu: 4 # Across all pods, sum of requests cannot exceed this
    requests.memory: 4Gi
    limits.cpu: 10 # Across all pods in the ns, sum of cpu limits cannot exceed this
    limits.memory: 10Gi # Across all pods in the ns, sum of memory limits cannot exceed this