Kubernetes: Taints and Tolerations

 

A taint marks a node with a specific characteristic, such as “gpu=true”. A taint consists of a key, value, and effect.By default, pods cannot be scheduled on tainted nodes unless they have a special permission called toleration.

Taint is like a reserved table with a side corner view having some booking number. If the guest has the same booking number, then they are allowed to sit at the reserved table and enjoy dinner.

When a toleration on a pod matches with the taint on the node then only that pod will be scheduled on that node.

We provide toleration on pod. Toleration allows a pod to say, “Hey, I can handle that taint. Schedule me anyway!” You define tolerations in the pod specification to let them bypass the taints.

Effects of Taints and Tolerance
1. NoSchedule (Newer Pods)
2. PreferNoSchedue (No Guaranty)
3. NoExecution (Existing/Newer Pods)

Taint a node using below command:

kubectl taint nodes node1 key1=value1:NoSchedule

kubectl taint node kind-worker2 gpu=true:NoSchedule
kubectl taint node kind-worker gpu=true:NoSchedule

Worker nodes: kind-worker,kind-worker2

The allowed values for the effect field are:

NoExecuteThis affects pods that are already running on the node as follows:

• Pods that do not tolerate the taint are evicted immediately
• Pods that tolerate the taint without specifying tolerationSeconds in their toleration specification remain bound forever
• Pods that tolerate the taint with a specified tolerationSeconds remain bound for the specified amount of time. After that time elapses, the node lifecycle controller evicts the Pods from the node.

NoScheduleNo new Pods will be scheduled on the tainted node unless they have a matching toleration. Pods currently running on the node are not evicted.

PreferNoSchedulePreferNoSchedule is a "preference" or "soft" version of NoSchedule. The control plane will try to avoid placing a Pod that does not tolerate the taint on the node, but it is not guaranteed.

 

kubectl run nginx - image=nginx
kubectl get pods
kubectl describe pod nginx

If we try to schedule pod on the tainted nodes, we will get below error:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  54s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 2 node(s) had untolerated taint {gpu: true}. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

 

Provide the tolerance as show in below to run a pod on node:

 

 

 

Now , pod can be schedule on the worker nodes,since pod has below toleration which matches with node taint.

 

tolerations:
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

Remove taint using — at the end of the command:

kubectl taint node kind-worker2 gpu=true:NoSchedule-  

 Taints applied at node level, Tolerations at pod level — This gives node ability to allow which pods to be scheduled on them. (Node centric approach)