Kubernetes: Nodes, Taints, and Tolerations Best Practices

What are Taints and Tolerations?

In Kubernetes, node taints and tolerations function in a manner similar to node affinity rules, though they take the almost opposite approach. Affinity rules are set for Pods to attract them to specific nodes. A tainted node repels pods that do not have tolerations for those nodes set. Together, taints and tolerations make sure that pods are not scheduled onto inappropriate nodes. A taint produces three possible effects: NoSchedule,  (Kubernetes will only schedule pods that tolerate the node’s taint), PreferNoSchedule (Kubernetes will avoid scheduling non-tolerant Pods on the Nodes, but may still do so), or NoExecute (Kubernetes will evict any running non-tolerant pods already running on a tainted node).

Why Use Taints and Tolerations?

Kubernetes taints and tolerations allow you to create special nodes that are reserved for specific uses or only run specific processes (Pods) that match the node. You may wish to keep workloads off or your Kubernetes management nodes and tainting nodes so that no workload Pod would have matching tolerations would keep them from being scheduled to those nodes.  You may have nodes with specialized hardware for specific jobs (e.g GPUs) and tainting such nodes can reserve it so that the Pods that specifically need that resource type can be scheduled to those nodes when needed.

Node Taints and Pod Tolerations

Applying Taints and Tolerations

Taints are applied to a node using kubectl, for example:

kubectl taint nodes machineLearningNode1=computer-vision:NoSchedule

You can then verify that this taint has been applied with the kubectl describe nodes machineLearningNode1 command and any applied taints, and there could be multiple, would be described in the Taints: section. In this example, any existing nodes would keep running on the tainted node, but no further Pods would be scheduled unless they have the following tolerations fields in their Podspec:

Since this toleration matches the tainted node, any pod with that spec could be deployed in the node machineLearningNode1. If you later wished to remove the taint on this node, the command kubectl taint nodes machineLearningNode1 computer-vision:NoSchedule untainted will remove it.

Using Multiple Taints

It is possible to apply more than one taint to a single node and more than one toleration to a single Pod. Multiple taints and tolerations are used by Kubernetes like a filter. Taint’s matching a Pod’s tolerations are ignored. The remaining taint effects are then applied to the Pod. Some of the effects include

  • Kubernetes will not schedule the Pod if at least one non-tolerated taint has a NoSchedule effect.
  • Kubernetes will try not to schedule the Pod on the node if at least one non-tolerated taint has a PreferNoSchedule effect.
  • A NoExecute taint will cause Kubernetes to evict the Pod if it is currently running on the node or will not schedule the Pod the node.

As an example, if you have a Node to which you’ve applied the following taints:

And you have a Pod with the following tolerations:

The Pod would not be scheduled to the node because it tolerates the first two taints but will be affected by the last, non-tolerated NoSchedule taint.  Now if the Pod was already running on the node when the last NoSchedule taint was added it would continue running on the pod.  In the case of the running Pod, if we then added also a NoExecute taint:

The running Pod would now be evicted from the Node.

You may have noticed another modifier in the toleration Podspec example, namely the operator value, that allows further modification of how Kubernetes evaluates the toleration against a taint.  The default value for operator is Equal so if the value for taint and toleration are indeed equal, the taint is tolerated. If the operator Exists, no value should be specified in the toleration. Two behaviors to be aware of are that an empty key with operator Exists will match all keys, values and effects and thus tolerate everything, and an empty effect matches all effects with key key.