When Kubernetes clusters grow larger and encompasses more tools, applications, services, etc., cluster management becomes crucial. The way in which the cluster expands lays the foundation for proper management, and to carry out this task in an automated and efficient manner, the following Kubernetes features come into play:

  • Node Selector

  • Affinity

  • Taint/Toleration

Those three features belong to the same umbrella of features that have for objective the scheduling of pods on the right node. It’s critical that you have a well thought strategy for each of your pods on where to run as you might have different node configurations depending on the application. Here are just two common use cases:

  • For instance you might host Hashicorp Vault on Kubernetes and you would obviously like to have a dedicated/isolated secured node to host all the workloads related to this application.

  • Or a common use case in analytics is to run data workloads as Kubernetes Pod Operators. As those tasks are ephemeral (data workload executes and then pod is killed), you might want to spin up all those pods on a node pool that leverages preemptible features.

In this article, we will go through the use of these features and provide some examples to easily understand their functionality and determine the scenarios in which each feature should be employed.

Node Selector


Schematic diagram representing a Node Selector in a Kubernetes Cluster, showing two pods assigned to two different nodes based on node selectors, with Pod 1 assigned to Node 1 and Pod 2 to Node 2, while Node 3 remains unassigned


This feature is the most basic and easy to understand, primarily relying on labels to establish connections between nodes and pods. In the pod configuration, a label is set, and nodes with this label become eligible for deploying the pod.

This feature is highly useful for ensuring that a pod is deployed precisely on the node or group of nodes needed. It facilitates application segregation, resource management, departmental organization, cost management, and more.

The configuration is straightforward — just assign a label to the node and, on the other hand, include the ‘nodeSelector’ feature in the pod.

To label the node, the kubectl label nodes command can be used as follows:

kubectl label nodes "node-name" label-key=label-value

And for the pod, you just need to add the ‘nodeSelector’ feature, specifying the correct label that matches the pod or group of pods:

apiVersion: v1kind: Podmetadata:name: temporalspec:containers:- image: nginx:alpinename: temporalnodeSelector:label-key: label-value


With these configurations, the pod will be deployed on the node pool that has that label.

Affinity


Diagram showing the 'Affinity' scheduling feature in a Kubernetes Cluster, with Pod 1 and Pod 2 preferring Node 1, while Pod 3 and Pod 4 are scheduled on Node 3, indicating selective placement based on defined affinities.


Affinity is another feature that provides us with much more flexibility and customization capability to fine-tune our deployment. Before delving into details, it’s worth mentioning that it can be configured in three flavors:

  • Available:
     - requiredDuringSchedulingIgnoredDuringExecution    : It is mandatory that the configured conditions are met; otherwise, the pod will not be deployed (if the label is removed and the pod is already deployed, nothing happens).
     - preferredDuringSchedulingIgnoredDuringExecution: Pods with this label can be scheduled on the node, but if there is no other available node, it will also accept pods without the specified toleration (if the label is removed and the pod is already deployed, nothing happens).

  • Planned:
     - requiredDuringSchedulingRequiredDuringExecution    : It is mandatory that a pod with the specified label exists. If the label is removed from the node while the pod is in execution, the pod will be stopped.

Once again, we will use the node label for pod affinity. Instead of ‘nodeSelector,’ we will employ ‘affinity’ as follows:

apiVersion: v1kind: Podmetadata:name: temporalspec:containers:- image: nginx:alpinename: temporalAffinity:nodeAffinity:RequiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: label-keyoperator: Invalues:- label-value


At first, it might seem more complicated, as this example would work the same as ‘nodeSelector.’ However, if we focus on ‘matchExpressions,’ this is where we can start to experiment and configure various things like:

More than one label across different nodes:

- matchExpressions:- key: label-keyoperator: Invalues:- label-value-1- label-value-2

To avoid deploying pods on a node with a specific label:

- matchExpressions:- key: label-keyoperator: NotInvalues:- label-value-3

Or, simply, we want them to be deployable on any node that has a label, regardless of the value of the label:

- matchExpressions:- key: label-keyoperator: Exists

Taint/Tolerations

A hand-drawn diagram showing taints and tolerations within a Kubernetes cluster. It outlines four pods, with Pod 1 tolerating Node A and Node B, Pod 2 tolerating Node B, Pod 3 tolerating Node A, and Pod 4 with no tolerances. Nodes A, B, and C each have a taint associated with them, affecting where pods can be scheduled

Finally, let’s discuss this feature, which essentially allows us to block nodes for specific pods.

On one hand, we will need to configure a taint on the node using the following command:

kubectl taint nodes "node-name" key=value:taint-effect

There are 3 types of effects:

  • NoSchedule: Marks the pod with this taint, preventing any pod with this toleration from being deployed.

  • PreferNoSchedule: A pod without toleration will try to deploy first on nodes without taints. However, if unavailable, this node will still be suitable for deploying the pod.

  • NoExecute: If a pod is running with toleration and the toleration is removed, the pod is evicted from the node.

In this example, we will configure a taint color=white

kubectl taint nodes "node-name" color=white:NoSchedule

On the other hand, we configure a pod with toleration to this taint:

apiVersion: v1kind: Podmetadata:name: temporalspec:containers:- image: nginx:alpinename: temporaltolerations:- key: "color"operator: "Equal"value: "white"effect: "NoSchedule"

In this case, by using ‘NoSchedule’, no pod can be hosted on this node without this toleration. Only the ‘temporal’ pod can be hosted here. However, without using affinity, this pod could also be hosted on another node that lacks the specified toleration.

This is why best practices recommend combining these features for perfect management. Even if a pod has a taint and a node has toleration, it is not mandatory for the pod to deploy on the node. But by adding affinity, it will be, and with the node having toleration, it won’t allow another pod to reside on it.

Conclusion

While this article has been mainly technical, the business impact of not leveraging those technical features well can be damaging. Some of your node pools could be overloaded and make your critical applications crash, some sensitive pods might be launched on less secure node pools and the list goes on.

Being in full control of where your pods are spun up is not just a technical detail; it’s the foundation for ensuring your node pools run the pods they need to run and nothing more.

Thank you

If you enjoyed reading this article, stay tuned as we regularly publish technical articles on Kubernetes. Follow Astrafy on LinkedIn to be notified for the next article ;).

If you are looking for support on Data Stack or Google Cloud solutions, feel free to reach out to us at sales@astrafy.io.