Skip to content

KubeDaemonSetMisScheduled

Meaning

A number of pods of daemonset are running where they are not supposed to run.

Impact

Service degradation or unavailability. Excessive resource usage where they could be used by other apps.

Diagnosis

Usually happens when specifying wrong pod nodeSelector/taints/affinities or node (node pools) were tainted and existing pods were not scheduled for eviction.

  • Check daemonset status via kubectl -n $NAMESPACE describe daemonset $NAME.
  • Check DaemonSet update strategy
  • Check the status of the pods which belong to the replica sets under the deployment.
  • Check pod template parameters such as:
  • pod priority - maybe it was evicted by other more important pods
  • affinity rules - maybe due to affinities and not enough nodes it is not possible to schedule pods
  • Check node taints and labels
  • Check logs for node-feature-discovery and other supporting tools such as gpu-feature-discovery

Mitigation

Update DaemonSet and apply change, delete pods manually.