AlertmanagerClusterCrashlooping

Meaning

Half or more of the Alertmanager instances within the same cluster are crashlooping.

Impact

Alerts could be notified multiple time unless pods are crashing to fast and no alerts can be sent.

Diagnosis

kubectl get pod -l app=alertmanager

NAMESPACE   NAME                    READY   STATUS              RESTARTS    AGE
default     alertmanager-main-0     1/2     CrashLoopBackOff    37107 2d
default     alertmanager-main-1     2/2     Running             0 43d
default     alertmanager-main-2     2/2     Running             0 43d 

Find the root cause by looking to events for a given pod/deployement

kubectl get events --field-selector involvedObject.name=alertmanager-main-0

Mitigation

Make sure pods have enough resources (CPU, MEM) to work correctly.