PrometheusTSDBCompactionsFailing

Meaning

Prometheus has issues compacting blocks.

Impact

Metrics and alerts may be missing or inaccurate.

Diagnosis

Check storage used by the pod. This can happen if there is a lot of going on in the cluster and prometheus did not manage to compact data.

Mitigation

At first just wait, it may fix itself after some time.

Increase Prometheus pod memory so that it caches more from disk. Try expanding volumes if they are too small or too slow. Change PVC storageClass to a more performant one.