Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KubeContainerOOMKilledCount never resolves because relies on terminated reason #888

Open
yene opened this issue May 16, 2024 · 5 comments
Open

Comments

@yene
Copy link

yene commented May 16, 2024

hi,

the rule KubeContainerOOMKilledCount keeps being active as long as the pod has containerStatuses.lastState.terminated.reason OOMKilled.

is this intended behaviour? Should the metric not exclude pods that are healthy again after some time.

thank you for your time

sum by (cluster,container,controller,namespace)(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} * on(cluster,namespace,pod) group_left(controller) label_replace(kube_pod_owner, "controller", "$1", "owner_name", "(.*)")) > 0
@vishiy
Copy link
Contributor

vishiy commented May 17, 2024

Hi @yene - Thanks for reporting. So are you saying if there is a container that was OOMkilled at some point, and then its running fine now, but the alert is active as its looking at last state being oomkilled ?

@yene
Copy link
Author

yene commented May 17, 2024

@vishiy correct. As an end result we have lots of Azure Alerts that are open.

@vishiy
Copy link
Contributor

vishiy commented May 17, 2024

ok, thank you. We get the issue, and will update to a different query. will keep this issue open to track. Thanks for reporting this.

Copy link

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Copy link

github-actions bot commented Jun 2, 2024

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants