-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Pods metrics missing from API #16839
Comments
K8s containers metrics are in "Kubernetes containers" I am not sure why you are using exporting in JSON format, but anyway: $ curl -sq "http://10.10.11.102:19999/api/v1/allmetrics?format=json" | grep cgroup_k8s_cntr | more
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_eth0",
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_operstate_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_operstate_eth0",
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_carrier_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_carrier_eth0",
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_mtu_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_mtu_eth0",
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_packets_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_packets_eth0",
"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_drops_eth0": {
"name":"cgroup_k8s_cntr_lab-httpd_httpd-78d959c55c-9tsrq_httpd.net_drops_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_operstate_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_operstate_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_carrier_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_carrier_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_mtu_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_mtu_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_packets_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_packets_eth0",
"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_drops_eth0": {
"name":"cgroup_k8s_cntr_lab-busybox-load_busybox-load-5b5b854dbb-4plpp_busybox-load.net_drops_eth0",
"cgroup_k8s_cntr_lab-redis_redis-66cf65f9b7-rnqsh_redis.net_eth0": {
...
... curl -sq "http://10.10.11.102:19999/api/v1/allmetrics?format=json" | grep cgroup_k8s_cntr | wc -l
6922 |
@ilyam8 Perhaps I can clarify through comparison. The metrics I am currently not receiving pertain to CPU and memory by pod on the control-plane. In the provided screenshot, you'll notice the server I am encountering issues with. I am specifically searching for netdata's child pod, and there are only two matches. In contrast, in the second screenshot featuring another server with the same setup, I find 62 matches, encompassing the metrics I am seeking, including CPU, as visible in the screenshot. |
@Garahk Unfortunately, I don't see how I can help you with the information provided, other than suggesting that you debug the problem yourself - I can't reproduce the problem. These metrics are collected by cgroups plugin - consider checking logs. If you want you can share your logs and we will check them too. |
@ilyam8 Apologies, I should've started sharing some logs. I've sent them to your inbox, and here's the same reply I wrote. Let me explain the logs..., Am attaching 2 child-pod logs, the one called last week was taken right after I installed netdata, so it has the top of the logs, when netdata is instantiated. While the other is from today, which logs are smaller, not sure why though. Does it have to do with these errors that keep showing in the logs, regarding the maximum number of CGROUPS (these are just 2 of several in the logs)?
And if so, is it something related to my setup? |
@ilyam8 Correct me if am wrong, but this section of the logs mention the collector cannot reach the metrics server:
|
i have the same thing |
Hello @ilyam8 I've upgraded my setup to v1.45.4, and keep missing those metrics in one of my servers RH 8.8: I would expect the Kubernetes Containers section, but is not there, like on this other server RH 7.9: I don't think it's related to the OS version, however I would much appreciate if you can check for any error in the logs of the child pods so I can narrow it down, let me know if you need the parent's logs. I've shared the pod's logs via mail. |
Bug description
Pod's CPU/MEM metrics are missing from the API.
In V1 dashboard cannot be seen:
Filtering on the Allmetrics API, if look for netdata's child pods only the k8s_kubelet.kubelet_pods_log_filesystem_used_bytes chart contains the keyword child:
I had version 1.42.1 installed in this server and it also had the issue, upgraded to v1.44.1 to see if issue cleared, however having the same issue.
Expected behavior
Pod's metrics available in API
Steps to reproduce
...
Installation method
helmchart (kubernetes)
System info
Netdata build info
Additional info
Cluster is k3s
The text was updated successfully, but these errors were encountered: