Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nethermind not pushing metrics to Pushgateway? #6916

Closed
fatstormtrooper opened this issue Apr 13, 2024 · 4 comments
Closed

Nethermind not pushing metrics to Pushgateway? #6916

fatstormtrooper opened this issue Apr 13, 2024 · 4 comments
Labels
bug low priority

Comments

@fatstormtrooper
Copy link

Description
Despite setting both flags: --Metrics.Enabled true --Metrics.PushGatewayUrl http://pushgateway:9091 metric data does not seem to be sent to Pushgateway and group/instance remains empty.

Steps to Reproduce
Following: https://docs.nethermind.io/monitoring/metrics/setting-up-local-metrics-infrastracture => https://github.com/NethermindEth/metrics-infrastructure
I'm using latest docker images for both Nethermind (1.25.4) and Pushgateway. (1.8.0)
When I set the above flags for Nethermind, the metric page in the Pushgateway interface remains empty.

Actual behavior
No errors/exceptions reported and Nethermind is able to connect to the PushGatewayUrl http endpoint just fine.
Stopping the Pushgateway container, does give exceptions in Nethermind about not being able to connect to the given PushGatewayUrl. (so its not a network connectivity issue between both containers)
Upping the logging level and sifting through the logs, I found this:

2024-04-13 05:36:25.2988|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (WaitingForDependencies) is WaitingForDependencies
2024-04-13 05:36:25.2988|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (WaitingForDependencies) is waiting for InitializeNetwork : InitializeNetwork (WaitingForExecution)
2024-04-13 05:36:25.2988|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (WaitingForDependencies) is WaitingForDependencies
2024-04-13 05:36:25.2988|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (WaitingForDependencies) is waiting for InitializeNetwork : InitializeNetwork (WaitingForExecution)
2024-04-13 05:36:25.5970|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (WaitingForDependencies) is WaitingForDependencies
2024-04-13 05:36:25.7588|DEBUG|Init.Steps.EthereumStepsManager|16|StartMonitoring : StartMonitoring (WaitingForExecution) stage changed to WaitingForExecution
2024-04-13 05:36:25.7588|DEBUG|Init.Steps.EthereumStepsManager|16|StartMonitoring : StartMonitoring (WaitingForExecution) is WaitingForExecution
2024-04-13 05:36:26.5550|DEBUG|Init.Steps.EthereumStepsManager|16|Executing step: StartMonitoring : StartMonitoring (WaitingForExecution)
2024-04-13 05:36:26.7192|INFO|Monitoring.MonitoringService|19|Started monitoring for the group: nethermind, instance: Mainnet
2024-04-13 05:36:26.7192|INFO|Init.Steps.StartMonitoring|11|System.Diagnostics.Metrics disabled
2024-04-13 05:36:26.7192|DEBUG|Init.Steps.EthereumStepsManager|11|Step StartMonitoring          executed in 164ms
2024-04-13 05:36:26.7192|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring          complete
2024-04-13 05:36:26.7192|DEBUG|Init.Steps.EthereumStepsManager|16|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:36:26.7192|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:36:26.7388|DEBUG|Init.Steps.EthereumStepsManager|11|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:36:26.7388|DEBUG|Init.Steps.EthereumStepsManager|19|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:37:08.5230|DEBUG|Init.Steps.EthereumStepsManager|19|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:37:08.5230|DEBUG|Init.Steps.EthereumStepsManager|19|StartMonitoring : StartMonitoring (Complete) is Complete
2024-04-13 05:37:08.6913|DEBUG|Init.Steps.EthereumStepsManager|19|StartMonitoring : StartMonitoring (Complete) is Complete

Expected behavior
Metrics pushed by Nehtermind to Pushgateway should be visible on the metrics page in the Pushgateway interface.
Manually posting data with curl (from the docker host) to Pushgateway does work successfully and appears on the metric page, using following command:

echo "my_metric{job=\"nethermind\", instance=\"Mainnet\"} 42" | curl --data-binary @- http://localhost:9091/metrics/job/nethermind/instance/Mainnet

Desktop (please complete the following information):

  • Operating System: Debian 12 bookworm
  • Installation Method: Docker compose with yml file
  • Docker 26.0.0, build 2ae903e
  • Nethermind 1.25.4
  • Pushgateway 1.8.0
  • Consensus Client: Lodestar v1.17.0
@brbrr
Copy link
Contributor

brbrr commented Apr 18, 2024

I was able to reproduce the issue on Mac with sedge. It feels like a configuration issue though, since the push gateway is working for our test nodes (although, we use a centralized instance). @stdevMac or @cbermudez97 maybe you can take a look at it?

@chrisemmanuel17
Copy link

chrisemmanuel17 commented May 2, 2024 •
edited

I also face similar issue, I'm running

  • Nethermind v1.26.0
  • Pushgateway v1.8.0
  • Consensus Client: Lighthouse v5.1.3-3058b96
  • Prometheus v2.51.2

Below is the scrape configs in my prometheus.yml file

  - job_name: "pushgateway"
    honor_labels: true
    static_configs:
    - targets: ["localhost:9091"]

I run Nethermind with this configs specified:

nethermind -c mainnet -dd /home/ubuntu/nethermind/data --JsonRpc.Enabled true --JsonRpc.Host 0.0.0.0 --JsonRpc.Port 8545 --JsonRpc.WebSocketsPort 8546 --JsonRpc.JwtSecretFile "/secrets/jwt.hex" --Metrics.Enabled true --Metrics.PushGatewayUrl http://localhost:9091

Here is an image that shows Pushgateway service active and running.
image

Update:
I was able to get the metrics working by doing this-

change "--Metrics.PushGatewayUrl http://127.0.0.1:9091 " in nethermind command to "--Metrics.PushGatewayUrl http://127.0.0.1:9091/metrics/"

@mvuk
Copy link

mvuk commented May 15, 2024 •
edited

I have the same issue. Grafana runs. Nethermind runs. But when I access my Nethermind Dashboard in Grafana I get "no sync data" and the page is blank.

Here is the output of localhost:9091/metrics/ for me:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.65e-05
go_gc_duration_seconds{quantile="0.25"} 3.2732e-05
go_gc_duration_seconds{quantile="0.5"} 3.7491e-05
go_gc_duration_seconds{quantile="0.75"} 4.3112e-05
go_gc_duration_seconds{quantile="1"} 4.3402e-05
go_gc_duration_seconds_sum 0.00025949
go_gc_duration_seconds_count 7
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 12
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.22.1"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.471168e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.079372e+07
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.453214e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 42042
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.034072e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.471168e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 6.529024e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.316608e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 3464
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.873664e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.1845632e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7157957340574412e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 45506
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 103840
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 114240
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 7.442032e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 991586
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 720896
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 720896
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.817524e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 9
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 0.12
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.9968e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71579523419e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.270448128e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP pushgateway_build_info A metric with a constant '1' value labeled by version, revision, branch, goversion from which pushgateway was built, and the goos and goarch for the build.
# TYPE pushgateway_build_info gauge
pushgateway_build_info{branch="HEAD",goarch="amd64",goos="linux",goversion="go1.22.1",revision="5aef87129c8d9ecebcc8b2fb0e97a31980680e8c",tags="unknown",version="1.8.0"} 1
# HELP pushgateway_http_requests_total Total HTTP requests processed by the Pushgateway, excluding scrapes.
# TYPE pushgateway_http_requests_total counter
pushgateway_http_requests_total{code="200",handler="static",method="get"} 32
pushgateway_http_requests_total{code="200",handler="status",method="get"} 9

@fatstormtrooper
Copy link
Author

Update: I was able to get the metrics working by doing this-

change "--Metrics.PushGatewayUrl http://127.0.0.1:9091 " in nethermind command to "--Metrics.PushGatewayUrl http://127.0.0.1:9091/metrics/"

This is indeed the solution for me, great find @chrisemmanuel17!
@mvuk you should see a lot more metrics once you add the /metrics to your PushGatewayUrl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug low priority
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mvuk @brbrr @chrisemmanuel17 @fatstormtrooper