Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Support Ticket]: Unable to access Web UI for monitoring stack using a remote machine #1774

Open
1 of 10 tasks
stuberman opened this issue Oct 24, 2023 · 8 comments
Open
1 of 10 tasks

[Support Ticket]: Unable to access Web UI for monitoring stack using a remote machine #1774

stuberman opened this issue Oct 24, 2023 · 8 comments
Assignees
@nonsense
Labels
support Support Ticket

Comments

@stuberman
Copy link

Boost component

  • boost daemon - storage providers
  • boost client
  • boost UI
  • boost data-transfer
  • boost index-provider
  • booster-http
  • booster-bitswap
  • LID Database - Yugabyte/LevelDB
  • boostd-data
  • Other

Boost Version

boostd version 2.1.0-rc2+mainnet+git.4502e03

Describe the problem

Following directions at https://boost.filecoin.io/tutorials/setting-up-a-monitoring-stack-for-boost
When I try to see Prometheus dashboard from another computer on LAN using: http://192.168.1.98:9090/
I get:
This site can’t be reached192.168.1.98 refused to connect.

When I use http://192.168.1.98:9190/ I get access but errors:

Endpoint State Labels Last Scrape Scrape Duration Error
http://boost:1288/metrics DOWN host="local"instance="boost:1288"job="boost" 8.316s ago 0.684ms Get "http://boost:1288/metrics": dial tcp 172.17.0.1:1288: connect: connection refused
http://booster-bitswap:9696/metrics UP host="local"instance="booster-bitswap:9696"job="booster-bitswap" 11.133s ago 6.796ms
prometheus (1/1 up)
http://localhost:9090/metrics UP host="local"instance="localhost:9090"job="prometheus" 5.873s ago 9.045ms

When I try to see Graphana dashboard from another computer on LAN using: http://192.168.1.98:3333/
I get a web page prompting me for a login, but the login always fails

Logging Information

grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:12.112078044Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:12.112196706Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=237.883µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:32.153671646Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:32.153792822Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=236.277µs size=102 referer= handler=/api/live/ws
tempo       | level=info ts=2023-10-24T21:43:34.287108296Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo       | level=info ts=2023-10-24T21:43:34.287153414Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:44.090236711Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:43:44.090304947Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=151.138µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:03.086328811Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:03.086424565Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=206.244µs size=102 referer= handler=/api/live/ws
tempo       | level=info ts=2023-10-24T21:44:04.287463829Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo       | level=info ts=2023-10-24T21:44:04.287513068Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
loki        | level=info ts=2023-10-24T21:44:04.722698646Z caller=table_manager.go:171 msg="uploading tables"
loki        | level=info ts=2023-10-24T21:44:04.722852089Z caller=table.go:336 msg="uploading table index_19654"
loki        | level=info ts=2023-10-24T21:44:04.722872483Z caller=table.go:363 msg="finished uploading table index_19654"
loki        | level=info ts=2023-10-24T21:44:04.722896928Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19654"
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:13.110959327Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.1.154 time_ms=0 duration=96.871µs size=29 referer= handler=/
grafana     | logger=http.server t=2023-10-24T21:44:13.127972574Z level=error msg="Failed to parse user ID" error="identifier is not initialized"
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:14.262713778Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:14.262800941Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=180.542µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:15.178396132Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:15.178533931Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=241.166µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:16.706304009Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:16.706378321Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=157.425µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:18.222278133Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:18.222379754Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=206.384µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:19.092356996Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:19.092440806Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=165.457µs size=102 referer= handler=/api/live/ws
grafana     | logger=authn.service t=2023-10-24T21:44:20.353097216Z level=info msg="Failed to authenticate request" client=auth.client.form error="[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found"
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:20.353199116Z level=info msg=Unauthorized error="[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:20.353260857Z level=info msg="Request Completed" method=POST path=/login status=401 remote_addr=192.168.1.154 time_ms=1 duration=1.78531ms size=107 referer=http://192.168.1.98:3333/login handler=/login
grafana     | logger=authn.service t=2023-10-24T21:44:22.293590383Z level=info msg="Failed to authenticate request" client=auth.client.form error="[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found"
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:22.293697661Z level=info msg=Unauthorized error="[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:22.293771554Z level=info msg="Request Completed" method=POST path=/login status=401 remote_addr=192.168.1.154 time_ms=1 duration=1.899642ms size=107 referer=http://192.168.1.98:3333/login handler=/login
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:22.470666577Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:22.470781328Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=201.495µs size=102 referer= handler=/api/live/ws
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:24.26036278Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:24.260458953Z level=info msg="Request Completed" method=POST path=/api/frontend-metrics status=401 remote_addr=192.168.1.154 time_ms=0 duration=148.415µs size=102 referer=http://192.168.1.98:3333/login handler=/api/frontend-metrics
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:29.132625677Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana     | logger=context userId=0 orgId=0 uname= t=2023-10-24T21:44:29.132721361Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=200.168µs size=102 referer= handler=/api/live/ws

Repo Steps

  1. Run either URL from a remote computer browser on LAN
  2. Do '...'
  3. See error '...'
    ...

Boostd, bitswap are running locally on boost node, lotus-miner and lotus node on a different server.
@stuberman stuberman added the support Support Ticket label Oct 24, 2023
@LexLuthr LexLuthr added this to the Boost v2 stability milestone Oct 25, 2023
@LexLuthr
Copy link
Collaborator

@stuberman Can you confirm if 192.168.1.98 is your server's IP (where Grafana is running) or if this is a docker internal IP. If it is a docker internal IP, I would recommend using server IP to access.

@stuberman
Copy link
Author

stuberman commented Oct 25, 2023 •
edited

Correct, I use the following IPs:

  • Boost/Yugabyte/Grafana/Docker/Prometheus/Bitswap = 192.168.1.98
  • Lotus-miner & Lotus node = 192.168.1.95
  • My Mac (browser) = 192.168.1.154
  • Lotus-worker = 192.168.1.96

@LexLuthr
Copy link
Collaborator

You are correct about prometheus port being 9190 instead of 9090. I have updated the docs. Grafana's default username and passwords are "admin/admin". Can you tell me what error you get on the screen when trying to login?

@stuberman
Copy link
Author

Using admin/admin - I now get a change password prompt

Login

After changing the default password, I get this screen:

Initial

Navigating to dashboards, I see three options, all of which are empty.

Logs in docker monitoring window:

loki | level=info ts=2023-10-26T12:55:04.762341791Z caller=table_manager.go:476 msg="creating table" table=index_18795
grafana | logger=context userId=0 orgId=0 uname= t=2023-10-26T12:55:11.286655146Z level=info msg= error="[auth.unauthorized] cannot authenticate request" remote_addr=192.168.1.154 traceID=
grafana | logger=context userId=0 orgId=0 uname= t=2023-10-26T12:55:11.286777859Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=192.168.1.154 time_ms=0 duration=226.149µs size=102 referer= handler=/api/live/ws
grafana | logger=cleanup t=2023-10-26T12:55:21.2861812Z level=info msg="Completed cleanup jobs" duration=22.166273ms
grafana | logger=grafana.update.checker t=2023-10-26T12:55:21.406862636Z level=info msg="Update check succeeded" duration=25.734866ms
grafana | logger=plugins.update.checker t=2023-10-26T12:55:21.460767559Z level=info msg="Update check succeeded" duration=51.739044ms
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T12:55:30.761020381Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.1.154 time_ms=5 duration=5.042056ms size=0 referer= handler=/api/live/ws
tempo | level=info ts=2023-10-26T12:55:34.287910802Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo | level=info ts=2023-10-26T12:55:34.287954314Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T12:56:01.715741092Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.1.154 time_ms=5 duration=5.883796ms size=0 referer= handler=/api/live/ws
tempo | level=info ts=2023-10-26T12:56:04.287842356Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0

@stuberman
Copy link
Author

Here is the default dashboard window (briefly see a message, unable to load default dashboard)

Dashboard

grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T12:59:11.556352129Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.1.154 time_ms=3 duration=3.326449ms size=0 referer= handler=/api/live/ws
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T12:59:31.645739767Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.1.154 time_ms=3 duration=3.114687ms size=0 referer= handler=/api/live/ws
tempo | level=info ts=2023-10-26T12:59:34.287807686Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo | level=info ts=2023-10-26T12:59:34.287861045Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
prometheus | ts=2023-10-26T13:00:00.849Z caller=compact.go:523 level=info component=tsdb msg="write block" mint=1698314400025 maxt=1698321600000 ulid=01HDP13ZVHMQQ36WX9TV1ETSVQ duration=96.439447ms
prometheus | ts=2023-10-26T13:00:00.855Z caller=head.go:1298 level=info component=tsdb msg="Head GC completed" caller=truncateMemory duration=5.022151ms
tempo | level=info ts=2023-10-26T13:00:04.241763881Z caller=poller.go:191 msg="writing tenant index" tenant=single-tenant metas=5 compactedMetas=2
tempo | level=info ts=2023-10-26T13:00:04.28387466Z caller=poller.go:191 msg="writing tenant index" tenant=single-tenant metas=5 compactedMetas=2
tempo | level=info ts=2023-10-26T13:00:04.287751605Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo | level=info ts=2023-10-26T13:00:04.287785897Z caller=compactor.go:114 msg="Compacting hash" hashString=single-tenant-0-471756
tempo | level=info ts=2023-10-26T13:00:04.287823682Z caller=retention.go:53 msg="marking block for deletion" blockID=e6b37ca3-3656-440e-b5be-b26d36987a0d tenantID=single-tenant
tempo | level=info ts=2023-10-26T13:00:04.287849524Z caller=compactor.go:144 msg="beginning compaction" traceID=2bc10d2585269a6d
tempo | level=info ts=2023-10-26T13:00:04.287932985Z caller=compactor.go:156 msg="compacting block" block="&{Version:vParquet BlockID:2a3bca20-c2a5-42d6-8930-85bf39a2b942 MinID:[0 0 0 0 0 0 0 0 3 86 103 128 102 23 254 249] MaxID:[0 0 0 0 0 0 0 0 126 41 215 207 95 8 128 211] TenantID:single-tenant StartTime:2023-10-26 12:47:34 +0000 UTC EndTime:2023-10-26 12:52:39 +0000 UTC TotalObjects:106 Size:30627 CompactionLevel:0 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5912}"
tempo | level=info ts=2023-10-26T13:00:04.287982015Z caller=retention.go:76 msg="deleting block" blockID=905815da-185b-4906-aeeb-e5d361e1ca06 tenantID=single-tenant
tempo | level=info ts=2023-10-26T13:00:04.288034606Z caller=compactor.go:156 msg="compacting block" block="&{Version:vParquet BlockID:0e59ed85-1aaf-402a-a6ca-862a96da9400 MinID:[0 0 0 0 0 0 0 0 1 113 77 138 43 216 204 78] MaxID:[0 0 0 0 0 0 0 0 125 78 214 143 39 204 155 170] TenantID:single-tenant StartTime:2023-10-26 12:52:44 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:113 Size:31763 CompactionLevel:0 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5914}"
tempo | level=info ts=2023-10-26T13:00:04.288458898Z caller=retention.go:76 msg="deleting block" blockID=acf6c3dd-6b65-42f7-b806-731010673037 tenantID=single-tenant
tempo | level=info ts=2023-10-26T13:00:04.300603214Z caller=compactor.go:253 msg="wrote compacted block" meta="&{Version:vParquet BlockID:e669e7b7-194a-474b-99a0-0add91b6cd3a MinID:[0 0 0 0 0 0 0 0 1 113 77 138 43 216 204 78] MaxID:[0 0 0 0 0 0 0 0 126 41 215 207 95 8 128 211] TenantID:single-tenant StartTime:2023-10-26 12:47:34 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:219 Size:47625 CompactionLevel:1 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5918}"
tempo | level=info ts=2023-10-26T13:00:04.300970724Z caller=compactor.go:223 msg="compaction complete" elapsed=13.075942ms block="&{Version:vParquet BlockID:e669e7b7-194a-474b-99a0-0add91b6cd3a MinID:[0 0 0 0 0 0 0 0 1 113 77 138 43 216 204 78] MaxID:[0 0 0 0 0 0 0 0 126 41 215 207 95 8 128 211] TenantID:single-tenant StartTime:2023-10-26 12:47:34 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:219 Size:47625 CompactionLevel:1 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5918}"
tempo | level=info ts=2023-10-26T13:00:04.301005017Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant
loki | level=info ts=2023-10-26T13:00:04.722328962Z caller=table_manager.go:171 msg="uploading tables"
loki | level=info ts=2023-10-26T13:00:04.722389166Z caller=table_manager.go:208 msg="syncing tables"
loki | level=info ts=2023-10-26T13:00:04.722509365Z caller=table.go:336 msg="uploading table index_19654"
loki | level=info ts=2023-10-26T13:00:04.722534718Z caller=table.go:363 msg="finished uploading table index_19654"
loki | level=info ts=2023-10-26T13:00:04.722557486Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19654"
loki | level=info ts=2023-10-26T13:00:04.72263131Z caller=table.go:336 msg="uploading table index_19655"
loki | level=info ts=2023-10-26T13:00:04.722652472Z caller=table.go:363 msg="finished uploading table index_19655"
loki | level=info ts=2023-10-26T13:00:04.722674961Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19655"
loki | level=info ts=2023-10-26T13:00:04.722764639Z caller=table.go:336 msg="uploading table index_19656"
loki | level=info ts=2023-10-26T13:00:04.72279516Z caller=table.go:363 msg="finished uploading table index_19656"
loki | level=info ts=2023-10-26T13:00:04.722825751Z caller=table.go:421 msg="cleaning up unwanted dbs from table index_19656"
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T13:00:14.241609463Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=192.168.1.154 time_ms=3 duration=3.449022ms size=0 referer= handler=/api/live/ws
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T13:00:25.376290029Z level=error msg="Failed to load home dashboard" error="open /var/lib/grafana/dashboards/exported_dashboard.json: no such file or directory" remote_addr=192.168.1.154 traceID=
grafana | logger=context userId=1 orgId=1 uname=admin t=2023-10-26T13:00:25.376380125Z level=error msg="Request Completed" method=GET path=/api/dashboards/home status=500 remote_addr=192.168.1.154 time_ms=2 duration=2.165879ms size=56 referer=http://192.168.1.98:3333/ handler=/api/dashboards/home
tempo | level=info ts=2023-10-26T13:00:34.287977065Z caller=compactor.go:101 msg="starting compaction cycle" tenantID=single-tenant offset=0
tempo | level=info ts=2023-10-26T13:00:34.288031402Z caller=compactor.go:114 msg="Compacting hash" hashString=single-tenant-1-471756
tempo | level=info ts=2023-10-26T13:00:34.288089231Z caller=compactor.go:144 msg="beginning compaction" traceID=58d7d30fee54d890
tempo | level=info ts=2023-10-26T13:00:34.288185125Z caller=compactor.go:156 msg="compacting block" block="&{Version:vParquet BlockID:a7ec18f5-c682-4388-b5a8-5ba2221c1194 MinID:[0 0 0 0 0 0 0 0 0 16 251 67 133 159 41 252] MaxID:[0 0 0 0 0 0 0 0 127 178 187 27 138 40 154 162] TenantID:single-tenant StartTime:2023-10-26 12:37:14 +0000 UTC EndTime:2023-10-26 12:47:29 +0000 UTC TotalObjects:212 Size:46737 CompactionLevel:1 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5916}"
tempo | level=info ts=2023-10-26T13:00:34.288410785Z caller=compactor.go:156 msg="compacting block" block="&{Version:vParquet BlockID:e669e7b7-194a-474b-99a0-0add91b6cd3a MinID:[0 0 0 0 0 0 0 0 1 113 77 138 43 216 204 78] MaxID:[0 0 0 0 0 0 0 0 126 41 215 207 95 8 128 211] TenantID:single-tenant StartTime:2023-10-26 12:47:34 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:219 Size:47625 CompactionLevel:1 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5918}"
tempo | level=info ts=2023-10-26T13:00:34.314774014Z caller=compactor.go:253 msg="wrote compacted block" meta="&{Version:vParquet BlockID:e162068f-e281-4b2b-8114-75ae819c617f MinID:[0 0 0 0 0 0 0 0 0 16 251 67 133 159 41 252] MaxID:[0 0 0 0 0 0 0 0 127 178 187 27 138 40 154 162] TenantID:single-tenant StartTime:2023-10-26 12:37:14 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:431 Size:81046 CompactionLevel:2 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5949}"
tempo | level=info ts=2023-10-26T13:00:34.315134958Z caller=compactor.go:223 msg="compaction complete" elapsed=26.997885ms block="&{Version:vParquet BlockID:e162068f-e281-4b2b-8114-75ae819c617f MinID:[0 0 0 0 0 0 0 0 0 16 251 67 133 159 41 252] MaxID:[0 0 0 0 0 0 0 0 127 178 187 27 138 40 154 162] TenantID:single-tenant StartTime:2023-10-26 12:37:14 +0000 UTC EndTime:2023-10-26 12:57:49 +0000 UTC TotalObjects:431 Size:81046 CompactionLevel:2 Encoding:none IndexPageSize:0 TotalRecords:1 DataEncoding: BloomShardCount:1 FooterSize:5949}"
tempo | level=info ts=2023-10-26T13:00:34.31516974Z caller=compactor.go:107 msg="compaction cycle complete. No more blocks to compact" tenantID=single-tenant

Overview

@LexLuthr
Copy link
Collaborator

You can ignore the default dashboard message. It looks like your data points might not be connected. Please check that metrics are being reported on the ports and these ports are reachable from the prometheus container. Also, match that the ports exposed are the ones being scraped for metrics.

@stuberman
Copy link
Author

Please explain how I do that, all of these configurations are inside Docker it seems and I am not familiar with configuring and troubleshooting Docker.

@nonsense nonsense removed this from the Boost v2 stability milestone Oct 30, 2023
@LexLuthr
Copy link
Collaborator

LexLuthr commented Nov 2, 2023

@stuberman We have made some change to the monitoring stack and the documentation as part of the new v2.1.0-rc3. Can you please check https://boost.filecoin.io/monitoring and see if the new information helps you get across the line?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nonsense nonsense

Labels
support Support Ticket
Projects
Boost
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nonsense @stuberman @LexLuthr