Dashboards for Forest website

[LesnyRumcajs]

Issue summary

A little bit of background - we plan to ship a Forest website later this year (see Forest Roadmap 2023). There, we plan to embed some dashboards that will show the status of our long-running nodes.

The dashboards on the website should be in the form of a summary for slightly-but-not-overly-technical people. That is to say; they should expect a small level of technical competence (e.g., understanding the difference between CPU and memory usage) and Filecoin arcana knowledge (epochs, peers), but not details (like the number of ChainExchange p2p messages).

At the moment, we have some basic New Relic dashboards migrated from Grafana. They are a good start, but we need more structure.

Beforehand, we must consider whether we can easily embed the dashboards in New Relic or Grafana on the website. If NR is not possible (to my understanding, it should be fine - given a web application, we could use NR API to request a PDF of the dashboard and embed it in the website), we can consider opening a Grafana endpoint (which would require additional work). Alternatively, the web application could manually generate the dashboards using raw NR API data, which would put more effort on the team responsible for the website.

Tasks

• Investigate if and how NR offers to make dashboards public. Summarize the findings in this thread.
• If the NR investigation proves that it's not feasible to do with NR, consider local Grafana (or Grafana Cloud?),
• Discuss with the team on the best approach,
• Change the dashboards. They should be more structured, with several pages, the first one being an executive summary with only the following metrics #236
  • Head epoch,
  • Tipsets validated per minute (this should be always ~2),
  • Number of connected peers,
  • Process uptime (single number)
  • CPU usage,
  • Memory usage (Process Resident Memory and NOT virtual memory),
  • Disk usage
    All metrics should be absolute, i.e., disk usage should not show 50% because it means nothing without knowledge of the disk size.
• Additional pages with more detailed metrics. This is not directly related to the website functionality, so we can do it later. For now, we can just put the current metrics there.
• Ensure that the results on every chart make sense, both in shape and value,
• Present the dashboard to the team; @lerajk acceptance is required. Implement requested changes.
• Exactly the same dashboard (with the executive summary) should be created for Lotus, so both nodes are easily comparable. #238
• Prepare a short guide for the website team on integrating the dashboard into the website.

Acceptance criteria

• Forest and Lotus dashboards are publicly available,
• Data retention period is at least six months, #239
• Dashboards are structured, with the first page/tab offering an executive summary of the node.
• A guide for the website team is ready.

Other information and links

Feel free to split this issue into subtasks. Some can be worked on in parallel.

[samuelarogbonlo]

Based on findings highlighted here in the New Relic community forum, directly publicizing an entire New Relic dashboard isn't possible. Given this, I've explored alternative strategies:

• One solution involves generating embeddable links for specific charts. By incorporating these links into a public web page, users can view individual charts on our site, eliminating the need to visit New Relic directly.

• Additionally, New Relic's NerdGraph, a GraphQL API, provides a nuanced method for data queries. we can use NR API to request a PDF of the dashboard and embed it in the website but the down side is that this is going be static dashboard.

[samuelarogbonlo]

Based on findings highlighted here in the New Relic community forum, directly publicizing an entire New Relic dashboard isn't possible. Given this, I've explored alternative strategies:

• One solution involves generating embeddable links for specific charts. By incorporating these links into a public web page, users can view individual charts on our site, eliminating the need to visit New Relic directly and this would required some manual intervention
• Additionally, New Relic's NerdGraph, a GraphQL API, provides a nuanced method for data queries. we can use NR API to request a PDF of the dashboard and embed it in the website but the down side is that this is going be static dashboard.

After considering everything, I don't believe we should adopt any of the proposed solutions. First, I'm doubtful about our ability to view these dashboards in real-time, especially when compared to the direct view on New Relic. Second, while we could use the New Relic API to fetch a PDF version of the dashboard for the forest website, this approach would make the dashboards feel more static.

With all that being said, i think a better approach is to consider using Grafana Cloud or Local Grafana( if Grafana Cloud free tier is not enough for our use case) and then configure new relic as a data source and opening the Grafana endpoint (which would require additional work).

[samuelarogbonlo]

Based on findings highlighted here in the New Relic community forum, directly publicizing an entire New Relic dashboard isn't possible. Given this, I've explored alternative strategies:

• One solution involves generating embeddable links for specific charts. By incorporating these links into a public web page, users can view individual charts on our site, eliminating the need to visit New Relic directly and this would required some manual intervention
• Additionally, New Relic's NerdGraph, a GraphQL API, provides a nuanced method for data queries. we can use NR API to request a PDF of the dashboard and embed it in the website but the down side is that this is going be static dashboard.

After considering everything, I don't believe we should adopt any of the proposed solutions. First, I'm doubtful about our ability to view these dashboards in real-time, especially when compared to the direct view on New Relic. Second, while we could use the New Relic API to fetch a PDF version of the dashboard for the forest website, this approach would make the dashboards feel more static.

With all that being said, i think a better approach is to consider using Grafana Cloud or Local Grafana( if Grafana Cloud free tier is not enough for our use case) and then configure new relic as a data source and opening the Grafana endpoint (which would require additional work).

So, one thing to keep in mind is that using New Relic as a data source has its drawbacks. It doesn’t give us all the system-level metrics like CPU usage and storage. That means we’ll probably need to come up with our own way of getting these metrics for our Grafana dashboard.