-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Create optional monitoring configuration for Mastodon using Grafana #12
base: main
Are you sure you want to change the base?
Conversation
…nd Prometheus. Includes automatic scraping of kubernetes node_exporter, Mastodon Statsd output, and Postgres data. Includes four default dashboards. Includes ingress changes to create routes for grafana.hostname
@ywwg It looks like Helm is trying to template your txt file 🙃 |
any chance you can re-trigger the workflow? I think I might have fixed it but I can't reproduce the issue locally so I'm not sure |
I am new to helm, but it appears that NOTES.txt is supposed to be a templateable yaml file that helm can process. I believe the version of helm in the CI is too old: prometheus-community/helm-charts#2723 updated helm to 3.7 to see if that fixes it |
@@ -67,5 +68,24 @@ spec: | |||
pathType: Exact | |||
{{- end }} | |||
{{- end }} | |||
- host: {{ printf "grafana.%s" .host | quote }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we shouldn't be setting up the ingress for Grafana, it's better to use their Values.ingress instead of having to maintain a copy of it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good point, I'll work on that
Doesn't it seem a little bit heavy to add 4 more subcharts here? |
That's one reason I wanted to post the in-progress chart, to get a sense of what the goal of this chart is and how full-featured y'all want it to be. For instance, if this chart exists just to deploy mastodon, then it's probably superfluous to include the monitoring stack. I was thinking this chart could be used by admins who want to deploy a solid mastodon instance and are maybe less familiar with running an important service like this. I have seen a lot of messages in my feeds by admins who were surprised to run out of disk, or that the server went down over night. In that light, this chart could be used to help these less-experienced admins set up a more robust instance without having to worry about configuring monitoring themselves. (And those who don't need the monitoring stack or want to roll their own can disable it easily). To answer your question specifically, in order to have a functional monitoring stack we need at least three of the charts: prometheus, which gathers and stores the metrics; the statsd exporter, to record metrics from mastodon; and grafana, to view the metrics. We could easily drop the postgresql chart if we don't care about providing access to postgres stats. It also might help to talk about what "heavy" means in this case -- do you mean the maintenance burden of keeping them functional? |
Hi there, and first of all thanks @ywwg for this PR! I am working on moving mastodon.social & mastodon.online to K8s. For now mastodon.online has been deployed using home-made K8s resources (we needed to move it fast) but we are looking into using the official chart for those servers. We have our existing monitoring deployment, and we dont want to have the Prometheus stack deployed using this chart. But it would probably a good idea to have a way to get the dashboards, and the various Would this be a valid alternative to what you have done?
On another topic, I see you chose to deploy the The last issue I am looking to solve is being able to host multiple Mastodon instances on the same cluster / Prometheus monitoring. This is important for us, so I am looking into adding a label to all Mastodon-related metrics to specify the Mastodon server/instance the metrics refer to, and patching the dashboard to be able to view metrics only from the selected instance. Does this makes sense to you? |
Hi all, this is a draft PR of a change to add optional Grafana+Prometheus monitoring to Mastodon instances deployed with this chart set. The goal is to enable instance admins to very quickly add monitoring with all of the tricky parts of wiring together the various scrapers and config files done automatically.
This is a work in progress, but is functional. The main question I'd like to ask with this PR is, is this a feature the Mastodon project is interested in including? I don't want to do the work of continuing to polish and refactor the config if it's not a direction you want to go. The current major issues are the hardcoded values inside values.yaml that I would like to replace with auto-generated URLs, but I am having trouble figuring out how to do that in Helm and given the available ways to configure the services.
Major TODOS: