docs(self-hosted): external storage configurations

[aldy505]

Sorry for the ping but I need your feedbacks on this. @hubertdeng123 @azaslavsky @stayallive

[vercel]

@aldy505 is attempting to deploy a commit to the Sentry Team on Vercel.

A member of the Team first needs to authorize it.

[stayallive]

Maybe we should generalize to "Data Storage" or something.

This way this document can explain where the data is stored by default and can list alternatives if there are any.

Could have the following sections:

• Sentry (with a general explanation about postgres, clickhouse and kafka maybe)
  • Filestore (Uploads, Replays)
    • Database
    • Object Storage
  • Nodestore (Event data)
    • Database
• Vroom (Profiles)
  • Docker volume
  • Object Storage

We should probably either rename those section to what the specifically store or explain that in the intro because "Vroom" is not very descriptive but if it's explained that that component is responsible for (ingest and) storing profiling data it makes a lot more sense.

Maybe with until someone else also chimes in before rewriting the whole thing in case I'm off base with this outline but this sounds like a document I would love to have had when I started my self-hosted adventures 👍

For the Object Storage thing we might want to link to the relevant documentation instead of adding examples for every option under the sun because otherwise there is no bound to the size of this document.

[hubertdeng123]

I think we can safely assume that people can find that out for themselves if they are looking at this page

[hubertdeng123]

External storage sounds fine. I don't think we should recommend people to use external postgres, redis, etc as that can introduce a lot of issues for people trying to set that up unless they really know what they're doing.

[aldy505]

I agree with "don't think we should recommend people to use external postgres, redis, etc", but if there are some people who wishes to do that... I don't know if there's any other better way to say to them that "they're on their own"

[hubertdeng123]

Just to make sure, have you tried the Azure/s3 compatible backend without issues? We're using GCS so wanted to make sure

[aldy505]

Yes! See my patch getsentry/self-hosted@master...teknologi-umum:sentry:teknum-patch

[azaslavsky]

nit: much easier to review if questions/comments like this are added in the GH comment system, rather than in-line in the PR.

[azaslavsky]

I think we should have a separate page that, rather than "External Services", says something like "Unsupported Workflows".

For example, S3 storage support technically exists, but is a. untested, and b. unused by Sentry internally. So there's no real pressure ensuring that is stays functional over time. Ultimately, what we have at the moment is a (very possibly bit-rotted) thin wrapper around Django's FileStore capabilities. We do not want to indicate to users that it is something we'll offer support for, because realistically we can't offer very good support for it, and folks will be left disappointed.

For this specific doc, we need to be very clear that this is provided as a rough best effort template, and that we offer very limited support for it.

[aldy505]

because realistically we can't offer very good support for it, and folks will be left disappointed.

I know this, but I think the current S3 support is good enough for selfhosted. Users can always pull out their own Django plugins though. Like @stayallive's S3 Nodestore plugin https://github.com/stayallive/sentry-nodestore-s3

we need to be very clear that this is provided as a rough best effort template

I agree. Need to take some time to come up with good enough copywriting for this lol.

[azaslavsky]

What exactly do you mean by "external storage"? Does that essentially mean "storage supplied by a cloud services provider like AWS/GCP/Azure"?

[aldy505]

I think it's something that's not strictly on the same filesystem as the sentry self-hosted instance. But it also excludes if you want to use something like NAS or external bind-mount storage to store Sentry data. For files, it's the blob storage provided by each cloud provider. For databases, it's external database that's either managed or unmanaged, but it should be separate to the sentry self-hosted instance. Do you have any suggestion on how to better phrase this out?

[azaslavsky]

nit:

Suggested change

	## Filestore
	## Django Filestore

Sentry (confusingly) maintains a separate service called Filestore which acts as an intermediate layer in front of ex GCP, though we don't really recommend this for self-hosted use.

[azaslavsky]

I would call the proposed page "Integrating with Major Cloud Providers" or something similar, just to make it clear that we are specifically referring to GCS/AWS/Azure.

[azaslavsky]

nit (here and elsewhere): change the value from foobar to something else, to make clear that the two keys will be different in practice. I would suggest something like your_secret_key or similar.

[azaslavsky]

My inclination is to remove Azure for now, and indicate somewhere in this document that there are no shims for it atm. Since we don't have a Filestore shim for Azure, in practice it will be very hard to run vroom on Azure, and users will likely work themselves into a corner if they try.