Seeking Advice: Integrating bullMQ with Durable Redis in a Highly Scalable System with Long-Term Task History Retention

[ramiloif]

Has anyone here used bullMQ in a highly scalable system and integrated it with durable Redis? Additionally, we have a requirement to retain task history for at least a month. If someone has experience with this setup, what type of persistence mechanism would you recommend for this scenario?

[manast]

I recommend AOF for persistence https://redis.io/docs/management/persistence/
You can configure BullMQ to keep up to a month of completed and/or failed jobs https://docs.bullmq.io/guide/queues/auto-removal-of-jobs#keep-jobs-based-on-their-age
Now the question is how many jobs you will keep in a month and how much memory you will need for the queues to not fill all the memory in your Redis instance. Try to keep the jobs data with as little data as possible, if you need to keep a lot, try to offload to a separate database where space is not an issue, and/or provision a Redis host with enough RAM, remember that persistence is used for failure, you still need to have enough RAM to keep all the jobs in memory, even the completed/failed ones.