Migrating from AgendaJS to BullMQ

[wootwoot1234]

I'm migrating from Agenda to BullMQ and I was hoping to get some feedback from others that have experience with both (or just BullMQ).

I have 3 reasons for moving over to BullMQ:

• BullMQ has better job failure features including retry on failure which I'm looking forward to.
• BullMQ seems to have some good management UI options which is another aspect I think Agenda is missing.
• Reduced the DB costs. I have about 100k jobs running, about 70k of them are one-time jobs scheduled as far as 2 years in advance and 30k of the jobs are repeating jobs that run every 5 min to every 24 hours depending on the job. I'm hoping that Redis will be less expensive than MongoDB (both hosted using managed hosting)

Will 100k jobs be an issue with Redis/BullMQ?

It seems like Redis is better designed for job management than MongoDB. I'm hoping it will translate into lower hosting fees. Will it?

Is there anything I can do to reduce DB costs? For example, only schedule jobs that are 30 days in advance and create a system to add jobs for anything great than 30 days when their execution date is within 30 days?

[manast]

100k per which unit of time? Also, the amount of jobs per second depends on how much work you do per job and your concurrency settings, it is not difficult to achieve several thousand per second. Let us know how it goes and if there are things we can improve to make the transition easier.

[wootwoot1234]

It's hard to say without adding them all up, there are many job types that run at different intervals and the Agenda UI doesn't give info like that.

If I look at the locked jobs on agenda (aka jobs that are running) there are generally about 300 at any given time.

My plan is to have 2 servers, a worker server with http endpoints for each job and a second server running BullMQ, it will receive the jobs then call the endpoints.

That way the worker server can scale with the load.

Is that a good way to set it up?

Any idea how large of a redis db I'll need to run a BullMQ with that many jobs?

[alko89]

I've been using Bull for a while and works great for non critical jobs.

Now I'm trying to expend my app to include jobs that will require to run at certain time every month.

What I'm worried with Bull using Redis is persistence, so I'm considering utilizing both at the same time: Bull for non critical jobs that will execute soon and Agenda for long lasting jobs that will run after a month or year and must be persistent in case of unexpected failures.

Would that be a good design, or should I stick to Bull and setup persistence in Redis instead?

[wootwoot1234]

Here are some of my take always from moving to BullMQ.

I'm not sure about persistence but I've found BullMQ to be much more reliable. One issue I have had with AjendaJS is that you can not easily retry failed scheduled jobs.

With BullMQ it's very clear which jobs are running, failed, etc. With Ajenda it's not as clear, The Ajendash UI often shows jobs are running when they are not.

The docs and support are better at BullMQ.

Database fees are lower but that could be due to design changes I've made as well.

[alko89]

You specifically mentioned scheduling jobs up to 2 years in advance. What is your strategy to ensure not loosing these jobs in case of Redis going down?

Documentation on persistence is not really extensive in Bull.

[wootwoot1234]

I’m changing the design. I actually just deployed it to the staging server. I have one job that creates jobs over the next 24 hours. It looks for any jobs that need to run and creates them. The create job job runs every hour so it would need to fail for a full day to not schedule a job. This design is still new to me but I like the idea of not filling the db with jobs that will run in 2 years as well as having to maintain them (ie edit/remove them when things change) I’m also using this for repeating jobs. One issue I had was evenly distributing repeating jobs to level out server load. By using the create job job, I can schedule all repeating jobs over the next 24 hours (as scheduled jobs) and I can evenly distribute the run times. I also don’t need to maintain a repeating job that might need to be removed/edited.

…

On Sat, Sep 10, 2022 at 5:44 AM Aleš Ferlan ***@***.***> wrote: You specifically mentioned scheduling jobs up to 2 years in advance. What is your strategy to ensure not loosing these jobs in case of Redis going down? Documentation on persistence is not really extensive in Bull. — Reply to this email directly, view it on GitHub <#1215 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADGOGIZSQIIRGQZWJBGNUDV5RRBPANCNFSM6AAAAAAQJHPK4E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alko89]

Hmm, I want to create a job for each user and run it every month since its creation. That's why I want the job to be persistent since it will only be removed when and if the user itself is removed.

The jobs would be created with the cron to run each month on day and hour (M H D * *) when the user registered so they would be non evenly distributed in this regard.

[wootwoot1234]

Every use case is different. I also have jobs for each user but depending on their setup each user has a significantly different number of jobs and that gets hard to maintain. I found myself creating jobs to check on jobs which isn’t great. What I found is that as I started to hit server limits causing them to scale it becomes important to distribute server loads by not running all the jobs in clusters which happens if you’re not some how distributing them but I’m still learning and might find issues with this new setup as I move it to production. You have a good point with reliability. It’s probably a good idea to setup some sort of notification when jobs don’t run as expected. It could be something as simple as a job that changes a date filed in the DB that is checked via an api endpoint regularly by a tool like Uptime Robot. It can them text/call you if it fails. Did you have a better idea? I’m really interested in this as my server fees are insane and reliability is very important. I would love to hear from others who have had similar issues and which configuration has worked for them. Feel free to check in with me in a month or two and I’ll report back how it’s going.

…

On Sat, Sep 10, 2022 at 7:03 AM Aleš Ferlan ***@***.***> wrote: Hmm, I want to create a job for each user and run it every month since its creation. That's why I want the job to be persistent since it will only be removed when and if the user itself is removed. — Reply to this email directly, view it on GitHub <#1215 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADGOGIBATEBTTMUMKR7AP3V5R2J3ANCNFSM6AAAAAAQJHPK4E> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alko89]

I've actually decided to store job configs in Mongo and add a disaster recovery endpoint which can be called in case Redis goes down to recover all jobs.

Btw you mentioned there is a UI for Bull. I can't find it, can you share the repo link?

[wootwoot1234]

I use https://taskforce.sh as my go-to UI for bullmq

…

On Fri, Sep 16, 2022 at 2:44 PM Aleš Ferlan ***@***.***> wrote: I've actually decided to store job configs in Mongo and add a disaster recovery endpoint which can be called in case Redis goes down to recover all jobs. Btw you mentioned there is a UI for Bull. I can't find it, can you share the repo link? — Reply to this email directly, view it on GitHub <#1215 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADGOGKHY2IA73HYOOZBPPTV6TEYJANCNFSM6AAAAAAQJHPK4E> . You are receiving this because you authored the thread.Message ID: ***@***.***>