How to ensure ordering of jobs in case of delayed retries.

[ksyd9821]

Hello,

for our current use case, we are adding jobs to a queue and would like them to be successfully processed in the same order they were added (FIFO).
So when a job fails, the desired outcome would be that the job is retried automatically after some delay before moving on to the next job on the queue.

For example, if we have these 3 jobs in our queue [1, 2, 3] with job 1 being the first added job, here is what a possible execution would look like:

• process job 1
• job 1 fails
• X amount of delay
• process job 1 again
• job 1 completed
• process job 2
• process job 3

How can we achieve this with bullmq?

Thank you in advance for the support!

[manast]

Thank you for your question. I am afraid that currently, when a job fails, the queue is not halted, so the other jobs waiting to be processed will be processed as soon as a worker is free.
How critical is this case for you? can you develop a bit more the whole scenario where this functionality is needed?

[hardcodet]

@manast

How critical is this case for you? can you develop a bit more the whole scenario where this functionality is needed?

It is critical, unfortunately. Our use case is a number of event queues for webhooks (each queue representing a customer's subscription), where we would like to submit events in proper order. We see that in practice, webhooks are sometimes not working (e.g. the customer's endpoint is temporary available) and need to be retried, but we can't have those events move to the back of the queue, because order matters.

As a dummy example: imagine two events occurring in this order:

1. The system is offline
2. The system is online

If we would send these events in inverted order, the outcome on the customer end would be completely wrong, since they assumed the system is offline, and might cease communication to it.

[manast]

Ok, so this function would be specific for groups, where a group would not continue processing new jobs until the previous one have been completely completed or failed, furthermore this feature would only make sense with concurrency equal 1.
We need to study to see how feasible this feature is in current design.

[hardcodet]

You're right. We're already using concurrency of 1 extensively to enforce sequential processing because there's a lot of cases for us that warrant that. Preserving order on retries is just one flavor more.

If it's a bad fit for BullMQ, we could work around the issue with the following strategy I guess:

• handle the error ourselves, and
  • pause the queue
  • mark the failed job as completed
  • create a new job with the same payload and enqueue it LIFO
• re-enable the queue after the retry delay

This is absolutely feasible for us. We just figured that ordered processing (including retries with backoff delays) would be a common scenario, so we wanted to discuss this with you first 👍

[hardcodet]

It wouldn't be specific for groups though: we thought about creating a queue for each customer (rather than groups with a customer ID), which would reduce the complexity for the retries remarkably compared to queues that still would have to process events for other groups.

[manast]

We are working on a solution for this in BullMQ and then we will extend it to groups as well, this is the PR: taskforcesh/bullmq#2465

[hardcodet]

You guys rock! Looking forward to the implementation :)