Clarify guarantees on message delivery

[gl-jkeys]

Is your feature request related to a problem? Please describe.

I do not see any description of the guarantees about message re-delivery.

If I attach an event listener to my worker, and the worker node crashes in the middle of processing a message, will the message get re-delivered to another worker? Or is that message lost forever?

Describe the solution you'd like

Please add some documentation around message delivery guarantees.

If messages are lost forever if the worker node crashes in the middle of message processing, this should be clearly documented and there should be recommendations for how to recover from missing messages.

Ideally, there should be an optional (if configured) synchronous ack method on the message, and the message should redeliver if this ack function is not called within the configured message lock period.

Describe alternatives you've considered

• enqueueing messages to a separate dead letter queue for re-processing if necessary
  • not a "correct" solution, because the message handler might throw between the time that the message is received and the call to enqueue to the dead letter message. This minimizes the problem but does not remove it.

Additional context
N/A

Edit: to clarify, when I say messages, I mean the job event messages that are emitted to the worker or QueueEvents event listeners. I'm not referring to Job redelivery, which is documented.

[manast]

It is documented here: https://docs.bullmq.io/
and here: https://docs.bullmq.io/guide/jobs/stalled

[gl-jkeys]

Hi, can you cite the passage that answers my questions? I don't think they are documented.

[gl-jkeys]

If you are saying that event messages -- e.g. worker.on('failed', msg => {/*...*/}) -- shares the same semantics as new Worker(namespace, handler, opts), then that's great, but it's not documented anywhere.

It's also not documented whether any re-delivered message would be re-delivered to any worker, or only workers attached to the same namespace.

[manast]

Events, as any events in Node, do not provide any guarantees. The processor in the workers is what gives you the mechanism to achieve at-least-one semantics. This is documented if you really care to look into it. There are also loads of tutorials with much more information here: https://blog.taskforce.sh/

[gl-jkeys]

What is the purpose of the event listeners, if BullMQ makes no semantic guarantees about redelivery or at-least-once delivery for them? It seems like they are not useful in that case.

The processor in the workers is what gives you the mechanism to achieve at-least-one semantics.

Okay, recapping to make sure I understand this:

• BullMQ guarantees that jobs will be re-delivered until success (or retry limit is reached, or UnrecoverableError);
• BullMQ does not guarantee that event listener messages about any given job will be re-delivered.
• QueueEvents / worker event listeners, despite being offered by BullMQ, should not be used for any transactional work, or work in which losing the message is very bad for the consistency of the system.
• instead, all transactional work must be done inside the job handler.
• based on the above, If the batch processor crashes in the middle of the job handler, after some initial "start" transaction has been committed (imagine that I have a long running process that is sandwiched between two transactions), then I need to have a separate rollback mechanism outside of BullMQ.
  • Or, I need to make my entire worker process idempotent.
  • Or, I would need to keep a single database transaction open for the entire worker handler lifetime.

A lot of this appears to be problematic for my system. In my case, I either need to rollback the initial "start" transaction outside of BullMQ somehow (I guess I'll have to build a cronjob into my system?), or I need to keep a transaction open for the entire time it takes to process the batch, which could be several minutes. The former requires building extra infrastructure, writing extra code, etc. The latter will likely lead to a myriad of database problems, such as overusing the precious pool of connections, deadlocks, and other problems that occur from having many long-running transactions concurrently.

(I also think that the lack of re-delivery of the Node events is an undocumented footgun. It's not obvious that transactional work should not be done inside the Node event listeners. I again suggest that the delivery guarantees are under-documented.)

My suggestion

It seems like it would be simpler for BullMQ to add the ability to ack messages delivered to queueEvents.on('failed') / worker.on('failed'), and default to re-delivering messages if they are not successfully acknowledged. This seems to be a common pattern in many queueing systems.

For instance, add this ack parameter to every event listener:

worker.on('failed', (job, err, ack) => {
try {
    // do some transactional cleanup with `job`, then:
   ack()
  } catch (e) {
    // handle error and/or do not ack, so the message is re-delivered
   }
})

[OsoianMarcel]

I have similar problems with my project.

[manast]

What is the problem exactly? To make things clear, the last post of @gl-jkeys just overcomplicate things, I don't really understand why this is the case. You get all the guarantees that you need by using the job handler, events are for purposes such as debugging, signaling other systems about statuses (where guarantees are not that important, for example a progress event for jobs in a UI, etc). If your job is composed of several operations you can divide the job in smaller steps and/or make every job idempotent... nothing special here in BullMQ really, same as you would do with any other queue system.

[OsoianMarcel]

According to the documentation:

The QueueEvents class is implemented using Redis streams. This has some nice properties, for example, it **provides guarantees** that the events are delivered and not lost during disconnections such as it would be the case with standard pub-sub.

The "QueueEvent" provides guarantees that the message will be delivered, but it does not provide a guarantee that the message will be re-delivered on failure. In my opinion, this moment should also be mentioned in the documentation. I & @gl-jkeys expect that the failure event will be redelivered on failure, so we can update the failure status in another places (like databases).

Also, I would also add to the documentation a mention, that the event will be delivered to all consumers. So, having 5 consumers, the event (for example, "failed") will be delivered to all 5 consumers. If we want to process the event only once, this needs to be taken into consideration.

---

What is the right way to handle the job failure and make sure the failure event is not lost in case of an unexpected temporary error?
Thank you.

[manast]

@OsoianMarcel The event guarantees are mostly useful if you want to implement for example a UI where you want to show some status and you want to be sure that you can show the status (progress for example) accurately without missing any event. This is because you can use the lastId of the stream to know which was the last event consumed (as specified by Redis streams).

Regarding failure handling. So basically if the fail event had some kind of retry mechanism, what would happen if that handling of the fail event also failed? you would need a fail event for the fail event and so on, you just move the problem further away but it is still there.

When dealing with unavoidable job failures, a practical solution involves wrapping the process function in a try/catch block and implementing a response in the catch handler, though this too is not foolproof as the catch handler could fail. The most effective strategy for handling jobs that fail permanently is to review these jobs in a separate system. This review can be conducted manually or through an automated process, such as using another job (potentially a repeatable one) to address each failed job in the failed set.

In practical terms, failed jobs should ideally be very infrequent occurrences. When they do occur, the appropriate response is to analyze the circumstances leading to the failure and address these underlying issues. Once the causes of the failures are rectified, the failed jobs can be retried manually, akin to the process used in a dead-letter queue pattern. This approach emphasizes problem resolution at its root, ensuring that similar failures are less likely to recur in the future