How to get around "nestjs queue job stalled more than allowable limit"

[jmobes]

Let me explain the problem we are trying to solve. Our application is used for managing cargo ship voyages. There are many data points the user wants to keep track of during a voyage (speed, fuel consumption, distance traveled, ETC). User wants to be able to select any number of ships in their fleet and export all this data into an excel spreadsheet within a given time range.

We are using bull to solve this problem by sending a job to the queue that has two consumers. One consumer to execute the api call(s) (1 api call for each ship selected), and one consumer to format the data and create an excel file.

This strategy works well most of the time. However, for large number of selected ships (300+), the job will not finish and return the error "nestjs queue job stalled more than allowable limit". We used to get an error regarding "missing job lock" but have not encountered that for a while after increasing lock duration. I think running all the api calls in parallel or formatting all the data, might be causing the CPU to work too hard and create a stalled job.

One potential solution that I am thinking is instead of running all api requests in parallel, perhaps creating a new queue job for each api call and then processing them one by one. Only after all of these api call jobs have been processed by the first consumer, does it then get sent to the other consumer that handles creating the xlsx file. This would require the queue to be able to identify which jobs belong to the same batch.

Does bull support this kind solution? Or is there another way to solve this issue? I am open to suggestions.

[roggervalf]

hi @jmobes, did you have the time to review flows docs https://docs.bullmq.io/guide/flows ? , in case you may need this kind of parent-child dependency, we also have a pattern for dynamic flows https://docs.bullmq.io/patterns/process-step-jobs#waiting-children