Make nightly job kick of child jobs and wait until all are finished before completing

[sbruinsje]

Is there a way to make a job kick-off child jobs dynamically, wait for those child jobs to finish and only then make the parent job finish? I know there's flows, but for what I understand with flows you define the child jobs statically. What I am looking for is way to define a job that iterates over documents in a database and create child jobs for some of those documents in another queue.

To give a bit more context:

• I have a queue for some nightly jobs that I run: lets call that queue nightly-jobs.
• Some of these nightly jobs are called: delete-obsolete-items, enrich-items, etc.
• The enrich-items job should run after delete-obsolete-items has finished, because otherwise I might be enriching the items eventhough they will be deleted anyway (enriching them has a cost).
• The delete-obsolete-items job is a job that finds items that are obsolete and then deletes them. This delete operation is rather complex and they require to be executed in isolation (e.g. one at a time). I want the delete-obsolete-items job to kick off a seperate child job for each item to be deleted so that they can be retried individually when they fail and to make sure only one operation runs at the same time. Lets call this queue isolated-operations.

How can I make the delete-obsolete-items kick of child jobs in the isolated-operations queue and wait for all child jobs to be finished before finishing itself?

[roggervalf]

Hi @sbruinsje, looks like you may need to check our step jobs patterns https://docs.bullmq.io/patterns/process-step-jobs#waiting-children

[sbruinsje]

Ah thanks, that indeed looks like it could be what I am looking for. I am not entirely sure I understand how it works though.

This page of the docs shows the state transitions for a job. Is it the case that by throwing the the WaitingChildrenError error, the job is moved back to the waiting-children state? If so, doesn't that mean that other jobs in the same queue (as my parent job) will start being processed while the job is waiting for the children to finish? Or does the job stay in the active state? In that case, I assume the worker's processor function is executed in its entirety again as soon as its children finish? Is the worker idling in the meantime (which is what I want)?

I am also not entirely sure what the job.moveToWaitingChildren() method does exactly. From the link you provided it looks like it just checks if there are any children to wait for and if that's the case the worker stops by throwing the WaitingChildrenError. The page also mentions that moveToWaitingChildrent "unlocks" the job, but wouldn't it make more sense to make the job unlock when throwing the WaitingChildrenError?

[roggervalf]

hi @sbruinsje, sorry for the delay. job.moveToWaitingChildren() method moves the job to waiting children only if there are no pending children, this is why this method returns a boolean, to know if we moved the parent job to waiting-children state. When it returns true, we must throw WaitingChildrenError to prevent calling moveToFinished script and get missing lock error as we already moved the job to another state (waiting-children) and we delete the lock, this is happening in this way because we only keep the lock when job is in active state. Having this in place, when moving the parent to waiting children, workers are not blocked and they can pick other jobs to be processed. As soon as children are finished, parent will return to wait state and then a worker will be able to pick it and pass it to active again. Hope this clarify a little bit how the example works

[theDanielJLewis]

Is this similar to what I'm looking for, too? #1986

[sbruinsje]

Yes indeed we seem to want/need the same thing but maybe for a slightly different reason. You need it for monitoring the total duration of the parent job. I need it because I don't want other jobs to be picked up from the queue of the parent job until the parent job is finished including all its children.

Possibly roggervalf's answer might be the solution, but I am not sure. That feature is barely documented so I don't understand exactly the state things are in when calling job.moveToWaitingChildren() and then throwing the WaitingChildrenError error.

[theDanielJLewis]

@sbruinsje, I implemented the "grandparent queue" approach I suggested in #1986 and it's working well for me. I'll put more details in there.