Dependent Repeat Queue

[leegunwoo98]

I am using the bullmq to repeatedly get user's social data through youtube, twitter tikok etc's official API. However, since there are multiple cases of failures (authentication failures etc) there could be cases where the job is impossible to finish even after multiple attempts. If the repeat continues, it would exhaust unneccessary quotas that we have on the socialNetwork API. In addition, if job A starts proccessing and job B is generated as the next repeated job, my current project requires that job B should not be processed before job A. In the current schema, since job B is not blocked by job A, it is possible that job B can be completed before job A is completed if there are multiple attempts on job A. As a result, we need a repeated queueing system that is dependent on the completion of the previous job in order to start processing.

One solution that we came up with is to have a flow so that the next repeated job is a parent of the previous repeated job. If this is the case, the next repeated job would have to wait until the current repeated job is finished. However, we had trouble getting the key of the next repeated job when the current job starts. I think it can be possible to implement this in the nextJobFromJobData function inside the worker class, and if there is an repeat option where dependOnComplete is true, it creates a flow to wait for the current job to finish.

We also tried throwing a delayedError (without a repeat option) instead of adding a new job. The reason for this was that there was no insurance that a callback event after job finished would always work to add a new job. However, we ran into an issue where the attempts number didn't change and when the attempts number accumulated due to random errors, it failed the entire job. It can work if we had an option to change the number of attempts.

[leegunwoo98]

If we have an option to implement this dependent repeated Queue system, it would greatly improve many system that implements third party APIs that are reliant on Quota systems as it can greatly reduce unnecessary calls.

[leegunwoo98]

@manast can you give a quick comment on this idea?