Jobs not being non-deterministically processed upon starting

[xinluh]

Hello, we have a fairly standard worker setup, starting the work by instantiating the worker object:

new Worker('queue_name', job => { console.log('job started'); ... },  { concurrency: 10, connection: {...} })
console.log('worker started')

The issue we ran into is, once in a blue moon, the worker would just not process any jobs at all - no error message, nada just silent (i.e. in the code sample above, we will just not get the job started log at all when this happens). Restarting will always fix it, but it makes our deployment very unreliable and someone needs to monitor the queue processing correctly at every deploy.

he mysterious thing is, we have 4 queues, and 1 queue never runs into this problem, the other 3 are problematic but not always at the same time (sometimes just 1 queue would fail). There is no difference on how the queues were set up, except that the non-problematic queue has only concurrency of 4 instead of 10 and we put significantly more (short) jobs put on it.

Do you have any pointer to where to start investigating this issue? I can confirm that the new Worker(...) line was ran because we do get the log line of worker started. We also subscribe to the worker error worker.on('error', ...) but we get no logs from that either.

[manast]

Well, it could be as simple as there are jobs that stay in active for ever, did you check that that is not the case? maybe your workers are never finishing and therefore no new jobs will be processed. Using a dashboard will help you in getting more insight on what is going on.

[xinluh]

@manast as I mentioned above, we do use taskforce.sh (and posted a screenshot above). But it doesn't quite help us with useful information in this case. I'm going to try to fix the not graceful worker shutdown problem next (it is problematic for other reasons though unclear to me that it could cause the non-deterministic not processing job problem).

And I can try to fully upgrade to bullmq queues just to eliminate one more potential cause - do you have suggestion on how to do this in a non-disruptive way? Sounds like we need to somehow spin up new queues to force the upgrade (even though we are using bullmq library)? We have jobs getting scheduled at every second and it's not an option to stop those jobs from being scheduled.

[manast]

From which version do you need to upgrade?

[xinluh]

We were on bull 4.9.0, then we upgraded to using bullmq (it was 2.1.0 when we upgraded, now we are on 3.6.6), pointing to the same redis and queue names. I didn't realize that that would not upgrade the underlying queue to bullmq queues (actually didn't even realize that the underlying queues are different between bull and bullmq).

[manast]

They are slightly different but besides the operation of pausing, old Bull queues will work with BullMQ.

[manast]

In any case, check the changelog as there are some breaking changes that you should be aware of even if they wont affect you if you are not using rate-limit group keys: https://docs.bullmq.io/changelog#3.0.0-2022-10-25