Number of active tasks drop and stay dropped

[chrismclennon]

I notice that sometimes the number of active tasks my workers will process will drop, and stay dropped, until I restart my servers. I charted this behavior by measuring queue.getJobCounts() on a regular basis, and found that the number of active tasks went from some reasonable number of active tasks to flatlining at 1 active task.

I do have an error listener on the "error" event that didn't print out any logs during this time, and I suspect that if the jobs were long running they would continue to report as active, but allow the queue to grow.

Any idea for what other angle I might have missed?

Thank you for the help!

[manast]

So in the graph above it is between 21:00 and something like 11:00 where it should not be 1 job right?
What version of BullMQ and concurrency factor are you using?
Also, I would not expect the number of active jobs to fluctuate like in your graph, it should stay mostly on the max concurrency factor as long as the workers are not CPU bounded. Could you provide a simple example code that produces a graph that fluctuates like that?

[chrismclennon]

Yep that's correct! I would've expected the active number of jobs to hold steady at the maximum as long as the queue had a backlog. It kicked back up at 11:00 and 15:00 after I restarted the instances, but this issue has been a recurring one for me.

BullMQ version is 3.10.0 with a concurrency factor of 1. The jobs can be long-running, spanning from a second to several minutes.

I suspect this is an issue with my processor code, but I'm not sure how best to diagnose. My initial suspicion was there was an unhandled error breaking a processor, but I would expect the event listener on "error" should have caught it. Do you have any recommendation of other ways I could investigate?

Beside my actual processor, I don't have a replicable code snippet (yet?). If I'm able to write one I'll follow-up.

[manast]

Let me see, so you use concurrency factor 1, but you have several workers running in different machines then? otherwise, with a concurrency factor of 1 and only 1 worker the strange thing would be to have more than 1 active job at a time 🤔

[manast]

In any case, any code that reproduces the issue where the number of active jobs does not match the concurrency factor or the number of workers would be quite interesting to analyze, so please let me know if you manage to create such a case.

[chrismclennon]

Yep that's correct, each worker has a concurrency factor of 1, but I run many workers. In my specific case, one machine runs several Node processes, so one machine may run something like 10 workers, and I have several machines.

I'll let you know if I'm able to make any interesting cases.