Understanding the round robin behavior between processes

[thetutlage]

I am just trying to understand how the Round robin works between processes. Quoting from the docs

What I understand is that if I run two Node processes and each creates a worker instance using new Worker with concurrency set to 1. Then new jobs will be sent to both the processes in round robin fashion.

• Job 1 ---> Process 1
• Job 2 ----> Process 2 (even if Process 1 is idle )
• And continues in the same fashion

However, testing it shows different results altogether.

In the shared video I have two processes running to process the jobs and then one process adds one job at a time to the queue. I am waiting for 4-5 seconds to ensure that the process which processed the job becomes idle.

round-robin-bull.mov

Code for adding a new job

const { Queue } = require('bullmq')

const name = process.argv.slice(2)
const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379')
videoQueue.add('wall', { name })

Code for processing the job

const { Worker } = require('bullmq')

const name = process.argv.slice(2)
console.log(name)
new Worker('video transcoding', function (job) {
  console.log(job.data)
  console.log({ name })
})

[manast]

Hmm. But in the video you are starting the workers manually. If you have a fleet of workers online waiting for jobs to arrive to the queue the jobs will be distributed in a round-robin fashion, this comes from the fact that idle workers are waiting for jobs using Redis BRPOPLPUSH command, and Redis do wake up the blocked connections in the same order as they were created (the blocked connection).

[thetutlage]

In the video the workers are already started in the right split. I am closing and starting the process which creates a new job

[manast]

ah, ok sorry I checked this way too quick. I will check this on my end, it Redis is working as advertised then it should be distributed in a round-robin fashion, I never verified this myself just assumed it was correct, will do it now.

[manast]

Ok, so the issue is that the default "drainDelay" is too low for this example to behave as you expect. If you change it to 30s it will behave correcty:

new Worker(
  "video transcoding",
  function (job) {
    console.log(job.data);
    console.log({ name });
  },
  { drainDelay: 30 }
);

[thetutlage]

Thanks for looking into it.

Can you please point me to the docs of drainDelay. Unfortunately I am not able find it

[manast]

There is not a lot of documentation for some options, unfortunately. Normally there is no need to change the default values: https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.workeroptions_2.draindelay.md

In any case, that is the "drainDelay" is the amount of time in seconds that the connection is blocked waiting for a job to arrive.