Dynamically change the concurrency of workers running across multiple node instances

[prolific]

Setup:
I have a queue named Paint and two node instances running the Worker for Paint queue with a concurrency of 10 each (i.e total 20 jobs are going to be processed parallel)

Problem
Now through a master node instance, I want to change the concurrency of these two Paint workers (running on two different node instances) from 10 to 20 so that total 40 jobs can be processed.

Possible Solutions with issue

1. I though of fetching the list of workers using queue.getWorkers() and then putting a job on each worker specific queue to change concurrency. To generate worker specific queue i thought I can somehow create dynamic queues with a name in the following pattern: workerqueue-${workerId} and then have each worker process from there own specific queue. But the id from getWorkers is not matching with the id returned by worker.id.
2. Another option i thought was to create uuid based queue name for each node instance so that the workers of that specific instance can process the jobs to self configure but the problem is I don't know how to notify the master server with the list of all node instances and the respective queue names. One thing I can do on the master server is to fetch the list of all queues and filter out instance specific queues by using a specific naming pattern.
3. Another solution is that i create instance specific queue at the time a new node instance is started by following a specific pattern and then add a repeatable job on it which continuously check of concurrency setting at repeated intervals and configure. But this obviously is kind of a polling mechanism which I am trying to avoid.

In short I am trying to configure the concurrency of a worker running on one node instance from another node instance (master instance). Is there a way to do this?

Also, is there a way to add meta information to each worker like use our own ID which can become some sort of identification for another node instance. Something we can fetch while calling queue.getWorkers ?

[manast]

As you have already suspected there is no built-in mechanism to send commands to workers. This would require an extra communication channel, and I feel it falls outside of the scope of BullMQ at the moment.

[prolific]

@manast I can understand.

But to help with building the communication channel like this can you please tell if it is possible to get the list of workers for a queue with an ID that the worker is also self aware of.

Currently the id returned by queue.getWorkers (from the master instance) does not match with the id returned by worker.id (in the worker instance).

getWorkers returns an integer id while worker.id returns a uuid.

Regarding the scope:
Though it is not necessary if not already available for me but wouldn't it be a nice feature to allow configuring workers across node instances from a single instance. Alternatively, the right words would be to allow setting global concurrency of workers.

[manast]

I have planned for a while that the worker's id can be used to identify both workers and jobs being currently processed by a given worker. Have not prioritized it up yet though.