Centralized Server with BullMQ vs. Direct Connections to Redis Across Microservices

[ramiloif]

My team and I are facing a dilemma with our system, which consists of numerous microservices, most of which are written in Node.js. We're planning to utilize BullMQ for task management across these services.

Our central question is: should each microservice directly connect to Redis and use the BullMQ library, or should we create a single centralized server that wraps BullMQ and dispatches messages to all other microservices?

A significant advantage we see with the centralized server solution is the ease it offers for integrating microservices written in languages other than Node.js.

However, we're curious if there are any potential disadvantages with this approach? Has anyone here implemented something similar? We're particularly interested in hearing about your experiences and any challenges you may have faced.

Thank you for your time, and we look forward to your insights.

[manast]

As you suggest, the advantage of implementing a "proxy" server is that other servers written in different languages can also interact with the queues. However, it is much easier to just use BullMQ directly connected to one or several Redis instances. I would start with the latter and then if needed in the future build a proxy service for the services that require it.

[ramiloif]

@manast Thank you for your kind answer.
We also may lose built-in abilities like a priority, if, for example, our proxy sends the message via a different messaging solution or at least we will need to "translate" the priority to that other MQ, as well as the worker concurrency feature...

When you wrote several Redis instances you meant it like a Redis cluster? or different instances for different queues?

[manast]

I meant different instances, but sure you can use cluster too.

[ramiloif]

Thank you for your help.
And this great project I am willing to contribute, will glad to get some help to contribute or maybe a simple issue to start with .

Btw in this dilema I think it's a philosophic question if you treat bullmq as a task management orchestrator or as a smart message queue ( let's say MQ 2.0 )
Wdyt?
Isn't that the reason it evolved from bull to bullMQ?

[manast]

It is queue, you can use it for job management or for messages, but most people use it for jobs.