Bigger project structure

[Dammmien]

Hi 👋

I need some advices about project structure, I read several examples and tutorials but use cases are always pretty simple.

How things are organized when a project grows up ?

Is it better to use queues as a router and so have a lot of queues each dedicated to a specific job or is it better to have a few queues and add a router logic based on job name ? For example if I have 10 different scheduled tasks: compute stats every month, send notifications every monday, update state of items in a DB every night, ... should I have one queue dedicated to scheduled jobs and process them differently based on their name or should i have 10 queues ?

Do I need a server (I mean an instance) dedicated to a specific queue or can I have an instance with multiple worker processing multiples queues ?

Generally speaking, do you have any examples of project structures with tens / hundreds of queues and jobs ?
If you have any readings or youtube video, I'm interested.

[manast]

I am not sure if you have read the tutorials at https://blog.taskforce.sh/, but that would be a good start if you haven't.
In general, the number of queues should be something manageable, I would say max dozens, but often much less than that. In reality, there are so many ways to architect your solution, that only with the requirements at hand it would be possible to give a bit of good advice. There is nothing wrong in using the same server for running workers for several queues, it all depends on the amount of volume, the amount of work that you need to process in every job, the redundancy requirements, and so on...

[Dammmien]

Hi @manast, thanks for the answer !

Yeah I read several tutorials but they are always pretty simple, it's always one specific use case but there is no articles about the architecture of several use cases together and how things are organized when a project grows up. It can be a good topic for an article at https://blog.taskforce.sh.

But as always in software engineering: it depends on the use case I guess 😅

If there is nothing wrong in having different kind of jobs in the same queue or consuming several queues with the same server, I'll keep it as simple as possible and as long as it works and that the code is not twisted it's fine 👌

[ccravens]

Hello @Dammmien I was wondering the same thing. I spent about a month trying various architectures but finally came up with something I feel is working pretty well for me. I tend to wrap libraries around "Service Objects" which tend to help me standardize how I interface with various services. So in my opinion my answer would be rather long because not only does it consider number of queues, standardizing how I interface with Jobs and Workers, but also standardizing modeling the data and converting from, for example, mongo collection to a Node object to a JSON that can be submitted as data along with the job and understood by the other workers. I make heavy use of generics as well as extending from various base classes to ensure things like ids are consistent, etc.

I don't know if any of this helps, but I would say a good architecture won't come overnight or even "standardized". My 2 cents is, wrap services around the raw BullMQ library objects such as Jobs or Workers to enforce an architecture that ensures you get compile time errors (rather than runtime errors) and enforcing those standards across the entire code base (easier said than done lol)