add possibility to update/replace jobs and filter options

[dberardo-com]

would be great to have the possibility to filter jobs by some kind of parameters in the queue.getJobs method (also in the doc is not clear which parameters they would be sort by, i guess that time ?)

and most importantly, how to edit an existing job ? i understand it is a combination between remove and add, but it would be better to have an update or replace method for that

[ajfranzoia]

Not having a filtering options makes it really hard for cases where there is a centralized consumer and multiple producers

[manast]

Not having a filtering options makes it really hard for cases where there is a centralized consumer and multiple producers

Filtering in getJobs would only be useful for debugging or visualization purposes, for multiple producers expecting to consume dedicated results, the best would be to define a result queue for every producer.

[dberardo-com]

visualization purposes,

this is what i am aiming for here ;)

[ajfranzoia]

Not having a filtering options makes it really hard for cases where there is a centralized consumer and multiple producers

Filtering in getJobs would only be useful for debugging or visualization purposes, for multiple producers expecting to consume dedicated results, the best would be to define a result queue for every producer.

That's not feasible, it would introduce coupling with tens of consumers, and the producer service would have to be modified everytime a new result queue is needed. Thanks anyway

[manast]

That's not feasible, it would introduce coupling with tens of consumers, and the producer service would have to be modified every time a new result queue is needed. Thanks anyway

Not sure what you mean by coupling here, the consumer would just place the result in a queue based on the job data for example.

[ajfranzoia]

@manast I apologize I didn't explain myself clearly enough. This is my use case, I have a service whose role is to generate audio files on demand. Multiple consumer services generate jobs, this service picks them up and generates the required audio file. The audio service does not care where it comes from.
If I properly understand your suggestion:

• either the audio service knows beforehand all of the possible result queues and registers them at startup (this is not possible since consumer services are dynamic and the audio service is not aware of them)
• or the audio service initializes a Bull Queue on completion, based on the name given by the job data, posts the result and closes the connection to it. Do you see this as feasible? Are there any caveats?

Thanks for your help

[manast]

• either the audio service knows beforehand all of the possible result queues and registers them at startup (this is not possible since consumer services are dynamic and the audio service is not aware of them)

Why can't you pass the name of the result queue in the job itself? the producer knows the name of its result queue right?

[ajfranzoia]

@manast I was thinking about the case of using one result queue per requester service. In that case I would pass the queue name in the job data and post to that queue on completion (I'm using Nest so I think I'd have to manually initialize and close the queue for this instead of registerQueue).

I discarded the idea of using just a single result queue, because if the audio service posts a results job to that queue, I could not find a way to have the proper requester service pick up that job. E.g.: Service A requests an audio, audio service generates the audio and posts the result to the result queue, but Service B is assigned by Bull and takes that job first.

[manast]

That's why I am suggesting to have several result queues instead. But as the queues may be dynamic, the workers will need to instantiate a queue every time they want to send a result, reusing a Redis connection would reduce the overhead of instantiating said queues.

[ajfranzoia]

That's why I am suggesting to have several result queues instead. But as the queues may be dynamic, the workers will need to instantiate a queue every time they want to send a result, reusing a Redis connection would reduce the overhead of instantiating said queues.

Coming from the NestJs module, I was too fixated on the static method registerQueue at startup. I understand your suggested approach now, I will try that. Thanks for your time @manast