Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client

[Ekeu]

Hello BullMQ Community,

I'm currently learning BullMQ, and I built a small REST API with Node.js and I am facing a challenge regarding the management of event listeners for job processing. Specifically, my use case involves responding to completed and failed events for individual jobs in a manner that's tied to specific HTTP requests.

Here's the challenge I'm facing and which I can't really explain why it's working

Below is the version of my controller that is having problems

  static async voteComment(req: Request, res: Response, next: NextFunction): Promise<void> {
    const { commentId } = req.params;
    const { value } = req.body;

    await voteCommentMQ.addJob(COMMENT_VOTE_JOB, {
      commentId,
      userId: req.currentUser?.userId as string,
      value,
    });

    voteCommentMQ.worker?.on("completed", (_, result: Record<string, boolean>) => {
      res.status(StatusCodes.OK).json({
        success: true,
        message: "Comment voted successfully",
        data: result,
      });
    });

    voteCommentMQ.worker?.on("failed", (_, error) => {
      if (!res.headersSent) {
        next(error);
      }
    });
  }

With this version, when I send a first request, I get a successful response. Then when I send a second request I get an error.

But now if I update it and instead have:

 voteCommentMQ.worker?.on("completed", async (_, result: Record<string, boolean>) => {
   if (!res.headersSent) {
       res.status(StatusCodes.OK).json({
        success: true,
        message: "Comment voted successfully",
        data: result,
      });
   }
 });

Everything works fine. I can't figure out what's happening exactly.

My clue is that every time the voteComment controller is called a new complete event listener is attached to the voteCommentMQ.worker. And since the listener is not removed at the end of the request, when I send the second request since there was already a previous complete event listener attached, the previous res is sent and now when the new res tries to be sent I get the error.

I guess it is the same with the failed event.

Is there a recommended pattern or practice within BullMQ for managing such scenarios? Any insights or examples of handling this type of scenario would be greatly appreciated.

Thank you in advance for your guidance!

[manast]

You are attaching new listeners for every new request, so that is going to create several problems, including the one you are experiencing (calling res.json multiple times and so on).
Furthermore, It seems you are implementing the "wait for job to complete" pattern, which is not a good approach, here I wrote about it and how you should think instead: https://blog.taskforce.sh/do-not-wait-for-your-jobs-to-complete/

[Ekeu]

Interesting!

Thank you for the blog post. So for such a scenario I guess I will have something like this:

async voteComment(req: Request, res: Response){
 
    const job = await voteCommentMQ.addJob(COMMENT_VOTE_JOB, {
      commentId,
      userId: req.currentUser?.userId as string,
      value,
    });

   res.redirect(`/get-job-res/${job.id}`) 
 }

async getJobRes(eq: Request, res: Response){
 const { jobId } = req.params
  const job = await myQueue.getJob(jobId);
    if (job) {
        const isCompleted = await job.isCompleted()
        if (isCompleted) {
            const result = await job.returnvalue;
            return result;
        }
    } else {
        // Do something else
    }
}

[manast]

yes, much better!