Best way to way for a job

[MichielvdVelde]

I'm looking for some advice about how to wait for a job to complete. I am adding a job through an API endpoint and I want to wait for the results.

I have used the following code:

/** Waits for a job to be processed and rejects if the job failed */
export async function waitForJob(queue: Queue, job: Job, timeout: number) {
  return new Promise<void>((resolve, reject) => {
    const timer = setTimeout(() => {
      unlistenCompleted()
      unlistenFailed()
      reject('waiting for job timed out')
    }, timeout)

    const unlistenCompleted = listen(queue, 'completed', (j: Job) => {
      if (j.id === job.id) {
        clearTimeout(timer)
        unlistenCompleted()
        unlistenFailed()
        resolve()
      }
    })

    const unlistenFailed = listen(queue, 'failed', (j: Job) => {
      if (j.id === job.id) {
        clearTimeout(timer)
        unlistenCompleted()
        unlistenFailed()
        reject(j.failedReason)
      }
    })
  })
}

const queue = new Queue('fit', {
      connection: await getConnection(),
      sharedConnection: true,
    })

    const job = await queue.add('some-name', {
      characterId: context.identity!.characterId,
      visibility: search.get('visibility') === 'private' ? 'private' : 'public',
    })

// ...
await waitForJob(queue, job, 10000)

However every time waitForJob times out. I have tried several variations, but nothing seems to be working. I know the job is executed because the console.logs it contains are printed.

Any insight would be appreciated. :)

[manast]

You could use this method: https://github.com/taskforcesh/bullmq/blob/master/docs/gitbook/api/bullmq.job.waituntilfinished.md

However, let me warn you that most likely you are using an anti-pattern, waiting for jobs to complete is not the best way to use queues, it does not scale well nor is the most robust way to handle jobs.

[MichielvdVelde]

Thank you for the link and your fast reply. I realize waiting for jobs will not scale, however I am not anticipating significant loads in this case. In any case, should this change, I have made sure it's a simple matter to remove it. For development though I'd like some direct feedback :)

[pguardiario]

the link is broken :(

[roggervalf]

https://api.docs.bullmq.io/classes/v4.Job.html#waitUntilFinished this is the new one @pguardiario

[usholanb]

Thanks. Is python version for this also available?

[TheRealFlyingCoder]

just a little tidbit that might be a better pattern for some people:

Wrap your queues up so that it returns typed helper functions, and include a direct path to the async job so that you can choose when you want to run it "inline".

My eCommerce use case, is that most jobs queue up automatically, but if staff want to manually adjust something in the backend I want them to see a loading screen while the job takes place... so they just run the async function directly instead of queueing it up

export const myRandomJob = queueExport<MyJobData>('myRandomJob', async job => {})

//Adds a job to queue
myRandomJob.addJob()

//Runs the job directly inline instead of queueing
await myRandomJob.runDirect() 

Alternatively you can just put the function that the job runs somewhere else so that you can just call it when you want and not have to duplicate the code