Nginx kills sockets older than 2 hours #31

oddlama · 2022-09-23T22:17:46Z

When running the same CLI tool instance for exactly 2 hours (by uploading many files), the nginx proxy will forcefully terminate the connection. I guess there is some kind of internal auto-kill. Maybe the socket should be reconnected after a while to prevent this from happening.

Failed to upload 16 files  [
  {
    file: '/mnt/m/images/missing/VID_20180321_110337.mp4',
    reason: Error: Request failed with status code 504
        at createError (/root/.local/share/npm-global/lib/node_modules/immich/node_modules/axios/lib/core/createError.js:16:15)
        at settle (/root/.local/share/npm-global/lib/node_modules/immich/node_modules/axios/lib/core/settle.js:17:12)
        at IncomingMessage.handleStreamEnd (/root/.local/share/npm-global/lib/node_modules/immich/node_modules/axios/lib/adapters/http.js:322:11)
        at IncomingMessage.emit (node:events:525:35)
        at endReadableNT (node:internal/streams/readable:1359:12)
        at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
      config: [Object],
      request: [ClientRequest],
      response: [Object],
      isAxiosError: true,
      toJSON: [Function: toJSON]
    },
    response: '<html>\r\n' +
      '<head><title>504 Gateway Time-out</title></head>\r\n' +
      '<body>\r\n' +
      '<center><h1>504 Gateway Time-out</h1></center>\r\n' +
      '<hr><center>nginx</center>\r\n' +
      '</body>\r\n' +
      '</html>\r\n'
  },
  # <-- [...] and 15 more

The text was updated successfully, but these errors were encountered:

alextran1502 · 2022-09-23T22:37:17Z

Does it continue to work when you re-invoke the command?

Additional context, is this through a DNS or local IP?

oddlama · 2022-09-23T22:48:35Z

Sure, works like a charm when restarting. This is through a non local endpoint although i'd argue DNS has long been resolved at that point. It happened exactly on the 2 hour mark. Is the process keeping a websocket open for the whole time?

alextran1502 · 2022-09-23T22:51:35Z

There isn't a web socket connection between CLI client and the server so I am not sure. With the constant data stream, I am not sure why does the client gets killed.

In the NGINX config file, we have the following lines to keep the connection opened

      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

Is there anyway you can test with the local endpoint or your server cannot be accessed through local IP?

oddlama · 2022-09-23T23:46:01Z

I'm unfortunately not sure what causes this either, but this kind of error after exactly 2 hours on the second cannot be a coincidence. I am also not using an additional nginx proxy, my deployment is fully native without docker - but with the same configuration as you provided to the docker images. (I just stripped docker from the equation). So the two lines you posted exist verbatim in my config.

I could test this locally, but I have no idea what to upload so it takes another 2 hours, I just barely scratched the 2 hour mark this time. I'm open to any ideas. And after, this isn't really a big issue, just a minor nuisance.

alextran1502 · 2022-09-24T00:11:15Z

Understood, I'm gonna give this some thought

bo0tzz · 2022-10-20T15:36:55Z

I would guess this is probably nginx' keepalive_time, although the default on that is 1h. Were you perhaps running with --threads=2 to cause the 2 hour timeout?

oddlama · 2022-10-21T12:12:14Z

Good catch, that's probably it. I don't remember the exact parameters anymore but there's a possibility that I had used --threads 2.

bo0tzz · 2022-10-21T12:15:02Z

IMO this issue is rare enough (only happens when uploading a very large amount of files) and very easy to work around (just run the upload command again) that we don't need to make any changes. We'll have this issue here documenting it in case anybody else encounters it in the future.

oddlama · 2022-10-21T12:18:55Z

Once files are checksummed on the client I think you are right, but currently it would take the same amount of time to retry, as all files will be reuploaded.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nginx kills sockets older than 2 hours #31

Nginx kills sockets older than 2 hours #31

oddlama commented Sep 23, 2022

alextran1502 commented Sep 23, 2022

oddlama commented Sep 23, 2022 •

edited

alextran1502 commented Sep 23, 2022 •

edited

oddlama commented Sep 23, 2022 •

edited

alextran1502 commented Sep 24, 2022

bo0tzz commented Oct 20, 2022

oddlama commented Oct 21, 2022

bo0tzz commented Oct 21, 2022

oddlama commented Oct 21, 2022

Nginx kills sockets older than 2 hours #31

Nginx kills sockets older than 2 hours #31

Comments

oddlama commented Sep 23, 2022

alextran1502 commented Sep 23, 2022

oddlama commented Sep 23, 2022 • edited

alextran1502 commented Sep 23, 2022 • edited

oddlama commented Sep 23, 2022 • edited

alextran1502 commented Sep 24, 2022

bo0tzz commented Oct 20, 2022

oddlama commented Oct 21, 2022

bo0tzz commented Oct 21, 2022

oddlama commented Oct 21, 2022

oddlama commented Sep 23, 2022 •

edited

alextran1502 commented Sep 23, 2022 •

edited

oddlama commented Sep 23, 2022 •

edited