Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve graceful shutdown error reporting #15572

Open
nholland94 opened this issue Apr 25, 2024 · 1 comment
Open

Improve graceful shutdown error reporting #15572

nholland94 opened this issue Apr 25, 2024 · 1 comment
Assignees

Comments

@nholland94
Copy link
Member

When the node encounters an error, that error is bubbled up to the highest level of async, where we then initiate our graceful shutdown logic for the node. This graceful shutdown logic is supposed to log the error, generate a crash report, and shut things down in a more graceful manner. However, we are seeing that it doesn't always do this well: there seems to be some code paths in which the error will not actually get logged sufficiently before the shutdown finishes. We just end up with logs that say graceful shutdown has started with no other information.

Below is an example of what this looks like in the logs (no error or useful information prior to this).

{"timestamp":"2024-04-21 10:57:00.890593Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 31, characters 4-15"},"message":"Running async shutdown handler: $description","metadata":{"description":"Close transition frontier, if exists","host":"46.4.27.171","peer_id":"12D3KooWBtktgL7kvikZScEAioJpv36DVx6vBMECKbMRWc25rT5R","pid":18,"port":8302}}
{"timestamp":"2024-04-21 10:57:00.890632Z","level":"Info","source":{"module":"Exit_handlers","location":"File \"src/lib/exit_handlers/exit_handlers.ml\", line 31, characters 4-15"},"message":"Running async shutdown handler: $description","metadata":{"description":"Remove daemon lockfile","host":"46.4.27.171","peer_id":"12D3KooWBtktgL7kvikZScEAioJpv36DVx6vBMECKbMRWc25rT5R","pid":18,"port":8302}}
@nholland94
Copy link
Member Author

Let's fix this in develop for now so that we don't introduce risk to berkeley or compatible so close to the upgrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: No status
Development

No branches or pull requests

2 participants