-
Notifications
You must be signed in to change notification settings - Fork 21.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possible race condition in postgresql type_map initialization #51780
Comments
We have the same issue. We haven't been able to reproduce it outside production. I will make another attempt this week 😢. |
We're also seeing the same problem. Ruby 3.2.4 and Rails 7.1.3.2 |
We faced the same issue after our DB ran out of memory. Ruby 3.3.1 and Rails 7.1.3.3 |
Same issue here. I can't find a way to reproduce it neither |
@nelhefni Are you able to share any graphs? Does it happen after a deploy or does the issue occur under load? |
We run into this issue periodically after upgrading to It happens during deployment of new pods, and sometimes will resolve it self after some time (Possibly only when the pod is restarted. I haven't been able to confirm). We have had a couple of variants of this error happen (only in production). First one matches the one from the ticket description:
And this one we've seen more often:
|
We got the same stack trace as @martinvium (the one ending with Here is the memory of our DB when the incident occured: We got our first instance of the error at 15:23 (13 mins after freeable memory reach a dangerous low), and the error continued for over an hour, despite the freeable memory increasing again. We had to restart the application to fix the issue. |
Steps to reproduce
Since upgrading from Rails 7.0 to 7.1 we've started to see the following exception:
I haven't been able to replicate the behavior outside of production where it occurs when the application briefly loses its connection to the postgres database while under load. This is a multi-threaded environment. The server never recovers after entering this state.
I know there was a change to type_map initialization at dc4420c, and it looks to me like it's possible there's a race condition here. I'll update this issue with steps to replicate if I can manage to trigger this outside of production.
Expected behavior
The application should recover from a brief database outage and reconnect.
Actual behavior
The server is never able to reconnect.
System configuration
Rails version: 7.1.3.2
Ruby version: 3.2.4
The text was updated successfully, but these errors were encountered: