How to update an openraft cluster to a newer application version? #319
-
|
So I have an application deployment on kubernetes that I want to update to a new version. I have it setup so that all of the RPC calls and internal state are versioned so old nodes can talk to new nodes without breaking. But the question is, how do I ensure that while I'm replacing nodes I don't accidentally cause any data loss by taking down a leader while it's replicating or taking client requests? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
|
Because the raft itself is something like a clock, whose duty is to determine orders of events(client request), if the clock itself is damaged and the time on it becomes wrong, it has no way to find it out. There has to be another clock as a reference to test this one: And finally, a test has to ensure every replica has identical logs. |
Beta Was this translation helpful? Give feedback.
-
|
Assuming an environment (kubernetes, nomad, etc) is pulling down nodes and replacing them one-by-one, once it arrives to replacing the leader, would triggering a voluntary leader step down and waiting for it to finish be a good way of ensuring that application state is preserved? |
Beta Was this translation helpful? Give feedback.
-
|
By design of raft, a node can be shut down at any time with consistency guaranteed. |
Beta Was this translation helpful? Give feedback.
Because the raft itself is something like a clock, whose duty is to determine orders of events(client request), if the clock itself is damaged and the time on it becomes wrong, it has no way to find it out.
There has to be another clock as a reference to test this one:
Recording all client events in an access log and replaying the access log is a way to determine the consistency roughly. jepson provides a more delicate way to test the consistency by calculating all of the possible states after applying a series of events.
And finally, a test has to ensure every replica has identical logs.