Add PostgreSQL node failover leases - #1000
Conversation
|
👋 Thanks for assigning @tnull as a reviewer! |
|
I’m primarily looking for a concept ACK on the overall direction at this stage. The code has already gone through several iterations, and it will probably need another pass once we agree that the lease, fencing, and process-exit approach is the right path.
|
629386d to
056be7a
Compare
|
Would be nice if this is abstracted out to also work with VSS. That way multi-device with a VSS backend can work |
|
The safety-critical part seems mostly backend-specific to me, since lease validation and the KV mutation must be atomic within the backend. What layer of this do you envision abstracting? If generalizing this later would not require much rewriting, I would prefer to leave any necessary code movement to a future VSS lease PR. |
|
Friendly ping @tnull. I would like to agree on the overall approach before doing further implementation cleanup. If you do not currently have the bandwidth to take a look, please let me know too. |
I'll try to get back to a first look some time this week, but review on PRs for v0.8 has priority right now. I think to begin with my first question is whether we don't need a more general version of this anyways, i.e., something a |
For an initial version, I think rebuilding can remain the responsibility of the infrastructure, for example Kubernetes. ldk-node reports the lease lost, ldk-server exits, and the infrastructure restarts the process. The wrapper you propose seems quite a bit broader, and I do not think it is needed to meet the initial objective of failover? As mentioned in #1000 (comment), making lease+mutation atomic is backend-specific, so a general wrapper would not replace that work. |
ce37bcf to
9d0c61a
Compare
Make PostgresStore acquire a table-scoped lease during construction and fence schema setup, migrations, and every mutation with it. Track renewals conservatively and fail closed when the local deadline elapses, including while database I/O is stalled. Treat runtime lease loss as process-fatal, reject startup after lease loss, and notify servers so they can exit without final persistence and restart from durable state.
9d0c61a to
0ff95ef
Compare
|
Pushed a few AI fixes in the mean time. Once we agree on general direction and scope, I can go in and make this PR merge ready. |
4 participants
Addresses #932 by adding minimal active/passive failover support for nodes using the built-in PostgreSQL store.
PostgresStoreacquires an exclusive lease for its KV table, renews it in the background, and fences every mutation by validating and renewing the lease in the same transaction. Lease loss is terminal for the node and is exposed to the application (ldk-server) so the process can exit and restart from persisted state.This initial implementation is intentionally limited to PostgreSQL. Generic stores and VSS are out of scope. The API, timing configuration, and backend support can be extended later if operational experience requires it.