Skip to content

How to prove CloudNativePG failover works for your application

Last updated

View as Markdown

Quick answer: Kill the primary while something is writing and time the gap in your own client, because the operator’s promotion is the half that already works, and it is not the half that takes the time. On a three-instance cluster the operator decided to fail over about a second after a pod delete and the new primary reported itself promoted 19.77 seconds after it, while the client-visible outage ranged from 3.4 to 186.0 seconds depending on how the clients were configured, including clients that were not yours. A node failure is a different test with a different answer, and where the operator pod is scheduled changes that answer by minutes.

“High availability” gets tested as “the operator promoted a new primary”. That is the easy half, and it is not the half your users experience.

Between the primary dying and your application writing again there is a chain of delays, and only the middle of it belongs to CloudNativePG:

  1. Kubernetes noticing the failure. For a pod this is immediate; for a node it is the largest term in the sum, and it is a kube-controller-manager setting.
  2. CloudNativePG deciding to act, which it gates on the primary pod’s Ready condition, on the primary lease, and on .spec.failoverDelay if you set one.
  3. The operator promoting a replica.
  4. The Service endpoint being repointed, and the dataplane converging on it.
  5. Your client giving up on the old address and connecting to the new one: its own DNS cache, its pool evicting broken connections, and its retry policy.

A pooler is not step 5. It sits in step 2, because a connection it is holding open stops the old primary shutting down and the operator will not promote until it has. The timings for that are further down.

Decide two numbers before you start, or the measurements below have nothing to mean:

  • RTO, the longest you can accept writes failing.
  • RPO, the amount of acknowledged data you can accept losing.

And measure a third, because it is the one people skip: how long until it stays up, not how long until it works once. Recovery flaps.

The numbers come from three clusters. Each number names its cluster, because the clusters disagree in ways that matter.

  • Cluster A: CloudNativePG 1.30.0, three PostgreSQL 18.1 instances, Kubernetes 1.35.5, with a PgBouncer pooler. The client-configuration and synchronous-replication numbers are from here.
  • Cluster B: CloudNativePG 1.30.0, three PostgreSQL 18.1 instances, Kubernetes 1.35.5 on k3d, three nodes, each instance on its own node, operator on a fourth. The node-failure timeline is from here.
  • Cluster C: CloudNativePG 1.30.0, three PostgreSQL 18.1 instances in ghcr.io/cloudnative-pg/postgresql:18.1 containers, Kubernetes 1.35.5 on k3d, three nodes, one instance each, operator pinned away from them, plus a two-pod PgBouncer Pooler in transaction mode. The pooler numbers are from here, on 2026-09-21.

All three ran the same probe: one row every 200 ms, a fresh connection each time, the outcome recorded with a timestamp before the attempt.

Anything that writes in a loop and records success or failure with a timestamp will do. The important properties are that it writes rather than reads, that it opens a fresh connection each time, so you are measuring the reconnect and not a lucky idle socket, and that it logs the time before the attempt.

This probe measures a floor. A fresh connection plus one autocommit statement is the simplest thing that can fail, and a pooled application fails differently: it holds broken sockets, it has transactions open across the promotion, and its retry policy is code somebody wrote. Run this one because it is five lines, then run a second probe that uses your driver, your pool, your transaction boundaries and your retry policy. The numbers below are from the simple probe and they are the optimistic end.

Terminal window
while true; do
t=$(date +%s.%N)
op=$(uuidgen)
if psql "host=app-db-rw user=app dbname=app connect_timeout=2" \
-q -t -A -c "INSERT INTO beat (op_id) VALUES ('$op')" >/dev/null 2>&1
then echo "$t OK $op"; else echo "$t FAIL $op"; fi
sleep 0.2
done

Create the table it writes to first:

CREATE TABLE beat (
id bigserial PRIMARY KEY,
op_id uuid NOT NULL UNIQUE,
t timestamptz NOT NULL DEFAULT now()
);

op_id is the client’s own identifier for the attempt, generated before the write. It is what makes the reconciliation at the bottom of this page exact rather than indicative, and it is what lets a retry be safe.

Then kill the primary:

kubectl delete pod $(kubectl get cluster app-db -o jsonpath='{.status.currentPrimary}') --grace-period=1

The first OK after the first FAIL is the recovery, and it is not the end of the outage. Measured on cluster B’s node kill, the probe’s first success came at 75.0 s and it failed once more at 78.8 s before settling. Record three numbers:

  • first FAIL, which is when the outage started.
  • first OK, which is when something worked once.
  • last FAIL, or better, the start of N consecutive successes over a fixed window. That is the one to put in front of anybody.

From the Kubernetes events on clusters A and B:

FailingOver Current primary isn't healthy, initiating a failover from app-db-1
FailoverTarget Failing over from app-db-1 to app-db-2

about one second after the delete. On a pod delete against a healthy cluster with caught-up replicas, the operator is not the slow part. That holds for this failure mode only. A replica with replay lag, synchronous settings that make promotion wait, slow storage, or a heavily loaded database all put time back into this step.

The decision is fast. The promotion is not. Do not carry “the operator decided quickly” over into “the database is back”. On cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1 containers, the delete recommended above, flag and all, gave:

03:17:22.979 kubectl delete pod --grace-period=1
03:17:23.893 operator: Current primary isn't healthy, initiating a failover
03:17:25.785 operator: Failing over
03:17:25.834 operator: Waiting for the new primary to notice the promotion request
03:17:42.041 new primary: I'm the target primary, applying WALs and promoting my instance
03:17:42.751 new primary: server promoted
03:17:45.845 the direct probe writes again

Decision at 0.91 s, server promoted at 19.77 s, and the client writing again at 22.87 s, all measured from that 2026-09-21 delete on the cluster C containers. The grace-period table further down says 22.5 s for this same delete, which is the same recovery counted from the direct probe’s last successful write at 03:17:23.322 rather than from the command: it kept committing for a third of a second after the delete returned. Count from the delete for an RTO, and from the last write for what the client lost.

The gap in the middle is sixteen seconds between the operator asking and the target noticing, and none of it is the client’s. Your RTO is made of the second number, not the first.

Same cluster, same delete, same probe. Only the connection string changed.

how the client connectedoutage
direct, no connect_timeout, run 13.4 s
direct, no connect_timeout, run 222.3 s
direct, no connect_timeout, run 322.6 s
direct, connect_timeout=2 target_session_attrs=read-write6.7 s

All four rows are from cluster A, and none of them had a pooler in the path. The pooler gets its own section below, because it changes the failover rather than the client.

The first three rows, three runs of the same configuration, are the finding. With no connect_timeout, libpq has no deadline of its own, so how long an attempt takes is decided by DNS resolution, which address it picks when there are several, routing, and whether the far end sends a reset or goes silent. An address that refuses immediately fails fast; one that has stopped answering waits on TCP retransmission. In run 3 a single connection attempt took 19.6 seconds on its own.

connect_timeout=2 does not make the good case better. It bounds each connection attempt, which is a different thing from bounding the outage: retries, backoff, TLS and authentication stalls, and a server that accepts the connection and then hangs are all still in front of you. What it removes is the 19.6 second single attempt, and that is worth the string change on its own.

It is also a libpq setting. If your client is not libpq, find the equivalent in your driver and check what it actually bounds, because they differ.

Time both paths in the same failover or you are comparing failovers, not paths. This matters because a pooler holding a connection changes how long the failover takes for every client in the cluster, including the ones that never touch the pooler.

On cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1, two copies of the probe ran at once, differing only in the host: app-db-rw for the direct one and app-db-pooler-rw for the pooled one.

Terminal window
# read-write is what makes a connection to a demoted primary fail rather than
# appear to work, and connect_timeout bounds each attempt
URL="postgresql://app@app-db-pooler-rw:5432/app?connect_timeout=2&target_session_attrs=read-write"
while true; do
if psql "$URL" -Atq -c 'INSERT INTO beat DEFAULT VALUES;' >/dev/null 2>&1
then echo "$(date -u +%H:%M:%S.%3N) ok"
else echo "$(date -u +%H:%M:%S.%3N) FAIL"; fi
sleep 0.2
done

Both probe pods sit on a node the test does not touch, or they die with the thing they are measuring. Every gap in this section is read out of those two logs the same way, as the last ok before a block of FAIL and the first ok after it:

Terminal window
kubectl logs writer | awk '
$2=="FAIL" { if(!n) last=prev; n++; next }
n { print last, "->", $1, n" failed attempts"; exit }
{ prev=$1 }'

The primary pod was then deleted at 02:56:27.275, on that same 2026-09-21 cluster C running CloudNativePG 1.30.0 with PostgreSQL 18.1 containers, with no --grace-period flag, which matters and is the next subsection:

pathlast write beforefirst write aftergap
direct to app-db-rw02:56:27.35102:59:33.327186.0 s
through the two-pod PgBouncer Pooler02:59:27.77902:59:43.61915.8 s

Read the middle columns rather than the last one. The pooled client did not recover faster. It kept committing, successfully, for three minutes after the primary had been told to shut down, and only lost its connection when the shutdown stopped being polite. Meanwhile the direct client, which opens a fresh connection each attempt, was refused from the first one.

Three logs say why, and they agree. The old primary’s:

02:56:27.483 Requesting smart shutdown of the PostgreSQL instance
02:56:27.579 received smart shutdown request
02:59:27.925 Error while handling the smart shutdown request
02:59:27.934 received fast shutdown request

A smart shutdown refuses new connections and waits for the existing ones to end. PgBouncer’s server connections do not end, so on that 2026-09-21 cluster C delete on CloudNativePG 1.30.0 with PostgreSQL 18.1 containers it waited out .spec.smartShutdownTimeout, which defaults to 180 and measured 180.4 s from the smart request to the fast one. The operator’s log, for the whole of that window:

02:56:27.931 Current primary isn't healthy, initiating a failover
02:56:28.034 Setting primary label to unhealthy in the old primary during failover
02:56:28.083 Waiting for all WAL receivers to be down to elect a new primary

On that same 2026-09-21 cluster C delete on CloudNativePG 1.30.0, it decided in 0.66 s, then logged that last line 192 times, the final one at 02:59:19.778, with nothing else in between. The new primary’s log on that same CloudNativePG 1.30.0 cluster on 2026-09-21 closes it: I'm the target primary, applying WALs and promoting my instance at 02:59:29.833, which is 1.9 seconds after the old primary took the fast shutdown at 02:59:27.934, and server promoted at 02:59:30.738, 2.8 seconds after it.

So the pooler is not slow in itself. The pooled sessions keep the old primary alive, the old primary keeps its WAL receivers up, and the operator will not promote while they are up.

It reproduces to the second, because what sets the length is a timeout rather than anything variable. A second delete on the same 2026-09-21 cluster C, CloudNativePG 1.30.0 with PostgreSQL 18.1, gave 185.1 s direct, 15.9 s pooled, and 180.5 s from the smart shutdown request to the fast one.

Prove it is the pooled sessions and not the pooler. Three deletes on cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1, same probe, none of them passing --grace-period, only the pooler’s state different:

pooler statethe direct client’s gap
scaled to zero6.19 s
two pods running, nothing using them3.91 s
two pods running, a client writing through them186.0 s

A deployed pooler costs nothing. A used one costs the difference. If you run one, either lower .spec.smartShutdownTimeout and accept that transactions in flight on the old primary get cut, or measure your failover with the pooler under load, because idle is the case that flatters it.

The --grace-period you pass decides which failure you measured

Section titled “The --grace-period you pass decides which failure you measured”

The kubectl delete pod above carries --grace-period=1, and everything in this section so far ran without it. That one flag changes the answer by three minutes.

CloudNativePG sets the instance pod’s terminationGracePeriodSeconds from .spec.stopDelay, which defaults to 1800. Leave the flag off and the smart shutdown has half an hour of room, so it takes its full .spec.smartShutdownTimeout. Pass --grace-period=1 and the kubelet kills the container a second in, before the smart shutdown has waited for anything. Both deletes on cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1 containers, same probes, same pooled client writing throughout:

deletedirectthrough the Pooler
no flag, the pod’s own 1800 second grace186.0 s15.8 s
--grace-period=122.5 s30.6 s

The bottom row is an abrupt loss, and it is the one PgBouncer pays server_login_retry for, twice, which CloudNativePG pooler or direct connections works through from PgBouncer’s own log. The top row is a cooperative shutdown, which is what a drain, a rolling update or a kubectl delete with the defaults gives you.

Neither is wrong. They are different failures, and your cluster does both: an OOM kill or a node loss looks like the bottom row, a node drain or an operator upgrade looks like the top. Test both, and say which one a number came from, because a page that says “failover took twenty-two seconds” without saying how the primary died has not told you anything you can plan with.

Nothing was lost either way. On that 2026-09-21 cluster C delete on CloudNativePG 1.30.0 with PostgreSQL 18.1 containers, the 686 rows the pooled client wrote during that window were all on the new primary afterwards, running to a last row timestamp of 02:59:27.774 on the database’s clock, the same write the table above logs at 02:59:27.779 on the probe pod’s clock, which stamps its line after psql returns, against 355 rows in the 87 seconds before the delete and 584 in the 87 seconds after recovery. Replication was not under load in that run, so read it as this fixture rather than as a promise.

There is one trap when you look for the operator’s side of this. kubectl logs on cnpg-controller-manager returned only the last forty seconds, because the operator is chatty enough that containerd had already rotated the file; the window above came off the node out of /var/log/pods/cnpg-system_cnpg-controller-manager-*/manager/. And do not time anything off the FailingOver Event: Kubernetes aggregates repeats, so its lastTimestamp is the most recent occurrence of that message and not the start of this failover.

This is a different failure with a different answer, and testing only the pod delete is how people end up believing their cluster recovers in seconds.

On cluster B, from stopping the container that is the node:

what happenedtime after the node stopped
node Ready becomes Unknown, primary pod Ready becomes False47 s
FailingOver Current primary isn't healthy, initiating a failover49 s
status.currentPrimary moves to the new instance70 s
the probe writes again75.0 s
the probe stops failing for good78.8 s

Seventy-five seconds, and the 300 second eviction toleration is not in it. CloudNativePG’s own documentation says so directly:

Pod eviction (actual deletion from the unreachable node) is a separate mechanism, driven by tolerationSeconds on the node.kubernetes.io/unreachable NoExecute taint (300s by default). That timer does not hold up the operator’s failover decision; CloudNativePG promotes a new primary as soon as the Ready condition flips.

CloudNativePG 1.30, Automated failover

There is one way to turn that 49 second decision into a 323 second one and be right about the clock and wrong about the cause. CloudNativePG ships one replica of cnpg-controller-manager. Put it on the node you stop and the sentence above still holds, because the toleration is not delaying the decision: it is delaying the arrival of anything that can make one. Measured on 2026-09-21 on this CloudNativePG 1.30.0 cluster with PostgreSQL 18.1 on Kubernetes 1.35.5, the same node stop with the operator on the failing node gave:

00:54:35 NodeNotReady Node is not ready
00:59:35 TaintManagerEviction Marking for deletion Pod cnpg-system/cnpg-controller-manager-...
00:59:56 LeaderElection cnpg-controller-manager-...-7kfrt became leader
00:59:58 FailingOver Current primary isn't healthy, initiating a failover

Until it is evicted the cluster reports Cluster in healthy state, because the thing that notices is the thing that is gone.

Five minutes between the node going and the operator being evicted, on the operator pod’s own stock 300 second toleration, then twenty-three seconds to elect a leader and decide. So pin the operator away from the instances before you measure anything, or you will publish a number about pod eviction and label it failover.

Check where yours is before you trust any failover number:

kubectl -n cnpg-system get pods -o wide

and give the operator deployment anti-affinity from your database nodes, or at least more than one replica.

The 47 seconds is the part worth tuning, and it is not CloudNativePG’s either. The same page:

With stock kube-controller-manager settings, the transition is governed by --node-monitor-grace-period (default 40s on Kubernetes 1.29-1.31, raised to 50s in 1.32 and later): after that window the controller marks the node Unknown and, in the same monitoring pass, issues a patch per pod on that node to flip the Ready condition.

So on Kubernetes 1.32 and later, expect roughly 50 to 55 seconds of detection before CloudNativePG has anything to react to. The 47 seconds measured here falls inside the 40 to 55 seconds the same page gives for the operator seeing the primary as unready. Managed distributions tune these values; check yours rather than assuming this number.

Lowering --node-monitor-grace-period shortens every node failure in the cluster and makes every transient API server hiccup look like one. It is a cluster-wide trade with partition behaviour on the other side of it, not a dial for one database.

A dead pod, a dead node, an unreachable node and a network partition are four different tests, and this page has measured two of them. The list, in the order they get skipped:

  • the primary process killed inside a healthy pod,
  • the primary pod deleted,
  • the node stopped,
  • the node partitioned from the control plane while clients can still reach it,
  • a zone partition, if the cluster spans zones,
  • the volume failing under the primary.

The fourth is the one where CloudNativePG’s safety machinery matters rather than its speed: the primary lease under .spec.primaryLease, the isolation liveness check, and failover quorum under .spec.postgresql.synchronous.failoverQuorum. Those decide whether a partition ends in a promotion or in two primaries, and none of them are exercised by kubectl delete pod.

Time is only half the question. Run the probe so it records the id of every row the server acknowledged, then compare that list against the table afterwards.

Counting will not do it. After cluster B’s node kill the table held 603 rows with a maximum id of 660. Neither number says whether an acknowledged id is missing from the middle, and the gaps in a bigserial are there whether or not anything was lost. Compare the sets:

Terminal window
kubectl exec app-db-2 -c postgres -- \
psql -U postgres -d app -tAc "copy (select op_id from beat) to stdout" \
| sort > present.txt
grep ' OK ' probe.log | awk '{print $3}' | sort > acknowledged.txt
comm -23 acknowledged.txt present.txt # acknowledged and gone: this is your RPO
comm -13 acknowledged.txt present.txt # present and never acknowledged

The second list is not an error. An INSERT can commit and the acknowledgement can be lost on the way back, which leaves the client believing it failed. Retry that blindly and you have a duplicate, so reconcile on the op_id the client generated rather than on the bigserial the server did, which is why the probe table above has one.

Four tests:

  • Pod deleted, asynchronous replication, cluster A. 382 acknowledged ids, 382 present. Nothing lost. The pod gets a signal, PostgreSQL shuts down cleanly, and the replica is caught up before the promotion. Note the shape of that test: --grace-period=1 gives the shutdown one second to finish, which on a quiet database is enough for a checkpoint and on a busy one is not, so it sits between a clean stop and an abrupt loss rather than being either.
  • Node stopped, asynchronous replication, cluster B. 356 acknowledged ids, 0 missing. The replicas were caught up when the node went away.
  • Node killed, asynchronous replication, cluster A. 500,000 rows acknowledged, 300,000 of them gone after the promotion. The replica that got promoted had itself rejoined the cluster 80 seconds earlier and was still catching up.
  • Node killed, synchronous replication, cluster A. 300,000 rows acknowledged, 300,000 present. Nothing lost.

Read the middle two together, because they are the same failure with opposite outcomes. Under asynchronous replication the primary can acknowledge a commit that no other instance holds, so your RPO is whatever the replay lag was at the moment the primary died. You can watch that number, but you cannot promise it. Synchronous replication is what lets you promise it.

Turning that on is one patch:

kubectl patch cluster app-db --type merge \
-p '{"spec":{"postgresql":{"synchronous":{"method":"any","number":1}}}}'

After which the standbys report sync_state = quorum:

SELECT application_name, state, sync_state, write_lag, flush_lag
FROM pg_stat_replication
ORDER BY application_name;

Synchronous replication has a cost. The same failover probe that took 6.7 seconds asynchronously took 22.4 seconds with ANY 1 configured, so the durability was paid for in outage time. Make that trade deliberately, after measuring both sides.

Be careful with the setting that sounds like it already does this:

SELECT current_setting('synchronous_commit') AS commit_mode,
current_setting('synchronous_standby_names') AS standbys;

synchronous_commit = on with synchronous_standby_names empty means flushed to the local disk and acknowledged. No standby was involved. That is the default in a three-instance CloudNativePG cluster.

Set connect_timeout in every connection string, and target_session_attrs where the driver supports it. That is the single cheapest improvement measured here, and it is a string change. Check what your driver’s equivalent bounds; not every client is libpq.

Test the node failure, not only the pod failure. With nothing holding a session open on the old primary, the pod deletes on this page put the client back in service in 3.4 to 22.6 seconds, the widest of them a run with no connect_timeout set; the node stop took 75.0, and the difference is node detection rather than anything the database does.

Measure with everything else connected too. The same pod delete was 3.91 seconds with the pooler idle and 186.0 with a client writing through it. A failover test against a quiet cluster measures a failure mode you do not have.

Check where the operator is scheduled. A controller on the same node as the primary turns a 75.0 second failover into one that does not begin until the controller’s own pod is evicted, five minutes in.

Run the test again after every operator upgrade and after any change to storage, pooling or replication. Three of the numbers above changed when only the client configuration changed.

Record what you ran it against. Operator version, operand image, Kubernetes version and distribution, the dataplane, the storage class, where each instance and the client sat, and the connection string. Without those the numbers are anecdotes, including these ones.

Decide synchronous or asynchronous with both numbers in front of you, rather than by reputation.

  • How long does CloudNativePG take to fail over?

    The decision comes about one to two seconds after a primary pod is deleted, measured on two separate CloudNativePG 1.30.0 clusters, but the promotion does not: on a third CloudNativePG 1.30.0 cluster with PostgreSQL 18.1 on 2026-09-21 the operator decided at 0.91 seconds and the new primary reported server promoted at 19.77, so quote the second number when someone asks how long failover takes. On a node failure it is the node detection that dominates: measured on Kubernetes 1.35.5, the primary pod was marked not ready 47 seconds after the node stopped and the operator began failing over 2 seconds later. Neither is the number your application sees. Client-visible write outages on those clusters and on a third one running a PgBouncer pooler ranged from 3.4 seconds to 186.0 seconds, and what moved them was how the clients connected rather than anything the operator did.

  • Why does my client see a longer outage than the database was down?

    Most often because libpq has no connection timeout by default, so an attempt against an address that has stopped answering waits on whatever DNS, routing and TCP retransmission do rather than on PostgreSQL. In one run a single attempt hung for 19.6 seconds. connect_timeout=2 bounds each attempt, not the whole outage.

  • Is failover slower through the pooler?

    The pooler is not the thing that is slow: it makes the failover slow for everyone. PgBouncer's held server connections keep the old primary in a smart shutdown until smartShutdownTimeout expires, and the operator will not promote until the old primary's WAL receivers are down. Timed in one pod delete on 2026-09-21 on a CloudNativePG 1.30.0 cluster with PostgreSQL 18.1, the pooled client kept committing for three minutes after the primary was told to shut down and then lost 15.8 seconds, while a direct client lost 186.0 seconds. With that same 2026-09-21 pooler deployed and idle on CloudNativePG 1.30.0, the direct client lost 3.91 seconds. That is a cooperative delete. On that same 2026-09-21 CloudNativePG 1.30.0 cluster, the same delete with kubectl --grace-period=1, which kills the container before the smart shutdown waits for anything, gave 22.5 seconds direct and 30.6 pooled, so always say how the primary died.

  • Does a node failure behave like a pod failure?

    No, and this is the measurement that matters most, but the gap is node detection rather than pod eviction. Measured on Kubernetes 1.35.5 with CloudNativePG 1.30.0: the node stopped, the primary pod was marked not ready at 47 seconds, the operator began failing over at 49 seconds, the new primary was serving at 70 seconds and the client wrote again at 75 seconds. The 300 second unreachable toleration is about deleting the pod and does not hold up the promotion.

  • Will I lose committed writes?

    Under asynchronous replication you can, and whether you do depends on how far behind the promoted replica was. One measured node kill lost 300,000 acknowledged rows because the promoted replica had rejoined 80 seconds earlier; another, with the replicas caught up, lost none of 356. Synchronous replication removes the question and cost 22.4 seconds of outage against 6.7. Reconcile the acknowledged ids against the table; count(*) and max(id) cannot tell you.