# How to prove CloudNativePG failover works for your application

> **Quick answer:** Kill the primary while something is writing and time the gap
> in your own client, because the operator's promotion is the half that already
> works, and it is not the half that takes the time. On a three-instance cluster
> the operator decided to fail over about a second after a pod delete and the new
> primary reported itself promoted 19.77 seconds after it, while the
> client-visible outage ranged from 3.4 to 186.0 seconds depending on how the
> clients were configured, including clients that were not yours. A node failure
> is a different test with a different answer, and where the operator pod is
> scheduled changes that answer by minutes.

## What is the problem?

"High availability" gets tested as "the operator promoted a new primary". That is
the easy half, and it is not the half your users experience.

Between the primary dying and your application writing again there is a chain of
delays, and only the middle of it belongs to CloudNativePG:

1. Kubernetes noticing the failure. For a pod this is immediate; for a node it is
   the largest term in the sum, and it is a `kube-controller-manager` setting.
2. CloudNativePG deciding to act, which it gates on the primary pod's `Ready`
   condition, on the primary lease, and on `.spec.failoverDelay` if you set one.
3. The operator promoting a replica.
4. The Service endpoint being repointed, and the dataplane converging on it.
5. Your client giving up on the old address and connecting to the new one:
   its own DNS cache, its pool evicting broken connections, and its retry policy.

A pooler is not step 5. It sits in step 2, because a connection it is holding
open stops the old primary shutting down and the operator will not promote until
it has. The timings for that are further down.

Decide two numbers before you start, or the measurements below have nothing to
mean:

- **RTO**, the longest you can accept writes failing.
- **RPO**, the amount of acknowledged data you can accept losing.

And measure a third, because it is the one people skip: **how long until it stays
up**, not how long until it works once. Recovery flaps.

### What produced these numbers

The numbers come from three clusters. Each number names its cluster, because the
clusters disagree in ways that matter.

- **Cluster A:** CloudNativePG 1.30.0, three PostgreSQL 18.1 instances,
  Kubernetes 1.35.5, with a PgBouncer pooler. The client-configuration and
  synchronous-replication numbers are from here.
- **Cluster B:** CloudNativePG 1.30.0, three PostgreSQL 18.1 instances,
  Kubernetes 1.35.5 on k3d, three nodes, each instance on its own node, operator
  on a fourth. The node-failure timeline is from here.
- **Cluster C:** CloudNativePG 1.30.0, three PostgreSQL 18.1 instances in
  `ghcr.io/cloudnative-pg/postgresql:18.1` containers, Kubernetes 1.35.5 on k3d,
  three nodes, one instance each, operator pinned away from them, plus a two-pod
  PgBouncer `Pooler` in transaction mode. The pooler numbers are from here, on
  2026-09-21.

All three ran the same probe: one row every 200 ms, a fresh connection each time, the
outcome recorded with a timestamp before the attempt.

## How do I confirm it?

### The probe

Anything that writes in a loop and records success or failure with a timestamp
will do. The important properties are that it writes rather than reads, that it
opens a fresh connection each time, so you are measuring the reconnect and not a
lucky idle socket, and that it logs the time before the attempt.

This probe measures a floor. A fresh connection plus one
autocommit statement is the simplest thing that can fail, and a pooled
application fails differently: it holds broken sockets, it has transactions open
across the promotion, and its retry policy is code somebody wrote. Run this one
because it is five lines, then run a second probe that uses your driver, your
pool, your transaction boundaries and your retry policy. The numbers below are
from the simple probe and they are the optimistic end.

```sh
while true; do
  t=$(date +%s.%N)
  op=$(uuidgen)
  if psql "host=app-db-rw user=app dbname=app connect_timeout=2" \
       -q -t -A -c "INSERT INTO beat (op_id) VALUES ('$op')" >/dev/null 2>&1
  then echo "$t OK $op"; else echo "$t FAIL $op"; fi
  sleep 0.2
done
```

Create the table it writes to first:

<!-- sql-check: postgres -->
<!-- sql-check: discard -->
```sql
CREATE TABLE beat (
  id    bigserial PRIMARY KEY,
  op_id uuid NOT NULL UNIQUE,
  t     timestamptz NOT NULL DEFAULT now()
);
```

`op_id` is the client's own identifier for the attempt, generated before the
write. It is what makes the reconciliation at the bottom of this page exact
rather than indicative, and it is what lets a retry be safe.

Then kill the primary:

```
kubectl delete pod $(kubectl get cluster app-db -o jsonpath='{.status.currentPrimary}') --grace-period=1
```

The first `OK` after the first `FAIL` is the recovery, and it is not the end of
the outage. Measured on cluster B's node kill, the probe's first success came at
75.0 s and it failed once more at 78.8 s before settling. Record three numbers:

- **first `FAIL`**, which is when the outage started.
- **first `OK`**, which is when something worked once.
- **last `FAIL`**, or better, the start of N consecutive successes over a fixed
  window. That is the one to put in front of anybody.

### What the operator did

From the Kubernetes events on clusters A and B:

```
FailingOver      Current primary isn't healthy, initiating a failover from app-db-1
FailoverTarget   Failing over from app-db-1 to app-db-2
```

about one second after the delete. On a pod delete against a healthy cluster with
caught-up replicas, the operator is not the slow part. That holds for this
failure mode only. A replica with replay lag, synchronous
settings that make promotion wait, slow storage, or a heavily loaded database all
put time back into this step.

The decision is fast. The promotion is not. Do not
carry "the operator decided quickly" over into "the database is back". On
cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1 containers,
the delete recommended above, flag and all, gave:

```
03:17:22.979  kubectl delete pod --grace-period=1
03:17:23.893  operator: Current primary isn't healthy, initiating a failover
03:17:25.785  operator: Failing over
03:17:25.834  operator: Waiting for the new primary to notice the promotion request
03:17:42.041  new primary: I'm the target primary, applying WALs and promoting my instance
03:17:42.751  new primary: server promoted
03:17:45.845  the direct probe writes again
```

Decision at 0.91 s, `server promoted` at 19.77 s, and the client writing again at
22.87 s, all measured from that 2026-09-21 delete on the cluster C containers.
The grace-period table further down says 22.5 s for this same delete, which is
the same recovery counted from the direct probe's last successful write at
03:17:23.322 rather than from the command: it kept committing for a third of a
second after the delete returned. Count from the delete for an RTO, and from the
last write for what the client lost.

The gap in the middle is sixteen seconds between the operator asking and the target
noticing, and none of it is the client's. Your RTO is made of the second number,
not the first.

### What the client saw

Same cluster, same delete, same probe. Only the connection string changed.

| how the client connected | outage |
|---|---|
| direct, no `connect_timeout`, run 1 | 3.4 s |
| direct, no `connect_timeout`, run 2 | 22.3 s |
| direct, no `connect_timeout`, run 3 | 22.6 s |
| direct, `connect_timeout=2 target_session_attrs=read-write` | 6.7 s |

All four rows are from cluster A, and none of them had a pooler in the path. The
pooler gets its own section below, because it changes the failover rather than the
client.

The first three rows, three runs of the same configuration, are the finding. With no
`connect_timeout`, libpq has no deadline of its own, so how long an attempt takes
is decided by DNS resolution, which address it picks when there are several,
routing, and whether the far end sends a reset or goes silent. An address that
refuses immediately fails fast; one that has stopped answering waits on TCP
retransmission. In run 3 a single connection attempt took 19.6 seconds on its
own.

`connect_timeout=2` does not make the good case better. It bounds each
connection attempt, which is a different thing from bounding the outage: retries,
backoff, TLS and authentication stalls, and a server that accepts the connection
and then hangs are all still in front of you. What it removes is the 19.6 second
single attempt, and that is worth the string change on its own.

It is also a libpq setting. If your client is not libpq, find the equivalent in
your driver and check what it actually bounds, because they differ.

### The pooler makes the failover slow

Time both paths in the same failover or you are comparing failovers, not paths.
This matters because a pooler holding a connection changes
how long the failover takes for every client in the cluster, including the ones
that never touch the pooler.

On cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1, two copies
of the probe ran at once, differing only in the host: `app-db-rw` for the direct
one and `app-db-pooler-rw` for the pooled one.

```sh
# read-write is what makes a connection to a demoted primary fail rather than
# appear to work, and connect_timeout bounds each attempt
URL="postgresql://app@app-db-pooler-rw:5432/app?connect_timeout=2&target_session_attrs=read-write"
while true; do
  if psql "$URL" -Atq -c 'INSERT INTO beat DEFAULT VALUES;' >/dev/null 2>&1
  then echo "$(date -u +%H:%M:%S.%3N) ok"
  else echo "$(date -u +%H:%M:%S.%3N) FAIL"; fi
  sleep 0.2
done
```

Both probe pods sit on a node the test does not touch, or they die with the thing
they are measuring. Every gap in this section is read out of those two logs the
same way, as the last `ok` before a block of `FAIL` and the first `ok` after it:

```sh
kubectl logs writer | awk '
  $2=="FAIL" { if(!n) last=prev; n++; next }
  n { print last, "->", $1, n" failed attempts"; exit }
  { prev=$1 }'
```

The primary pod was then deleted at 02:56:27.275, on that same 2026-09-21 cluster
C running CloudNativePG 1.30.0 with PostgreSQL 18.1 containers, with no
`--grace-period` flag, which matters and is the next subsection:

| path | last write before | first write after | gap |
|---|---|---|---|
| direct to `app-db-rw` | 02:56:27.351 | 02:59:33.327 | 186.0 s |
| through the two-pod PgBouncer `Pooler` | 02:59:27.779 | 02:59:43.619 | 15.8 s |

**Read the middle columns rather than the last one.** The pooled client did not
recover faster. It kept committing, successfully, for three minutes after the
primary had been told to shut down, and only lost its connection when the shutdown
stopped being polite. Meanwhile the direct client, which opens a fresh connection
each attempt, was refused from the first one.

Three logs say why, and they agree. The old primary's:

```
02:56:27.483  Requesting smart shutdown of the PostgreSQL instance
02:56:27.579  received smart shutdown request
02:59:27.925  Error while handling the smart shutdown request
02:59:27.934  received fast shutdown request
```

A smart shutdown refuses new connections and waits for the existing ones to end.
PgBouncer's server connections do not end, so on that 2026-09-21 cluster C delete
on CloudNativePG 1.30.0 with PostgreSQL 18.1 containers it waited out
`.spec.smartShutdownTimeout`, which defaults to 180 and measured 180.4 s from the
smart request to the fast one. The operator's log, for the whole of that window:

```
02:56:27.931  Current primary isn't healthy, initiating a failover
02:56:28.034  Setting primary label to unhealthy in the old primary during failover
02:56:28.083  Waiting for all WAL receivers to be down to elect a new primary
```

On that same 2026-09-21 cluster C delete on CloudNativePG 1.30.0, it decided in
0.66 s, then logged that last line 192 times, the final one at 02:59:19.778, with
nothing else in between. The new primary's log on that same CloudNativePG 1.30.0
cluster on 2026-09-21 closes
it: `I'm the target primary, applying WALs and promoting my instance` at
02:59:29.833, which is 1.9 seconds after the old primary took the fast shutdown
at 02:59:27.934, and `server promoted` at 02:59:30.738, 2.8 seconds after it.

So the pooler is not slow in itself. The pooled sessions keep the old primary
alive, the old primary keeps its WAL receivers up, and the operator will not
promote while they are up.

It reproduces to the second, because what sets the length is a timeout rather than
anything variable. A second delete on the same 2026-09-21 cluster C, CloudNativePG
1.30.0 with PostgreSQL 18.1, gave 185.1 s direct, 15.9 s pooled, and 180.5 s from
the smart shutdown request to the fast one.

**Prove it is the pooled sessions and not the pooler.** Three deletes on cluster
C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1, same probe, none of
them passing `--grace-period`, only the pooler's state different:

| pooler state | the direct client's gap |
|---|---|
| scaled to zero | 6.19 s |
| two pods running, nothing using them | 3.91 s |
| two pods running, a client writing through them | 186.0 s |

A deployed pooler costs nothing. A used one costs the difference. If you run one,
either lower `.spec.smartShutdownTimeout` and accept that transactions in flight
on the old primary get cut, or measure your failover with the pooler under load,
because idle is the case that flatters it.

### The `--grace-period` you pass decides which failure you measured

The `kubectl delete pod` above carries `--grace-period=1`, and everything in this
section so far ran without it. That one flag changes the answer by three
minutes.

CloudNativePG sets the instance pod's `terminationGracePeriodSeconds` from
`.spec.stopDelay`, which defaults to 1800. Leave the flag off and the smart
shutdown has half an hour of room, so it takes its full
`.spec.smartShutdownTimeout`. Pass `--grace-period=1` and the kubelet kills the
container a second in, before the smart shutdown has waited for anything. Both
deletes on cluster C on 2026-09-21, CloudNativePG 1.30.0 with PostgreSQL 18.1
containers, same probes, same pooled client writing throughout:

| delete | direct | through the `Pooler` |
|---|---|---|
| no flag, the pod's own 1800 second grace | 186.0 s | 15.8 s |
| `--grace-period=1` | 22.5 s | 30.6 s |

The bottom row is an abrupt loss, and it is the one PgBouncer pays
`server_login_retry` for, twice, which
[CloudNativePG pooler or direct connections](/learn/postgres/cloudnativepg-pooler-or-direct-connections/)
works through from PgBouncer's own log.
The top row is a cooperative shutdown, which is what a drain, a rolling update or
a `kubectl delete` with the defaults gives you.

Neither is wrong. They are different failures, and your cluster does both: an OOM
kill or a node loss looks like the bottom row, a node drain or an operator upgrade
looks like the top. Test both, and say which one a number came from, because a
page that says "failover took twenty-two seconds" without saying how the primary
died has not told you anything you can plan with.

Nothing was lost either way. On that 2026-09-21 cluster C delete on CloudNativePG
1.30.0 with PostgreSQL 18.1 containers, the 686 rows the pooled client wrote
during that window were all on the new primary afterwards, running to a last row
timestamp of 02:59:27.774 on the database's clock, the same write the table above
logs at 02:59:27.779 on the probe pod's clock, which stamps its line after psql
returns, against 355 rows in the 87 seconds before the delete
and 584 in the 87 seconds after recovery. Replication was not under load in that
run, so read it as this fixture rather than as a promise.

There is one trap when you look for the operator's side of this. `kubectl logs` on
`cnpg-controller-manager` returned only the last forty seconds, because the
operator is chatty enough that containerd had already rotated the file; the window
above came off the node out of
`/var/log/pods/cnpg-system_cnpg-controller-manager-*/manager/`. And do not time
anything off the `FailingOver` Event: Kubernetes aggregates repeats, so its
`lastTimestamp` is the most recent occurrence of that message and not the start of
this failover.

### Now kill a node instead

This is a different failure with a different answer, and testing only the pod
delete is how people end up believing their cluster recovers in seconds.

On cluster B, from stopping the container that is the node:

| what happened | time after the node stopped |
|---|---|
| node `Ready` becomes `Unknown`, primary pod `Ready` becomes `False` | 47 s |
| `FailingOver  Current primary isn't healthy, initiating a failover` | 49 s |
| `status.currentPrimary` moves to the new instance | 70 s |
| the probe writes again | 75.0 s |
| the probe stops failing for good | 78.8 s |

Seventy-five seconds, and the 300 second eviction toleration is not in it.
CloudNativePG's own documentation says so directly:

> Pod eviction (actual deletion from the unreachable node) is a separate
> mechanism, driven by `tolerationSeconds` on the
> `node.kubernetes.io/unreachable` NoExecute taint (300s by default). That timer
> does not hold up the operator's failover decision; CloudNativePG promotes a new
> primary as soon as the `Ready` condition flips.

[CloudNativePG 1.30, Automated failover](https://cloudnative-pg.io/docs/1.30/failover/)

**There is one way to turn that 49 second decision into a 323 second one and be
right about the clock and wrong about the cause.** CloudNativePG ships one replica of
`cnpg-controller-manager`. Put it on the node you stop and the sentence above
still holds, because the toleration is not delaying the decision: it is delaying
the arrival of anything that can make one. Measured on 2026-09-21 on this
CloudNativePG 1.30.0 cluster with PostgreSQL 18.1 on Kubernetes 1.35.5, the same
node stop with the operator on the failing node gave:

```
00:54:35  NodeNotReady           Node is not ready
00:59:35  TaintManagerEviction   Marking for deletion Pod cnpg-system/cnpg-controller-manager-...
00:59:56  LeaderElection         cnpg-controller-manager-...-7kfrt became leader
00:59:58  FailingOver            Current primary isn't healthy, initiating a failover
```

Until it is evicted the cluster reports `Cluster in healthy state`, because the
thing that notices is the thing that is gone.

Five minutes between the node going and the operator being evicted, on the
operator pod's own stock 300 second toleration, then twenty-three seconds to
elect a leader and decide. So pin the operator away from the instances before you
measure anything, or you will publish a number about pod eviction and label it
failover.

Check where yours is before you trust any failover number:

```
kubectl -n cnpg-system get pods -o wide
```

and give the operator deployment anti-affinity from your database nodes, or at
least more than one replica.

The 47 seconds is the part worth tuning, and it is not CloudNativePG's either.
The same page:

> With stock kube-controller-manager settings, the transition is governed by
> `--node-monitor-grace-period` (default 40s on Kubernetes 1.29-1.31, raised to
> 50s in 1.32 and later): after that window the controller marks the node
> `Unknown` and, in the same monitoring pass, issues a patch per pod on that node
> to flip the `Ready` condition.

So on Kubernetes 1.32 and later, expect roughly 50 to 55 seconds of detection
before CloudNativePG has anything to react to. The 47 seconds measured here falls
inside the 40 to 55 seconds the same page gives for the operator seeing the primary
as unready. Managed distributions tune these values; check yours rather
than assuming this number.

Lowering `--node-monitor-grace-period` shortens every node failure in the cluster
and makes every transient API server hiccup look like one. It is a cluster-wide
trade with partition behaviour on the other side of it, not a dial for one
database.

### The failures worth testing

A dead pod, a dead node, an unreachable node and a network partition are four
different tests, and this page has measured two of them. The list, in the order
they get skipped:

- the primary process killed inside a healthy pod,
- the primary pod deleted,
- the node stopped,
- the node partitioned from the control plane while clients can still reach it,
- a zone partition, if the cluster spans zones,
- the volume failing under the primary.

The fourth is the one where CloudNativePG's safety machinery matters rather than
its speed: the primary lease under `.spec.primaryLease`, the isolation liveness
check, and failover quorum under `.spec.postgresql.synchronous.failoverQuorum`.
Those decide whether a partition ends in a promotion or in two primaries, and
none of them are exercised by `kubectl delete pod`.

## Did it lose anything?

Time is only half the question. Run the probe so it records the id of every row
the server acknowledged, then compare that list against the table afterwards.

**Counting will not do it.** After cluster B's node kill the table held 603 rows
with a maximum id of 660. Neither number says whether an acknowledged id is
missing from the middle, and the gaps in a `bigserial` are there whether or not
anything was lost. Compare the sets:

```sh
kubectl exec app-db-2 -c postgres -- \
  psql -U postgres -d app -tAc "copy (select op_id from beat) to stdout" \
  | sort > present.txt
grep ' OK ' probe.log | awk '{print $3}' | sort > acknowledged.txt

comm -23 acknowledged.txt present.txt   # acknowledged and gone: this is your RPO
comm -13 acknowledged.txt present.txt   # present and never acknowledged
```

The second list is not an error. An `INSERT` can commit and the acknowledgement
can be lost on the way back, which leaves the client believing it failed. Retry
that blindly and you have a duplicate, so reconcile on the `op_id` the client
generated rather than on the `bigserial` the server did, which is why the probe
table above has one.

Four tests:

- **Pod deleted, asynchronous replication, cluster A.** 382 acknowledged ids, 382
  present. Nothing lost. The pod gets a signal, PostgreSQL shuts down cleanly, and
  the replica is caught up before the promotion. Note the shape of that test:
  `--grace-period=1` gives the shutdown one second to finish, which on a quiet
  database is enough for a checkpoint and on a busy one is not, so it sits
  between a clean stop and an abrupt loss rather than being either.
- **Node stopped, asynchronous replication, cluster B.** 356 acknowledged ids, 0
  missing. The replicas were caught up when the node went away.
- **Node killed, asynchronous replication, cluster A.** 500,000 rows
  acknowledged, 300,000 of them gone after the promotion. The replica that got
  promoted had itself rejoined the cluster 80 seconds earlier and was still
  catching up.
- **Node killed, synchronous replication, cluster A.** 300,000 rows
  acknowledged, 300,000 present. Nothing lost.

Read the middle two together, because they are the same failure with opposite
outcomes. Under asynchronous replication the primary can acknowledge a commit
that no other instance holds, so your RPO is whatever the replay lag was at the
moment the primary died. You can watch that number, but you cannot promise it.
Synchronous replication is what lets you promise it.

Turning that on is one patch:

```
kubectl patch cluster app-db --type merge \
  -p '{"spec":{"postgresql":{"synchronous":{"method":"any","number":1}}}}'
```

After which the standbys report `sync_state = quorum`:

<!-- sql-check: postgres -->
```sql
SELECT application_name, state, sync_state, write_lag, flush_lag
FROM pg_stat_replication
ORDER BY application_name;
```

Synchronous replication has a cost. The same failover probe that took 6.7 seconds
asynchronously took 22.4 seconds with `ANY 1` configured, so the durability was paid
for in outage time. Make that trade deliberately, after measuring both sides.

Be careful with the setting that sounds like it already does this:

<!-- sql-check: postgres -->
```sql
SELECT current_setting('synchronous_commit')      AS commit_mode,
       current_setting('synchronous_standby_names') AS standbys;
```

`synchronous_commit = on` with `synchronous_standby_names` empty means flushed to
the local disk and acknowledged. No standby was involved. That is the default in
a three-instance CloudNativePG cluster.

## How do I stop it regressing?

**Set `connect_timeout` in every connection string**, and `target_session_attrs`
where the driver supports it. That is the single cheapest improvement measured
here, and it is a string change. Check what your driver's equivalent bounds; not
every client is libpq.

**Test the node failure, not only the pod failure.** With nothing holding a
session open on the old primary, the pod deletes on this page put the client back
in service in 3.4 to 22.6 seconds, the widest of them a run with no
`connect_timeout` set; the node stop took 75.0, and the difference is node
detection rather than anything the database does.

**Measure with everything else connected too.** The same pod delete was 3.91
seconds with the pooler idle and 186.0 with a client writing through it. A
failover test against a quiet cluster measures a failure mode you do not have.

**Check where the operator is scheduled.** A controller on the same node as the
primary turns a 75.0 second failover into one that does not begin until the
controller's own pod is evicted, five minutes in.

**Run the test again after every operator upgrade** and after any change to
storage, pooling or replication. Three of the numbers above changed when only the
client configuration changed.

**Record what you ran it against.** Operator version, operand image, Kubernetes
version and distribution, the dataplane, the storage class, where each instance
and the client sat, and the connection string. Without those the numbers are
anecdotes, including these ones.

**Decide synchronous or asynchronous with both numbers in front of you**, rather
than by reputation.

## How DBGorilla helps

DBGorilla connects read-only and samples the instance continuously, so a
failover leaves a shape in the data rather than a gap you have to reconstruct:
connection counts and per-session state either side of the promotion, workload
by query fingerprint before and after, and topology snapshots with their
history. All of that was verified in a live session.
[Get started free →](https://app.dbgorilla.com/signup)

## Related

- [Why a pod restart is not a clean PostgreSQL recovery](https://www.dbgorilla.com/learn/postgres/why-a-pod-restart-is-not-a-clean-postgres-recovery/)
- [CloudNativePG pooler or direct connections?](https://www.dbgorilla.com/learn/postgres/cloudnativepg-pooler-or-direct-connections/)
- [Is CloudNativePG ready to replace RDS, Aurora or AlloyDB?](https://www.dbgorilla.com/learn/postgres/is-cloudnativepg-ready-to-replace-rds-aurora-or-alloydb/)