Skip to content

Should WAL have its own PVC on CloudNativePG?

Last updated

View as Markdown

Quick answer: Usually yes, but check the storage class first. If a volume fills, it is better for it to hold WAL than tables. Two PVCs are often two directories on the same node disk, which provides naming isolation but no storage isolation. Adding walStorage later is a rolling restart, not a rebuild, so you can defer the decision.

PostgreSQL writes two different streams. Table and index changes go to the data directory in scattered pages of the default 8 kB block size, which the preset options documentation, read on 2026-09-19, gives for every supported version. The write-ahead log goes to pg_wal as a single sequential append, and it is the record that makes a crash survivable.

By default they share a filesystem. If either one fills it, the server stops accepting writes, and a PostgreSQL instance that cannot write WAL cannot do anything at all.

CloudNativePG lets you split them:

spec:
storage:
size: 2Gi
walStorage:
size: 1Gi

This creates a second PVC per instance named <cluster>-N-wal, plus a symlink inside the data directory:

/var/lib/postgresql/data/pgdata/pg_wal -> /var/lib/postgresql/wal/pg_wal

PostgreSQL sees no difference. The only change is which volume can fill.

Check this before you decide anything else

Section titled “Check this before you decide anything else”

Two PVCs are not necessarily two disks. Measured on 2026-09-19 on CloudNativePG 1.30.0 running PostgreSQL 18.4, on a k3d cluster with the default local-path provisioner, after adding walStorage to a running cluster:

$ df -h /var/lib/postgresql/data /var/lib/postgresql/wal
Filesystem Size Used Avail Use% Mounted on
/dev/root 495G 459G 37G 93% /var/lib/postgresql/data
/dev/root 495G 459G 37G 93% /var/lib/postgresql/wal

Same device, same filesystem, same free space. local-path creates a directory on the node, so the two claims are two paths into one disk. That gives you the symlink and the separate PVC object, but not capacity isolation, fault isolation or independent throughput. If that node disk fills, both volumes are full at the same instant.

It is also why a requested PVC size may not be a limit. A PVC request is a request to a provisioner. Whether the provisioner enforces it depends on that provisioner, not Kubernetes. On the same cluster, both claims asked for 2Gi:

$ kubectl get pvc -o custom-columns=NAME:.metadata.name,\
REQ:.spec.resources.requests.storage,SC:.spec.storageClassName
NAME REQ SC
walsplit-1 2Gi local-path
walsplit-1-wal 2Gi local-path
walsplit-2 2Gi local-path
walsplit-2-wal 2Gi local-path

Four claims of 2Gi each, all of them backed by the one 495G filesystem in the df above. The number in the manifest is a request that this provisioner records but does not enforce, as the next measurement shows.

With max_slot_wal_keep_size at -1, one abandoned physical slot created with immediately_reserve, and six bursts of 400,000 rows each, the WAL directory on walsplit-1-wal went 673M, 977M, 1.4G, 1.9G, 2.3G, 2.8G. Nothing stopped it at the 2Gi request. Measured on 2026-09-19 on the same CloudNativePG 1.30.0 and PostgreSQL 18.4 cluster, at the end of the sixth burst:

$ kubectl exec walsplit-1 -c postgres -- du -sh /var/lib/postgresql/wal/pg_wal
3.0G /var/lib/postgresql/wal/pg_wal
$ kubectl get pvc -o custom-columns=NAME:.metadata.name,\
REQ:.spec.resources.requests.storage,CAP:.status.capacity.storage
NAME REQ CAP
walsplit-1-wal 2Gi 2Gi
$ kubectl exec walsplit-1 -c postgres -- psql -U postgres -c \
"SELECT slot_name, active, wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))
FROM pg_replication_slots"
slot_name | active | wal_status | pg_size_pretty
-----------+--------+------------+----------------
abandoned | f | extended | 3027 MB

Measured on 2026-09-19 on the k3d bed, CloudNativePG 1.30.0 running PostgreSQL 18.4: 3.0G in a volume that both claims and reports a 2Gi capacity, and the pod was still Running with zero restarts. The limit that was going to be hit was the free space left on the node filesystem, which the df above put at 37G of a 495G device already 93 percent full, not the PVC’s 2Gi. Treat a PVC size on local-path as a label, not as a cap, and check yours rather than carrying that conclusion across storage classes.

Run that df on your own storage class before relying on the arguments for splitting. On a class that provisions a real volume per claim, with its own capacity and its own IOPS policy, the separation is real and so are the benefits. On local-path it is a naming convention.

The version qualifiers matter throughout this page, and there are two test beds behind it rather than one. Everything about CloudNativePG, PVCs and local-path was run on a k3d cluster, CloudNativePG 1.30.0, PostgreSQL 18.4. Everything about replication slot semantics was run on stock postgres:18.6 containers, because those are PostgreSQL behaviours and the operator is not involved. Both were measured on 2026-09-19, on Linux 6.17.0 under Docker 29.1.3. Every measurement below names which of the two it came from.

PostgreSQL semantics are portable. CloudNativePG behaviour is tied to 1.30.0 and was tested there. Storage class behaviour is specific to your environment and is not portable.

An abandoned replication slot is a common cause and has its own section below, but no single setting closes the question. WAL is retained by:

  • Write rate against checkpoint behaviour. max_wal_size is a soft trigger, not a ceiling.
  • wal_keep_size, which retains segments for standbys with no slot involved. The PostgreSQL 18 replication configuration documentation is careful that this “sets only the minimum size of segments retained in pg_wal; the system might need to retain more segments for WAL archival or to recover from a checkpoint.”
  • A physical replica that is lagging, with or without a slot.
  • Logical decoding consumers, whose loss is not the same event as losing a physical standby. A rebuilt read replica is an afternoon. A CDC pipeline that missed a window may be an application-level reconciliation.
  • Archiving failures, through either archive_command or archive_library. Both exist in 18 and pg_settings lists both.
  • Restartpoint behaviour during recovery.
  • Long-running backups that require a particular WAL range.

A slot cap and a separate PVC address two of those.

A new slot holds nothing, until it is used once

Section titled “A new slot holds nothing, until it is used once”

Measured on 2026-09-19 on a stock postgres:18.6 container. Two slots, one created the default way and one with immediate reservation:

SELECT pg_create_physical_replication_slot('slot_default');
SELECT pg_create_physical_replication_slot('slot_reserved', true);
SELECT slot_name, active, restart_lsn, wal_status, safe_wal_size
FROM pg_replication_slots ORDER BY 1;
slot_name | active | restart_lsn | wal_status | safe_wal_size
---------------+--------+-------------+------------+---------------
slot_default | f | | |
slot_reserved | f | 0/1761918 | reserved |

The second argument is immediately_reserve, and at the protocol level it is RESERVE_WAL. Without it, the slot reserves WAL when a streaming client connects, not when the statement runs. restart_lsn IS NULL is the direct test for that, and it is better than reading wal_status, which is NULL here only as a consequence.

That NULL only means no client has connected yet. Connect a streaming client for six seconds and disconnect it:

$ timeout 6 pg_receivewal -h 127.0.0.1 -U postgres -D /tmp/wal -S slot_default --no-loop
slot_name | active | restart_lsn | wal_status | safe_wal_size
---------------+--------+-------------+------------+---------------
slot_default | f | 0/1000000 | reserved |
slot_reserved | f | 0/1761918 | reserved |

The inactive slot is now pinning WAL, and at an older position than the one created to reserve immediately. A slot that was used once and then abandoned is the dangerous case, because slots usually outlive the consumer that created them.

The cap acts at a checkpoint, not when the limit is crossed

Section titled “The cap acts at a checkpoint, not when the limit is crossed”

This determines whether max_slot_wal_keep_size protects you when needed. The PostgreSQL 18 documentation for max_slot_wal_keep_size says it specifies “the maximum size of WAL files that replication slots are allowed to retain in the pg_wal directory at checkpoint time”.

Measured on 2026-09-19 on a stock postgres:18.6 container, with max_slot_wal_keep_size=32MB, max_wal_size=8GB and checkpoint_timeout=3600 so that no automatic checkpoint could interfere:

pointwal_statussafe_wal_sizecheckpointsWAL segments
slot createdreserved40 MB01
after ~200 MB of WALunreserved-177 MB015
after ~400 MB of WALunreserved-394 MB028
after one CHECKPOINTlostNULL128

Measured on 2026-09-19 on the slot-semantics bed, a stock postgres:18.6 container on Linux 6.17.0 under Docker 29.1.3, not on the k3d CloudNativePG cluster. The slot sat 394 MB beyond a 32 MB cap, across two write bursts, with WAL accumulating the whole time, and nothing invalidated it. A cap of 32 MB did not bound pg_wal to 32 MB. It limited what the slot could hold when the next checkpoint ran.

On a cluster with a long checkpoint_timeout and a quiet write pattern, that gap can be long. Size the volume for the gap, not for the cap.

The states are documented and they are not a two-way switch. From the PostgreSQL 18 pg_replication_slots documentation: reserved “means that the claimed files are within max_wal_size”, extended “means that max_wal_size is exceeded but the files are still retained”, unreserved “means that the slot no longer retains the required WAL files and some of them are to be removed at the next checkpoint”, and lost “means that this slot is no longer usable”. The same page adds that unreserved “can return to reserved or extended”. A consumer that reconnects before the next checkpoint saves its slot.

The PostgreSQL 18 pg_replication_slots documentation defines safe_wal_size as “The number of bytes that can be written to WAL such that this slot is not in danger of getting in state ‘lost’”, and adds that it “is NULL for lost slots, as well as if max_slot_wal_keep_size is -1”. So a NULL safe_wal_size means either “no limit configured” or “already too late”, which are opposite situations. Read it with wal_status, never alone. It also goes negative before invalidation, which makes it a usable alert: alert on safe_wal_size < 0, not on it reaching zero.

The parameter name suggests a shared pool, but the limit applies to each slot. Measured on 2026-09-19 on the same stock postgres:18.6 container with the same 32MB cap, one slot left far behind and a second created after the write burst:

slot_name | restart_lsn | wal_status | safe_wal_size
-------------+-------------+------------+---------------
fresh | 0/7682B6E0 | reserved | 28 MB
old_laggard | | lost |

One runaway slot does not consume an allowance that the others then go without. The PostgreSQL 18 documentation for max_slot_wal_keep_size matches: the limit applies “if restart_lsn of a replication slot falls behind the current LSN by more than the given size”, which is a per-slot distance.

What is cluster-wide is that there is one number for every consumer. A disposable read replica and a CDC pipeline with a four-hour recovery objective get the same budget, and you cannot express the difference. The response to that constraint is an inventory: every slot, its owner, its consumer type, and what losing it costs.

Invalidating a slot does not shrink pg_wal

Section titled “Invalidating a slot does not shrink pg_wal”

Measured on 2026-09-19 on the slot-semantics bed, a stock postgres:18.6 container rather than the k3d CloudNativePG cluster, because this is a PostgreSQL behaviour and the operator is not involved. Two checkpoints after the slot above went lost, pg_wal still held 28 segments and 448 MB, with the oldest file named at the current write position. The segments were recycled, not removed: preallocated for future reuse, because max_wal_size was 8GB and PostgreSQL had no reason to give the space back.

Invalidating the slot frees the WAL only in one sense. It removes the requirement. Whether disk usage falls depends on max_wal_size and min_wal_size. If you are watching a full volume recover, watch free space, not slot status.

An invalidated slot costs you its consumer. An uncapped slot can cost you the primary. That is a real trade and the cap is usually worth setting, but two qualifications belong with it.

The consumer is not always a replica you can rebuild in an afternoon. Price the loss per consumer before choosing one number for all of them.

A cap is one guardrail, not the disk-exhaustion control. The list at the top of this page has seven entries and max_slot_wal_keep_size bounds one of them. A volume can still fill with archiving stalled and no slot involved at all.

ALTER SYSTEM is off by default and is an opt-in, not a prohibition

Section titled “ALTER SYSTEM is off by default and is an opt-in, not a prohibition”

Measured on 2026-09-19 on CloudNativePG 1.30.0 running PostgreSQL 18.4. Out of the box:

$ psql -tAc "SHOW allow_alter_system"
off
$ psql -c "ALTER SYSTEM SET max_slot_wal_keep_size = '1GB'"
ERROR: ALTER SYSTEM is not allowed in this environment

The operator exposes a switch for it. After kubectl patch cluster walsplit --type merge -p '{"spec":{"postgresql":{"enableAlterSystem":true}}}':

$ psql -tAc "SHOW allow_alter_system"
on
$ psql -c "ALTER SYSTEM SET max_slot_wal_keep_size = '1GB'"
ALTER SYSTEM

The change was not reverted. A common claim is that the next reconcile undoes a hand-made change. In this test it did not. Measured on 2026-09-19 on CloudNativePG 1.30.0 running PostgreSQL 18.4, on a k3d cluster with local-path:

$ kubectl exec walsplit-2 -c postgres -- \
psql -tAc "SELECT name, setting, source FROM pg_settings
WHERE name='max_slot_wal_keep_size'"
max_slot_wal_keep_size|-1|default
$ kubectl delete pod walsplit-2
pod "walsplit-2" deleted from default namespace
$ kubectl exec walsplit-2 -c postgres -- \
cat /var/lib/postgresql/data/pgdata/postgresql.auto.conf
# Do not edit this file manually!
# It will be overwritten by the ALTER SYSTEM command.
max_slot_wal_keep_size = '1GB'
$ kubectl exec walsplit-2 -c postgres -- \
psql -tAc "SELECT name, setting, source FROM pg_settings
WHERE name='max_slot_wal_keep_size'"
max_slot_wal_keep_size|1024|configuration file

Read the source column twice. Before the pod was deleted the running server was still on the default, because ALTER SYSTEM writes the file and does not reload it. After the operator recreated the pod the hand-made value was live. Deleting the pod did not undo the change. It applied it.

It also survived a reconcile driven by the manifest. Patching .spec.postgresql.parameters with work_mem: 5MB brought the cluster back to Cluster in healthy state with both values in place:

max_slot_wal_keep_size|1024|configuration file
work_mem|5120|configuration file

It persists in postgresql.auto.conf on the PGDATA volume and the operator leaves it there.

This has a drawback. A setting applied that way is invisible in the Cluster manifest, is not in your Git history, and will not be corrected for you. Patch the manifest:

kubectl patch cluster app-db --type merge \
-p '{"spec":{"postgresql":{"parameters":{"max_slot_wal_keep_size":"256MB"}}}}'
SELECT slot_name,
slot_type,
active,
wal_status,
pg_size_pretty(safe_wal_size) AS safe_wal,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS retained
FROM pg_replication_slots
ORDER BY retained DESC;

And the settings that determine the calculation:

SELECT name, setting, unit, source
FROM pg_settings
WHERE name IN ('max_wal_size', 'min_wal_size', 'wal_keep_size',
'max_slot_wal_keep_size', 'archive_mode', 'archive_command',
'archive_library', 'archive_timeout')
ORDER BY name;

max_wal_size defaults to 1GB and the PostgreSQL 18 WAL configuration documentation, read on 2026-09-19, is explicit that it is not a hard limit. So a multiple of it is not a sizing method: it is a number that happens to be larger, derived from a parameter that describes checkpoint pacing rather than how long a consumer might be away.

Size from your own numbers:

WAL capacity >= (peak WAL rate x longest interruption you intend to survive)
+ checkpoint and recycling headroom
+ filesystem and operational headroom

The interruption has to be the real worst case: an archive destination outage, a replica or network outage, a maintenance window, and the time to notice plus the time to act.

Worked through with numbers you choose rather than numbers measured here: at a peak of 40 MiB/s and a 30 minute tolerated interruption, that is about 70 GiB of WAL before any headroom. Against that, a 1Gi WAL PVC is incompatible with the objective. The case against 1Gi is not that it is small next to a 1GB max_wal_size. It is that 1Gi cannot survive any interruption worth naming.

Then set max_slot_wal_keep_size as a deliberate portion of that budget, below the amount that would exhaust the PVC, and remember from the measurement above that enforcement waits for a checkpoint.

If archiving is configured and failing, WAL is retained until it succeeds, whether archiving runs through archive_command or archive_library.

SELECT archived_count,
last_archived_wal,
last_archived_time,
failed_count,
last_failed_wal,
last_failed_time
FROM pg_stat_archiver;

Read it as two questions rather than one number. failed_count is cumulative, so a non-zero value says something failed at some point, not that anything is failing now. Measured on 2026-09-19 on the same CloudNativePG 1.30.0 and PostgreSQL 18.4 cluster, by writing and forcing four WAL switches on the primary and sampling either side:

T0 archived=8 failed=1 last_archived=000000020000000000000009.partial last_failed=00000003.history
T1 archived=12 failed=1 last_archived=00000003000000000000000C last_failed=00000003.history

Four more segments archived, last_archived_wal advancing, and failed_count unchanged on a .history file. A history file failing matters for timeline recovery, and it did not stop a single segment from being archived. An alert on failed_count > 0 would have been firing throughout.

The risky condition is a sustained absence of progress while WAL is being generated: last_archived_time not advancing, last_failed_wal naming a real segment, and pg_wal growing. Alert on that combination and on WAL filesystem utilisation, not on failed_count alone.

Decide walStorage on the storage class, not by default. Run the df check above. If both claims land on the same device, a separate PVC buys you a symlink, and adequate shared capacity with a slot cap and archive monitoring may be the simpler design. Where the class gives each claim real capacity and a real failure domain, use the isolation.

A second volume adds a dependency. It is a second attachment that has to mount for the instance to start. On a healthy multi-instance cluster that is manageable, but it is still a trade.

You are not locked out by deciding later. Measured on 2026-09-19 on CloudNativePG 1.30.0 with PostgreSQL 18.4, patching walStorage into a running two-instance cluster took about 35 seconds end to end: the operator provisioned walsplit-1-wal and walsplit-2-wal, restarted each instance in turn, logged the cluster phase Primary instance is being restarted without a switchover, moved the existing segments and repointed pg_wal at the new volume. The PGDATA PVCs were untouched and no data was re-cloned. It is a planned rolling restart with real availability implications, not a rebuild.

Inventory every slot, with an owner and a consumer type. The cap is one number for all of them, so the differences have to live somewhere, and the only place left is a list somebody maintains.

Alert on safe_wal_size < 0 and on wal_status IN ('unreserved','lost'), with WAL filesystem utilisation and archive progress beside them. Pair each with an action: find the consumer, restore it, drop the orphan, or reseed.

Drop slots you are finished with. Most abandoned slots come from a replica deleted without its slot, or a logical subscriber turned off.

  • Does a separate WAL volume make PostgreSQL faster?

    It depends entirely on whether the two PVCs land on different devices, and often they do not. Measured on 2026-09-19 on CloudNativePG 1.30.0 with the local-path provisioner, both PVCs reported the same filesystem and the same 495G device, so the separation was by directory only. On a storage class that gives each claim its own volume with its own IOPS policy it can matter. Check yours rather than assuming either way.

  • How big should the WAL PVC be?

    Size it from your own peak WAL rate multiplied by the longest archive or replication interruption you intend to survive, plus headroom for checkpoint recycling. Worked through with numbers you choose rather than numbers measured here: at a peak of 40 MiB/s and a 30 minute tolerated outage that is about 70 GiB before headroom. A number expressed as a multiple of max_wal_size is not a sizing method, because max_wal_size is a soft checkpoint trigger and says nothing about how long a consumer might be away.

  • What actually fills a WAL volume?

    An abandoned replication slot is one cause and it is a common one, but it is not the only one. Write rate against checkpoint behaviour, wal_keep_size, a lagging physical replica, logical decoding consumers, archive failures through archive_command or archive_library, and restartpoint behaviour during recovery all retain WAL too. Setting a slot cap and adding a PVC does not close the question.

  • How do I stop a slot filling the disk?

    max_slot_wal_keep_size, understanding that it acts at a checkpoint and not when the limit is crossed. Measured on 2026-09-19 on a stock postgres:18.6 container with checkpoint_timeout at one hour, a slot sat 394 MB past a 32MB cap in wal_status = unreserved, with WAL still accumulating, across two write bursts and zero checkpoints. One explicit CHECKPOINT moved it to lost. Between those two points the cap was protecting nothing.

  • Can I just run ALTER SYSTEM to set it?

    By default no: CloudNativePG 1.30.0 starts PostgreSQL with allow_alter_system off and ALTER SYSTEM returns an error. It is an opt-in, not a prohibition: setting .spec.postgresql.enableAlterSystem to true turned it on and the statement succeeded. Measured on 2026-09-19 on CloudNativePG 1.30.0 with PostgreSQL 18.4, a value set that way then survived both a pod recreation and a Cluster spec change, so it is not corrected by reconciliation. Patch the manifest anyway, because a setting only the container knows about is one nobody will find.