My pg_wal directory filled the disk, what is safe to delete?
Quick answer: Nothing in
pg_walis safe to delete by hand. Removing WAL files can leave the database unable to recover and can permanently break replicas and backups. WAL piles up because something is pinning it, usually an inactive replication slot, a failingarchive_command, orwal_keep_sizeset too high. Fix the cause and PostgreSQL recycles the files itself at the next checkpoint.
Rule zero: do not rm anything in pg_wal
Section titled “Rule zero: do not rm anything in pg_wal”The write-ahead log is PostgreSQL’s durability mechanism, not a cache or a log file in the syslog sense. On crash recovery PostgreSQL replays WAL from the last checkpoint forward; if a segment it needs is missing, the cluster will not start, and the remediation is restoring from backup. Delete the wrong file and you have turned a full disk into data loss.
pg_wal is also not a directory that “grows out of control” on its own. Under
normal operation PostgreSQL removes or recycles segments at every checkpoint. If
files are accumulating, some component has told the server “do not remove those
yet.” Find that component and revoke its hold.
(On PostgreSQL 9.6 and earlier this directory is named pg_xlog. It was renamed
because too many people assumed a directory called “log” was safe to
clear out.)
Step 1: Confirm pg_wal is actually what filled the disk
Section titled “Step 1: Confirm pg_wal is actually what filled the disk”Bloated tables, runaway temp files in base/pgsql_tmp, and unrotated
server logs fill disks too.
df -h /var/lib/postgresql # adjust to your data directory mountdu -sh /var/lib/postgresql/*/main/* | sort -h | tail -20From inside the database, without shell access:
SELECT pg_size_pretty(sum(size)) AS pg_wal_total, count(*) AS segment_countFROM pg_ls_waldir();Default segment size is 16 MB, so segment count × 16 MB should match. A steady-state
cluster typically holds somewhere around max_wal_size worth plus a margin:
SHOW max_wal_size; -- soft target for WAL between checkpointsSHOW min_wal_size;SHOW wal_keep_size; -- extra segments held for replicas without slotsSHOW archive_mode;SHOW archive_command;SHOW archive_library; -- if set, the module archives, not archive_commandCheck archive_library before you spend any time on archive_command. The two
are mutually exclusive: when archive_library is set to a non-empty value that
module does the archiving. On PostgreSQL 15 archive_command is then ignored
entirely; from PostgreSQL 16, setting both is an error and the archiver exits
instead of archiving anything. Otherwise you can spend twenty minutes at
3am debugging a shell command the server never runs. If archive_library is set, the
failure and its logging belong to that module, so check its own diagnostics rather
than archive_command.
If total pg_wal size is much larger than max_wal_size, something is
pinning WAL. Move to step 2.
A quick check is to count the archive backlog directly:
SELECT count(*) FROM pg_ls_dir('pg_wal/archive_status') WHERE pg_ls_dir ~ '\.ready$';(pg_ls_waldir needs superuser or membership in pg_monitor. pg_ls_dir
needs superuser unless someone has granted EXECUTE on it. The unaliased pg_ls_dir in the WHERE is the
function’s own output column. It is valid, though an alias reads better.)
Thousands of .ready files means archiving is failing or far behind, and every one
of those segments is being retained until it succeeds.
Step 2: Find what is pinning the WAL
Section titled “Step 2: Find what is pinning the WAL”Several holds are common. Check all of them, because they can coexist.
2a. inactive replication slots
Section titled “2a. inactive replication slots”Inactive replication slots are the most common cause.
SELECT slot_name, slot_type, database, active, active_pid, restart_lsn, pg_size_pretty( pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) ) AS wal_retainedFROM pg_replication_slotsORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;| What you see | What it means |
|---|---|
active = false, wal_retained in the tens of GB | A dead consumer is holding your disk. This is the cause |
active = true, wal_retained large and growing | A live consumer that is lagging badly, such as a replica that cannot keep up or a stalled CDC pipeline |
wal_retained small on every slot | Slots are not the problem; check archiving |
On versions that expose them, wal_status and safe_wal_size are more direct:
SELECT slot_name, active, wal_status, pg_size_pretty(safe_wal_size) AS safe_wal_sizeFROM pg_replication_slots;wal_status progresses reserved → extended → unreserved → lost. Anything
past reserved means the slot is beyond the normal retention limit; lost means
required WAL has already been removed and that consumer cannot resume. It needs
rebuilding regardless of what you do next.
Why does a slot pin WAL at all? A replication slot is a durable promise. The consumer, whether a physical standby or a logical decoding client like Debezium, tells the primary “I have processed up to LSN X.” The primary must then retain every WAL segment from that point onward, because the consumer is allowed to disconnect and reconnect later and resume exactly where it left off with no gap. That guarantee protects replication when a replica reboots. The cost is that a consumer which never comes back never advances its confirmed position, so the promise is never released, and WAL accumulates forever unless you have configured a ceiling.
2b. archiving is failing
Section titled “2b. archiving is failing”If archive_mode = on, PostgreSQL will not remove a WAL segment until it has
been successfully archived. Whether the archiving is done by a shell
archive_command or by an archive module named by archive_library, a failure
means WAL retention grows without bound. Typical causes are expired cloud
credentials, a full archive destination, a mistyped path, or an unmounted NFS share.
The pg_stat_archiver view below works either way. The remediation differs:
if archive_library is set, the module does the archiving, so fix the module’s
configuration rather than the shell command.
SELECT archived_count, last_archived_wal, last_archived_time, failed_count, last_failed_wal, last_failed_time, stats_resetFROM pg_stat_archiver;| Signal | Diagnosis |
|---|---|
failed_count high, last_failed_time recent | Archiving is broken right now. This is the cause |
last_archived_time hours old while writes continue | Archiving is stalled or extremely slow |
last_failed_wal equals a segment that still exists | It is retrying the same segment and failing |
Then look at the server log for the actual error text. archive_command failed with exit code N lines tell you whether it is permissions, network, or disk.
2c. wal_keep_size held too high
Section titled “2c. wal_keep_size held too high”SHOW wal_keep_size; -- named wal_keep_segments on older versionsThis is a flat floor of WAL retained at all times, independently of max_wal_size, and regardless of whether any replica exists or uses a slot. It exists to let a slotless replica catch up, but the retention itself is
unconditional, so do not rule it out just because all your replicas use slots.
If someone set it to a large value to paper over a replica that kept falling behind, that value now permanently occupies disk. It is less common than the other causes but easy to miss because it looks like a deliberate setting.
2d. less common: checkpoints not completing
Section titled “2d. less common: checkpoints not completing”If checkpoints are stalled, WAL since the last checkpoint cannot be released.
{/* sql-check: pg 17+ */}
-- PostgreSQL 17+: checkpoint counters moved out of pg_stat_bgwriter into-- their own view. num_done was added in PostgreSQL 18, omit it on 17.SELECT num_timed, num_requested, write_time, sync_time, stats_resetFROM pg_stat_checkpointer;{/* sql-check: pg -16 */}
-- PostgreSQL 16 and earlier: the same counters live in pg_stat_bgwriterSELECT checkpoints_timed, checkpoints_req, checkpoint_write_time, checkpoint_sync_time, stats_resetFROM pg_stat_bgwriter;SELECT pg_current_wal_lsn(), pg_walfile_name(pg_current_wal_lsn());A cluster with very large max_wal_size and a heavy write burst can also legitimately
hold a lot of WAL without anything being wrong, because it is sized bigger than the
volume it lives on. That is a capacity problem rather than an incident cause, but rule
it in or out before you look for a hold.
Step 3: The safe remediation for each cause
Section titled “Step 3: The safe remediation for each cause”Dead replication slot
Section titled “Dead replication slot”If you have confirmed the consumer is gone permanently:
-- DANGEROUS for the consumer. Safe for the database.SELECT pg_drop_replication_slot('old_standby_slot');What this breaks, explicitly:
- A physical standby using that slot loses its retention guarantee. If it comes
back and the WAL it needs has since been recycled, it cannot catch up and must be
re-seeded from a fresh base backup (or
pg_rewind, if applicable). - A logical replication / CDC consumer loses its decoding position entirely. Slots are the only record of where a logical consumer was. Dropping it means that pipeline must be re-snapshotted, which can take hours for a large table, and downstream systems will see a gap or a full reload.
You cannot drop a slot while it is active; you must stop the consumer first. Do
not reflexively terminate the consumer’s backend to force the drop: if it is a
live replica that is simply lagging, killing it makes the problem worse. Confirm the
consumer is truly abandoned first.
Another option for a logical slot you want to keep but need to unpin:
-- DANGEROUS: skips ahead; the consumer will never see the skipped changes.-- Replace the LSN with one you chose from the query above.SELECT pg_replication_slot_advance('cdc_slot', '<target_lsn>'::pg_lsn);This releases the retained WAL without dropping the slot, at the cost of a data gap for that consumer. Only do this if you can reconcile downstream. An analytics sink can usually be backfilled; a system of record usually cannot.
WAL is not freed the instant you run the command. A checkpoint has to happen. You can force one:
CHECKPOINT; -- safe; causes an I/O burst, so expect a latency bumpBroken archive_command
Section titled “Broken archive_command”The fix is to make archiving succeed. If the destination is reachable and it is a credentials or permissions problem, fix that and archiving drains on its own. WAL frees at the following checkpoint.
If you cannot fix the destination fast enough and the disk is about to fill, you can neutralize the command temporarily:
# postgresql.conf, DANGEROUS: creates a hole in your archivearchive_command = '/bin/true'SELECT pg_reload_conf();The trade-off: every segment “archived” this way is never actually stored, which means your point-in-time recovery window is broken from this moment until you take a fresh base backup. That gap in your recovery capability is permanent. Do it only to keep production alive, write down the exact time you did it, restore the real command immediately after, and take a new base backup as the very next task.
Turning archive_mode = off requires a restart and has the same recovery
consequence, so the /bin/true route is usually preferred under pressure.
Wal_keep_size too high
Section titled “Wal_keep_size too high”wal_keep_size = 1GB # or 0 if every replica uses a slotSELECT pg_reload_conf();Safe, provided your replicas use replication slots (which give them a stronger guarantee anyway). If a slotless replica is currently behind by more than the new value, lowering it can strand that replica, so check replica lag first.
Step 4: Emergency space when the disk is already full
Section titled “Step 4: Emergency space when the disk is already full”If the volume is at 100%, PostgreSQL may have already PANICed and shut down, and it may refuse to start because it cannot write WAL. You need headroom before anything else works.
In rough order of preference:
-
Delete things that are not PostgreSQL data. Old server logs (not WAL), rotated logs, package caches, core dumps, forgotten tarballs in
/tmp. Start here. -
Delete a ballast file if you have one. A pre-allocated multi-GB dummy file on the data volume exists for this situation. If you do not have one, create one after this incident.
fallocate -l 10G /var/lib/postgresql/BALLASTcosts nothing and guarantees you a way to free space. -
Grow the volume. On cloud block storage this is often an online operation and is the safest real fix. Do this while you diagnose, not after.
-
Move the archive destination or old backups off the volume, if they inadvisedly share it.
-
Last resort: relocate
pg_walto another filesystem. With the server cleanly stopped, move the directory and symlink it back:Terminal window # Server MUST be stopped. Verify with pg_ctl status. Do not do this while running.# Get the real path from the cluster itself, do not assume a distribution# layout, and make sure it is the cluster you think it is. Run this BEFORE# stopping the server:# psql -Atc 'SHOW data_directory'PGDATA=$(psql -Atc 'SHOW data_directory') # e.g. /var/lib/postgresql/18/mainmv "$PGDATA/pg_wal" /bigdisk/pg_walln -s /bigdisk/pg_wal "$PGDATA/pg_wal"chown -h postgres:postgres "$PGDATA/pg_wal"This is a supported layout, but it adds a second filesystem to your durability path: if
/bigdiskdisappears, so does your database. Treat it as temporary unless you plan it properly.
Why you do not just rm the oldest segments: the oldest segments are the ones
being retained deliberately. They are the files the archiver has not stored, that a
slot still points at, or that recovery would need to replay from. Here the oldest
segments are the most needed, not the most disposable.
The pg_archivecleanup tool removes WAL, but its intended target is the
archive destination, where it cleans segments older than a restart point. Pointing it at
a live cluster’s pg_wal is an expert operation with the same failure modes as
rm, and it should not be run at 3am on the strength of a search result.
Once you have space, start the server, then work through steps 2 and 3 to fix the actual cause. Freeing space without fixing the cause only delays the next full disk.
How do I stop it coming back?
Section titled “How do I stop it coming back?”Set max_slot_wal_keep_size. Left unset (unlimited), a
single forgotten slot can fill any disk. When it is set, PostgreSQL invalidates a slot that
exceeds the limit rather than let it take the cluster down.
max_slot_wal_keep_size = 100GB(Available in PostgreSQL 13 and later. On older versions there is no such ceiling, so slot monitoring is mandatory there.)
How to size it. It is a trade between two failure modes: too small and a replica that reboots for 20 minutes gets invalidated and needs a full re-seed; too large and it cannot prevent the disk from filling. Work it out from your actual numbers:
-
Measure your WAL generation rate. Sample
pg_current_wal_lsn()an hour apart:-- run, wait an hour, run again, then:SELECT pg_size_pretty(pg_wal_lsn_diff('<later_lsn>', '<earlier_lsn>')) AS wal_per_hour;Do this during a busy hour, and again during your heaviest batch job, because a nightly bulk load can generate more WAL in 20 minutes than the whole rest of the day.
-
Decide your tolerable consumer outage. How long should a replica or CDC pipeline be able to be down and still resume without a rebuild? For a physical standby with fast automated re-seeding, a couple of hours may be plenty. For a logical slot feeding a warehouse where re-snapshotting takes six hours, you want a day or more.
-
Multiply, using the peak rate, then sanity-check against free disk. If peak WAL is 10 GB/hour and you want to survive a 12-hour outage, that is 120 GB, and you need to have 120 GB of headroom on the WAL volume plus room for
max_wal_sizeand archiving lag on top. If you do not, meeting that retention target requires more storage, not a config change. Set the limit to what the disk can absorb (say 60 to 70% of free space) and accept that a long outage means a rebuild.
Alert on all four signals. Each can be monitored and can prevent a 3am page:
-- Slots pinning significant WAL, or inactive at allSELECT slot_name, active, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS bytes_retainedFROM pg_replication_slotsWHERE active = false OR pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 10 * 1024^3;
-- Archiver failingSELECT failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver;
-- pg_wal total sizeSELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();The fourth is plain free disk percentage on the WAL volume, from outside the database, so you still get the page when PostgreSQL itself is the thing that is stuck.
Give pg_wal its own volume if you can. Then a WAL surge degrades replication
instead of taking down the entire cluster, and you get a separate metric to alarm on.
Clean up slots as part of decommissioning. The most common origin of this incident is a replica or CDC connector that was torn down months ago and whose slot nobody dropped. Make “drop the replication slot” an explicit line item in the runbook for removing any WAL consumer.
Why does AI-generated code cause this?
Section titled “Why does AI-generated code cause this?”It does so less directly than most database incidents. An AI coding agent
will readily scaffold a logical replication setup or a CDC
connector (CREATE_REPLICATION_SLOT, a Debezium config, a pg_recvlogical
sidecar), because that is a well-documented, locally-correct pattern. What it
does not add is the operational half: a max_slot_wal_keep_size ceiling,
monitoring on slot lag, or teardown logic that drops the slot when the consumer
is removed.
The same gap shows up in generated infrastructure code: an experimental replica spun up in a branch environment, torn down by deleting the pod or the VM, with the slot left behind on the primary, retaining WAL indefinitely. The code that created the problem is correct; the code that would have prevented it was never in scope.
Can I delete files from pg_wal to free disk space?
No. Manually deleting files from pg_wal can make the database unable to recover after a crash and can permanently break replicas and point-in-time recovery. WAL accumulates because something is pinning it. Remove the cause and PostgreSQL recycles or removes the files itself at the next checkpoint.
Why is my pg_wal directory growing?
Almost always one of three things: an inactive replication slot whose consumer stopped confirming progress, a failing archive_command that leaves WAL marked not-yet-archived, or wal_keep_size set high. Less commonly, a very high write rate combined with a large max_wal_size or checkpoints that cannot keep up.
Why does a replication slot pin WAL?
A slot is a durable promise. The primary must retain every WAL segment from the slot restart_lsn onward until the consumer confirms it has processed that far, so the consumer can disconnect and resume without data loss. If the consumer never comes back, the promise is never released and WAL accumulates without limit unless max_slot_wal_keep_size is set.
Is it safe to drop a replication slot?
It frees the WAL immediately, but it is destructive for that consumer. A standby using the slot can no longer catch up if the WAL it still needs is gone, and a logical consumer such as a CDC pipeline loses its position and must be re-seeded. Drop a slot only when you have confirmed the consumer is gone for good or you accept rebuilding it.
Does WAL free up immediately after I fix the cause?
No. Removal and recycling happen at a checkpoint. You can run CHECKPOINT to force one; it is safe but causes an I/O burst.
What is the one setting that prevents pg_wal filling the disk?
max_slot_wal_keep_size, on PostgreSQL 13 and later. It caps how much WAL a slot may pin, invalidating the slot instead of filling the disk. Size it from your measured peak WAL rate times the consumer outage you want to survive, bounded by real free disk.