Skip to content

My pg_wal directory filled the disk, what is safe to delete?

Last updated

View as Markdown

Quick answer: Nothing in pg_wal is safe to delete by hand. Removing WAL files can leave the database unable to recover and can permanently break replicas and backups. WAL piles up because something is pinning it, usually an inactive replication slot, a failing archive_command, or wal_keep_size set too high. Fix the cause and PostgreSQL recycles the files itself at the next checkpoint.

The write-ahead log is PostgreSQL’s durability mechanism, not a cache or a log file in the syslog sense. On crash recovery PostgreSQL replays WAL from the last checkpoint forward; if a segment it needs is missing, the cluster will not start, and the remediation is restoring from backup. Delete the wrong file and you have turned a full disk into data loss.

pg_wal is also not a directory that “grows out of control” on its own. Under normal operation PostgreSQL removes or recycles segments at every checkpoint. If files are accumulating, some component has told the server “do not remove those yet.” Find that component and revoke its hold.

(On PostgreSQL 9.6 and earlier this directory is named pg_xlog. It was renamed because too many people assumed a directory called “log” was safe to clear out.)

Step 1: Confirm pg_wal is actually what filled the disk

Section titled “Step 1: Confirm pg_wal is actually what filled the disk”

Bloated tables, runaway temp files in base/pgsql_tmp, and unrotated server logs fill disks too.

Terminal window
df -h /var/lib/postgresql # adjust to your data directory mount
du -sh /var/lib/postgresql/*/main/* | sort -h | tail -20

From inside the database, without shell access:

SELECT pg_size_pretty(sum(size)) AS pg_wal_total,
count(*) AS segment_count
FROM pg_ls_waldir();

Default segment size is 16 MB, so segment count × 16 MB should match. A steady-state cluster typically holds somewhere around max_wal_size worth plus a margin:

SHOW max_wal_size; -- soft target for WAL between checkpoints
SHOW min_wal_size;
SHOW wal_keep_size; -- extra segments held for replicas without slots
SHOW archive_mode;
SHOW archive_command;
SHOW archive_library; -- if set, the module archives, not archive_command

Check archive_library before you spend any time on archive_command. The two are mutually exclusive: when archive_library is set to a non-empty value that module does the archiving. On PostgreSQL 15 archive_command is then ignored entirely; from PostgreSQL 16, setting both is an error and the archiver exits instead of archiving anything. Otherwise you can spend twenty minutes at 3am debugging a shell command the server never runs. If archive_library is set, the failure and its logging belong to that module, so check its own diagnostics rather than archive_command.

If total pg_wal size is much larger than max_wal_size, something is pinning WAL. Move to step 2.

A quick check is to count the archive backlog directly:

SELECT count(*) FROM pg_ls_dir('pg_wal/archive_status') WHERE pg_ls_dir ~ '\.ready$';

(pg_ls_waldir needs superuser or membership in pg_monitor. pg_ls_dir needs superuser unless someone has granted EXECUTE on it. The unaliased pg_ls_dir in the WHERE is the function’s own output column. It is valid, though an alias reads better.)

Thousands of .ready files means archiving is failing or far behind, and every one of those segments is being retained until it succeeds.

Several holds are common. Check all of them, because they can coexist.

Inactive replication slots are the most common cause.

SELECT
slot_name,
slot_type,
database,
active,
active_pid,
restart_lsn,
pg_size_pretty(
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
) AS wal_retained
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;
What you seeWhat it means
active = false, wal_retained in the tens of GBA dead consumer is holding your disk. This is the cause
active = true, wal_retained large and growingA live consumer that is lagging badly, such as a replica that cannot keep up or a stalled CDC pipeline
wal_retained small on every slotSlots are not the problem; check archiving

On versions that expose them, wal_status and safe_wal_size are more direct:

SELECT slot_name, active, wal_status, pg_size_pretty(safe_wal_size) AS safe_wal_size
FROM pg_replication_slots;

wal_status progresses reserved → extended → unreserved → lost. Anything past reserved means the slot is beyond the normal retention limit; lost means required WAL has already been removed and that consumer cannot resume. It needs rebuilding regardless of what you do next.

Why does a slot pin WAL at all? A replication slot is a durable promise. The consumer, whether a physical standby or a logical decoding client like Debezium, tells the primary “I have processed up to LSN X.” The primary must then retain every WAL segment from that point onward, because the consumer is allowed to disconnect and reconnect later and resume exactly where it left off with no gap. That guarantee protects replication when a replica reboots. The cost is that a consumer which never comes back never advances its confirmed position, so the promise is never released, and WAL accumulates forever unless you have configured a ceiling.

If archive_mode = on, PostgreSQL will not remove a WAL segment until it has been successfully archived. Whether the archiving is done by a shell archive_command or by an archive module named by archive_library, a failure means WAL retention grows without bound. Typical causes are expired cloud credentials, a full archive destination, a mistyped path, or an unmounted NFS share.

The pg_stat_archiver view below works either way. The remediation differs: if archive_library is set, the module does the archiving, so fix the module’s configuration rather than the shell command.

SELECT archived_count,
last_archived_wal,
last_archived_time,
failed_count,
last_failed_wal,
last_failed_time,
stats_reset
FROM pg_stat_archiver;
SignalDiagnosis
failed_count high, last_failed_time recentArchiving is broken right now. This is the cause
last_archived_time hours old while writes continueArchiving is stalled or extremely slow
last_failed_wal equals a segment that still existsIt is retrying the same segment and failing

Then look at the server log for the actual error text. archive_command failed with exit code N lines tell you whether it is permissions, network, or disk.

SHOW wal_keep_size; -- named wal_keep_segments on older versions

This is a flat floor of WAL retained at all times, independently of max_wal_size, and regardless of whether any replica exists or uses a slot. It exists to let a slotless replica catch up, but the retention itself is unconditional, so do not rule it out just because all your replicas use slots.

If someone set it to a large value to paper over a replica that kept falling behind, that value now permanently occupies disk. It is less common than the other causes but easy to miss because it looks like a deliberate setting.

2d. less common: checkpoints not completing

Section titled “2d. less common: checkpoints not completing”

If checkpoints are stalled, WAL since the last checkpoint cannot be released.

{/* sql-check: pg 17+ */}

-- PostgreSQL 17+: checkpoint counters moved out of pg_stat_bgwriter into
-- their own view. num_done was added in PostgreSQL 18, omit it on 17.
SELECT num_timed, num_requested, write_time, sync_time, stats_reset
FROM pg_stat_checkpointer;

{/* sql-check: pg -16 */}

-- PostgreSQL 16 and earlier: the same counters live in pg_stat_bgwriter
SELECT checkpoints_timed, checkpoints_req, checkpoint_write_time,
checkpoint_sync_time, stats_reset
FROM pg_stat_bgwriter;
SELECT pg_current_wal_lsn(), pg_walfile_name(pg_current_wal_lsn());

A cluster with very large max_wal_size and a heavy write burst can also legitimately hold a lot of WAL without anything being wrong, because it is sized bigger than the volume it lives on. That is a capacity problem rather than an incident cause, but rule it in or out before you look for a hold.

Step 3: The safe remediation for each cause

Section titled “Step 3: The safe remediation for each cause”

If you have confirmed the consumer is gone permanently:

-- DANGEROUS for the consumer. Safe for the database.
SELECT pg_drop_replication_slot('old_standby_slot');

What this breaks, explicitly:

  • A physical standby using that slot loses its retention guarantee. If it comes back and the WAL it needs has since been recycled, it cannot catch up and must be re-seeded from a fresh base backup (or pg_rewind, if applicable).
  • A logical replication / CDC consumer loses its decoding position entirely. Slots are the only record of where a logical consumer was. Dropping it means that pipeline must be re-snapshotted, which can take hours for a large table, and downstream systems will see a gap or a full reload.

You cannot drop a slot while it is active; you must stop the consumer first. Do not reflexively terminate the consumer’s backend to force the drop: if it is a live replica that is simply lagging, killing it makes the problem worse. Confirm the consumer is truly abandoned first.

Another option for a logical slot you want to keep but need to unpin:

-- DANGEROUS: skips ahead; the consumer will never see the skipped changes.
-- Replace the LSN with one you chose from the query above.
SELECT pg_replication_slot_advance('cdc_slot', '<target_lsn>'::pg_lsn);

This releases the retained WAL without dropping the slot, at the cost of a data gap for that consumer. Only do this if you can reconcile downstream. An analytics sink can usually be backfilled; a system of record usually cannot.

WAL is not freed the instant you run the command. A checkpoint has to happen. You can force one:

CHECKPOINT; -- safe; causes an I/O burst, so expect a latency bump

The fix is to make archiving succeed. If the destination is reachable and it is a credentials or permissions problem, fix that and archiving drains on its own. WAL frees at the following checkpoint.

If you cannot fix the destination fast enough and the disk is about to fill, you can neutralize the command temporarily:

# postgresql.conf, DANGEROUS: creates a hole in your archive
archive_command = '/bin/true'
SELECT pg_reload_conf();

The trade-off: every segment “archived” this way is never actually stored, which means your point-in-time recovery window is broken from this moment until you take a fresh base backup. That gap in your recovery capability is permanent. Do it only to keep production alive, write down the exact time you did it, restore the real command immediately after, and take a new base backup as the very next task.

Turning archive_mode = off requires a restart and has the same recovery consequence, so the /bin/true route is usually preferred under pressure.

wal_keep_size = 1GB # or 0 if every replica uses a slot
SELECT pg_reload_conf();

Safe, provided your replicas use replication slots (which give them a stronger guarantee anyway). If a slotless replica is currently behind by more than the new value, lowering it can strand that replica, so check replica lag first.

Step 4: Emergency space when the disk is already full

Section titled “Step 4: Emergency space when the disk is already full”

If the volume is at 100%, PostgreSQL may have already PANICed and shut down, and it may refuse to start because it cannot write WAL. You need headroom before anything else works.

In rough order of preference:

  1. Delete things that are not PostgreSQL data. Old server logs (not WAL), rotated logs, package caches, core dumps, forgotten tarballs in /tmp. Start here.

  2. Delete a ballast file if you have one. A pre-allocated multi-GB dummy file on the data volume exists for this situation. If you do not have one, create one after this incident. fallocate -l 10G /var/lib/postgresql/BALLAST costs nothing and guarantees you a way to free space.

  3. Grow the volume. On cloud block storage this is often an online operation and is the safest real fix. Do this while you diagnose, not after.

  4. Move the archive destination or old backups off the volume, if they inadvisedly share it.

  5. Last resort: relocate pg_wal to another filesystem. With the server cleanly stopped, move the directory and symlink it back:

    Terminal window
    # Server MUST be stopped. Verify with pg_ctl status. Do not do this while running.
    # Get the real path from the cluster itself, do not assume a distribution
    # layout, and make sure it is the cluster you think it is. Run this BEFORE
    # stopping the server:
    # psql -Atc 'SHOW data_directory'
    PGDATA=$(psql -Atc 'SHOW data_directory') # e.g. /var/lib/postgresql/18/main
    mv "$PGDATA/pg_wal" /bigdisk/pg_wal
    ln -s /bigdisk/pg_wal "$PGDATA/pg_wal"
    chown -h postgres:postgres "$PGDATA/pg_wal"

    This is a supported layout, but it adds a second filesystem to your durability path: if /bigdisk disappears, so does your database. Treat it as temporary unless you plan it properly.

Why you do not just rm the oldest segments: the oldest segments are the ones being retained deliberately. They are the files the archiver has not stored, that a slot still points at, or that recovery would need to replay from. Here the oldest segments are the most needed, not the most disposable.

The pg_archivecleanup tool removes WAL, but its intended target is the archive destination, where it cleans segments older than a restart point. Pointing it at a live cluster’s pg_wal is an expert operation with the same failure modes as rm, and it should not be run at 3am on the strength of a search result.

Once you have space, start the server, then work through steps 2 and 3 to fix the actual cause. Freeing space without fixing the cause only delays the next full disk.

Set max_slot_wal_keep_size. Left unset (unlimited), a single forgotten slot can fill any disk. When it is set, PostgreSQL invalidates a slot that exceeds the limit rather than let it take the cluster down.

max_slot_wal_keep_size = 100GB

(Available in PostgreSQL 13 and later. On older versions there is no such ceiling, so slot monitoring is mandatory there.)

How to size it. It is a trade between two failure modes: too small and a replica that reboots for 20 minutes gets invalidated and needs a full re-seed; too large and it cannot prevent the disk from filling. Work it out from your actual numbers:

  1. Measure your WAL generation rate. Sample pg_current_wal_lsn() an hour apart:

    -- run, wait an hour, run again, then:
    SELECT pg_size_pretty(pg_wal_lsn_diff('<later_lsn>', '<earlier_lsn>')) AS wal_per_hour;

    Do this during a busy hour, and again during your heaviest batch job, because a nightly bulk load can generate more WAL in 20 minutes than the whole rest of the day.

  2. Decide your tolerable consumer outage. How long should a replica or CDC pipeline be able to be down and still resume without a rebuild? For a physical standby with fast automated re-seeding, a couple of hours may be plenty. For a logical slot feeding a warehouse where re-snapshotting takes six hours, you want a day or more.

  3. Multiply, using the peak rate, then sanity-check against free disk. If peak WAL is 10 GB/hour and you want to survive a 12-hour outage, that is 120 GB, and you need to have 120 GB of headroom on the WAL volume plus room for max_wal_size and archiving lag on top. If you do not, meeting that retention target requires more storage, not a config change. Set the limit to what the disk can absorb (say 60 to 70% of free space) and accept that a long outage means a rebuild.

Alert on all four signals. Each can be monitored and can prevent a 3am page:

-- Slots pinning significant WAL, or inactive at all
SELECT slot_name, active,
pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS bytes_retained
FROM pg_replication_slots
WHERE active = false
OR pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 10 * 1024^3;
-- Archiver failing
SELECT failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver;
-- pg_wal total size
SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

The fourth is plain free disk percentage on the WAL volume, from outside the database, so you still get the page when PostgreSQL itself is the thing that is stuck.

Give pg_wal its own volume if you can. Then a WAL surge degrades replication instead of taking down the entire cluster, and you get a separate metric to alarm on.

Clean up slots as part of decommissioning. The most common origin of this incident is a replica or CDC connector that was torn down months ago and whose slot nobody dropped. Make “drop the replication slot” an explicit line item in the runbook for removing any WAL consumer.

It does so less directly than most database incidents. An AI coding agent will readily scaffold a logical replication setup or a CDC connector (CREATE_REPLICATION_SLOT, a Debezium config, a pg_recvlogical sidecar), because that is a well-documented, locally-correct pattern. What it does not add is the operational half: a max_slot_wal_keep_size ceiling, monitoring on slot lag, or teardown logic that drops the slot when the consumer is removed.

The same gap shows up in generated infrastructure code: an experimental replica spun up in a branch environment, torn down by deleting the pod or the VM, with the slot left behind on the primary, retaining WAL indefinitely. The code that created the problem is correct; the code that would have prevented it was never in scope.

  • Can I delete files from pg_wal to free disk space?

    No. Manually deleting files from pg_wal can make the database unable to recover after a crash and can permanently break replicas and point-in-time recovery. WAL accumulates because something is pinning it. Remove the cause and PostgreSQL recycles or removes the files itself at the next checkpoint.

  • Why is my pg_wal directory growing?

    Almost always one of three things: an inactive replication slot whose consumer stopped confirming progress, a failing archive_command that leaves WAL marked not-yet-archived, or wal_keep_size set high. Less commonly, a very high write rate combined with a large max_wal_size or checkpoints that cannot keep up.

  • Why does a replication slot pin WAL?

    A slot is a durable promise. The primary must retain every WAL segment from the slot restart_lsn onward until the consumer confirms it has processed that far, so the consumer can disconnect and resume without data loss. If the consumer never comes back, the promise is never released and WAL accumulates without limit unless max_slot_wal_keep_size is set.

  • Is it safe to drop a replication slot?

    It frees the WAL immediately, but it is destructive for that consumer. A standby using the slot can no longer catch up if the WAL it still needs is gone, and a logical consumer such as a CDC pipeline loses its position and must be re-seeded. Drop a slot only when you have confirmed the consumer is gone for good or you accept rebuilding it.

  • Does WAL free up immediately after I fix the cause?

    No. Removal and recycling happen at a checkpoint. You can run CHECKPOINT to force one; it is safe but causes an I/O burst.

  • What is the one setting that prevents pg_wal filling the disk?

    max_slot_wal_keep_size, on PostgreSQL 13 and later. It caps how much WAL a slot may pin, invalidating the slot instead of filling the disk. Size it from your measured peak WAL rate times the consumer outage you want to survive, bounded by real free disk.