My pg_wal directory filled the disk, what is safe to delete?

Quick answer: Nothing in pg_wal is safe to delete by hand. Removing WAL files can leave the database unable to recover and can permanently break replicas and backups. WAL piles up because something is pinning it, usually an inactive replication slot, a failing archive_command, or wal_keep_size set too high. Fix the cause and PostgreSQL recycles the files itself at the next checkpoint.

Rule zero: do not rm anything in pg_wal

Say it out loud before you touch the shell.

The write-ahead log is not a cache and it is not a log file in the syslog sense. It is the durability mechanism. On crash recovery PostgreSQL replays WAL from the last checkpoint forward; if a segment it needs is missing, the cluster will not start, and the remediation is restoring from backup. Delete the wrong file and you have turned a full disk into data loss.

pg_wal is also not a directory that "grows out of control" on its own. Under normal operation PostgreSQL removes or recycles segments at every checkpoint. If files are accumulating, some component has told the server "do not remove those yet." Your job in this incident is to find out which one and revoke the hold, not to fight the filesystem.

(On PostgreSQL 9.6 and earlier this directory is named pg_xlog. It was renamed specifically because too many people assumed a directory called "log" was safe to clear out.)

Step 1: Confirm pg_wal is actually what filled the disk

Do not assume. Bloated tables, runaway temp files in base/pgsql_tmp, and unrotated server logs fill disks too.

df -h /var/lib/postgresql          # adjust to your data directory mount
du -sh /var/lib/postgresql/*/main/*  | sort -h | tail -20

From inside the database, without shell access:

SELECT pg_size_pretty(sum(size)) AS pg_wal_total,
       count(*)                  AS segment_count
FROM pg_ls_waldir();

Default segment size is 16 MB, so segment count × 16 MB should match. A steady-state cluster typically holds somewhere around max_wal_size worth plus a margin:

SHOW max_wal_size;      -- soft target for WAL between checkpoints
SHOW min_wal_size;
SHOW wal_keep_size;     -- extra segments held for replicas without slots
SHOW archive_mode;
SHOW archive_command;
SHOW archive_library;   -- if set, archive_command is IGNORED

Check archive_library before you spend any time on archive_command. The two are mutually exclusive: when archive_library is set to a non-empty value that module does the archiving and archive_command is ignored entirely. Debugging a shell command the server never runs is a classic way to lose twenty minutes at 3am. If archive_library is set, the failure and its logging belong to that module, check its own diagnostics, not archive_command.

If total pg_wal size is dramatically larger than max_wal_size, something is pinning WAL. Move to step 2.

Also check the archive backlog directly. This is a fast tell:

SELECT count(*) FROM pg_ls_dir('pg_wal/archive_status') WHERE pg_ls_dir ~ '\.ready$';

(pg_ls_dir and pg_ls_waldir need superuser, or membership in pg_monitor or pg_read_server_files. The unaliased pg_ls_dir in the WHERE is the function's own output column, valid, though aliasing reads better.)

Thousands of .ready files means archiving is failing or far behind, and every one of those segments is being retained until it succeeds.

Step 2: Find what is pinning the WAL

There are three common holds. Check all three. They can coexist.

2a. inactive replication slots

This is the number one cause.

SELECT
    slot_name,
    slot_type,
    database,
    active,
    active_pid,
    restart_lsn,
    pg_size_pretty(
        pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
    ) AS wal_retained
FROM pg_replication_slots
ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC NULLS LAST;

What you see	What it means
active = false, wal_retained in the tens of GB	A dead consumer is holding your disk. This is your culprit
active = true, wal_retained large and growing	A live but badly lagging consumer, a replica that cannot keep up, or a stalled CDC pipeline
wal_retained small on every slot	Slots are not the problem; check archiving

On versions that expose them, wal_status and safe_wal_size are more direct:

SELECT slot_name, active, wal_status, pg_size_pretty(safe_wal_size) AS safe_wal_size
FROM pg_replication_slots;

wal_status progresses reserved → extended → unreserved → lost. Anything past reserved means the slot is beyond the normal retention limit; lost means required WAL has already been removed and that consumer cannot resume, it needs rebuilding regardless of what you do next.

Why does a slot pin WAL at all? A replication slot is a durable promise. The consumer, a physical standby, or a logical decoding client like Debezium, tells the primary "I have processed up to LSN X." The primary must then retain every WAL segment from that point onward, because the consumer is allowed to disconnect and reconnect later and resume exactly where it left off with no gap. That guarantee is the entire point of slots; it is what makes replication safe against a replica rebooting. The cost is that a consumer which never comes back never advances its confirmed position, so the promise is never released, and WAL accumulates forever unless you have configured a ceiling.

2b. archiving is failing

If archive_mode = on, PostgreSQL will not remove a WAL segment until it has been successfully archived. Whatever is doing that archiving, a shell archive_command, or an archive module named by archive_library, a failure means WAL retention grows without bound. Typical causes: expired cloud credentials, a full archive destination, a mistyped path, an unmounted NFS share.

The pg_stat_archiver view below works either way. Only the remediation differs: if archive_library is set, archive_command is ignored, so fix the module's configuration rather than the shell command.

SELECT archived_count,
       last_archived_wal,
       last_archived_time,
       failed_count,
       last_failed_wal,
       last_failed_time,
       stats_reset
FROM pg_stat_archiver;

Signal	Diagnosis
failed_count high, last_failed_time recent	Archiving is broken right now. This is your cause
last_archived_time hours old while writes continue	Archiving is stalled or extremely slow
last_failed_wal equals a segment that still exists	It is retrying the same segment and failing

Then look at the server log for the actual error text. archive_command failed with exit code N lines tell you whether it is permissions, network, or disk.

2c. wal_keep_size held too high

SHOW wal_keep_size;   -- named wal_keep_segments on older versions

This is a flat floor of WAL retained at all times, independently of max_wal_size, and regardless of whether any replica exists or uses a slot. Its purpose is to let a slotless replica catch up, but the retention itself is unconditional, so do not rule it out just because all your replicas use slots.

If someone set it to a large value to paper over a replica that kept falling behind, that value is now a permanent tax on your disk. It is the least common of the three but the easiest to miss, because it looks like a deliberate setting.

2d. less common: checkpoints not completing

If checkpoints are stalled, WAL since the last checkpoint cannot be released.

-- PostgreSQL 17+: checkpoint counters moved out of pg_stat_bgwriter into
-- their own view. num_done was added in PostgreSQL 18, omit it on 17.
SELECT num_timed, num_requested, write_time, sync_time, stats_reset
FROM pg_stat_checkpointer;

-- PostgreSQL 16 and earlier: the same counters live in pg_stat_bgwriter
SELECT checkpoints_timed, checkpoints_req, checkpoint_write_time,
       checkpoint_sync_time, stats_reset
FROM pg_stat_bgwriter;

SELECT pg_current_wal_lsn(), pg_walfile_name(pg_current_wal_lsn());

A cluster with very large max_wal_size and a heavy write burst can also legitimately hold a lot of WAL without anything being wrong. It is just sized bigger than the volume it lives on. That is a capacity mistake, not an incident cause, but it is worth ruling in before you go hunting.

Step 3: The safe remediation for each cause

Dead replication slot

If you have confirmed the consumer is gone permanently:

-- DANGEROUS for the consumer. Safe for the database.
SELECT pg_drop_replication_slot('old_standby_slot');

What this breaks, explicitly:

• A physical standby using that slot loses its retention guarantee. If it comes back and the WAL it needs has since been recycled, it cannot catch up and must be re-seeded from a fresh base backup (or pg_rewind, if applicable).
• A logical replication / CDC consumer loses its decoding position entirely. Slots are the only record of where a logical consumer was. Dropping it means that pipeline must be re-snapshotted, for a large table that can be hours, and downstream systems will see a gap or a full reload.

You cannot drop a slot while it is active; you must stop the consumer first. Do not reflexively terminate the consumer's backend to force the drop, if it is a live replica that is simply lagging, killing it makes the problem worse. Confirm the consumer is truly abandoned first.

A middle path for a logical slot you want to keep but need to unpin:

Do not copy the LSN below

0/A0000000 is a placeholder, not a value to run. The target LSN is the decision, every change between the slot's current restart_lsn and wherever you advance it is discarded, permanently, with no way to replay it. Pasting a stranger's LSN at 3am is how you silently drop a day of CDC events.

Read the slot's current position first, and choose a target deliberately:

SELECT slot_name, restart_lsn, confirmed_flush_lsn,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained
FROM pg_replication_slots WHERE slot_name = 'cdc_slot';

-- DANGEROUS: skips ahead; the consumer will never see the skipped changes.
-- Replace the LSN with one you chose from the query above.
SELECT pg_replication_slot_advance('cdc_slot', '<target_lsn>'::pg_lsn);

This releases the retained WAL without dropping the slot, at the cost of a data gap for that consumer. Only do this if you can reconcile downstream, for an analytics sink you can backfill, for a system of record you usually cannot.

Either way, WAL is not freed the instant you run the command. A checkpoint has to happen. You can force one:

CHECKPOINT;   -- safe; causes an I/O burst, so expect a latency bump

Broken archive_command

The fix is to make archiving succeed. If the destination is reachable and it is a credentials or permissions problem, fix that and archiving drains on its own. WAL frees at the following checkpoint.

If you cannot fix the destination fast enough and the disk is about to fill, you can neutralize the command temporarily:

# postgresql.conf, DANGEROUS: creates a hole in your archive
archive_command = '/bin/true'

SELECT pg_reload_conf();

Understand exactly what you are trading: every segment "archived" this way is never actually stored, which means your point-in-time recovery window is broken from this moment until you take a fresh base backup. That is a real, permanent gap in your recovery capability, not a temporary inconvenience. Do it only to keep production alive, write down the exact time you did it, restore the real command immediately after, and take a new base backup as the very next task.

Turning archive_mode = off requires a restart and has the same recovery consequence, so the /bin/true route is usually preferred under pressure.

Wal_keep_size too high

wal_keep_size = 1GB    # or 0 if every replica uses a slot

SELECT pg_reload_conf();

Safe, provided your replicas use replication slots (which give them a stronger guarantee anyway). If a slotless replica is currently behind by more than the new value, lowering it can strand that replica, check replica lag first.

Step 4: Emergency space when the disk is already full

If the volume is at 100%, PostgreSQL may have already PANICed and shut down, and it may refuse to start because it cannot write WAL. You need headroom before anything else works.

In rough order of preference:

1. Delete things that are not PostgreSQL data. Old server logs (not WAL), rotated logs, package caches, core dumps, forgotten tarballs in /tmp. Always the first move.

2. Delete a ballast file if you have one. A pre-allocated multi-GB dummy file on the data volume exists precisely for this moment. If you do not have one, create one after this incident.fallocate -l 10G /var/lib/postgresql/BALLAST costs nothing and buys you a guaranteed escape hatch.

3. Grow the volume. On cloud block storage this is often an online operation and is by far the safest real fix. Do this while you diagnose, not after.

4. Move the archive destination or old backups off the volume, if they inadvisedly share it.

5. Last resort: relocate pg_wal to another filesystem. With the server cleanly stopped, move the directory and symlink it back:

   # Server MUST be stopped. Verify with pg_ctl status. Do not do this while running.
   # Get the real path from the cluster itself, do not assume a distribution
   # layout, and make sure it is the cluster you think it is. Run this BEFORE
   # stopping the server:
   #   psql -Atc 'SHOW data_directory'
   PGDATA=$(psql -Atc 'SHOW data_directory')   # e.g. /var/lib/postgresql/18/main
   
   mv "$PGDATA/pg_wal" /bigdisk/pg_wal
   ln -s /bigdisk/pg_wal "$PGDATA/pg_wal"
   chown -h postgres:postgres "$PGDATA/pg_wal"

   This is a supported layout, but you have now added a second filesystem to your durability path, if /bigdisk disappears, so does your database. Treat it as temporary unless you plan it properly.

Why you do not just rm the oldest segments: the oldest segments are precisely the ones being retained deliberately. They are the file the archiver has not stored, or the file a slot is still pointing at, or the file recovery would need to replay from. "Oldest" is a proxy for "most load-bearing" here, not "most disposable."

There is a tool, pg_archivecleanup, that removes WAL, its intended target is the archive destination, cleaning segments older than a restart point. Pointing it at a live cluster's pg_wal is an expert operation with the same failure modes as rm, and it is not the move to make at 3am on the basis of a search result.

Once you have space, start the server, then work through steps 2 and 3 to fix the actual cause. Space alone just resets the timer.

How do I stop it coming back?

max_slot_wal_keep_size is the guard that matters. Left unset (unlimited), a single forgotten slot can fill any disk. Set, PostgreSQL will invalidate a slot that exceeds the limit rather than let it take the cluster down.

max_slot_wal_keep_size = 100GB

(Available in PostgreSQL 13 and later. On older versions there is no such ceiling, which is exactly why slot monitoring is non-negotiable there.)

How to size it. It is a trade between two failure modes: too small and a replica that reboots for 20 minutes gets invalidated and needs a full re-seed; too large and it cannot prevent the disk from filling. Work it out from your actual numbers:

1. Measure your WAL generation rate. Sample pg_current_wal_lsn() an hour apart:

   -- run, wait an hour, run again, then:
   SELECT pg_size_pretty(pg_wal_lsn_diff('<later_lsn>', '<earlier_lsn>')) AS wal_per_hour;

   Do this during a busy hour, and again during your heaviest batch job, a nightly bulk load can generate more WAL in 20 minutes than the whole rest of the day.

2. Decide your tolerable consumer outage. How long should a replica or CDC pipeline be able to be down and still resume without a rebuild? For a physical standby with fast automated re-seeding, a couple of hours may be plenty. For a logical slot feeding a warehouse where re-snapshotting is a six-hour ordeal, you want a day or more.

3. Multiply, using the peak rate, then sanity-check against free disk. If peak WAL is 10 GB/hour and you want to survive a 12-hour outage. That is 120 GB, and you need to actually have 120 GB of headroom on the WAL volume plus room for max_wal_size and archiving lag on top. If you do not, the honest answer is that your retention target is a storage purchase, not a config value. Set the limit to what the disk can genuinely absorb (say 60–70% of free space) and accept that a long outage means a rebuild.

Alert on all four signals, because each is cheap and each one is a 3am call avoided:

-- Slots pinning significant WAL, or inactive at all
SELECT slot_name, active,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS bytes_retained
FROM pg_replication_slots
WHERE active = false
   OR pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) > 10 * 1024^3;

-- Archiver failing
SELECT failed_count, last_failed_wal, last_failed_time FROM pg_stat_archiver;

-- pg_wal total size
SELECT pg_size_pretty(sum(size)) FROM pg_ls_waldir();

Plus plain free disk percentage on the WAL volume, from outside the database, so you still get the page when PostgreSQL itself is the thing that is stuck.

Give pg_wal its own volume if you can. Then a WAL surge degrades replication instead of taking down the entire cluster, and you get an isolated metric to alarm on.

Clean up slots as part of decommissioning. The single most common origin story for this incident is a replica or CDC connector that was torn down months ago and whose slot nobody dropped. Make "drop the replication slot" an explicit line item in the runbook for removing any WAL consumer.

Why does aI-generated code cause this?

Less directly than most database incidents, but it happens. An AI coding agent (Claude Code, Cursor) will happily scaffold a logical replication setup or a CDC connector.CREATE_REPLICATION_SLOT, a Debezium config, a pg_recvlogical sidecar, because that is a well-documented, locally-correct pattern. What it does not add is the operational half: a max_slot_wal_keep_size ceiling, monitoring on slot lag, or teardown logic that drops the slot when the consumer is removed.

The same gap shows up in generated infrastructure code: an experimental replica spun up in a branch environment, torn down by deleting the pod or the VM, with the slot left behind on the primary quietly retaining WAL forever. The code that created the problem is correct. The code that would have prevented it was never in scope.

How DBGorilla helps

DBGorilla connects read-only and gives your AI coding agent (Claude Code, Cursor) the state that makes this diagnosable instead of guessable: which replication slots exist, which are inactive, how much WAL each is pinning, whether the archiver is failing and on which segment, and how pg_wal size compares to your retention settings. Your agent can then tell you which of the three causes you are actually looking at, and which of your services owns the slot that is doing it. It surfaces and explains the data; it does not delete files, drop replication slots, change configuration, or touch your WAL. Get started free →

FAQ

Can I delete files from pg_wal to free space? No. Deleting WAL can leave the cluster unable to recover after a crash and can permanently break replicas and point-in-time recovery. Remove the cause of the retention instead; PostgreSQL then recycles the files itself at the next checkpoint.

Why is pg_wal growing? Almost always an inactive replication slot, a failing archive_command, or wal_keep_size set high. Check pg_replication_slots, pg_stat_archiver, and your config in that order.

Why does a replication slot pin WAL? The slot is a durable promise that the consumer can disconnect and resume without a gap, so the server must retain WAL from the slot's restart_lsn until the consumer confirms it has processed past it. A consumer that never returns never releases it.

Is dropping a replication slot safe? It is safe for the database and destructive for the consumer. A physical standby may need a full re-seed; a logical/CDC consumer loses its position and must be re-snapshotted. Only drop a slot whose consumer is genuinely gone.

Does WAL free up immediately after I fix the cause? No, removal and recycling happen at a checkpoint. You can run CHECKPOINT; to force one; it is safe but causes an I/O burst.

What is the one setting that prevents this? max_slot_wal_keep_size (PostgreSQL 13 and later). It caps how much WAL a slot may pin, invalidating the slot instead of filling the disk. Size it from your measured peak WAL rate times the consumer outage you want to survive, bounded by real free disk.

Related

• Why is autovacuum not keeping up (and how do I safely make it more aggressive)?
• How to fix idle in transaction connections in PostgreSQL
• What is table bloat in PostgreSQL and how do I fix it?
• Product: DBGorilla docs