Fill a WAL Volume on Purpose Before ENOSPC Turns Your Health Check Green

A service reports healthy because its process is alive, while WAL writes fail with ENOSPC. Log rotation also fails, retries amplify load, and deleting one file does not prove durable writes resumed.

Bounded local fault injection

mkdir -p ./disk-fixture
dd if=/dev/zero of=./disk-fixture/filler bs=1M count=900
# Run only inside a disposable volume with a declared size.

Enter fullscreen mode Exit fullscreen mode

Drive a fixed write workload and record free bytes, successful writes, failed writes by error class, WAL/checkpoint age, retry count, and health state. Never run this against a workstation root or shared production volume.

Threshold Action 20% free warn and stop optional writes 10% free reject new work, retain read path first ENOSPC fail readiness, stop retries space restored integrity check before readiness

A useful readiness probe performs a bounded durable-write check or observes a recent successful checkpoint; a process-only liveness probe should remain separate.

inject -> ENOSPC -> readiness false -> bounded cleanup
       -> integrity/checkpoint -> test write -> readiness true

Enter fullscreen mode Exit fullscreen mode

Test log rotation failure, deletion of an open file that does not release space, and a restart before checkpoint. Cleanup must remove the fixture, confirm file descriptors release blocks, run integrity checks, and prove a new write survives restart.

This local container drill does not model cloud-volume burst credits, remote filesystems, or every database WAL. Which signal should remove your instance from service first: free bytes, checkpoint age, or the first failed durable write?