I Kept Deleting Logs for 48 Hours. The Inodes Were Already Gone.

Have you ever watched a two-kilobyte write fail with No space left on device while df -h still showed free gigabytes? I did, and I spent the next forty-eight hours cleaning the wrong evidence. This is the reconstructed field notebook from that session, including the commands I ran, the ones that misled me, and the checklist I now run before I blame the disk. Nothing here is a benchmark, a quota promise, or a claim about hardware I did not measure.

I was iterating on a small Python worker that dumped JSON sidecars next to each run. The worker itself was unremarkable. The failure mode was not.

Hour 0: the write that should have been boring

The first traceback looked like a disk problem, so I treated it like a disk problem. Would you have done anything else with ENOSPC staring at you from a three-line stack? I would not, and that is exactly how the next two days started.

OSError: [Errno 28] No space left on device: 'runs/2026-09-05T07-12-04.json'

Enter fullscreen mode Exit fullscreen mode

I ran the obvious command, got a comforting number, and closed the wrong investigation. df -h reported plenty of space on the root filesystem, and /tmp looked equally relaxed. I even created a dummy file in $HOME by hand, which succeeded, so I told myself the worker path was special.

df -h
df -h /tmp /var /home
touch ~/probe-ok.txt && ls -l ~/probe-ok.txt

Enter fullscreen mode Exit fullscreen mode

That last touch was the trap. Can a filesystem accept a file in one directory and refuse a tiny file in another while still having blocks to spare? Yes, and inode exhaustion is the boring reason. I did not ask that question for twelve hours.

What I tried first, and why it felt reasonable

I treated the symptom as log rot, because that is the story operators tell each other. I truncated worker logs, deleted old JSON sidecars I could see, and reran the job with a smaller batch. The write still failed, sometimes on file number twenty, sometimes on file number four.

• Truncated worker.log and debug.log with : > file instead of deleting the path.
• Removed a handful of large .jsonl files I could find with du -sh.
• Restarted the process, assuming a leaked temp file would vanish on exit.
• Asked myself whether the path was on a different mount than $HOME.

: > worker.log
du -sh runs logs /tmp 2>/dev/null
mount | awk '{print $1, $3}'
find runs -type f -name '*.json' | wc -l

Enter fullscreen mode Exit fullscreen mode

The file count in runs/ was already larger than I expected, but it was not millions. So I stopped counting files and went back to watching bytes. That was the first mistake I would not repeat.

Hour 12: cleaning made the symptom worse

Have you noticed how a cleanup script can create more metadata than it destroys? Mine did, because I wrote a “safe delete” helper that copied each file to runs/.trash/ before unlinking it. Copying a tiny JSON file is cheap in bytes and expensive in inodes when you keep both copies.

I also kept pytest cache directories, __pycache__ trees, and editor swap files, because none of them looked large in du -sh. Why would they? They are thousands of small nodes, not a single fat log. The more I “tidied,” the closer the filesystem sat to its inode ceiling.

# Reconstruct the cleanup that backfired.
mkdir -p runs/.trash
find runs -type f -name '*.json' -print0 |
  xargs -0 -I{} cp {} runs/.trash/

Enter fullscreen mode Exit fullscreen mode

The worker then failed faster. I blamed the code generator for “writing too much,” which was emotionally satisfying and technically incomplete. The generator was writing small files. The filesystem was out of names, not out of blocks.

The command I had not run

Around hour twenty I finally ran the command that should have been second, not twentieth. df -i reports inode use, and it does not care how comforting df -h looked an hour ago. On that mount, inodes were effectively full.

df -i
df -ih /
# Compare blocks vs inodes on the same mount.
paste <(df -h / | tail -1) <(df -i / | tail -1)

Enter fullscreen mode Exit fullscreen mode

Once I could see inode pressure, the rest of the trail was ordinary Linux. I counted files by directory, then by extension, then by cache folder. The noisy directories were not the logs I had been truncating. They were caches and sidecars.

# Top directories by file count, not by bytes.
find . -xdev -type f | awk -F/ 'NF>1{print $2}' | sort | uniq -c | sort -nr | head

# Extension histogram.
find . -xdev -type f | sed 's/.*\.//' | sort | uniq -c | sort -nr | head

# Classic tiny-file nests.
find . -type d \( -name '__pycache__' -o -name '.pytest_cache' -o -name '.mypy_cache' \) | wc -l

Enter fullscreen mode Exit fullscreen mode

Would du -sh have shown this? Not in a way I would have trusted at 2 a.m. Bytes and inodes answer different questions, and I had only asked one of them.

A reproducible inode trap

I wanted a lab I could rerun without inventing a production outage. The snippet below is labeled as a local demonstration. It creates many tiny files, shows df -i moving, and then stops before it can harm a real mount. Run it only inside a throwaway directory on a filesystem you own.

# demo_inode_trap.py — demonstration only, not a production test suite.
from pathlib import Path
import os
import sys

TARGET = Path(os.environ.get("INODE_LAB", "/tmp/inode-lab"))
N = int(os.environ.get("INODE_N", "20000"))

def main() -> None:
    TARGET.mkdir(parents=True, exist_ok=True)
    if not os.path.isdir(TARGET):
        raise SystemExit("refusing to run without a directory")
    print(f"writing {N} tiny files under {TARGET}")
    for i in range(N):
        p = TARGET / f"sidecar-{i:06d}.json"
        p.write_text("{}", encoding="utf-8")
        if i % 5000 == 0 and i:
            print(f"wrote {i}", flush=True)
    print("done; now run: df -i", TARGET.anchor)

if __name__ == "__main__":
    if "--i-understand" not in sys.argv:
        raise SystemExit("refusing to run without --i-understand")
    main()

Enter fullscreen mode Exit fullscreen mode

mkdir -p /tmp/inode-lab
python3 demo_inode_trap.py --i-understand
df -i /tmp
find /tmp/inode-lab -type f | wc -l
rm -rf /tmp/inode-lab

Enter fullscreen mode Exit fullscreen mode

I also hit a cousin of the same bug: a directory with so many entries that ls felt hung while df -h still smiled. That is not inode exhaustion of the filesystem, but it is the same class of mistake. I was measuring the wrong scarce resource.

Diagnostic artifact: one script, four scarce resources

The next time a write fails with ENOSPC or “read-only” vibes, I want one script that asks four questions in a fixed order. Blocks, inodes, directory density, and open-file limits are different ceilings. The script prints a guess. It does not repair anything.

#!/usr/bin/env bash
# why_enospc.sh — read-only diagnostics for mysterious ENOSPC.
set -euo pipefail
PATH_TO_CHECK="${1:-.}"
MOUNT_SRC=$(df -P "$PATH_TO_CHECK" | awk 'NR==2{print $6}')

echo "== blocks =="
df -h "$MOUNT_SRC"

echo "== inodes =="
df -i "$MOUNT_SRC"

echo "== can we create one file here? =="
PROBE="$PATH_TO_CHECK/.enospc-probe-$$"
if touch "$PROBE" 2>/tmp/enospc-probe.err; then
  echo "touch ok at $PROBE"
  rm -f "$PROBE"
else
  echo "touch failed:"
  cat /tmp/enospc-probe.err
fi

echo "== densest directories (file count) =="
find "$PATH_TO_CHECK" -xdev -type d -print0 2>/dev/null |
  while IFS= read -r -d '' d; do
    n=$(find "$d" -mindepth 1 -maxdepth 1 -printf '.' 2>/dev/null | wc -c)
    echo "$n $d"
  done | sort -nr | head -n 15

echo "== open files for this user =="
lsof -u "$(id -un)" 2>/dev/null | wc -l || true
ulimit -n

echo "== tiny-file suspects =="
for pat in '__pycache__' '.pytest_cache' '.mypy_cache' '.cache' 'runs' 'logs'; do
  if [ -e "$PATH_TO_CHECK/$pat" ]; then
    echo -n "$pat files: "
    find "$PATH_TO_CHECK/$pat" -xdev -type f 2>/dev/null | wc -l
  fi
done

Enter fullscreen mode Exit fullscreen mode

Run it like this, then read the decision table before deleting anything.

chmod +x why_enospc.sh
./why_enospc.sh /path/to/workspace

Enter fullscreen mode Exit fullscreen mode

Decision table

• df -h near 100%, df -i calm: you are actually out of blocks. Truncate or rotate large files first.
• df -h calm, df -i near 100%: stop deleting “big” logs. Count files, then remove caches and sidecars.
• both calm, touch fails in one directory only: check mount options, quotas, and directory entry limits.
• touch works, the app still raises ENOSPC: the process chdir‘d, or it writes under /tmp, not under the path you inspected.
• lsof huge and ulimit -n close: you may be looking at EMFILE, which people misread as disk because the traceback is ugly.

That last row bit me on a later rerun. Different errno, same panic. I now print e.errno explicitly in the worker instead of logging only the message string.

import errno
import os
from pathlib import Path

def write_sidecar(path: Path, payload: str) -> None:
    try:
        path.write_text(payload, encoding="utf-8")
    except OSError as e:
        # Field note: message text is not enough.
        print({
            "path": str(path),
            "cwd": os.getcwd(),
            "errno": e.errno,
            "enospc": e.errno == errno.ENOSPC,
            "emfile": e.errno == errno.EMFILE,
        }, flush=True)
        raise

Enter fullscreen mode Exit fullscreen mode

Where a free model and a free server actually showed up

Disclosure: This article was prepared as part of MonkeyCode’s product outreach.

I reproduced the mess on a free server option, then used free model access to draft the first version of why_enospc.sh. That pairing mattered because the failure is about metadata pressure, not about needing a larger paid box on day one. The model was useful for enumerating cache directories I forget under stress. It was not useful when I asked it to “just make the write succeed,” because it suggested more retries, which created more tiny files.

I had to keep the prompt boring on purpose. I asked for a read-only diagnostic script, a decision table, and a refusal to delete files. When I asked for a cleanup script instead, it happily generated the copy-to-trash pattern that made inode use worse. The lesson was about the question, not about a magic model name I am not going to invent here.

If you already have a small shared workspace and a worker that loves sidecar files, that same free server plus free model loop is enough to practice this checklist. I would not use it as a substitute for df -i on a machine you cannot afford to fill.

What broke for real

Three things broke, and only one of them was the filesystem.

1. My mental model of ENOSPC was “disk full in gigabytes,” which is incomplete on any inode-backed filesystem.
2. My cleanup copied files, doubling inode use while du -sh went down a little and fooled me.
3. The worker logged the exception message without errno, so I could not distinguish ENOSPC from later EMFILE noise.

The model-generated helper also assumed ./runs was small enough to ls without flags. After enough sidecars, even listing the directory became a time sink. That felt like a hang. It was a directory that had become a database I did not mean to build.

What I would repeat

I would run df -h and df -i as a pair, every time, before deleting evidence. I would print cwd, errno, and the target path in every write helper. I would budget sidecar files the way I already budget log bytes, because a two-kilobyte file is not free if you create it fifty thousand times.

• Pair df -h with df -i in the first five minutes, not at hour twenty.
• Count files in runs/, __pycache__/, and test caches before truncating logs.
• Refuse copy-on-delete cleanup unless the trash directory lives on another filesystem.
• Cap sidecar retention with a hard file count, not only with age.
• Keep diagnostic scripts read-only until a human has read the decision table.

# Retention by count, reconstructed from the later fix.
max=5000
extra=$(find runs -type f -name '*.json' | wc -l)
if [ "$extra" -gt "$max" ]; then
  find runs -type f -name '*.json' -printf '%T@ %p\n' |
    sort -n | head -n $((extra - max)) | awk '{print $2}' |
    xargs -r rm -f
fi

Enter fullscreen mode Exit fullscreen mode

Would I still use a model to draft the boring script? Yes, with a prompt that forbids deletes and forbids retries. Would I let it invent a cleanup? Not after this notebook.

Who should not use this approach

Do not run the inode trap on a shared mount you do not own. Do not point find at a network filesystem if you cannot afford the metadata storm. Do not take this checklist as a capacity plan for a production fleet, because I am not publishing hardware specs, quotas, or durability claims. If your write failures come with EROFS, snapshot layers, or container overlay surprises, this notebook is the wrong first document.

The script also will not help if the process is writing into a different mount than the one you inspected. In that case you need cwd, /proc/<pid>/cwd, and the absolute path in the traceback more than you need df. I lost a couple of hours there too, and I am not going to pretend otherwise.

readlink /proc/$(pgrep -n python)/cwd || true
ls -l /proc/$(pgrep -n python)/fd | head

Enter fullscreen mode Exit fullscreen mode

Limitations

This is a field notebook, not a kernel talk. Inode layout differs across ext4, xfs, overlayfs, and tmpfs, and I did not measure those differences here. File-count histograms are slow on large trees. lsof can be slow too. The demonstration script can hurt a disk if you remove the safety flag and raise INODE_N without thinking.

I also did not verify anyone else’s fleet, customer, or free-tier ceiling. If a host uses project quotas, df -i can look healthy while a user quota is not. Check quota -s when that is in play, and do not collapse those layers into one story.

Forty-eight hours is a long time to learn that df has two meters. I only needed two commands, one errno print, and a refusal to copy files as a form of deletion. Next time the traceback says there is no space left, I will ask which scarce resource it means before I delete a single log line.