Your README code examples are silently lying — I built a CLI to detect documentation drift using only AST, no LLM

Your README code examples are silently lying — I built a CLI to detect documentation drift using only AST, no LLM

“Did you know? This README example doesn’t work.”

Have you ever had a junior engineer tell you that? Or worse — have you added a line to your code, forgot the docs, and broke it yourself?

This time I want to tell you about doc-drift, a CLI I built that detects this “documentation drift” — the silent gap between code examples in your Markdown and the actual codebase — without using an LLM at all.

https://github.com/sunnydachs/doc-drift

What it does

You point it at a repository. It scans your Markdown files for fenced code blocks, extracts the functions and classes defined there, and checks whether they still match the real codebase.

# scan the repo (read-only)
doc-drift

# another repo, machine-readable output
doc-drift /path/to/repo --json

Enter fullscreen mode Exit fullscreen mode

Example output:

doc-drift — scanned /home/dev/myproject
  markdown files: 42 | python blocks checked: 87
README.md:153  ⚠️ SIGNATURE DRIFT
    block defines (url, retries) but repository defines (url, timeout)
summary: {"signature_drift": 3, "missing": 1, "unparseable": 5} | clean blocks: 41

Enter fullscreen mode Exit fullscreen mode

It reports three main findings:

• SIGNATURE DRIFT — the documented function exists, but its arguments have diverged from the real code.
• MISSING — the documented function/class no longer exists anywhere in the repo (renamed? deleted?).
• UNPARSEABLE — the block isn’t valid Python (pseudo-code, placeholders) — informational only.

A “function exists, but arguments differ” bug is one that IDEs and static analysis can never catch. Docs aren’t executed — so nothing in the toolchain ever checks them. That’s the hole.

Why deliberately not use an LLM

This was the core design decision. An AI reading the natural-language docs could grasp more “context.” But I deliberately built it on nothing but Python’s standard ast (Abstract Syntax Tree). Three reasons:

1. Deterministic. Same input → same output, every time. I don’t want a tool in CI whose verdict flips between runs.
2. Read-only and safe. It never imports or executes your code — it compares at the syntax-tree level. Your files stay untouched.
3. Zero dependencies. Python 3.11+ is enough. Install is pip install git+...

This is one answer to the question I keep asking myself when automating: how much should I hand to an AI? Deterministic work deserves deterministic tools.

The rule: docs may omit, but they must never invent

The detail I obsessed over: code examples in docs are almost always simplified versus the real code. Omitting an argument should be fine. So the rule is:

• Omit arguments — allowed.
• Invent arguments or functions that don’t exist in the code — forbidden.

That single line (“省略は許す、発明は許さない” / omit is fine, inventing is not) eliminated the vast majority of false positives. Classes work the same way: every documented method must exist on the real class, but unmentioned methods are treated as omitted.

Testing against real-world data

Not a toy README — a real, large open-source repository:

• Scanned 1,692 Markdown files / 4,451 code blocks.
• Found one genuine drift: a function documented with 2 arguments whose implementation had since moved to 1 argument.
• Also surfaced our own over-eager default exclusion that caused false positives — so we fixed the configuration.

The real win wasn’t just “it works” — it was confirming in practice that the tool does not degrade into a noisy false-positive spammer.

Honest limitations

• It can’t tell “documentation” from “illustration.” A hypothetical example in a top-level README (code never meant to exist) reports as MISSING. It shines in docs/ trees whose examples mirror the codebase.
• Python only (for now). Other languages are counted but not checked.
• Name-based comparison. Default values and type annotations are ignored — it catches renames, removals, and arity/name changes, not semantic drift.

Wrap-up

Docs-vs-code drift only gets more dangerous as a project grows. And these “quiet but sharp-toothed” problems are exactly where deterministic tools earn their keep.

This is one in a series I’m building on the same principles (deterministic / read-only / dry-run-first). A sibling, plan-drift, applies the same approach to analytics tracking plans.

https://github.com/sunnydachs/doc-drift

Feedback, and especially false-positive reports from your own repos, are the most useful thing you can give me — drop them in the repo issues.