The rule for documentation is one line: pull the API surface out of the source mechanically, put the extraction in the context, and reject any draft containing an identifier that is not in it. Two short scripts make that a build failure rather than a good intention.
Extract first, write second, never the reverse
The order that fails is the natural one: describe what the function does, then check the details. It fails because checking is where the effort is and describing is where the enthusiasm is, and because a confident sentence about a parameter that does not exist reads exactly like a confident sentence about one that does.
The order that works inverts it. A script produces a machine-readable list of every module, class, function, signature and docstring. That list goes into the prompt as the only permitted source of names. A second script then checks the output against the same list, and the check is a set difference rather than a judgement.
Why documentation is the worst place for this failure
A fabricated detail costs more here than anywhere else in writing, for three reasons that compound.
- The reader cannot tell. Someone reading your docs is reading them because they do not know the API. They have no basis for scepticism about a parameter name, which is the entire reason they came.
- The failure is expensive and silent. A wrong flag produces a confusing error, or worse, silently does nothing. The reader assumes they made the mistake and spends an hour before suspecting the documentation.
- It propagates. Documentation is copied into tutorials, answers, internal wikis and other models’ training data. An invented parameter acquires a small literature.
There is a fourth, structural reason. A model asked to documentprocess_batch has seen thousands of functions with that sort of name and knows what parameters they usually take. The most probable continuation after “Parameters:” is a plausible parameter list, not your parameter list, and it will be produced with the same fluency either way — the mechanism is the same one behind every other confabulation.
Extracting the surface from the source
For a Python package this is ast and nothing else — no import, so no side effects and no dependency on the package being installable.
# extract_api.py — Python 3.9+, standard library only.
# Usage: python extract_api.py ./mypackage > api.json
# Parses source without importing it, so nothing executes.
import ast, json, os, sys
def signature(node):
a = node.args
parts = []
defaults = [None] * (len(a.args) - len(a.defaults)) + list(a.defaults)
for arg, default in zip(a.args, defaults):
s = arg.arg
if arg.annotation is not None:
s += ": " + ast.unparse(arg.annotation)
if default is not None:
s += " = " + ast.unparse(default)
parts.append(s)
if a.vararg:
parts.append("*" + a.vararg.arg)
for arg, default in zip(a.kwonlyargs, a.kw_defaults):
s = arg.arg
if arg.annotation is not None:
s += ": " + ast.unparse(arg.annotation)
if default is not None:
s += " = " + ast.unparse(default)
parts.append(s)
if a.kwarg:
parts.append("**" + a.kwarg.arg)
out = "(" + ", ".join(parts) + ")"
if node.returns is not None:
out += " -> " + ast.unparse(node.returns)
return out
api = []
for root, dirs, files in os.walk(sys.argv[1]):
dirs[:] = [d for d in dirs if not d.startswith((".", "_")) or d == "__init__"]
for name in sorted(f for f in files if f.endswith(".py")):
path = os.path.join(root, name)
tree = ast.parse(open(path, encoding="utf-8").read(), filename=path)
for node in ast.walk(tree):
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
if node.name.startswith("_") and not node.name.startswith("__"):
continue
api.append({
"kind": "function",
"file": path,
"name": node.name,
"signature": node.name + signature(node),
"docstring": ast.get_docstring(node) or "",
"decorators": [ast.unparse(d) for d in node.decorator_list],
})
elif isinstance(node, ast.ClassDef):
api.append({
"kind": "class",
"file": path,
"name": node.name,
"bases": [ast.unparse(b) for b in node.bases],
"docstring": ast.get_docstring(node) or "",
})
json.dump(api, sys.stdout, indent=2)
Enter fullscreen mode Exit fullscreen mode
Parsing rather than importing is the important choice. Importing runs module-level code, which in a real package means configuration loading, connections and occasionally a side effect nobody remembers writing.ast.parse reads the file as text.
For a web API the equivalent artefact already exists: the OpenAPI document. Use it directly, and if it is generated from the code rather than maintained by hand, it has the same property this script gives you — it cannot describe an endpoint that is not there.
Writing against the extraction
You are writing reference documentation for one function. The JSON
below is the complete and only source of truth about this API. It was
extracted mechanically from the source.
Absolute constraints:
- Every function, class, parameter, attribute and return type you name
must appear in the JSON. Nothing else exists.
- Do not state a default value unless it appears in the signature.
- Do not state what exceptions are raised unless the docstring says so.
- Do not describe behaviour the signature and docstring do not support.
If the reader would need to know something that is not here, write
[GAP: what a reader needs and where it would have to come from].
- Do not invent an example that calls anything not in the JSON.
Write:
1. One sentence on what it does.
2. A parameter list, in signature order, each with its type and default
exactly as extracted.
3. What it returns.
4. One minimal example using only names from the JSON.
5. Any [GAP: ...] notes.
API
{ ...the relevant slice of api.json... }
Enter fullscreen mode Exit fullscreen mode
The [GAP: ...] convention is what makes this workable rather than merely safe. Reference documentation written strictly from signatures is thin, and the gaps are real — what the units are, what happens on an empty input, whether it is safe to call concurrently. Marking them produces a to-do list for the engineer who knows, which is exactly the artefact a documentation project normally lacks. Grep for GAP: before publishing.
Pass one function at a time with a small slice of the JSON. A whole package in the context invites cross-contamination, where a parameter from a neighbouring function migrates into the one being documented because both are in the window.
Failing the build on an invented identifier
The check is a set difference: every code-like token in the draft must be a name the extraction knows, a Python builtin, or on a small allow-list.
# check_identifiers.py — Python 3.9+, standard library only.
# Usage: python check_identifiers.py api.json draft.md
# Exit 1 if the draft names anything the API does not contain.
import builtins, json, re, sys
api = json.load(open(sys.argv[1], encoding="utf-8"))
draft = open(sys.argv[2], encoding="utf-8").read()
known = set(dir(builtins))
for item in api:
known.add(item["name"])
sig = item.get("signature", "")
# parameter names: everything before a colon, equals or comma
inside = sig[sig.find("(") + 1 : sig.rfind(")")] if "(" in sig else ""
for part in inside.split(","):
part = part.strip().lstrip("*")
if part:
known.add(re.split(r"[:=\s]", part)[0])
ALLOW = {"self", "cls", "None", "True", "False", "import", "from", "return",
"def", "class", "async", "await", "json", "str", "int", "float",
"bool", "list", "dict", "print"}
# Only inspect code: inline spans and fenced blocks.
fence = chr(96)
code_spans = re.findall(fence + r"([^" + fence + r"\n]+)" + fence, draft)
code_blocks = re.findall(fence * 3 + r".*?\n(.*?)" + fence * 3, draft, re.S)
unknown = {}
for chunk in code_spans + code_blocks:
for token in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", chunk):
if token not in known and token not in ALLOW:
unknown[token] = unknown.get(token, 0) + 1
for token, n in sorted(unknown.items(), key=lambda kv: -kv[1]):
print(f"unknown identifier: {token} ({n}×)")
print(f"\n{len(unknown)} unknown identifiers")
sys.exit(1 if unknown else 0)
Enter fullscreen mode Exit fullscreen mode
It over-reports, deliberately. Local variable names in examples will appear as unknown, and the correct response is to add them to the allow-list once rather than to loosen the check. A verifier tuned to produce no false positives will also produce no true ones, and the whole value here is that a fabricated parameter cannot pass silently.
Run it in the same place the tests run, alongside the style checker — both are the same pattern, a mechanical rule enforced downstream instead of asked for in a prompt. A check a human remembers to run is a check that catches errors on the days nothing was wrong.
Examples are executed, not read
Every code example in documentation must be run, and the best way to guarantee that is to store examples as files the test suite executes and include them by reference rather than by copying.
- Put each example in a file under a directory the test runner walks.
- Assert its output rather than merely that it did not raise. An example that runs and prints the wrong thing is still wrong documentation.
- Include it in the docs by transclusion. A copied snippet is a fork, and a fork goes stale on the first refactor.
- Pin versions in the example where behaviour depends on them, and say in the prose which version was used.
A generated example that has not been executed is the highest-risk artefact in the whole document, because it is the part readers copy without reading. Whoever holds the gate before publication should treat “did this example run?” as the documentation equivalent of checking a quotation against the recording.
What it is genuinely good at
- The narrative layer. Reference documentation follows from the code; the tutorial that gets someone from nothing to a working call does not, and turning a sequence of steps into readable prose is a real strength.
- The “why” paragraph, drafted from an engineer’s rough notes. Engineers routinely explain a design decision perfectly in a message and cannot face writing it up.
- Terminology audits. “List every term used for the same concept across these forty pages” is a search task over supplied text, it is reliable, and it finds the four names your product has for one thing.
- Structural consistency. Checking that every reference page has the same sections in the same order.
- Reading diffs for documentation impact. Given a changed signature and the current page, asking which sentences are now false is bounded, checkable and genuinely saves time.
- The first pass on error messages. Given the code path that raises, drafting a message that says what happened and what to do about it.
What it must never write
- Version numbers and compatibility claims. “ Available since 2.4” is either in the changelog or it is not true.
- Default values. From the signature, always. This is the single most commonly confabulated fact in generated documentation, because defaults are highly predictable from convention.
- Error strings and status codes. Copy from the source. Readers search for these literally, so an approximation is worse than an omission.
- Rate limits, quotas, prices and retention periods. From the system of record, not from the model.
- Security guidance. Plausible-sounding security advice is the most dangerous output in this entire category, because it is confidently phrased and rarely tested by the reader.
- Migration instructions. These must be executed on a real system before publication. Every one of them.
답글 남기기