The projector showed a green test suite. The room relaxed. Then a student opened the diff and found a brand-new redis import the assignment had never named.
That is the failure this lab is built to make visible. An AI coding assistant can repair a missing feature by inventing infrastructure. The patch reads as decisive. Local tests may even pass. The repository, meanwhile, now depends on a service nobody offered to run.
This article is a two-hour workshop outline, not a product tour. A facilitator can rerun every file on a laptop. Students leave with a written assumption contract, a small feature-flag module, and a checker that fails a patch when the model smuggles in a cache, a database, or a SaaS SDK.
The Tuesday scene
A borrowed classroom, twenty laptops, one shared repo. The starter code is a JSON feature-flag reader. The homework looks modest: add a percentage rollout, deterministic by user_id. Several assistants answer with vendor SDK keys, Redis counters, or a sidecar process. One answer even writes a Docker Compose file.
The green tests hid the real grade. The assignment was never “make rollout production-grade.” It was “change this module without changing the operating model.” The rest of the session exists to put that constraint in writing, then enforce it with a boring script.
Lab timing
The facilitator treats the two hours as four movements, each with a deliverable. Minutes 0–15 replay a canned failure: a patch that adds Redis and still satisfies a naive unit test. Minutes 15–40 are for writing assumptions.json in pairs. Minutes 40–85 are for prompting an assistant against that contract and saving the diff. Minutes 85–120 are for running the gate, reading false positives, and tightening the contract. The clock is part of the pedagogy. Without it, the discussion slides into model gossip.
Starter code students can rerun
The application is deliberately small. A short file keeps the model from drowning in context, and it keeps human review on one screen.
# flags.py
from __future__ import annotations
import json
from pathlib import Path
from typing import Optional
FLAGS_PATH = Path(__file__).with_name("flags.json")
def load_flags():
with FLAGS_PATH.open(encoding="utf-8") as handle:
return json.load(handle)
def enabled(name: str, user_id: Optional[str] = None) -> bool:
flags = load_flags()
value = flags.get(name, False)
if isinstance(value, bool):
return value
raise TypeError(f"flag {name!r} must be a boolean in this starter")
Enter fullscreen mode Exit fullscreen mode
{
"beta_checkout": false
}
Enter fullscreen mode Exit fullscreen mode
# test_flags.py
from flags import enabled
def test_missing_flag_is_off():
assert enabled("does_not_exist") is False
def test_beta_checkout_reads_json():
assert enabled("beta_checkout") is False
Enter fullscreen mode Exit fullscreen mode
Commands stay ordinary on purpose. The lab should survive a machine that has only Python and pip.
python3 -m venv .venv
source .venv/bin/activate
pip install pytest
pytest -q
Enter fullscreen mode Exit fullscreen mode
The assignment text, pasted into the prompt after the contract, is one paragraph. Extend enabled so a flag may be an integer 0–100 meaning percent rollout. Hash user_id with a stable function from the standard library. Do not add services, files outside the allowed set, or environment variables.
The contract
Students often want to write a novel. The lab asks for a page that a script can parse. The JSON below is the worked example, not a universal policy.
{
"goal": "Percentage rollout for feature flags, deterministic by user_id.",
"allowed_files": ["flags.py", "flags.json", "test_flags.py"],
"allowed_imports": ["json", "pathlib", "hashlib", "typing"],
"forbidden_substrings": [
"redis",
"sqlite",
"postgres",
"docker",
"LAUNCHDARKLY",
"os.environ",
"requests",
"httpx"
],
"assumptions_the_model_may_make": [
"flags.json remains the only persistence",
"user_id is a non-empty string when a percentage flag is evaluated",
"hashlib.sha256 is available in the standard library"
],
"assumptions_the_model_must_not_make": [
"a cache or database will be deployed beside the module",
"secret API keys exist in the environment",
"a background worker can precompute rollouts"
]
}
Enter fullscreen mode Exit fullscreen mode
The contract is the curriculum. Pairs argue about hashlib versus a homemade hash. They argue about whether test_flags.py may grow new cases. Those arguments are the point. An unconstrained assistant will skip them and ship a platform.
Prompting without handing over the keys
The facilitator does not need a particular vendor. Any chat completion endpoint that accepts a system message and a user message will do. Facilitators who want a hosted place to run the same prompts without standing up extra hardware can use MonkeyCode, an open source coding-assistant project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode’s product outreach. The lab remains intact if that host is absent. A local model or a school-provided endpoint can drive the same files.
Students paste a template rather than inventing prompt folklore under time pressure.
System: You are a patch generator. Output a unified diff and nothing else.
You must obey assumptions.json. If the task requires a forbidden assumption,
refuse with a one-line reason instead of a diff.
User: Repository files:
<paste flags.py>
<paste flags.json>
<paste assumptions.json>
Task: Support integer 0-100 rollouts in enabled(name, user_id).
Keep boolean flags working. Add tests. Stay inside allowed_files.
Enter fullscreen mode Exit fullscreen mode
They save the model output as incoming.patch even when it is prose. The gate will fail prose, which is a useful lesson. A refusal is a passing academic outcome. A silent Redis cluster is not.
The gate
The checker is intentionally blunt. It is a teaching instrument, not a security product. It reads the patch as text, then applies the contract before anyone runs git apply.
# gate.py
from __future__ import annotations
import json
import sys
from pathlib import Path
ROOT = Path(__file__).parent
def load_contract():
return json.loads((ROOT / "assumptions.json").read_text(encoding="utf-8"))
def fail(message: str) -> int:
print(f"GATE FAIL: {message}")
return 1
def paths_in_diff(patch: str) -> list[str]:
paths = []
for line in patch.splitlines():
if line.startswith("+++ b/"):
paths.append(line[6:].strip())
return paths
def main(argv: list[str]) -> int:
if len(argv) != 2:
return fail("usage: python gate.py incoming.patch")
patch = Path(argv[1]).read_text(encoding="utf-8", errors="replace")
contract = load_contract()
if not patch.strip().startswith("diff") and "---" not in patch:
return fail(
"output is not a unified diff; refusals must be recorded, not ignored"
)
for token in contract["forbidden_substrings"]:
if token.lower() in patch.lower():
return fail(f"forbidden substring {token!r} appears in the patch")
for path in paths_in_diff(patch):
if path not in contract["allowed_files"]:
return fail(f"patch touches {path}, which is outside allowed_files")
print("GATE PASS")
return 0
if __name__ == "__main__":
raise SystemExit(main(sys.argv))
Enter fullscreen mode Exit fullscreen mode
python gate.py incoming.patch
git checkout -b lab-rollout
git apply incoming.patch
pytest -q
Enter fullscreen mode Exit fullscreen mode
A worked patch that should pass stays inside hashlib and JSON. Students can type it by hand if the model refuses, which proves the assignment was never model-dependent.
# excerpt students may compare against their diff
from __future__ import annotations
import hashlib
import json
from pathlib import Path
from typing import Optional
FLAGS_PATH = Path(__file__).with_name("flags.json")
def load_flags():
with FLAGS_PATH.open(encoding="utf-8") as handle:
return json.load(handle)
def enabled(name: str, user_id: Optional[str] = None) -> bool:
flags = load_flags()
value = flags.get(name, False)
if isinstance(value, bool):
return value
if isinstance(value, int):
if not 0 <= value <= 100:
raise ValueError(f"flag {name!r} percent must be 0..100")
if not user_id:
raise ValueError("percentage flags require user_id")
digest = hashlib.sha256(f"{name}:{user_id}".encode("utf-8")).hexdigest()
bucket = int(digest[:8], 16) % 100
return bucket < value
raise TypeError(f"flag {name!r} must be bool or int")
Enter fullscreen mode Exit fullscreen mode
A corresponding flags.json value such as "beta_checkout": 25, plus two tests with frozen user_id strings, completes the example. The facilitator should lock those identifiers in the test file so reruns do not drift across Python versions. One user must always fall inside the bucket. The other must always fall outside it.
What the debrief needs to name
The gate will produce false positives. The substring requests will punish a comment that says the module never requests a network. Students then learn to keep comments dull. The gate will also produce false negatives. A model can hide Redis behind a homemade wrapper named faststore. That miss is the debrief, not a scandal. The contract still forced the assistant to declare a storage assumption in the open, where a human can reject it.
The second debrief theme is refusal. A one-line “this needs a cache I am not allowed to add” is a better lab result than a passing suite that changed the ops story. Graders should score the contract and the diff, not the chat window’s confidence.
Limitations
This method does not prove program correctness. It does not replace code review, type checking, or production observability. Substring bans are evasion-prone. They belong in a classroom or an inner-loop linter, not in a compliance story. Teams that ship regulated software should not present a passing gate as evidence that a model followed policy. People under a production incident should not run this lab as a hotfix protocol. The clock and the pedagogy assume a calm repo.
The approach also wastes time when the task truly needs new infrastructure. If the homework is “add durable storage,” forbidding SQLite is theater. The facilitator should pick tasks whose operating model is already settled.
Who should skip it
Skip the lab when students cannot install Python 3.10 or newer, when every model endpoint is blocked and no local weights are available, or when the course already grades architecture proposals rather than patches. Skip it when the staff cannot read diffs. The whole point is watching the diff, not watching the assistant talk.
The applause at the green bar was not wrong. It was incomplete. A two-hour assumption gate trains a cheaper instinct: before the patch lands, write down what the model is allowed to believe. Then make disbelief automatic.