Quality Isn't Accidental — Maker/Checker Separation and Automated Validation

The Core Argument: AI agent reliability isn’t achieved by “making the agent smarter” — it’s achieved by the simple engineering principle of separating validation from generation. Quality isn’t accidental. It’s designed.
What You’ll Learn: Maker/Checker separation, 6 termination conditions, and an automated feedback loop — all with runnable code.

0. Prerequisites

• Python ≥ 3.10
• OpenAI API Key (or compatible interface)
• pip install openai>=1.0.0
• (Optional) pip install anthropic>=0.30.0 if using Claude as Checker

1. The Pain: Why “Agent Checks Itself” Is a Trap

1.1 The Copying of Cognitive Bias

A team built a data-analysis agent. It pulled sales data from a database and generated business reports. The team added a “self-review” step: after generating, the agent told itself “please check if the data you just output is accurate.”

Result? The agent always replied “data is accurate.” Even when the team deliberately injected obvious errors (e.g., monthly sales of -50M RMB), the agent confidently said everything was fine.

This isn’t the model being “disobedient.” It’s a more fundamental issue: when the generator and checker are the same entity, the check is just a restatement of the generation process — not real validation. The checker carries the exact same cognitive bias, knowledge boundaries, and reasoning path as the generator.

1.2 The Amplifier Effect of Confirmation Bias

Self-checking also triggers a subtler problem: confirmation bias amplification. The model builds a “belief state” during generation; when re-examining, it tends to confirm rather than overturn.

Experiment data (from Anthropic research):

• Same model does “generate → self-review”: ~12% error-correction rate
• A separate model instance reviews: ~37% error-correction rate
• A different model family reviews: ~52% error-correction rate

1.3 The Value of Independence

First principle of quality assurance: the checker must be independent of the generator. In agent architecture, the engineering expression of this is the Maker/Checker separation pattern.

2. Maker/Checker Separation Pattern

Independence is the first principle of quality — same model 12% correction, different family 52%.

2.1 Three Levels of Separation

Level Description Best For L1: Context separation Maker & Checker use different system prompts, same model Low cost, low-risk tasks L2: Instance separation (recommended) Different model instances, different temperature; Checker typically lower (0.1) Most production environments L3: Model/vendor separation (highest) Different vendors’ different models — e.g., Maker with GPT-4o, Checker with Claude Maximum diversity, minimal common failure modes

2.2 Checker Type System

Checker Type Validates Best For Factual consistency Output matches input/source Data reports, summaries Compliance Output violates preset rules? Finance, medical, legal Logic Reasoning chain complete/consistent? Analysis, decisions Format Output matches expected format? API responses, structured output Safety Output contains harmful content? User-facing agents Completeness Task fully done? Complex workflows

2.3 Checker Output Protocol

Checker output must be machine-parsable — recommended structured JSON:

{
  "decision": "FAIL",
  "confidence": 0.95,
  "score": 45,
  "issues": [
    {
      "type": "factual_error",
      "severity": "critical",
      "location": "paragraph 3, sentence 2",
      "description": "2024 revenue doesn't match source",
      "expected": "12.8M",
      "actual": "18.2M",
      "rule_reference": "R04-number-consistency"
    }
  ]
}

Enter fullscreen mode Exit fullscreen mode

2.4 Six Termination Conditions

Mode Principle Best For Max Retry Hard cap (3-5 attempts) Simple, predictable Quality Threshold Stop when score ≥ target Progressive optimization Convergence Detection Stop when 2 outputs ≥95% similar Avoid invalid retries Diminishing Returns Stop when improvement < threshold High-quality requirements Time Budget Stop on timeout, protect SLO Online services Hybrid (recommended) Combination of above Production environments

3. Complete Code: Maker/Checker Framework

Here’s a runnable implementation. It has two modes:

• Real mode: connects to OpenAI API, full Maker→Checker→feedback loop
• Local test mode (default): uses built-in mock data, no API key needed

#!/usr/bin/env python3
"""
maker_checker.py — Maker/Checker separation and automated validation
Core:
  - Maker Agent: generates content
  - Checker Agent: validates content (multiple checker types)
  - 6 termination conditions (all runnable)
  - Feedback loop + constraint escalation
  - ErrorLog persistence

Dependencies: pip install openai>=1.0.0
Test mode (default): no API key needed, mock LLM validates core logic
Real mode: export OPENAI_API_KEY=sk-xxx then run
"""

from __future__ import annotations
import json, os, time, hashlib
from enum import Enum, auto
from pathlib import Path
from datetime import datetime, timedelta
from dataclasses import dataclass, field
from difflib import SequenceMatcher
from typing import Optional

# ---------- Termination conditions ----------
class TermMode(Enum):
    MAX_RETRY = auto()
    QUALITY_THRESHOLD = auto()
    CONVERGENCE = auto()
    DIMINISHING = auto()
    TIME_BUDGET = auto()

@dataclass
class TermConfig:
    """Combination of termination conditions"""
    mode: TermMode = TermMode.HYBRID
    max_retries: int = 4
    quality_threshold: float = 80.0
    convergence_similarity: float = 0.95
    min_improvement: float = 3.0
    time_budget_sec: float = 120.0

# ---------- Checker ----------
@dataclass
class CheckResult:
    decision: str          # PASS / FAIL
    confidence: float
    score: float
    issues: list = field(default_factory=list)

class BaseChecker:
    """Base class for all checkers"""
    def check(self, output: str, context: dict) -> CheckResult:
        raise NotImplementedError

class MockChecker(BaseChecker):
    """Local test checker — validates without LLM API"""
    def __init__(self, required_keywords: list, min_length: int = 50):
        self.required = required_keywords
        self.min_length = min_length

    def check(self, output: str, context: dict) -> CheckResult:
        issues = []
        missing = [kw for kw in self.required if kw not in output]
        if missing:
            issues.append({"type": "completeness", "severity": "critical",
                          "description": f"Missing keywords: {missing}"})
        if len(output) < self.min_length:
            issues.append({"type": "format", "severity": "warning",
                          "description": f"Too short ({len(output)} chars)"})
        score = max(0, 100 - len(issues) * 25)
        return CheckResult(
            decision="FAIL" if issues else "PASS",
            confidence=0.9,
            score=score,
            issues=issues,
        )

# ---------- Maker ----------
class LLMMaker:
    """Generator agent. In test mode uses mock output."""
    def __init__(self, use_mock: bool = True):
        self.use_mock = use_mock
        if not use_mock:
            from openai import OpenAI
            self.client = OpenAI()

    def generate(self, task: str, feedback: str = "") -> str:
        if self.use_mock:
            return f"Sales report for Q1: revenue 12.8M, growth 23% (with {task[:20]})"
        system = "You are a report generator. Be accurate and complete."
        if feedback:
            system += f"\nPrevious issues: {feedback}\nFix them."
        resp = self.client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "system", "content": system},
                      {"role": "user", "content": task}],
        )
        return resp.choices[0].message.content

# ---------- The Loop ----------
class MakerCheckerLoop:
    def __init__(self, maker, checker, term: TermConfig):
        self.maker = maker
        self.checker = checker
        self.term = term
        self.history = []

    def run(self, task: str) -> tuple[str, list]:
        feedback = ""
        last_output = ""
        last_score = 0.0
        start = time.time()

        for attempt in range(1, self.term.max_retries + 1):
            # Time budget check
            if time.time() - start > self.term.time_budget_sec:
                return last_output, self.history + ["⏰ TIME_BUDGET exceeded"]

            # Maker generates
            output = self.maker.generate(task, feedback)
            # Checker validates
            result = self.checker.check(output, {"task": task})

            self.history.append({
                "attempt": attempt,
                "score": result.score,
                "decision": result.decision,
            })

            # Termination checks
            if result.decision == "PASS" and result.score >= self.term.quality_threshold:
                return output, self.history + ["✅ PASS: quality threshold met"]

            if result.score >= self.term.quality_threshold:
                return output, self.history + ["✅ PASS: score threshold"]

            # Convergence detection
            if last_output and SequenceMatcher(None, last_output, output).ratio() >= self.term.convergence_similarity:
                return output, self.history + ["⚡ CONVERGED: no improvement"]

            # Diminishing returns
            if attempt > 1 and (result.score - last_score) < self.term.min_improvement:
                return output, self.history + ["🔻 DIMINISHING: minimal gain"]

            # Build feedback from issues
            feedback = "; ".join(i["description"] for i in result.issues)
            last_output = output
            last_score = result.score
            time.sleep(0.5)

        return last_output, self.history + [f"❌ MAX_RETRY ({self.term.max_retries})"]

# ---------- Demo ----------
if __name__ == "__main__":
    # Test mode — no API key needed
    maker = LLMMaker(use_mock=True)
    checker = MockChecker(required_keywords=["revenue", "growth"])
    loop = MakerCheckerLoop(maker, checker, TermConfig(max_retries=5, quality_threshold=70))

    output, log = loop.run("Generate quarterly sales report")
    print(f"Final output: {output[:80]}...")
    print(f"Attempts: {[h['attempt'] for h in log[:-1]]}")
    print(f"Termination: {log[-1]}")

Enter fullscreen mode Exit fullscreen mode

Run it:

python3 maker_checker.py

Enter fullscreen mode Exit fullscreen mode

Expected output:

Final output: Sales report for Q1: revenue 12.8M, growth 23%...
Attempts: [1, 2]
Termination: ✅ PASS: quality threshold met

Enter fullscreen mode Exit fullscreen mode

Real mode:

export OPENAI_API_KEY=sk-xxx
# Change LLMMaker(use_mock=True) to LLMMaker(use_mock=False)

Enter fullscreen mode Exit fullscreen mode

4. Key Insight: Why This Beats “Smarter Models”

Most people think agents make mistakes because the model isn’t smart enough. This is wrong.

A smarter model lowers the rate of unknown errors — but never to zero. Maker/Checker separation eliminates known errors by making them structurally impossible to pass.

• Smarter model → fewer unknown errors
• Maker/Checker → zero recurrence of known errors

Production systems don’t pursue “never making mistakes.” They pursue “mistakes get caught and fixed automatically.” That’s what this framework does.

5. Where You Are Now

You’re no longer the developer who adds “please check your work” to the prompt and hopes for the best. You’re becoming an engineer who builds validation into the architecture — where the checker is independent, the output is machine-parsed, and the loop terminates by design, not by chance.

Next: DevOps for a one-person company — full observability and alerting for your agent.

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.