Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor
Cloud Swords Mod — Blog 10
In Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But “works” isn’t a metric.
How do I know the AI is making good decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?
I needed an evaluation framework. So I built one. And the judge? Another AI.
The Problem: Non-Deterministic Decisions
The agent uses gpt-oss:20b. Same input can produce slightly different outputs between calls. You can’t write assert response == expected because there’s no single correct answer.
What you CAN evaluate:
- Is the action valid JSON? (deterministic check)
- Is the action within bounds? (1-5 minions, valid buff types)
- Is the response proportional to the threat? (requires judgment)
- Is the reasoning coherent? (requires judgment)
The first two are code. The last two need an LLM.
The Eval Dataset: Real Interactions
Remember interactions.json from the mock server? Every time the agent makes a decision, it’s logged:
{
"timestamp": "2026-05-19T19:17:34",
"payload": {
"player": "Carlos",
"weapon": "sword_of_lambda",
"health": 5,
"nearby_mobs": 10,
"biome": "nether",
"dimension": "the_nether"
},
"decision": {
"action": "summon",
"count": 5,
"duration": 10,
"reason": "low health, many mobs"
},
"source": "agent"
}
Enter fullscreen mode Exit fullscreen mode
I took 20+ real interactions and added expected behavior ranges:
eval_dataset = [
{
"context": {"health": 5, "nearby_mobs": 10, "biome": "nether", "dimension": "the_nether"},
"expected_action": "summon",
"expected_count_range": [4, 5],
"expected_reasoning": "Should recognize critical threat (low HP + many mobs + dangerous biome)",
"tags": ["critical", "nether"]
},
{
"context": {"health": 20, "nearby_mobs": 0, "biome": "plains", "dimension": "overworld"},
"expected_action": "none",
"expected_count_range": [0, 0],
"expected_reasoning": "Full health, no threats — should do nothing",
"tags": ["safe", "overworld"]
},
{
"context": {"health": 12, "nearby_mobs": 3, "biome": "dark_forest", "dimension": "overworld"},
"expected_action": "summon",
"expected_count_range": [1, 3],
"expected_reasoning": "Moderate threat — proportional response",
"tags": ["moderate", "overworld"]
},
# ... 17 more scenarios
]
Enter fullscreen mode Exit fullscreen mode
Level 1: Code Evaluators (Free, Instant)
def eval_valid_json(response: str) -> bool:
"""Can we parse the response as JSON?"""
try:
json.loads(response)
return True
except:
return False
def eval_action_in_bounds(decision: dict) -> bool:
"""Is the action within allowed bounds?"""
action = decision.get("action")
if action == "summon":
count = decision.get("count", 0)
duration = decision.get("duration", 0)
return 1 <= count <= 5 and 5 <= duration <= 15
elif action == "buff":
valid_effects = {"regeneration", "resistance", "strength", "speed"}
return decision.get("effect") in valid_effects and 3 <= decision.get("duration", 0) <= 10
elif action == "none":
return True
return False
def eval_has_reason(decision: dict) -> bool:
"""Does the decision include a reason?"""
reason = decision.get("reason", "")
return len(reason) > 3 and len(reason) < 50
def eval_proportional(decision: dict, context: dict, expected: dict) -> bool:
"""Is the response roughly proportional to the threat?"""
if expected["expected_action"] == "none" and decision.get("action") == "none":
return True
if expected["expected_action"] == "summon":
count = decision.get("count", 0)
low, high = expected["expected_count_range"]
return low <= count <= high
return decision.get("action") == expected["expected_action"]
Enter fullscreen mode Exit fullscreen mode
These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.
Level 2: LLM-as-Judge
For the subjective stuff — “is this a good decision?” — I use the same model as a judge:
JUDGE_PROMPT = """You are evaluating an AI combat advisor for a Minecraft mod.
Given the player context and the AI's decision, score it on:
1. **Threat Assessment** (1-10): Did the AI correctly gauge the danger level?
2. **Proportionality** (1-10): Is the response scaled to the threat?
3. **Reason Quality** (1-10): Is the reasoning clear and accurate?
Context: {context}
AI Decision: {decision}
Expected behavior: {expected}
Respond ONLY with JSON:
{{"threat_assessment": X, "proportionality": X, "reason_quality": X, "feedback": "one sentence"}}"""
def judge_decision(context: dict, decision: dict, expected: dict) -> dict:
prompt = JUDGE_PROMPT.format(
context=json.dumps(context),
decision=json.dumps(decision),
expected=expected["expected_reasoning"]
)
result = agent(prompt)
return json.loads(str(result))
Enter fullscreen mode Exit fullscreen mode
The meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the expected behavior as reference — it’s not evaluating in a vacuum.
The Eval Runner
class CombatAdvisorEval:
def __init__(self, agent):
self.agent = agent
self.results = []
def run(self, dataset: list[dict]) -> dict:
for case in dataset:
# Get agent decision
decision = decide(self.agent, case["context"])
# Level 1: Code checks
checks = {
"valid_json": eval_valid_json(json.dumps(decision)),
"in_bounds": eval_action_in_bounds(decision),
"has_reason": eval_has_reason(decision),
"proportional": eval_proportional(decision, case["context"], case),
}
# Level 2: LLM Judge
judge_scores = judge_decision(case["context"], decision, case)
self.results.append({
"context": case["context"],
"decision": decision,
"checks": checks,
"judge_scores": judge_scores,
"tags": case["tags"],
})
return self._summary()
def _summary(self) -> dict:
total = len(self.results)
return {
"total_cases": total,
"code_pass_rate": sum(
all(r["checks"].values()) for r in self.results
) / total,
"avg_threat_assessment": sum(
r["judge_scores"]["threat_assessment"] for r in self.results
) / total,
"avg_proportionality": sum(
r["judge_scores"]["proportionality"] for r in self.results
) / total,
"avg_reason_quality": sum(
r["judge_scores"]["reason_quality"] for r in self.results
) / total,
}
Enter fullscreen mode Exit fullscreen mode
A/B Testing System Prompts
The real power: I can test different system prompts and measure which one produces better decisions.
PROMPT_A = """You are a combat advisor. Given context, decide one action...""" # Original
PROMPT_B = """You are a combat advisor. IMPORTANT: Always consider the biome danger level.
Nether and End are inherently more dangerous than Overworld.
Scale your response accordingly...""" # Biome-aware variant
# Run eval with both
agent_a = Agent(model=model, system_prompt=PROMPT_A)
agent_b = Agent(model=model, system_prompt=PROMPT_B)
results_a = CombatAdvisorEval(agent_a).run(eval_dataset)
results_b = CombatAdvisorEval(agent_b).run(eval_dataset)
print(f"Prompt A — proportionality: {results_a['avg_proportionality']:.1f}/10")
print(f"Prompt B — proportionality: {results_b['avg_proportionality']:.1f}/10")
Enter fullscreen mode Exit fullscreen mode
Prompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes.
The Feedback Loop
Observe (interactions.json)
↓
Evaluate (code checks + LLM judge)
↓
Adjust (modify system prompt)
↓
Measure (re-run eval, compare scores)
↓
Deploy (update prompt in mock server)
↓
Observe again...
Enter fullscreen mode Exit fullscreen mode
This is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod. The pattern is universal.
Results: What I Found
After running evals across 20 scenarios:
Metric Prompt v1 Prompt v2 (biome-aware) Code pass rate 95% 95% Threat assessment 7.2/10 8.4/10 Proportionality 6.8/10 8.1/10 Reason quality 7.5/10 7.8/10The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.
The 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust (find first {, last }).
Cloud/AI Concepts Taught
Eval Concept What It Teaches Code evaluators Input validation, schema enforcement LLM-as-Judge AI evaluating AI (meta-evaluation) Eval dataset Test cases for non-deterministic systems A/B testing prompts Experimentation, data-driven decisions Feedback loop Continuous improvement, observability Scoring thresholds Quality gates, SLOs for AI interactions.json Observability, audit trailsWhat I’d Do Next
- Automated eval in CI — every prompt change triggers eval, blocks if scores drop
- Player feedback — thumbs up/down after each cloud invocation, feeds back into dataset
- Seasonal prompts — different system prompts for different game phases (early/mid/endgame)
- Multi-model comparison — test gpt-oss:20b vs Claude Haiku vs Nova Lite for cost/quality tradeoff
The Full Picture
Across 10 blog posts, this mod went from “7 swords with cloud names” to a full system with:
- Custom textures generated with Python + AI
- Energy systems with BFS algorithms
- Multiblock machines with custom GUIs
- A spell system with 28 spells
- Armor sets with dodge mechanics
- Wandering merchants with elite progression
- A serverless AI backend
- An evaluation framework for AI decisions
All teaching cloud computing through gameplay. All open source.
This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading.
Code:
- Mod: cloud-swords-mod-1.20.1
- Backend: cloud-swords-backend-cdk
- Agent: cloud-swords-strands-agent
I’m Carlos Cortez — and this is Breaking the Cloud.
답글 남기기