Beyond Passing Tests: 상황에 맞는 AI 코딩 에이전트를 평가하기 위한 100렌즈 프레임워크 🤖

AI coding agents are getting better at writing code.

But I think we are approaching a more difficult question:

How do we know that an AI agent made the right engineering decision for the current state of a software system?

Passing tests is important.

But passing tests alone does not necessarily tell us whether an agent understood:

• the current architecture,
• project constraints,
• previous engineering decisions,
• repository conventions,
• dependency relationships,
• security requirements,
• or why an existing implementation looks the way it does.

This becomes particularly important as AI systems move from generating isolated code snippets toward modifying real repositories.

The Problem: Correct Code Is Not Always Correct Engineering

Consider a simple example.

A project initially has:

Architecture v1

API
 ↓
Service
 ↓
Database

Enter fullscreen mode Exit fullscreen mode

An AI agent is asked to add a feature.

It studies the repository, follows the existing pattern, writes the code, and all tests pass.

Then the architecture changes:

Architecture v2

API
 ↓
Event Bus
 ↓
Service
 ↓
Database

Enter fullscreen mode Exit fullscreen mode

The same task is requested again.

If the agent still generates code based on the old architecture, the implementation may be:

✓ Valid syntax
✓ Compiles
✓ Existing tests pass
✗ Violates current architecture
✗ Ignores current constraints

Enter fullscreen mode Exit fullscreen mode

So we have an important distinction:

Functional Correctness
        ≠
Contextual Correctness
        ≠
System-Level Correctness

Enter fullscreen mode Exit fullscreen mode

This is the problem I want to explore.

This Is Already Becoming a Real Engineering Problem

This isn’t simply speculation about future AI systems.

Modern coding agents already depend on repository-level context.

OpenAI’s documentation for Codex recommends using persistent repository instructions such as AGENTS.md for naming conventions, business logic, known quirks, dependencies, and other information that may not be inferable directly from code. It also recommends providing file paths, component names, diffs, and documentation when describing tasks.

OpenAI has also described a broader approach where repository knowledge becomes a structured source of truth rather than one giant instruction document, explicitly noting that context management is one of the biggest challenges for agents working on large and complex tasks.

That leads to an interesting conclusion:

If context materially affects agent performance, context should also become part of agent evaluation.

Existing Benchmarks Are Already Moving Toward Real Software

SWE-bench was created to evaluate AI systems on real software-engineering issues from GitHub repositories.

The agent receives a repository and an issue, modifies the code, and is evaluated using tests. SWE-bench Verified was later created as a human-validated subset after OpenAI and the SWE-bench authors found problems with some benchmark tasks. 500 tasks were selected after professional developers screened the data.

But benchmark methodology itself is evolving.

In February 2026, OpenAI reported that SWE-bench Verified had become increasingly contaminated and recommended newer evaluations such as SWE-bench Pro.

In July 2026, OpenAI also reported that its audit of SWE-bench Pro found widespread task-quality problems and estimated roughly 30% of tasks were broken.

That matters because it demonstrates a broader lesson:

Evaluating AI systems is itself an engineering problem.

A benchmark can produce a number without necessarily producing a reliable measurement.

Context Reuse Is Already Being Studied

This is also not an isolated idea.

A 2026 research benchmark called SWE-ContextBench specifically investigates whether coding agents can reuse relevant experience across related software-engineering tasks.

The benchmark augments SWE-bench Lite with related tasks derived from dependency and reference relationships between GitHub issues and pull requests. It evaluates prediction accuracy, time efficiency, and cost efficiency. The authors report that appropriately selected summarized experience can improve resolution accuracy while reducing runtime and token cost, whereas poorly selected experience can provide limited or negative benefits.

That suggests something important:

More context
      ≠
Better result

Enter fullscreen mode Exit fullscreen mode

The real question is:

Relevant context
        +
Correct retrieval
        +
Correct interpretation
        ↓
Better decision

Enter fullscreen mode Exit fullscreen mode

So What Should We Actually Measure?

I propose thinking about agent evaluation as a multi-dimensional problem.

Instead of:

Task
 ↓
Agent
 ↓
Code
 ↓
Tests
 ↓
Pass / Fail

Enter fullscreen mode Exit fullscreen mode

we could evaluate:

                 ┌───────────────┐
                 │     TASK      │
                 └───────┬───────┘
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
         Repository   Constraints   History
             │           │           │
             └───────────┼───────────┘
                         ▼
                  AI CODING AGENT
                         │
                         ▼
                      DECISION
                         │
          ┌──────────────┼──────────────┐
          ▼              ▼              ▼
     Code Quality   Constraints    Context Fit
          │              │              │
          └──────────────┼──────────────┘
                         ▼
                    Final Score

Enter fullscreen mode Exit fullscreen mode

Context-Shift Testing

This is the idea I find particularly interesting.

Keep the:

Model
Task
Repository

Enter fullscreen mode Exit fullscreen mode

as constant as possible.

Then change one meaningful part of the context.

For example:

Scenario A

Database:
PostgreSQL

Architecture:
Repository pattern

Constraint:
All database access must go through repositories.

Enter fullscreen mode Exit fullscreen mode

The agent produces:

Controller
   ↓
Service
   ↓
Repository
   ↓
PostgreSQL

Enter fullscreen mode Exit fullscreen mode

Good.

Scenario B

Change one relevant constraint:

Database:
PostgreSQL

Architecture:
Event-driven

Constraint:
Services must communicate through events.

Enter fullscreen mode Exit fullscreen mode

Now the appropriate implementation should change.

If the agent continues producing the old architecture, we can measure a context adaptation failure.

But There Is a Second Test

We shouldn’t reward an agent merely for changing its answer.

Suppose we change something irrelevant:

README formatting

Enter fullscreen mode Exit fullscreen mode

The architecture hasn’t changed.

The agent should ideally make the same engineering decision.

So:

Relevant context changes
        ↓
Decision SHOULD change

Enter fullscreen mode Exit fullscreen mode

while:

Irrelevant context changes
        ↓
Decision SHOULD remain stable

Enter fullscreen mode Exit fullscreen mode

This gives us two useful properties.

Context Adaptation

Does the agent react when relevant context changes?

Context Stability

Does the agent remain stable when irrelevant context changes?

A Possible Benchmark

A practical benchmark could use paired or grouped scenarios:

Task T
Context C1
      ↓
   Agent
      ↓
Decision D1

Task T
Context C2
      ↓
   Agent
      ↓
Decision D2

Enter fullscreen mode Exit fullscreen mode

Where:

C1 → C2

Enter fullscreen mode Exit fullscreen mode

contains a controlled change.

Then evaluate:

Was the change relevant?
        ↓
Should the decision change?
        ↓
Did the agent change?
        ↓
Was the new decision correct?

Enter fullscreen mode Exit fullscreen mode

And separately:

Was the context change irrelevant?
        ↓
Should the decision remain stable?
        ↓
Did the agent unnecessarily change?

Enter fullscreen mode Exit fullscreen mode

Possible Metrics

I wouldn’t claim these are established industry-standard metrics. They are a proposed framework that would need experimental validation.

1. Functional Correctness

Did the implementation satisfy the task?

2. Constraint Adherence

Did the implementation respect explicit constraints?

3. Context Adaptation Rate

When relevant context changed, how often did the agent make the appropriate change?

4. Context Stability

When irrelevant context changed, how often did the agent preserve the appropriate decision?

5. Repository Consistency

Does the change follow the project’s established architecture and conventions?

6. Regression Rate

Did the change break previously working behavior?

7. Context Retrieval Efficiency

How much context did the agent need to retrieve to make the correct decision?

This could eventually produce something like:

Agent Reliability Score
│
├── Functional Correctness
├── Constraint Adherence
├── Context Adaptation
├── Context Stability
├── Repository Consistency
├── Regression Resistance
└── Context Efficiency

Enter fullscreen mode Exit fullscreen mode

Why “More Context” Isn’t the Answer

A common reaction might be:

“Just give the model the entire repository.”

But that’s not necessarily a solution.

OpenAI’s own engineering discussion around Codex describes the problem with extremely large instruction documents: context is limited, important information can be crowded out, stale instructions can accumulate, and humans may stop maintaining them. Their approach is instead to use a concise map pointing toward deeper sources of truth.

So the problem isn’t simply:

How much context?

Enter fullscreen mode Exit fullscreen mode

It is:

Which context?
When?
From where?
How current?
How reliable?
How relevant?

Enter fullscreen mode Exit fullscreen mode

That is a much more interesting systems problem.

Context Has a Lifecycle

I think repository context should be treated as something that changes over time:

Initial Decision
      ↓
Implementation
      ↓
New Requirement
      ↓
Architecture Change
      ↓
Dependency Change
      ↓
Security Change
      ↓
New Decision

Enter fullscreen mode Exit fullscreen mode

An agent working on a long-lived repository therefore needs something closer to:

Current State
+
Historical Decisions
+
Active Constraints
+
Repository Structure
+
Relevant Documentation

Enter fullscreen mode Exit fullscreen mode

rather than simply:

Prompt + Code

Enter fullscreen mode Exit fullscreen mode

The “Why” Behind Code Matters

Two implementations can be functionally equivalent while only one fits the project.

For example:

# Implementation A
cache_result()

Enter fullscreen mode Exit fullscreen mode

versus:

# Implementation B
await cache_result()

Enter fullscreen mode Exit fullscreen mode

Both might pass a narrow test.

But the correct choice could depend on:

• concurrency model,
• performance requirements,
• architectural conventions,
• API contracts,
• previous design decisions,
• runtime environment.

The code itself doesn’t always contain the complete explanation.

Sometimes the most important information is why the code was designed that way.

From Static Benchmarks to Dynamic Benchmarks

Traditional benchmark thinking often looks like:

Fixed Task
   ↓
Fixed Dataset
   ↓
Fixed Evaluation
   ↓
Score

Enter fullscreen mode Exit fullscreen mode

But real repositories look more like:

Task
 ↓
Repository evolves
 ↓
Requirements change
 ↓
Dependencies change
 ↓
Architecture changes
 ↓
Security constraints change
 ↓
Agent receives new task

Enter fullscreen mode Exit fullscreen mode

Therefore, a future benchmark could intentionally introduce controlled environmental changes.

For example:

Version 1
   ↓
Agent decision

Version 2
   ↓
Architecture changed

Version 3
   ↓
Security policy changed

Version 4
   ↓
Dependency changed

Enter fullscreen mode Exit fullscreen mode

Then measure whether the agent adapts correctly.

A 2×2 Evaluation Model

One simple way to visualize the experiment:

Decision Should Stay Same Decision Should Change Agent stays same ✅ Stable ❌ Adaptation failure Agent changes ❌ Instability ✅ Adaptation success

This is interesting because it separates two failure modes that ordinary pass/fail evaluation can hide.

The Bigger Research Question

The question isn’t:

“Can AI write code?”

We’re already measuring that.

The more difficult question is:

“Can an AI agent maintain correct engineering judgment as the software environment changes?”

That includes:

Architecture
Requirements
Dependencies
Security
Performance
Business Rules
Repository History
Team Conventions

Enter fullscreen mode Exit fullscreen mode

This is closer to how real software development works.

And This Is Where the 100 Thinking Lenses Come In

When investigating a complex AI engineering problem, I don’t think one reasoning style is enough.

Sometimes we need a diagram.

Sometimes a benchmark.

Sometimes a root-cause analysis.

Sometimes a comparison.

Sometimes a threat model.

Sometimes a timeline.

Sometimes a first-principles explanation.

So I compiled a reusable set of 100 visual-thinking, explanation, analysis, and strategy lenses.

These aren’t claims about AI capability. They are ways to structure thinking and communicate technical problems.

100 Visual Thinking, Explanation, Analysis & Strategy Lenses

# Shortcut Lens 1 /handwritten Notebook-style handwritten notes 2 /visualize Turn ideas into visual explanations 3 /stickynotes One idea per sticky note 4 /infographic Infographic layout 5 /diagram Draw a concept diagram 6 /flowchart Step-by-step flowchart 7 /mindmap Create a mind map 8 /xray Show internal structure 9 /blueprint Technical blueprint 10 /explodedview Break object into components 11 /thenvsnow Compare past vs present 12 /timeline Chronological timeline 13 /beforeafter Transformation comparison 14 /cutaway Cutaway illustration 15 /anatomy Explain all parts 16 /layers Layer-by-layer architecture 17 /ecosystem Show all connected players 18 /journey Show end-to-end journey 19 /process Explain a complete process 20 /cycle Visualize recurring cycles 21 /roadmap Learning or execution roadmap 22 /dashboard Dashboard with KPIs 23 /comparison Side-by-side comparison 24 /versus Head-to-head comparison 25 /scale Compare sizes visually 26 /evolution Show evolution over time 27 /future Imagine future scenarios 28 /inside Reveal inner workings 29 /microscopic Zoom into microscopic detail 30 /macroscopic Zoom out to system level 31 /crosssection Cross-sectional illustration 32 /map Geographic or conceptual map 33 /heatmap Show intensity 34 /network Show relationships 35 /architecture Software/system architecture 36 /wireframe Website/app layout 37 /mockup Realistic product preview 38 /prototype Early product concept 39 /schematic Simple technical schematic 40 /isometric 3D isometric illustration 41 /birdseye Top-down view 42 /360view All-angle visualization 43 /storyboard Scene-by-scene explanation 44 /comic Explain through comic panels 45 /poster Poster design 46 /cover Book/report cover 47 /adcreative Advertising concept 48 /thumbnail YouTube thumbnail concept 49 /carousel Instagram/LinkedIn carousel 50 /socialvisual Social media graphic 51 /quotevisual Quote as shareable visual 52 /eli5 Explain simply 53 /expert Expert-level explanation 54 /firstprinciples Break down to fundamentals 55 /deepdive Comprehensive explanation 56 /simplify Simplify difficult content 57 /analogy Explain through analogy 58 /socratic Teach through questions 59 /teachme Structured tutoring 60 /cheatsheet Quick-reference notes 61 /flashcards Study flashcards 62 /quiz Generate a quiz 63 /viva Viva preparation 64 /interview Mock interview 65 /devilsadvocate Challenge assumptions 66 /factcheck Verify claims 67 /mythvsfact Separate myths from facts 68 /proscons Advantages vs disadvantages 69 /swot SWOT analysis 70 /pestle PESTLE analysis 71 /fiveforces Porter’s Five Forces 72 /rootcause Find root cause 73 /fivewhys Five Whys analysis 74 /decisionmatrix Weighted decision matrix 75 /scenario Scenario planning 76 /simulate Simulation exercise 77 /roleplay Assume an expert role 78 /consultant Consulting-style advice 79 /executivebrief Executive summary 80 /insights Extract insights 81 /recommendations Provide recommendations 82 /prioritize Rank by priority 83 /benchmark Benchmark comparison 84 /marketmap Industry landscape 85 /strategy Strategic planning 86 /businessmodel Business model explanation 87 /pitch Investor/startup pitch 88 /investor Investor perspective 89 /redteam Stress-test a plan 90 /premortem Assume failure and analyze why 91 /reverseengineer Break down success 92 /promptengineer Optimize prompts 93 /research Structured research 94 /sources Find reliable sources 95 /summarize Summarize content 96 /extract Extract key information 97 /table Convert into a table 98 /presentation Presentation outline 99 /dashboardanalysis Analyze dashboards 100 /actionplan Create step-by-step action plan

The Important Distinction

These 100 lenses are not 100 claims that an AI model is more intelligent when using them.

They are simply structured ways of looking at a problem.

For AI-agent research, different lenses can answer different questions:

/architecture
        ↓
What is the system structure?

/xray
        ↓
What is happening internally?

/timeline
        ↓
How did the system change?

/thenvsnow
        ↓
What changed between versions?

/benchmark
        ↓
How should we measure it?

/factcheck
        ↓
Which claims have evidence?

/redteam
        ↓
How can the evaluation fail?

/rootcause
        ↓
Why did the agent fail?

/decisionmatrix
        ↓
Which approach is better?

/actionplan
        ↓
What should we build next?

Enter fullscreen mode Exit fullscreen mode

This is especially useful when researching complex AI systems because no single representation captures the entire problem.

What I Would Test First

If I were turning this idea into an actual research experiment, I’d start small.

Dataset

Create 50–100 repository-level tasks.

Context perturbations

For each task, create controlled variants:

Architecture change
Requirement change
Security constraint change
Dependency change
Performance constraint change
Documentation change
Irrelevant formatting change

Enter fullscreen mode Exit fullscreen mode

Evaluation

Run:

Same Model
Same Task
Different Context

Enter fullscreen mode Exit fullscreen mode

Then measure:

Functional correctness
Context adaptation
Context stability
Constraint adherence
Regression
Token usage
Runtime

Enter fullscreen mode Exit fullscreen mode

Human validation

For ambiguous cases, use experienced developers to verify whether the changed decision was actually appropriate.

This is important because benchmark design itself can introduce errors. OpenAI’s SWE-bench work demonstrates why human validation and benchmark auditing matter when interpreting agent performance.

The Hypothesis

My current hypothesis is:

A reliable coding agent should not simply produce correct code. It should produce decisions that are appropriate for the current context, adapt when relevant context changes, and remain stable when irrelevant context changes.

That’s a much stronger definition of reliability.

And importantly, it is something we can attempt to measure.

Final Thought

AI coding agents are moving from:

Code Completion

Enter fullscreen mode Exit fullscreen mode

toward:

Software Engineering Agents

Enter fullscreen mode Exit fullscreen mode

As that transition happens, our evaluation methods need to evolve too.

The future benchmark may not simply ask:

“Did the code pass?”

It may need to ask:

“Did the agent understand the current system well enough to make the right engineering decision?”

That is the problem I find most interesting.

And I don’t think we have completely solved it yet.

What would you add to a context-aware coding-agent benchmark?

Architecture changes?
Security constraints?
Dependency changes?
Business requirements?
Repository history?

I’d genuinely like to hear how other developers would design it.

References & Further Reading

• OpenAI — How OpenAI Uses Codex: repository context, AGENTS.md, task specification, and development-environment guidance.
• OpenAI — Harness Engineering: repository knowledge, context management, structured documentation, and agent-first development.
• OpenAI — SWE-bench Verified: human validation of software-engineering benchmark tasks and evaluation methodology.
• OpenAI — Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities: benchmark contamination and evaluation limitations.
• OpenAI — Separating Signal From Noise in Coding Evaluations: 2026 analysis of benchmark quality and broken tasks.
• SWE-ContextBench: research on context and experience reuse in coding agents.

AI #AIAgents #SoftwareEngineering #LLM #CodingAgents #MachineLearning #ArtificialIntelligence #DevTools #Benchmarking #AIResearch #OpenSource