I had a system that looked ready.
One AI model designed the architecture.
Another implemented it.
Additional models reviewed the output, checked the logic, and produced validation documents.
Synthetic legal cases passed several internal checks.
Then I tested the system with documents from a real legal case.
It confused the parties.
It misread the meaning of monetary figures.
It mixed opposing claims.
It inferred relationships that were not supported by the source documents.
It omitted information that was clearly important.
The internal review chain had not caught the failures that mattered most.
That morning, the workflow moved from:
Synthetic PASS
→ Real-case validation
→ Critical FAIL
→ Launch HOLD
→ Patch
→ Independent retest
Enter fullscreen mode Exit fullscreen mode
The most important lesson was simple:
Agreement between multiple AI models is not real-world validation.
This article explains the operating method I used, why it failed, and how I changed it.
I call the method the CHIMERA SYSTEM.
What the CHIMERA SYSTEM is
A chimera combines parts from different creatures.
The CHIMERA SYSTEM applies the same idea to AI work.
Instead of expecting one model to perform every task equally well, I assign different responsibilities to different models and keep a human in control of the workflow.
A simplified version looks like this:
Human-defined objective
→ Architecture AI
→ Execution AI
→ Adversarial review AI
→ Validation AI
→ Real-world data test
→ Human judgment
→ Error and revision log
Enter fullscreen mode Exit fullscreen mode
One model may be stronger at architecture and reasoning.
Another may be better at implementation.
Another may be better at criticism, comparison, or finding inconsistencies.
The human decides the objective, controls the handoffs, checks the result against reality, and decides whether the workflow should continue.
The system is not tied to one model provider.
Models can be replaced without changing the overall operating method.
What it is not
The CHIMERA SYSTEM is not a claim that I invented multi-agent AI.
Role separation, evaluator-optimizer patterns, shared memory, orchestration, and human-in-the-loop workflows already exist.
It is also not a fully autonomous multi-agent system.
There is no custom orchestrator automatically creating agents and routing tasks.
There is no server coordinating every transition.
There is no complex API integration.
The workflow is better described as a human-directed, document-based multi-model system.
The practical question behind it is:
Can a non-developer operate a useful multi-model AI team without first building an agent platform?
My answer so far is yes—but only if the workflow is transparent, the roles are explicit, and real-world testing can overrule AI consensus.
Why I used Google Drive
In this system, Google Drive is not simply a place to store finished files.
It acts as:
- shared memory between models,
- a handover channel,
- a decision log,
- an error record,
- a version trail,
- and a recovery point for the project.
Each model leaves a document for the next participant.
The next model can inspect the previous output, challenge it, revise it, or reject it.
The human can see the entire trail.
That matters because chat history is fragile.
Important decisions become buried.
Context windows end.
Models summarize earlier work incorrectly.
A new model may not know why an earlier decision was made.
Documents make the state of the project visible.
This connects closely with a lesson Anthropic described in its article on building a multi-agent research system. Anthropic noted that specialized agents can write persistent artifacts to external systems rather than passing everything through a coordinator. That reduces information loss and the “game of telephone” across multi-stage workflows.
In my version, the external artifact system is simply Google Drive.
It is not technically sophisticated.
But it is accessible.
A non-developer can open the folder, read the record, stop the workflow, replace a model, or return to an earlier decision without rebuilding infrastructure.
Why the approach seemed to work
I applied the CHIMERA structure to a legal-document organization project.
The product goal was narrow.
It was not supposed to give legal advice, predict outcomes, or replace a lawyer.
Its job was to organize information such as:
- parties,
- dates,
- amounts,
- claims,
- evidence,
- conflicting statements,
- deadlines,
- and source positions.
The workflow separated architecture, implementation, review, and validation.
Synthetic cases were created to test known conditions.
The system passed several of those checks.
The output looked structured.
The validation documents looked serious.
Different models reviewed one another’s work.
From inside the system, the process appeared disciplined.
That appearance was misleading.
What the real case exposed
Real legal documents were far messier than the synthetic cases.
The same person could appear under different labels.
Amounts could refer to principal, damages, deposits, repayments, or disputed figures.
The same event could be described differently by opposing parties.
Dates could represent filing dates, occurrence dates, contract dates, or later recollections.
Important facts could be distributed across multiple documents rather than stated cleanly in one place.
The system did not merely make cosmetic mistakes.
It made structural mistakes.
Examples included:
1. Party attribution errors
Statements from one side were assigned to the other side.
In legal work, that is not a minor formatting problem.
It changes the meaning of the entire record.
2. Monetary meaning errors
The system extracted numbers but misunderstood what they represented.
Correct extraction is not enough when the semantic role of the amount is wrong.
3. Claim merging
Opposing accounts were combined into one apparently coherent narrative.
The output became easier to read but less faithful to the source material.
4. Unsupported inference
The system created relationships or causal explanations that were not explicitly supported by the documents.
5. Critical omission
Important source information disappeared during summarization or restructuring.
These failures revealed a weakness in the earlier evaluation design.
The synthetic tests mainly checked whether the system could handle the conditions we had anticipated.
The real case introduced combinations we had not imagined.
Why multiple AI reviewers were not enough
It is tempting to believe that one AI can write, another can review, and a third can verify the result.
Sometimes that improves quality.
But it does not create truth automatically.
Models may share similar assumptions.
They may reward outputs that are internally coherent even when the source attribution is wrong.
A reviewer may focus on structure and miss factual alignment.
A validator may check the generated output against another generated summary rather than returning to the original evidence.
Multiple models can therefore produce a stronger-looking mistake.
Anthropic’s engineering guidance is relevant here.
In Building Effective Agents, Anthropic writes that successful implementations often use “simple, composable patterns” rather than unnecessarily complex frameworks. It also recommends starting with the simplest workable approach and adding complexity only when it measurably improves outcomes.
That supports the basic CHIMERA idea: role separation does not need to begin with a large framework.
But Anthropic’s multi-agent research system also demonstrates the harder side of the problem.
Multi-agent systems introduce new difficulties in coordination, evaluation, reliability, state management, and error propagation.
Most importantly, Anthropic states that:
Human evaluation catches what automation misses.
That was exactly the turning point in my project.
The real case was not just another input.
It was a human-grounded evaluation that invalidated the system’s internal confidence.
The revised operating loop
After the failure, I changed the workflow.
The new sequence is:
1. Define the human objective and product boundary
2. Design the architecture
3. Execute the implementation
4. Perform independent adversarial review
5. Validate against original source documents
6. Test with real-world cases
7. Classify every meaningful failure
8. Convert failures into regression tests
9. Patch the engine
10. Rerun the regression suite
11. Rerun the real case
12. Keep launch on hold until critical errors stop recurring
Enter fullscreen mode Exit fullscreen mode
The most important change is that AI review no longer closes the loop.
Reality closes the loop.
The system must return to original documents and real cases.
A PASS produced by models is provisional.
A real-world contradiction has higher authority.
Six operating principles
The current CHIMERA method follows six principles.
1. Separate architecture, execution, and validation
The same model should not define the standard, produce the result, and declare that the result passed.
Role separation reduces self-confirmation.
2. Transfer important decisions through documents
Do not rely only on chat history or memory.
Store the objective, assumptions, changes, open questions, and test results in persistent files.
3. Require independent challenge
The next model should not simply continue the previous model’s reasoning.
Its role should include disagreement, source checking, and failure discovery.
4. Prioritize evidence over model consensus
Three models agreeing does not outweigh one source document proving them wrong.
5. Convert failures into permanent tests
A failure should not remain a one-time correction.
It should become a regression case that the system must pass in future versions.
6. Keep a human able to stop the system
The human is not a ceremonial approver.
The human must be able to pause deployment, reject the result, change the architecture, or redefine the task.
This also aligns with Anthropic’s work on trustworthy agents, which places meaningful human control among the core principles for agent systems.
Why this matters for non-developers
Most public explanations of multi-agent systems begin with:
- APIs,
- orchestration frameworks,
- vector databases,
- tool schemas,
- memory services,
- state machines,
- and deployment infrastructure.
Those components are useful.
But they also create the impression that multi-model collaboration belongs only to developers.
The CHIMERA SYSTEM begins somewhere simpler:
Give models different roles.
Make them work from the same documents.
Require visible handovers.
Force independent review.
Test the result against reality.
Keep the human in control.
Enter fullscreen mode Exit fullscreen mode
That can be done manually before it is automated.
Manual operation is slower.
But it has an advantage: the process remains observable.
The human can see where a decision entered the system.
The human can compare versions.
The human can replace one model without rebuilding the entire workflow.
The human can identify which stage created the failure.
For early-stage or high-risk work, that visibility may be more valuable than maximum automation.
Where this pattern may be useful
The method is most relevant when:
- an error has meaningful consequences,
- source attribution matters,
- multiple documents conflict,
- the task has distinct reasoning and execution stages,
- auditability is important,
- or the workflow needs repeated improvement.
Potential applications include:
- legal document organization,
- financial and market research,
- software architecture and implementation,
- research synthesis,
- compliance workflows,
- publishing and editorial review,
- and internal business analysis.
It is unnecessary for every task.
A short translation, simple calculation, or low-risk draft does not need a multi-model workflow.
Complexity should be earned by the risk and structure of the problem.
Current limitations
The CHIMERA SYSTEM still has obvious limitations.
It is not fully automated
A human controls the handoffs.
That reduces speed and increases operational effort.
Google Drive is not a real orchestration engine
It does not enforce state transitions, permissions, dependencies, or structured execution.
The discipline comes from naming rules, document templates, and human oversight.
Multiple models can share the same blind spots
Model diversity does not guarantee independent reasoning.
Real-world validation can expose privacy and security risks
Sensitive documents should not be uploaded casually to external systems.
In my legal-document project, actual case originals must remain local and any reusable regression material must be anonymized and minimized.
The method does not remove domain experts
A document workflow cannot replace professional legal judgment, security engineering, or domain accountability.
The system organizes and tests work.
It does not inherit a professional license.
What I am testing next
The next phase is not to add more agents.
It is to improve the evaluation loop.
That means:
- expanding anonymized regression cases,
- checking attribution at the source level,
- measuring critical error recurrence,
- separating extraction accuracy from semantic interpretation,
- testing with legal professionals,
- and documenting where the system must refuse or defer.
If the workflow becomes reliable, parts of it may later be automated through APIs or agent frameworks.
But automation should come after the operating logic is understood.
Not before.
Final thought
The most interesting part of this experiment was not that several AI models could collaborate.
That is already well established.
The important part was discovering that a polished multi-model review process could still fail on the first serious real-world case.
The right question is not:
How many AI models worked on this?
The better question is:
When reality proves the system wrong, can it preserve the failure, turn it into a test, and change direction?
That is the question I want the CHIMERA SYSTEM to answer.
답글 남기기