I built a governance layer for AI agent skills — with an AI agent as my pair programmer

AI coding agents are becoming remarkably capable.

They can explore repositories, generate implementation code, write tests, refactor modules, and propose architectural changes.

But after using them across more than a few isolated sessions, I kept running into the same frustrating problem.
They forgot.

Not literally. The agent still had a context window, repository files, and a set of instructions.

But every new session required me to reconstruct part of the project’s mental model:

• why a particular architectural decision had been made;
• which conventions were mandatory;
• which approaches had already failed;
• what “production-ready” meant in this repository;
• which operations required approval;
• and how a task should be completed safely.

Instead of moving forward, I repeatedly rebuilt context.
At first, I assumed this was a prompting problem.
Maybe I needed a better system prompt.
A longer AGENTS.md.
More detailed instructions.
More examples.
Eventually, I realized I was asking the wrong question.

Instead of asking:

How can I make the agent remember more?

I started asking:

What if the project itself could remember?

That question led me to build the Skill Governance Toolkit: an open-source governance layer for AI agent skills.

And, somewhat recursively, I built most of it together with an AI agent acting as my pair programmer.

This is Part 1 of a three-part series about governing AI agent skills.

• Part 1: Why project memory became the real bottleneck
• Part 2: How I designed the governance architecture
• Part 3: Why AI agent skills should be reviewed like executable code

The problem is no longer just code generation

The first generation of AI coding discussions focused on one question:

Can AI generate working code?

Today, that is often the least interesting question.

The harder questions are:

• Can the agent behave consistently across sessions?
• Can another engineer review the instructions it follows?
• Can project knowledge survive beyond one conversation?
• Can the same workflow operate across different AI runtimes?
• Can a team distinguish a safe skill from one that merely worked during a demo?
• Can an agent understand not only what to do, but also what it must never do?

An agent can generate a correct function while still behaving unreliably at the project level.

It can solve a local task while violating a repository-wide convention.
It can produce a technically valid implementation that repeats a mistake discussed two weeks ago.
It can use an instruction that works in one runtime and behaves differently in another.

The model may be capable.
The surrounding system may still be immature.

Bigger prompts do not solve bigger projects

Like many developers, I initially responded by adding more instructions.

First came basic repository conventions.
Then architecture notes.
Then testing rules.
Then deployment instructions.
Then examples.
Then warnings about approaches that should not be used.

Eventually, the instruction file started to resemble a compressed engineering handbook.

That feels productive, but it introduces new problems.

Most context is irrelevant to the current task

A UI-testing task does not need deployment procedures.
A documentation change does not need the full database migration guide.
A small refactoring does not need every architectural decision ever made.
Loading everything into every conversation wastes context and makes the important instructions harder to identify.

Large instruction files become difficult to maintain

Rules begin to overlap.
Examples become outdated.
Two sections contradict each other.
Nobody knows which instruction is still active and which one describes a decision that was reversed months ago.

Knowledge remains passive

A document may explain what should happen, but it does not necessarily define:

• when a procedure should be selected;
• what inputs it expects;
• what output it should produce;
• what permissions it requires;
• how to validate its result;
• what happens after a partial failure;
• or whether it is safe to run twice.

More text does not automatically create a better operational contract.

From conversation memory to project memory

The workflow I had looked roughly like this:

The conversation was carrying too much responsibility.
Important knowledge lived in temporary chat history rather than in durable, reviewable project assets.

So I started working toward a different model:

In this model, the project does not expect the agent to remember everything.

Instead, it provides the right knowledge when the knowledge becomes relevant.
That is a subtle but important change.
The goal is not an infinitely large prompt.
The goal is a project that can assemble appropriate context for a specific task.

The folder of skills I did not trust

The idea became concrete when I looked at my growing collection of SKILL.md files.

Some were mine.
Some came from other libraries.
Some had been created late at night to unblock a task.
They all “worked” in the narrow sense that an agent could load them and do something.
But I could not confidently answer basic questions:

• Which skills were safe to share with another developer?
• Which were actually portable?
• Which depended on behavior specific to one AI runtime?
• Which contained hidden assumptions?
• Which were safe to execute repeatedly?
• Which had a dangerous command buried inside a code block?
• Which deserved to be called production-ready?

That was the real origin of the project.
Not a grand platform strategy.
A folder of skills I did not fully trust.

AI skills are more than reusable prompts

A useful agent skill is not just a paragraph of instructions.

It may contain:

• activation conditions;
• step-by-step procedures;
• shell commands;
• file references;
• integration configuration;
• permission assumptions;
• output contracts;
• validation steps;
• and recovery behavior.

Once an agent can act on those instructions, the skill becomes operational.

That led me to a central principle:

AI skills should be managed more like source code and less like disposable prompts.

A mature skill should be:

• versioned, so changes remain visible;
• reviewable, so another engineer can understand its behavior;
• discoverable, so humans and agents can find it;
• portable, so runtime-specific assumptions are explicit;
• testable, so “it worked once” is not the acceptance criterion;
• auditable, so risky behavior can be detected before execution;
• maintainable, so rules do not silently drift apart.

This was the conceptual foundation of the Skill Governance Toolkit.

What I built

The toolkit currently contains six meta-skills.

They are called meta-skills because they operate on other skills rather than directly on application code.

Skill Purpose skill-find Finds relevant skills across project, personal, and shared libraries skill-evaluate Evaluates one skill against explicit quality and safety criteria skill-compare Compares related skill versions and recommends reuse or merging library-audit Audits an entire library for consistency and risk skill-build-portable Converts runtime-specific instructions into a more portable skill integration-init Bootstraps optional integrations for a project

Under those conversational workflows sits a deterministic audit engine.

The same engine can be invoked through:

• a CLI for local development and CI;
• an MCP server for use inside agent workflows;
• a sandboxed Docker image.

This diagram is deliberately simplified.

In Part 2, I will explain why the toolkit separates deterministic enforcement from LLM-assisted judgment and how the six meta-skills cooperate.

Building it with the agent being governed

There is an obvious recursive element in the project.

I used an AI coding agent to help build a system intended to govern future AI-assisted work.
Claude Code produced much of the implementation and Markdown.
My role was different.
I made decisions about:

• architecture;
• scope;
• safety boundaries;
• acceptable false-positive rates;
• test strategy;
• release readiness;
• and what “good enough” meant at each stage.

The agent was especially effective at:

• generating implementation variants;
• writing fixtures;
• expanding repetitive test coverage;
• finding contradictions;
• synchronizing documentation;
• and turning architectural decisions into code quickly.

But faster implementation did not remove the need for engineering judgment.
It increased its importance.
When code can be produced rapidly, a poor decision can also spread rapidly.

The bottleneck moves from typing to:

• framing the problem;
• choosing constraints;
• evaluating trade-offs;
• verifying behavior;
• and deciding what should not be built.

By the numbers

The initial development period produced:

🧩 6 governance meta-skills

🛡️ 17 deterministic rule IDs

📐 9 quality dimensions

🚀 42 versioned releases in 18 days

✅ 37 self-CI phases

🧪 384 automated tests by v1.18.2

📦 CLI, MCP, and Docker delivery modes

Enter fullscreen mode Exit fullscreen mode

The release history was highly concentrated.

Twenty-four releases landed during the first three days.

Another major development burst happened roughly two weeks later.

The interesting point is not simply that the project moved quickly.
The interesting point is what made that speed survivable.
Every behavioral change followed the same pattern:

Failing test
    ↓
Implementation
    ↓
Targeted verification
    ↓
Full self-CI
    ↓
Commit

Enter fullscreen mode Exit fullscreen mode

Red first, then green.
Every time.
AI accelerated implementation, but tests and governance prevented that acceleration from turning into uncontrolled change.

The most important architectural boundary

While building the toolkit, one design decision became more important than any individual feature.
Not every question should be answered by an LLM.
And not every question should be forced into a regular expression.

Some checks are deterministic:

• Does the file contain a destructive command?
• Is a secret written inline?
• Does a reference escape the governed directory?
• Is there an invisible Unicode control character?
• Does an unsafe hook modify the execution environment?

Other checks require interpretation:

• Is the skill’s purpose clear?
• Does it explain when it should be activated?
• Is its portability claim credible?
• Does it define sensible failure behavior?
• Is the blast radius explicit?
• Is it safe to run twice?

The toolkit therefore uses two governance surfaces:

The rule of thumb became:

A semantic judgment forced into a deterministic gate becomes a false-positive machine.

And the inverse:

A security invariant left entirely to an LLM is not a reliable gate.

I will unpack this architecture in detail in Part 2.

What surprised me most

The biggest surprise was not that AI could write code.
I already knew that.
The surprise was how much more reliable the agent became when project knowledge was treated as structured, reviewable software.
The model did not change.
The context architecture did.
This led me to several conclusions.

AI does not always need a larger prompt

It often needs better knowledge selection.

Documentation is becoming operational

When an agent can execute instructions, natural-language documents inherit many of the risks of code.

Portability must be designed

A skill is not cross-platform merely because its Markdown can be opened everywhere.
Runtime semantics matter.

Faster coding increases the value of governance

When implementation becomes cheaper, poor decisions can propagate faster.
Review, boundaries, and verification become more important—not less.

An AI pair programmer still needs an engineer

The agent can generate.

The engineer still has to decide:

• what problem is worth solving;
• what constraints are real;
• what evidence is sufficient;
• what risks are acceptable;
• and when the result is ready for somebody else to trust.

Before and after

The change in my workflow can be summarized like this:

Before After Repeating instructions in every session Reusing versioned skills Loading one oversized prompt Selecting task-relevant knowledge Trusting skills because they worked once Evaluating them against explicit criteria Assuming portability Declaring and testing compatibility Reviewing Markdown as documentation Treating it as an operational asset Letting the LLM judge everything Separating judgment from deterministic gates Project knowledge living in conversations Project knowledge living with the project

Where the series goes next

This article focused on the problem and the change in engineering mindset.
The next two parts go deeper.

Part 2: Designing the governance architecture

In Part 2, I explain:

• how the six meta-skills cooperate;
• how deterministic and semantic governance are separated;
• why the same engine is exposed through CLI, MCP, and Docker;
• how portability is evaluated;
• how interrupted audits resume from checkpoints;
• and why the toolkit audits itself.

Part 3: Treating skills like executable code

In Part 3, I examine:

• hidden Unicode and homoglyph attacks;
• dangerous shell instructions;
• secret leakage;
• unsafe hooks and MCP configuration;
• blast radius;
• prompt injection;
• authorization assumptions;
• rollback;
• and idempotency.

Final thoughts

I do not think the future of AI-assisted development is one enormous prompt containing everything an agent might need.
I think the project itself should carry structured, discoverable, versioned knowledge.
Prompts are temporary.
Project memory should not be.

The Skill Governance Toolkit is my attempt to explore what that could look like.

If you have a collection of agent instructions that work—but that you would hesitate to hand to another engineer—you may already have the same problem that started this project.

I would be especially interested to hear:

• How do you keep AI coding agents consistent across weeks or months?
• Where does project knowledge live in your workflow?
• Do you review agent skills as documentation or as executable assets?
• Which governance checks should be deterministic?
• Which ones should remain engineering judgment?

The repository is open source, and critical feedback is very welcome.

Repository: artemrudenko/skill-governance-toolkit

Next: How I Designed a Governance Layer for AI Agent Skills