I Asked an AI to Author the Same Policy Tests 50 Times. It Hit Every Boundary in 49 Valid Runs.

I have been experimenting with a specific question: can an AI independently author the test cases needed to challenge structured business decision logic?

Consider a vendor-screening policy with sanctions rules, country restrictions, personal-data conditions, and risk thresholds. If that policy is converted into executable decision logic, someone still needs to test the difficult boundaries – exactly 70 versus just below 70, whether an embargo-list member is handled correctly, or whether a condition changes when personal data is involved.

In the earlier Judgment Pack evaluator experiments, I gave an independent AI model the policy and asked it to author those kinds of records without seeing the Judgment Pack or the defects that would later be tested.

Before the authoring call, six classes of potential defects had already been committed.

The model authored records that covered all 6 of 6 classes.

That was encouraging, but it answered only one question:

Can the model do it?

It did not answer the more important reliability question:

If I ask it to do the same job again, how often will it cover those boundaries?

A single successful generation could simply be a lucky draw.

So Study 011 repeated the same blinded authoring task 50 times.

The result, for this particular experimental setup, was striking:

• 49 runs passed the preregistered pipeline checks
• 1 run was rejected before scoring
• all 49 valid runs covered all six boundary classes
• every valid run produced exactly 16 accepted records
• 784 of 784 accepted records agreed with the reference policy semantics
• all 49 valid completions were different

For this one prompt, one model, and one synthetic policy, the answer to “how often?” was:

Every valid time we observed.

That sounds like a perfect result.

It isn’t.

And understanding why is the interesting part of the experiment.

The problem: one good generation is not a reliability result

I am working on an open-source project called the Judgment Pack Specification.

One area I am exploring is whether AI can help humans author the structured test cases around organizational decision logic.

Imagine a vendor-screening policy containing rules such as:

• sanctions matches create a hard stop
• vendors registered in embargoed countries require a different outcome
• a risk score of 70 crosses a high-risk boundary
• handling personal data changes the threshold at 40

Those rules produce boundaries.

And boundaries are where subtle errors tend to hide.

For example:

risk = 69.99
risk = 70
risk = 70.01

Enter fullscreen mode Exit fullscreen mode

can represent three very different cases if the policy says:

risk >= 70

Enter fullscreen mode Exit fullscreen mode

The same issue appears around a personal-data threshold:

39.99
40
40.01

Enter fullscreen mode Exit fullscreen mode

If AI is going to help author tests for these rules, I do not only care whether it notices the boundary once.

I want to know how consistently it notices it.

Study 011 asked “how often?”

The experiment used one fixed cell:

• one model – gpt-5.6-sol
• one fixed prompt
• one fixed synthetic policy
• one pinned CLI and binary
• fresh isolated environment for each call
• 50 sequential authoring calls
• six preregistered coverage classes

The prompt, policy family, binary, environment capture, and preregistration were all digest-pinned.

The scoring rules were fixed before looking at the batch.

There were no competing arms and no hypothesis test.

This study was simply estimating a frequency.

What counts as coverage?

Six boundary classes were registered before the batch.

Class 0 – exact high-risk boundary

No sanctions hit
Non-embargoed country
risk = 70

Enter fullscreen mode Exit fullscreen mode

This catches a common implementation error:

risk > 70

Enter fullscreen mode Exit fullscreen mode

instead of:

risk >= 70

Enter fullscreen mode Exit fullscreen mode

The model independently authored records with names such as:

exact-high-risk-threshold
northstar-risk-seventy
alpine-exact-seventy
summit-exactly-seventy

Enter fullscreen mode Exit fullscreen mode

Every valid run included a correctly labelled record at this boundary.

Coverage: 49/49

95% exact interval:

[0.9275, 1.0000]

Enter fullscreen mode Exit fullscreen mode

Class 1 – the off-by-one region above 70

The next class tested:

70 <= risk < 71

Enter fullscreen mode Exit fullscreen mode

Why?

Imagine someone accidentally changes the threshold from:

>= 70

Enter fullscreen mode Exit fullscreen mode

to:

>= 71

Enter fullscreen mode Exit fullscreen mode

Testing only 70 and 80 might miss useful information about that region.

The authored records included cases represented by names such as:

fraction-above-high-risk-threshold
atlas-risk-seventy-decimal
harbor-above-seventy
baltic-risk-above-seventy

Enter fullscreen mode Exit fullscreen mode

Again:

Coverage: 49/49

Class 2 – personal-data boundary at 40

The policy also changes behavior when a vendor handles personal data.

This class tested:

handles personal data
40 <= risk < 41

Enter fullscreen mode Exit fullscreen mode

It is designed to expose a lower-bound change such as:

risk >= 41

Enter fullscreen mode Exit fullscreen mode

instead of:

risk >= 40

Enter fullscreen mode Exit fullscreen mode

Examples of independently authored records included:

personal-data-exact-threshold
harbor-data-risk-forty
bluebell-personal-data-at-forty
maple-data-forty

Enter fullscreen mode Exit fullscreen mode

Coverage: 49/49

Class 3 – the interior decision region

This class tested:

40 <= risk < 70

Enter fullscreen mode Exit fullscreen mode

with either value of the personal-data flag.

This was intentionally broader.

The model repeatedly generated multiple records across this space, including examples labelled:

personal-data-mid-band
no-personal-data-mid-band
fjord-data-midrange
andes-no-data-moderate-risk

Enter fullscreen mode Exit fullscreen mode

Coverage: 49/49

Class 4 – membership rather than arithmetic

Not every policy defect is a numeric threshold.

Another class tested vendors registered in Syria:

registered country = SY

Enter fullscreen mode Exit fullscreen mode

This was designed to catch a mutation where SY disappears from an embargo list.

The model repeatedly produced examples such as:

syria-registration-high-risk
damascus-embargo-high-risk
levant-syria-high-risk
levant-cloud-embargo

Enter fullscreen mode Exit fullscreen mode

So the authoring behavior was not limited to discovering numeric boundaries.

Coverage: 49/49

Class 5 – the interesting one

The final class tested:

handles personal data
39 <= risk < 40

Enter fullscreen mode Exit fullscreen mode

This class deserves special attention.

The synthetic policy text explicitly mentions the threshold at 40.

It does not explicitly say:

Please test the hidden family boundary at 39.

Yet all 49 valid runs produced a record inside this interval.

The actual values were extremely revealing:

• 28 runs used 39.99
• 21 runs used 39.999

So the model was consistently generating a “just below 40” case.

Examples included:

personal-data-just-below-forty
fjord-data-below-forty
maple-data-below-threshold
cedar-data-below-forty

Enter fullscreen mode Exit fullscreen mode

This technically covers the registered class.

But it does not prove that the model discovered some hidden semantic threshold at 39.

A much simpler explanation fits the evidence:

The prompt asked for borderline cases around the stated threshold of 40, and the model generated values immediately below 40.

That distinction matters.

Coverage is real.

The interpretation must remain narrow.

The headline result

Across the six classes:

Class Valid runs covering it Rate Exact risk = 70 49 / 49 100% 70 <= risk < 71 49 / 49 100% Personal data + 40 <= risk < 41 49 / 49 100% 40 <= risk < 70 49 / 49 100% Registered in SY 49 / 49 100% Personal data + 39 <= risk < 40 49 / 49 100%

Each rate has the same exact 95% Clopper-Pearson interval:

[0.9275, 1.0000]

Enter fullscreen mode Exit fullscreen mode

That interval is important.

Observing 49 successes out of 49 valid runs does not establish a true rate of 100%.

With this sample size, the experiment cannot distinguish:

“always”

from something like:

“at least about 92.75% under this experimental cell”

That is why I prefer reporting:

49 / 49
95% CI [0.9275, 1.0000]

Enter fullscreen mode Exit fullscreen mode

rather than simply saying:

100% reliable

Enter fullscreen mode Exit fullscreen mode

Every output was different

Another result surprised me.

All 49 valid completions were distinct.

The largest group of byte-identical outputs was:

1

Enter fullscreen mode Exit fullscreen mode

So this was not one cached completion accidentally counted 49 times.

The model varied names, examples, and record composition across runs.

For example, the exact-70 case appeared under names including:

exact-high-risk-threshold
northstar-risk-seventy
atlas-risk-threshold
alpine-security-threshold
granite-risk-seventy
summit-risk-threshold

Enter fullscreen mode Exit fullscreen mode

The surface form changed.

The boundary coverage did not.

That is an interesting form of consistency:

Different generations converged on the same important semantic regions.

But even this has a limitation.

Different outputs do not prove statistically independent draws.

Provider-side cross-session behavior is not observable from the retained artifacts.

The study records that limitation rather than pretending it can prove independence.

784 accepted records, zero observed label disagreements

Each valid run produced exactly:

16 records

Enter fullscreen mode Exit fullscreen mode

Across 49 runs:

49 x 16 = 784 accepted records

Enter fullscreen mode Exit fullscreen mode

Against the study’s policy mirror:

policy-concordant records: 784
mislabelled records:        0
dropped records:            0

Enter fullscreen mode Exit fullscreen mode

So pooled label agreement was:

784 / 784 = 1.000

Enter fullscreen mode Exit fullscreen mode

I deliberately do not attach a binomial confidence interval to that number.

The 16 records inside a single completion are not independent trials.

Treating all 784 records as independent observations would make the precision look much stronger than it really is.

The run remains the meaningful repeated unit.

One run failed – and I think that is useful

Run 026 was not scored.

The model process itself exited successfully.

But the pre-prompt developer context did not match the locked golden context.

The pipeline therefore returned:

transcript-refused

Enter fullscreen mode Exit fullscreen mode

and excluded the run before looking at its authored records.

The observed pipeline-invalid rate was:

1 / 50 = 2%

Enter fullscreen mode Exit fullscreen mode

with a 95% interval of:

[0.05%, 10.65%]

Enter fullscreen mode Exit fullscreen mode

The likely explanation is service-side boilerplate variation.

But the study deliberately does not inspect the refused transcript and say:

This difference looks harmless, let’s count it anyway.

That would undermine the point of having an allowlist.

The rule was:

context matches the registered environment
        -> admit

context differs
        -> refuse

Enter fullscreen mode Exit fullscreen mode

So it refused.

I actually find this result useful.

The model-authoring behavior looked extremely stable.

The experimental pipeline was not perfect.

Those are two different reliability questions.

What about human review?

Before running the batch, the study registered a deliberately simple mapping from observed coverage to review depth.

It uses the lower bound of the confidence interval:

lower >= 0.80  -> LIGHT review
lower >= 0.40  -> STANDARD review
lower < 0.40   -> FULL review

Enter fullscreen mode Exit fullscreen mode

A high mislabelling rate can escalate the tier.

All six classes ended with:

lower bound = 0.9275
mislabel share = 0
tier = LIGHT

Enter fullscreen mode Exit fullscreen mode

So under the preregistered mapping, every class receives:

LIGHT review.

That does not mean AI-authored policy artifacts no longer require humans.

The review-tier mapping itself is experimental and has not been validated operationally.

But it suggests an interesting direction.

Instead of treating every AI-authored rule or test case as equally trustworthy, perhaps review depth can eventually depend on empirical evidence about the authoring process.

Something like:

AI authors candidate records
        |
        v
measure historical coverage / error rates
        |
        v
assign confidence
        |
        +------ high confidence ------> light review
        |
        +------ uncertain ------------> standard review
        |
        +------ weak evidence --------> full review

Enter fullscreen mode Exit fullscreen mode

That is much closer to what I ultimately want from AI-assisted authoring.

Not:

“The model wrote it, trust it.”

And not:

“The model wrote it, manually redo everything.”

Instead:

“We have measured evidence about where this authoring process is strong and where humans should spend their review time.”

What this study does NOT show

This is the most important section.

The result does not mean:

AI can reliably author arbitrary organizational judgment.

It means:

This prompt, with this model, on this small synthetic policy, repeatedly produced correctly labelled records reaching these six registered classes.

Nothing more transfers automatically.

It is one model

Only one model configuration was tested.

A different model may behave differently.

It is one prompt

The prompt explicitly asks for borderline cases.

That probably contributes substantially to the ceiling result.

It is one small synthetic policy

The policy has a few thresholds, one membership rule, sanctions logic, and a personal-data condition.

Real organizational policies can contain:

• interacting exceptions
• multiple documents
• ambiguous language
• precedence
• temporal conditions
• missing evidence
• conflicting authorities
• human approval requirements

Those may produce very different rates.

The mirror is not ground truth

“Correctly labelled” means the authored outcome agrees with the study’s deterministic mirror.

The same policy supplied to the model is implemented in that mirror.

So:

784 / 784

Enter fullscreen mode Exit fullscreen mode

measures consistency with the supplied policy semantics.

It does not independently prove those semantics are correct in the real world.

Coverage is not defect detection

Study 011 never evaluates a mutated Judgment Pack.

It asks whether an independently authored record lands in a region that could expose a registered defect.

Study 010 performed the single defect-detection draw.

Study 011 measures the repeatability of the authorship coverage behind it.

Those are deliberately separate claims.

The result changed how I think about AI authoring

Before this experiment, I was mostly thinking about AI-assisted authoring as:

policy
  |
  v
AI
  |
  v
candidate structured artifact
  |
  v
human review

Enter fullscreen mode Exit fullscreen mode

I now think a better model might be:

                 historical evaluations
                         |
                         v
policy -> AI author -> candidate artifact
                         |
                         v
                 automated validation
                         |
                         v
                 confidence estimate
                         |
              +----------+----------+
              |          |          |
            LIGHT     STANDARD     FULL
            review      review      review
              |
              v
        authorized approval
              |
              v
          versioned artifact

Enter fullscreen mode Exit fullscreen mode

The interesting question becomes less:

Can AI author the artifact?

Frontier models increasingly can.

The harder question is:

How much review does this particular generated artifact deserve?

That is something we may be able to measure.

What I want to test next

The obvious next experiment is not another 50 runs of this same cell.

This one is already sitting at the ceiling.

The useful next variable is the thing this study deliberately held constant.

For example:

Change the policy

Use a larger policy with interacting rules and exceptions.

Does coverage remain near the ceiling?

Change the model

Run the same registered task across multiple model families.

Do the boundary classes remain stable?

Remove the boundary-forward wording

If the prompt stops explicitly asking for borderline cases, what happens to classes like:

39 <= risk < 40

Enter fullscreen mode Exit fullscreen mode

That would help separate general semantic discovery from prompt-induced test generation.

Introduce ambiguity

What happens when the policy itself contains:

• conflicting language
• incomplete evidence requirements
• inconsistent descriptions and executable conditions
• unclear precedence

That is much closer to the real authoring problem.

The takeaway

Study 010 told me:

The independent author can cover all six classes.

Study 011 tells me something stronger:

Under this exact experimental setup, that coverage was the typical observed behavior, not a lucky single run.

But the more important lesson is methodological.

One impressive AI output is an anecdote.

Repeated outputs give you a rate.

A rate gives you uncertainty.

And uncertainty can start informing how much human review an AI-generated artifact deserves.

For this experiment:

50 calls
49 valid
49 / 49 valid runs covered all six classes
784 accepted records
0 observed label disagreements
49 distinct completions
1 pipeline refusal

Enter fullscreen mode Exit fullscreen mode

Promising?

Yes.

Proof that AI can safely author organizational judgment without review?

Absolutely not.

The next job is to find where these numbers stop being this clean.

The experiment, preregistration, retained artifacts, scorer, and analysis are all part of the open-source Judgment Pack evaluator experiments repository.

I would especially welcome adversarial policy examples that you think would break this authoring approach.