Canary Agentic Autofix With Failure Classes and Reliability Gates

GitHub announced agentic autofix for code scanning alerts in public preview on July 10, 2026.

Primary source: GitHub Changelog, July 10, 2026.

The wrong metric is “percentage of alerts with a generated patch.” Generation is only the first transition:

alert -> candidate -> build -> tests -> security oracle
      -> human review -> merge -> post-merge observation

Enter fullscreen mode Exit fullscreen mode

This is an evaluation proposal, not a benchmark or assessment of GitHub’s preview.

Choose a bounded canary

Start with repositories that have active owners, deterministic builds, relevant isolated tests, reversible releases, and no automatic production deployment from candidate patches. Exclude abandoned code, safety-critical paths, and repositories with unreliable tests.

Assign the canary deterministically—for example, hash a stable alert ID into a fixed percentage. Do not move difficult results out of the cohort after seeing them.

Record every attempt, including abstentions and failures:

attempt_id: "<id>"
alert_class: "<normalized class>"
base_revision: "<commit>"
outcome:
  generated: true
  applied_cleanly: true
  build_passed: true
  tests_passed: false
  security_oracle_passed: false
  human_decision: "rejected"
failure_class: "semantic_incomplete"
escaped_to_default_branch: false

Enter fullscreen mode Exit fullscreen mode

The schema is local evaluation metadata; it does not imply that GitHub exposes these fields.

Classify the earliest broken invariant

Class Meaning No candidate Tool abstained Scope violation Unrelated or forbidden paths changed Apply failure Patch does not apply to recorded base Build failure Patched revision cannot build Regression Existing behavior broke Semantic incomplete Alert changed but security property remains broken Overcorrection Valid behavior was blocked Test manipulation Validation was weakened or removed Stale base Result targeted another revision Review ambiguity Human cannot establish why the patch is safe Infrastructure Evaluation could not complete Post-merge escape Later evidence disproved acceptance

Use one primary class and optional secondary classes. Otherwise one attempt contaminates several denominators.

Add security oracles

Existing unit tests may not encode the alert’s security property. For authorization, require unauthorized denial, authorized success, alternate-entry coverage, and preserved audit behavior. Run the oracle on the unpatched baseline first; if it cannot expose the problem, a patched pass proves little.

Predeclare gates such as zero critical scope violations, zero test-manipulation events, zero automatic merges, bounded infrastructure failure, and a minimum completed sample. Segment results by alert class, language, repository tier, and patch size. Report confidence intervals rather than promoting after a lucky week.

Stop immediately for privilege expansion, weakened policy, revision mismatch, secret exposure, or unexplained production impact. Rollback includes disabling new attempts, freezing open candidates, reviewing already merged canary patches, and preserving evidence.

Offline fixtures are imperfect, alerts are correlated, and escapes may surface late. That is why wider deployment should be earned through bounded evidence, explicit denominators, and safety events that outweigh an attractive average.