I was building a small Spring Boot demo for Exevra, a JUnit test-execution check. I wanted a failure that was boring enough to be believable.
The project has two test classes:
GreetingControllerTest
GreetingFormatterTest
Enter fullscreen mode Exit fullscreen mode
On the main branch, mvn test discovers both classes and Surefire writes a JUnit XML file for each one. The GitHub Actions workflow runs the same Maven command.
Then I added this to pom.xml on a separate branch:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<includes>
<include>**/GreetingControllerTest.java</include>
</includes>
</configuration>
</plugin>
Enter fullscreen mode Exit fullscreen mode
GreetingFormatterTest no longer runs.
Maven still passes. It is doing exactly what the include filter asks it to do. The controller tests pass, so the command exits with code 0. The CI job stays green too.
That is not a Maven bug. It is a different question from the one Maven answers.
Maven answers: did the tests I ran pass?
For some repositories, CI also needs to answer: did this run the tests we expected it to run?
A green report can still be incomplete
The broken branch still produces a valid JUnit report for GreetingControllerTest. A normal test-reporting step has something to read, and it can accurately say that the tests in that report passed.
What it cannot know is whether a second report was expected. An absent report might be deliberate in another repository. Test suites can be split across modules, profiles, operating systems, or jobs. A generic JUnit reporter should not treat every missing file as an error.
That is why this kind of failure can look so ordinary in a pull request. There is no red test. There is no malformed XML. The workflow contains a report for passing tests. A change in the test command or Surefire configuration changed the shape of the run without making the command fail.
What I considered instead
Code review is the first defense. The Surefire include is visible in the diff. A careful reviewer can spot it and ask why the formatter tests are gone.
Test counts are useful too. In this demo, the count drops when one class disappears. A count check would catch it. Counts have a limit, though: a run can replace tests and keep the same total.
Coverage can also expose a suspicious change, but it measures executed code, not the intended test set. A coverage threshold may stay unchanged after a removed test, especially in a small or overlapping suite.
None of those are bad options. They answer different parts of the problem. I wanted to see what happened if the expected execution itself became a checked artifact.
Making the expected run explicit
Exevra records a known-good JUnit run in a committed baseline. Its configuration names the report files that the demo expects:
version: 1
baseline: .exevra/baseline.json
command: mvn -B test
reports:
- target/surefire-reports/TEST-io.exevra.demo.GreetingControllerTest.xml
- target/surefire-reports/TEST-io.exevra.demo.GreetingFormatterTest.xml
policy:
default:
min_executed: 1
max_drop_percent: 0
identity: enforce
identity_details: counts
Enter fullscreen mode Exit fullscreen mode
The Action runs the configured Maven command and compares the fresh reports with that baseline:
- uses: Exevra/[email protected]
with:
config: .exevra.yml
mode: enforce
Enter fullscreen mode Exit fullscreen mode
On the broken branch, the formatter report is absent. Exevra fails the workflow even though Maven succeeds.
The baseline is intentional. If a team adds, renames, or removes tests on purpose, it needs to update the expected run in the same reviewed change. That is extra maintenance. It is also the point: a change to the test-execution contract becomes visible instead of hiding inside a green build.
I would not add that contract to every repository. A project with rapidly changing generated tests or deliberately variable test selection may find it noisy. It makes more sense where the expected JUnit suites are stable and the cost of running less than intended is high.
The demo is small on purpose
The full example is public at Exevra/demo-spring-boot. The main branch runs both test classes. The demo/broken-test-discovery branch adds the Surefire include and keeps Maven green while dropping the formatter suite.
It is only a two-class project. That makes the failure easy to inspect, not less real. A test filter, changed report path, or runner configuration can have the same effect in a larger build, where a passing job is more likely to get a quick glance and a merge.
Exevra does not prove that the tests are good or that the application is correct. It does not replace test review, coverage, or the test runner. It checks a narrower claim: whether a CI run still matches the JUnit execution the repository chose to expect.