Quick take
Before you label a failing test flaky, write down the dates it failed. Flaky has no rhythm. Yours might.
“Flaky” is the most expensive word in CI. It closes the investigation. Nobody looks at a flaky test again until it fails on the day of a release.
Here is a cheap way to earn the label instead of assuming it. Pull the dates of the last failures and look at the gaps:
gh run list --workflow=ci.yml --status=failure \
--limit 50 --json createdAt \
--jq '.[].createdAt' | sort
Enter fullscreen mode Exit fullscreen mode
Random failures scatter. If the gaps land on the same number over and over, something on a timer is doing this to you, and it is not the test.
Seven days is the number worth knowing by heart: GitHub Actions evicts a cache entry that has not been touched for a week. A dependency that is only ever restored, never rebuilt, disappears on a schedule — and the first run after that is slow, cold, and sometimes fails on a timeout that was always marginal.
That failure is not random. It has a calendar. Yours might be a certificate, a token, a log rotation or a nightly database refresh. Same shape, different clock.
The long version — how we found ours, and why the fix was not a longer timeout — is here: Your CI is not flaky. Your cache expires every seven days.
I build cachly — persistent memory for AI coding assistants, over MCP. Your assistant re-reads your codebase every morning. It does not have to.
Free tier, hosted in the EU: cachly.dev