I’ve seen enough postmortems to know that a large percentage of them are essentially polite fiction.
We sit in a meeting, everyone is exhausted from the outage, and we agree on a narrative. We write down that ‘the database connection pool was exhausted’ or ‘a bad deploy caused an error spike.’ Then we add an action item like ‘add more monitoring’ or ‘improve testing,’ and we move on to the next feature request.
Three months later, the exact same thing happens. The same service, the same error, the same fatigue. We didn’t solve anything; we just documented our failure with slightly better prose.
The problem is that most postmortems fail at the fundamental level of investigation. They stop at the symptoms. They treat human error as a root cause—which is lazy engineer shorthand for ‘we don’t want to fix the system.’ And they treat vague timelines as acceptable data, which makes reconstruction impossible when you’re trying to correlate logs from different subsystems.
This is why I became interested in using MCP (Model Context Protocol) not just to give agents access to my tools, but to act as an auditor for these processes. Most people use LLMs to summarize what happened. That’s useless. You don’t need a summary; you need someone to tell you where your investigation is weak.
I’ve been working with the Incident Postmortem Prover (https://vinkius.com/mcp/incident-postmortem-prover) because it doesn’t try to be a scribe. It acts as an adversarial auditor for SREs and engineers. It uses semantic trap lists designed to catch exactly the kind of hand-wavy logic that ruins investigations.
The Death of the ‘Vague Timeline’
One of the most common failures is what I call TIMELINE_INCOMPLETE. You’ll see things like: ‘Around 3 PM, we noticed a spike in errors. By 4 PM, everything was back to normal.’
That isn’t a timeline; it’s an anecdote. An actual investigation needs minute-by-minute reconstruction in UTC. What happened at 15:02? Who acknowledged the PagerDuty alert? When did the first rollout attempt begin? If there is a five-minute gap in your history that you cannot account for, that gap is where your process failure lives. The Prover rejects these narrative summaries because they hide the blast radius and the latency of our response.
Moving Past Symptom-Level Analysis
Then there’s ROOT_CAUSE_SHALLOW. This is the classic ‘the server crashed due to OOM’ trap.
An Out Of Memory error is a symptom, not a root cause. The Prover forces you through 5 Whys until you reach something systemic or organizational.
‘Why did it crash?’ -> OOM.
‘Why was there an OOM?’ -> A memory leak in the new payment service path.
‘Why was there a leak?’ -> No integration tests were run on that specific path during deployment.
‘Why no integration tests?’ -> The deployment pipeline is configured to skip them for certain ‘low-risk’ tags to save time.
Now we have an actual engineering problem: the configuration of our CI/CD pipeline. That is a fixable, systemic root cause. ‘The engineer forgot to update the config’ is just blame, and blaming people doesn’t make systems more resilient.
Separating Cause from Contributor
A nuance that even senior engineers miss—and what this tool explicitly checks for—is CAUSES_CONFLATED.
We often point at a Friday afternoon deployment as the ’cause.’ It wasn’t. The cause was likely a specific code change or configuration error. The Friday deploy was a contributing factor that amplified the impact because staffing was low and visibility was reduced. If you conflate these, you miss the opportunity to fix the actual vulnerability in your code/config, and instead, you just end up creating ‘No-Deploy Fridays,’ which is a cultural band-aid that doesn’t actually improve software quality.
The ODAS Framework for Action Items
Finally, we have ACTIONS_UNACCOUNTABLE. This is where most engineering managers lose the plot. If I see an action item in a postmortem that says ‘Improve monitoring,’ I immediately know it will never happen.
The Prover enforces what I call ODAS: [Owner] [Deadline] [Specific Action] [Success Metric].
Instead of ‘improve monitoring,’ you need something like: ‘@alice [By: 2024-11-30] [Deploy connection pool metric alert at 80% threshold] [Success: Alert fires in staging within 30 seconds of threshold breach].’
It’s harder to write, but it’s the only way to ensure that the postmortem actually changes the state of your infrastructure. An action item without an owner and a metric is just a wish.
You can find this MCP server on Vinkius here: https://vinkius.com/mcp/incident-postmortem-prover. It’s built to be used within Claude or Cursor, providing that layer of SRE-grade rigor when you’re documenting the hard lessons learned from your outages.
Stop writing fiction. Start investigating.
MCPs are the music of AI Agents. We built the catalog. Discover Vinkius MCP Catalog.
답글 남기기