Automatic top-up — the feature that charges a saved card when a customer’s balance falls below their threshold — could never have succeeded for anybody. The invoice was constructed in the wrong currency, and every attempt would have failed in a way that told the customer their card was bad. This is the whole write-up, in the shape we would want any incident written in.
Summary
An invoice does not take its currency from the line items attached to it. It takes it from the customer’s default currency, or failing that from the Stripe account’s — which is EUR for a Dutch business. Every price in this product is denominated in USD. Finalising the invoice therefore failed with a currency-conflict error, on every automatic top-up, unconditionally.
The manual top-up path was never affected, because a Checkout Session takes its currency from the first line item rather than from the customer record. That difference is why the bug could exist in a product whose payment flow demonstrably worked.
Impact
Dimension Description Customers affected None. The defect was found before the path carried real traffic. This is stated plainly rather than omitted, because a postmortem that lets a near miss read as an outage is as dishonest as one that hides an outage. What would have happened Every automatic top-up fails. The failure surfaces as a payment error, which the failure counter records as a strike, and after three strikes the customer’s automatic top-up is switched off entirely. What the customer would have concluded That their card was declined. The message they receive says the saved card could not be charged. They would have gone and fixed a card that was working perfectly. Secondary effect A customer relying on automatic top-up to keep a production integration serving would have run out of credit silently, at whatever hour their traffic happened to cross the threshold.The second and third rows are what make this worth writing up. A defect that fails loudly and correctly is a bug; a defect that fails while blaming the user is a different category, because it consumes the customer’s time and trust on the way past.
Timeline
- The feature ships as part of a batch that also included profile pictures, two-factor authentication and email change. It is exercised by hand and by unit tests, both of which pass.
- A webhook-half harness is written. It drives a genuinely signed request through the real handler against a throwaway database copy, and asserts the things that lose money quietly: that a settled invoice credits exactly the amount bought, that a replay moves nothing, that two simultaneous deliveries of the same invoice move the balance once, that three declines switch the rule off. All pass.
- That harness names what it cannot prove, in the file itself: that Stripe accepts the invoice we build, that tax is applied to it, and that the mandate is accepted off-session. All three need a test-mode key and a card.
- A second check is written to close exactly that gap — the full path against real Stripe in test mode.
- Its first run fails immediately, at invoice finalisation, with a currency conflict. The defect had been present since the feature shipped.
The interesting entry is the third. The gap was known, written down in the place a reader would find it, and closed. That is the mechanism that found this — not vigilance, and not luck.
What actually happened
The direct cause is a mismatch between two currency defaults that never meet in any other code path:
Manual top-up (worked)
Checkout Session
-> line_items[0].price_data.currency = "usd"
-> the SESSION currency is set from that line
-> no conflict, ever
Automatic top-up (could never work)
Invoice Item (currency: usd)
+ Invoice (currency: taken from customer.currency,
else from the platform account default = eur)
-> finalize
-> "You cannot combine currencies on a single invoice ...
usd conflicts with the invoice currency eur"
Enter fullscreen mode Exit fullscreen mode
There is no configuration in which this succeeds. It is not a race, not an edge case, not dependent on the amount. Two ways of creating a charge that look equivalent from the application’s side have different rules about where the currency comes from, and only one of them was ever exercised against the real service.
Why it was invisible
Four independent reasons, and they are the transferable part of this write-up.
- The tested half was ours. The webhook harness covered the half of the path where our code decides things — crediting, deduplication, replay, the failure counter. That is genuinely the half that loses money quietly, and it was the right half to test first. It is also the half that cannot see a defect on the other side of the boundary.
- The working sibling path created false confidence. Manual top-up worked, in production, repeatedly. Anyone reasoning about “can this product charge a card” had a demonstrably true answer to a slightly different question.
- The default came from account configuration, not code. No source file anywhere contains the string that caused this. The currency was inherited from a business registered in the Netherlands, which is not a fact any code review would surface.
- The failure had a plausible innocent explanation. An invalid-request error from a payment provider, mapped to “the saved card could not be charged”, is exactly what a genuinely declining card looks like. Had this reached production, the monitoring would have shown a card-failure rate, not a bug.
The fourth is the one that generalises furthest. When an error path maps several distinct causes onto one customer-facing message, the least likely cause becomes indistinguishable from the most likely one, and the most likely one is what everybody investigates.
What was changed
- The currency is now set explicitly on the invoice rather than inherited, so the value is a property of the code rather than of an account setting somebody may change later.
- The new check drives the whole path against real Stripe in test mode. Stripe accepts the invoice, pays it off-session with a test payment-method token, and then the real invoice payload is signed and fed to the real webhook handler, which credits the ledger exactly once. It also covers a declining card, the failure counter and the cooldown.
- One assertion in the new check was wrong rather than the code. It asserted a tax field on the invoice item and on the finalised line; both report nothing, because the value appears on a different object. It was replaced with the invariant that holds in both modes — total equals subtotal plus tax — which is a stronger assertion than the one it replaced, since it is what actually guarantees the credit granted is not reduced by tax.
That third step belongs in the postmortem rather than being quietly fixed. A new test failing is not automatically a found bug, and recording which of the two was wrong is what stops the next person rewriting a test to match broken behaviour.
What is still not proved
The test account has no tax registration in the relevant jurisdiction, so the tax calculation returns zero and reports itself as not collecting. On the live account it must report as standard-rated. Until somebody runs the equivalent against live, that remains unproved and is recorded as unproved.
A postmortem that ends with everything closed is usually a postmortem that stopped looking. The open item above is the most likely source of the next incident on this path, and writing it down is the only thing that makes it findable.
The template
The structure above is reusable, and it is deliberately shorter than most incident templates. Eight headings, and the two that are usually missing are the last two.
Copy this
# <One sentence naming the defect, in the past tense>
## Summary
What broke, mechanically. Two paragraphs. No blame, no adjectives.
## Impact
- Customers affected: a number, or "none" — say which
- What would have / did happen: the failure as the system experiences it
- What the customer concluded: the failure as the CUSTOMER experiences it
- Secondary effects: what else moved
## Timeline
Numbered, factual, from the change that introduced it to the moment it
was found. Include the events that DID NOT find it — they are the ones
that tell you where the detection gap is.
## What actually happened
The mechanism, with the code or the payload. If someone cannot
reproduce it from this section, it is not finished.
## Why it was invisible
The most valuable section, and the one most often skipped. One bullet
per independent reason. "Nobody checked" is not a reason; it is the
absence of one.
## What was changed
Numbered. Include any change that turned out to be wrong, and say so.
## What is still not proved
The open items, named. A postmortem with none is usually incomplete.
Enter fullscreen mode Exit fullscreen mode
Three rules for using it
- Blameless means causal, not vague. “The currency was inherited from account configuration” is blameless and precise. “Mistakes were made in the billing layer” is neither.
- Separate the system’s view from the customer’s. Most incidents cost more in trust than in downtime, and the trust cost is only visible in the second view.
- Write it for a stranger. The audience is somebody who hits the same wall in eighteen months and searches for the error string. Include the literal error text.
The wider practice around this — severity levels, who is paged, how the loop is closed — is in incident response for AI systems.
답글 남기기