Designing a cross-border payment system

Cross-post. Original: stellarbytecapital.com/blog/cross-border-payment-system-design

A payment system has one job that dwarfs all the others: never lose track of money. Features, UI, and even uptime are negotiable in a pinch — a lost or duplicated transaction is not. Cross-border adds currencies, multiple payment channels, settlement delays, and regulators on top. Get the money-safety core right and everything else is ordinary engineering; get it wrong and no amount of polish saves you.

The ledger is the system

The single most important decision is to make an append-only, double-entry ledger the source of truth — not a mutable balance column you increment. Every movement of money is recorded as balanced entries (a debit and a matching credit) that sum to zero. A user’s balance is derived from the ledger, never stored as the primary fact.

• Immutable entries. You never edit or delete a posting. A mistake is corrected with a new reversing entry, so history is a complete, auditable trail.
• Balances always reconcile. Every entry is balanced, so the whole system sums to zero at all times. If it doesn’t, you have a bug — detectable immediately, not months later in an audit.
• Every entry has a reason. Each posting references the event that caused it, so you can always answer “why is this number what it is?”

A mutable balance is a number you hope is right. A ledger is a number you can prove is right.

Idempotency: the network will retry, so must you survive it

Money movement crosses networks that time out, drop, and duplicate. The classic failure: your service calls a payment channel, the channel processes it, but the response is lost — so a retry charges the user twice. The defense is idempotency, end to end.

• Client-supplied idempotency keys. Every money-moving write carries a unique key. The server records the key with the result; a repeat returns the original outcome instead of doing the work again.
• Exactly-once at the boundary. Calls to external channels are wrapped so a retry never means a second real charge — the same discipline that keeps a trading bot from double-submitting orders.
• Transactional writes. The ledger entry and the state change commit together, in one database transaction. Partial writes are the enemy.

Payment state as an explicit machine

A payment is never simply “done.” Model it as an explicit state machine — initiated → pending → settled, with failed and reversed branches — and persist every transition.

• Everything is async. Channels confirm out of band, sometimes hours later. Hold a payment in pending and resolve it on a callback or poll; never assume synchronous success.
• The external channel is the source of truth for its leg. Your local “I think it succeeded” means nothing until the channel confirms. Reconcile against the channel; trust the channel.

Reconciliation: assume drift, detect it daily

No matter how careful the writes, your records and the channels’ records will drift — missed callbacks, timing gaps. Reconciliation is a core scheduled job, not optional cleanup:

• Pull each channel’s settlement report and match it line-by-line against your ledger.
• Flag every mismatch into an exceptions queue a human can work.
• Track a reconciliation watermark so you always know the last point the books were provably correct.

Multi-channel and multi-currency without chaos

• A channel abstraction. Each provider sits behind a common interface (initiate, query, handle-callback, reconcile). Adding a channel is implementing that interface, not rewiring the core.
• Currency as first-class data. Every amount carries its currency, stored in minor units as integers — never floats. FX conversions are themselves ledger events, so the books stay balanced across currencies.

What to avoid

• A mutable balance column as the truth — the original sin; you can’t prove correctness or cleanly reconcile.
• Floats for money — rounding errors compound into unexplainable discrepancies. Integers in minor units, always.
• Assuming synchronous success — how double-charges and phantom balances happen.
• Skipping reconciliation until there’s a problem — by then the drift is large, old, and expensive.

We’re Xingyao Byte — building payment platforms, quant trading systems, and secure AI-execution layers. Remote, async-first → stellarbytecapital.com