Imagine an auction website similar to eBay.
A seller lists an item:
Vintage Camera
Starting bid: $500
Enter fullscreen mode Exit fullscreen mode
Other users can open the auction page and place bids:
User A → $550
User B → $600
User C → $700
Enter fullscreen mode Exit fullscreen mode
Everyone currently watching the auction should see the highest bid change in real time.
Eventually, the auction ends. The highest valid bidder becomes the winner, receives a notification, and gets a limited amount of time to complete payment.
That sounds simple.
But once we have millions of users, thousands of active auctions, real-time bid updates, concurrent bids, auction expiration, and payment failures, the design becomes an interesting distributed-systems problem.
This article builds the system from the ground up and gradually addresses those problems.
1. Understanding the Auction
Let’s first define what an auction actually means in our system.
A seller creates an auction for an item.
Other users can:
View auction
↓
See current highest bid
↓
Place a higher bid
Enter fullscreen mode Exit fullscreen mode
The important rule is that an auction does not necessarily end at a fixed clock time.
Instead, in this design:
The auction closes when there has been no higher bid for one hour.
For example:
10:00 → User A bids $500
10:20 → User B bids $550
10:45 → User C bids $600
Enter fullscreen mode Exit fullscreen mode
The one-hour timer is effectively extended by the latest bid.
If no higher bid arrives after:
10:45 + 1 hour
Enter fullscreen mode Exit fullscreen mode
the auction can be closed and User C becomes the winner.
After that:
Winner
↓
Payment notification
↓
10-minute payment window
↓
Payment succeeds → Auction succeeds
Payment fails/expires → Auction fails
Enter fullscreen mode Exit fullscreen mode
2. Requirements
Before designing the system, let’s make the rules explicit.
A user should be able to:
Create an auction
View an active auction
Place a bid
See the current highest bid in real time
Enter fullscreen mode Exit fullscreen mode
The system should:
Close an auction after one hour without a higher bid
Determine the winner
Notify the winner
Give the winner 10 minutes to pay
Enter fullscreen mode Exit fullscreen mode
There are also a few important business rules.
If two users submit the same bid amount, the first bid wins.
A bidder can only have one active bid in a particular auction.
However, the bidder can increase that bid later.
For example:
User A → $500
User A → $600
User A → $700
Enter fullscreen mode Exit fullscreen mode
The latest higher bid becomes the user’s current bid.
At the same time, we still keep the complete bid history.
So the system remembers:
$500
$600
$700
Enter fullscreen mode Exit fullscreen mode
rather than keeping only $700.
For simplicity, we won’t require a separate user-provided TTL for auctions that never receive a bid.
Search, payment processing, and inventory are treated as separate systems. Our focus is the auction service itself.
3. What Does the System Need to Guarantee?
The system has two very different kinds of data.
The first is the live bidding experience.
A customer watching an auction wants to see:
Current highest bid: $700
Enter fullscreen mode Exit fullscreen mode
If the UI briefly shows $650 while another bid of $700 is being processed, that isn’t necessarily catastrophic.
The live-bidding path can therefore tolerate some eventual consistency.
The second is winner selection.
When the auction closes, we must be absolutely certain who won.
We cannot have:
Client A → Winner: User A
Client B → Winner: User B
Enter fullscreen mode Exit fullscreen mode
for the same auction.
Therefore:
Live bid display
↓
Eventual consistency acceptable
Winner determination
↓
Strong consistency required
Enter fullscreen mode Exit fullscreen mode
This distinction is one of the most important ideas in the design.
4. Estimating the Scale
Let’s use the assumptions from the original design.
Suppose the platform has:
1 billion daily active users
100,000 auctions created per day
10% of users place one bid per day
Enter fullscreen mode Exit fullscreen mode
That gives roughly:
100M bids/day
Enter fullscreen mode Exit fullscreen mode
The average bid rate is approximately:
100M / 86,400
≈ 1,157 bids/sec
Enter fullscreen mode Exit fullscreen mode
But average traffic isn’t enough.
Traffic will be bursty.
Some auctions will attract almost no attention:
Auction A → 3 bidders
Enter fullscreen mode Exit fullscreen mode
while a popular auction might attract enormous traffic:
Auction B → millions of viewers
Auction B → thousands of concurrent bids
Enter fullscreen mode Exit fullscreen mode
The design therefore needs to handle both:
Large overall traffic
+
Hot individual auctions
Enter fullscreen mode Exit fullscreen mode
The original design also assumes roughly a:
10:1 read-to-write ratio
Enter fullscreen mode Exit fullscreen mode
That means viewing auction state is much more common than placing bids.
This becomes important when we design the real-time update path.
5. The Basic Architecture
At a high level, the system looks like this:
USERS
|
▼
Load Balancer
|
┌───────────────┼────────────────┐
↓ ↓ ↓
Auction Service Bid Update Fulfillment
| Service Service
| | |
↓ ↓ ↓
Auction DB Dispatcher Auction DB
|
↓
Cache
|
↓
Kafka
|
┌─────┴──────┐
↓ ↓
Notifications Reconciliation
Enter fullscreen mode Exit fullscreen mode
There are several important components:
Auction Service
↓
Creates auctions and accepts bids
Auction DB
↓
Durable source of truth
Cache
↓
Fast access to current auction state
Bid Update Service
↓
Maintains real-time connections to viewers
Dispatcher
↓
Routes bid updates to the correct Bid Update Service
Fulfillment Service
↓
Detects auctions that should end and processes winners
Notification Service
↓
Notifies winners
Reconciliation Service
↓
Detects and repairs abnormal states
Enter fullscreen mode Exit fullscreen mode
Let’s build these pieces one by one.
6. Creating an Auction
Creating an auction is relatively straightforward.
The client sends:
POST /api/v1/auctions
Enter fullscreen mode Exit fullscreen mode
with information such as:
{
"itemId": "item-123"
}
Enter fullscreen mode Exit fullscreen mode
The Auction Service creates a row in the auction database.
A simplified model is:
Auction
-------------------------
auction_id
owner_id
item_id
status
created_at
updated_at
expire_at
winner_id
winner_bid_id
winner_price
payment_expire_at
Enter fullscreen mode Exit fullscreen mode
The important fields are:
status
expire_at
Enter fullscreen mode Exit fullscreen mode
because they control the auction lifecycle.
A newly created auction starts as:
ACTIVE
Enter fullscreen mode Exit fullscreen mode
and its initial expiration time is established according to the auction’s bidding rules.
The service also places the auction state into the cache.
7. Why Make the Auction Service Stateless?
The Auction Service does not need to remember auction state inside its own process.
Instead:
Auction Service
↓
Cache / DB
Enter fullscreen mode Exit fullscreen mode
Any Auction Service instance can process a request.
For example:
User A
↓
Auction Service #1
User B
↓
Auction Service #7
User C
↓
Auction Service #12
Enter fullscreen mode Exit fullscreen mode
All of them can access the same external state.
This is what makes the stateless design easy to scale horizontally.
If one instance fails:
Request
↓
Another Auction Service instance
Enter fullscreen mode Exit fullscreen mode
The auction data is still available.
8. The Auction Database and Cache
We need durable storage and fast access.
The database contains the complete auction state and bid history.
The cache contains the information we need frequently.
A useful cache entry is:
auction:{auction_id}
{
status,
highest_bid,
highest_bidder_id,
updated_at,
expire_at
}
Enter fullscreen mode Exit fullscreen mode
For example:
auction:A123
status = ACTIVE
highest_bid = $700
highest_bidder_id = U456
updated_at = 10:45:12
expire_at = 11:45:12
Enter fullscreen mode Exit fullscreen mode
The cache is useful because thousands or millions of users may repeatedly ask:
"What is the current highest bid?"
Enter fullscreen mode Exit fullscreen mode
We don’t want every one of those reads to hit the database.
But an important rule is:
The cache is not the ultimate source of truth.
The Auction DB contains the durable record.
9. The First Consistency Problem
Suppose we successfully write a bid to the database:
DB write → SUCCESS
Enter fullscreen mode Exit fullscreen mode
but then the cache update fails:
Cache update → FAILURE
Enter fullscreen mode Exit fullscreen mode
Now we have:
Database → $700
Cache → $650
Enter fullscreen mode Exit fullscreen mode
This is a cache inconsistency.
We can retry the cache update.
But what if the retry also fails?
We therefore need mechanisms to detect stale cache entries.
That is why the cache stores:
updated_at
Enter fullscreen mode Exit fullscreen mode
The system can use that timestamp to determine whether cached information is sufficiently fresh.
When necessary, it can read the database and repair the cache.
This is essentially a form of read repair.
10. How Do Users Receive Live Bid Updates?
Now we reach one of the most interesting parts.
Imagine 50,000 people are watching the same auction.
When somebody bids:
User A → $700
Enter fullscreen mode Exit fullscreen mode
we need to push the update to all those viewers.
We have several possible technologies.
The main choices are:
HTTP polling
Long polling
WebSocket
Server-Sent Events (SSE)
Enter fullscreen mode Exit fullscreen mode
Polling would mean:
Client
↓
"Any new bid?"
↓
Server
↓
Client
↓
"Any new bid?"
↓
Server
Enter fullscreen mode Exit fullscreen mode
This creates unnecessary traffic.
Long polling is better, but still requires repeated HTTP requests.
WebSocket provides a persistent bidirectional connection.
SSE provides a persistent one-way connection:
Server
↓
Client
Enter fullscreen mode Exit fullscreen mode
For this auction system, the client mainly needs to receive updates.
The bid itself can still be sent through a normal HTTP request.
Therefore SSE is a natural fit.
11. Why SSE Works Well Here
The two directions are different.
When a user places a bid:
Client → Server
Enter fullscreen mode Exit fullscreen mode
we can use:
POST /api/v1/auctions/{auctionId}/bids
Enter fullscreen mode Exit fullscreen mode
When the server tells users that somebody else has placed a higher bid:
Server → Client
Enter fullscreen mode Exit fullscreen mode
we can use SSE.
So:
Bid placement
↓
HTTP
Live updates
↓
SSE
Enter fullscreen mode Exit fullscreen mode
WebSocket would also work, especially if the product later requires richer bidirectional real-time communication.
But for simple one-way live updates, SSE is less complex.
12. Connecting a User to a Bid Update Service
When a user opens an active auction page, the client first gets the auction details:
GET /api/v1/auctions/{auctionId}
Enter fullscreen mode Exit fullscreen mode
If the auction is still active, the browser opens an SSE connection.
A load balancer may route the connection to any Bid Update Service instance.
For example:
User U1 → bus1
User U2 → bus1
User U3 → bus2
User U4 → bus3
Enter fullscreen mode Exit fullscreen mode
Each Bid Update Service keeps an in-memory subscription table.
For example:
bus1
auction A1 → [U1, U2]
auction A2 → [U5]
Enter fullscreen mode Exit fullscreen mode
This tells the service:
These users are currently watching these auctions.
13. Why Do We Need a Dispatcher?
Suppose a bid arrives for auction A1.
The Auction Service knows:
Auction = A1
New highest bid = $700
Enter fullscreen mode Exit fullscreen mode
But which Bid Update Service has the viewers?
It might be:
bus1
Enter fullscreen mode Exit fullscreen mode
The Auction Service should not need to know the internal connection state of every Bid Update Service.
So we introduce a Dispatcher.
The Dispatcher maintains another subscription table:
Dispatcher
A1 → bus1
A2 → bus2
A3 → bus1
Enter fullscreen mode Exit fullscreen mode
Now the flow becomes:
User places bid
↓
Auction Service
↓
Dispatcher
↓
Correct Bid Update Service
↓
Connected viewers
Enter fullscreen mode Exit fullscreen mode
This separates responsibilities:
Auction Service
→ process business logic
Dispatcher
→ route update
Bid Update Service
→ maintain client connections
Enter fullscreen mode Exit fullscreen mode
14. The Full Bid Update Flow
Suppose:
Auction A1
Current bid = $600
Enter fullscreen mode Exit fullscreen mode
User U10 places:
$700
Enter fullscreen mode Exit fullscreen mode
The request goes:
U10
↓
Auction Service
Enter fullscreen mode Exit fullscreen mode
The Auction Service:
Checks auction status
↓
Writes bid to DB
↓
Updates highest bid in cache
↓
Sends update to Dispatcher
Enter fullscreen mode Exit fullscreen mode
The Dispatcher checks:
A1 → bus1
Enter fullscreen mode Exit fullscreen mode
and forwards the update:
Dispatcher
↓
bus1
Enter fullscreen mode Exit fullscreen mode
The Bid Update Service checks:
A1 → [U1, U2, U3, U4]
Enter fullscreen mode Exit fullscreen mode
and sends:
New highest bid = $700
Enter fullscreen mode Exit fullscreen mode
to those SSE connections.
The complete flow is:
Bidder
↓
Auction Service
↓
Auction DB + Cache
↓
Dispatcher
↓
Bid Update Service
↓
SSE
↓
All viewers
Enter fullscreen mode Exit fullscreen mode
15. Why Not Just Poll the Database?
A naive design would be:
Bid Update Service
↓
Poll Auction DB
↓
Find new bids
↓
Push to users
Enter fullscreen mode Exit fullscreen mode
This sounds simple.
But imagine:
100,000 active auctions
Enter fullscreen mode Exit fullscreen mode
and each auction is being polled every few seconds.
The database would receive enormous numbers of unnecessary queries.
Most queries would return:
Nothing changed.
Enter fullscreen mode Exit fullscreen mode
Instead of constantly asking the database:
"Did something happen?"
Enter fullscreen mode Exit fullscreen mode
we push the event when something actually happens.
That is much more efficient.
16. Making the Dispatcher Highly Available
The Dispatcher is stateful because it maintains:
auction → Bid Update Service
Enter fullscreen mode Exit fullscreen mode
If the Dispatcher fails, bid updates cannot be routed to viewers.
The actual auction may still work, but the live experience breaks.
There are several ways to make it resilient.
One option is to maintain a write-ahead log and snapshots:
Subscription changes
↓
WAL
↓
Snapshot
Enter fullscreen mode Exit fullscreen mode
If the Dispatcher crashes:
Snapshot
+
WAL
↓
Rebuild subscription table
Enter fullscreen mode Exit fullscreen mode
Another option is to replicate the state into an external key-value store.
A third option is an active-standby design:
Primary Dispatcher
↓
Standby Dispatcher
Enter fullscreen mode Exit fullscreen mode
If the primary fails:
Standby → becomes primary
Enter fullscreen mode Exit fullscreen mode
17. Could We Remove the Dispatcher?
Yes.
Instead of:
Auction Service
↓
Dispatcher
↓
Bid Update Service
Enter fullscreen mode Exit fullscreen mode
we could maintain the subscription mapping in a distributed key-value or coordination service.
Then:
Bid Update Service
↓
Coordination Store
Enter fullscreen mode Exit fullscreen mode
and the Auction Service can look up which Bid Update Service is responsible for an auction.
There is a trade-off.
With a Dispatcher:
Pros:
- Auction Service has less responsibility
- Dispatcher can scale independently
- Retry logic is centralized
Enter fullscreen mode Exit fullscreen mode
But:
Cons:
- Additional component
- More operational complexity
Enter fullscreen mode Exit fullscreen mode
Without a Dispatcher:
Pros:
- Simpler architecture
- Fewer components
Enter fullscreen mode Exit fullscreen mode
But:
Cons:
- Auction Service handles forwarding
- Retry logic becomes its responsibility
Enter fullscreen mode Exit fullscreen mode
The right choice depends on how much complexity the system can justify.
18. What Happens If a Bid Update Is Lost?
Real-time systems sometimes lose messages.
Suppose:
$700 bid happens
Enter fullscreen mode Exit fullscreen mode
but one client never receives the update.
Is the auction broken?
Not necessarily.
During an active auction, another bid may soon arrive:
$700
↓
$750
↓
$800
Enter fullscreen mode Exit fullscreen mode
The missing $700 event becomes less important because newer updates overwrite the displayed state.
The dangerous case is the last bid.
Suppose:
$700
Enter fullscreen mode Exit fullscreen mode
is the final bid, and the client never receives it.
A useful recovery mechanism is to have the client periodically check for stale updates.
For example:
No bid update for a while
↓
Hard pull
↓
GET /auctions/{id}
↓
Retrieve current authoritative state
Enter fullscreen mode Exit fullscreen mode
This combines:
Fast push
+
Occasional authoritative pull
Enter fullscreen mode Exit fullscreen mode
and makes the system resilient to lost live events.
19. Placing a Bid
Now let’s look more closely at the actual bid request.
The client sends:
POST /api/v1/auctions/{auctionId}/bids
Enter fullscreen mode Exit fullscreen mode
with:
{
"bidAmount": 700,
"requestId": "req-123"
}
Enter fullscreen mode Exit fullscreen mode
The Auction Service first checks:
Does auction exist?
Is status ACTIVE?
Enter fullscreen mode Exit fullscreen mode
It can check the cache first:
Cache
↓
ACTIVE?
Enter fullscreen mode Exit fullscreen mode
If the cache doesn’t contain the auction, it can fall back to the database.
This cache miss should normally be a corner case.
20. Recording Bid History
Once the auction is confirmed to be active, the bid is written to the bid table.
A useful schema is:
Bid
-------------------------
bid_id
auction_id
bidder_id
amount
request_id
created_at
Enter fullscreen mode Exit fullscreen mode
The original design uses an append-only pattern.
That means:
User A → $500
User A → $600
User A → $700
Enter fullscreen mode Exit fullscreen mode
creates three records.
We don’t overwrite the previous rows.
This gives us a complete audit trail.
The latest valid bid for a bidder can be treated as their current bid.
The request ID or insertion timestamp can help determine ordering more robustly than relying only on client timestamps.
21. Updating the Highest Bid
After the bid is persisted, the service checks whether it is higher than the current cached bid.
Suppose:
Cache:
highest_bid = $600
Enter fullscreen mode Exit fullscreen mode
and the new bid is:
$700
Enter fullscreen mode Exit fullscreen mode
Then:
highest_bid
↓
$700
highest_bidder_id
↓
U10
Enter fullscreen mode Exit fullscreen mode
The cache is updated.
Then the Dispatcher is notified so that live viewers receive the new value.
If the new bid is lower than the current highest bid, it is still stored in the bid history but does not change the current highest-bid state.
Under our business rule, a bidder can only increase their own bid.
22. Why the Append-Only Bid Table Is Useful
An append-only design gives us several advantages.
It provides:
Complete history
Auditability
High write throughput
Simple writes
Enter fullscreen mode Exit fullscreen mode
For example:
Bid 101 → U1 → $500
Bid 102 → U2 → $550
Bid 103 → U1 → $600
Bid 104 → U3 → $700
Enter fullscreen mode Exit fullscreen mode
We can later reconstruct what happened.
It also avoids repeatedly modifying one large bid record.
23. A Hot Auction Creates a Hot Key
There is an important scaling problem.
Most auctions may have:
5–10 bidders
Enter fullscreen mode Exit fullscreen mode
but one extremely popular auction could have:
Millions of viewers
Thousands of concurrent bids
Enter fullscreen mode Exit fullscreen mode
All of these operations revolve around:
auction_id = A123
Enter fullscreen mode Exit fullscreen mode
If our cache partitions by auction ID, all updates may land on the same partition.
This is a classic hot-key problem.
A single auction can become a bottleneck even when the overall system has plenty of capacity.
24. Handling Hot Auctions
There are several approaches.
One option is to use a lease or lock mechanism to coordinate concurrent updates.
Conceptually:
Bid request
↓
Acquire lease for auction A123
↓
Update highest bid
↓
Release lease
Enter fullscreen mode Exit fullscreen mode
The advantage is that concurrent writers are coordinated.
The disadvantage is that a request may have to retry if another writer currently owns the lease.
Another approach is replicated storage with quorum-style behavior.
If the system is designed so that:
Higher bid always wins
Enter fullscreen mode Exit fullscreen mode
then conflict resolution becomes relatively simple.
For example:
Replica A → $700
Replica B → $750
Enter fullscreen mode Exit fullscreen mode
The conflict resolver can choose:
max($700, $750)
= $750
Enter fullscreen mode Exit fullscreen mode
This works particularly well because the business rule does not allow a bidder to reduce their own bid.
25. Auction Expiration Is a Scheduling Problem
Eventually, the auction must end.
We don’t want every Auction Service instance constantly scanning every auction.
Instead, we can use a Fulfillment Service.
Its job is similar to a scheduler.
It periodically looks for auctions whose:
status = ACTIVE
Enter fullscreen mode Exit fullscreen mode
and:
expire_at <= now
Enter fullscreen mode Exit fullscreen mode
The cache can make this check efficient because it already contains:
status
expire_at
Enter fullscreen mode Exit fullscreen mode
26. Determining the Winner
The Fulfillment Service finds an auction that appears ready to close.
But we should not blindly trust the cache.
The cache might be stale.
So the service asks the Auction DB to verify the winner.
Conceptually:
Fulfillment Service
↓
"Is this really the current winning bid?"
↓
Auction DB
Enter fullscreen mode Exit fullscreen mode
If the cache was stale:
Cache → $700
DB → $750
Enter fullscreen mode Exit fullscreen mode
the Fulfillment Service can repair the cache.
This is another example of read repair.
27. Moving the Auction to Payment
Once the winner is confirmed, the auction transitions from:
ACTIVE
Enter fullscreen mode Exit fullscreen mode
to:
PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
The database records:
winner_id
winner_bid_id
winner_price
payment_expire_at
Enter fullscreen mode Exit fullscreen mode
These values should be updated together as one logical state transition.
The cache is updated as well.
Then the notification system sends a message to the winner:
Congratulations!
You won the auction for $750.
Please complete payment within 10 minutes.
Enter fullscreen mode Exit fullscreen mode
The live viewers can also receive an auction-closed update through the Dispatcher.
28. Payment Completion
Payment itself is handled by a separate payment system.
The auction does not need to own the payment implementation.
The flow is:
Auction
↓
PAYMENT_PENDING
↓
Payment Service
↓
Payment succeeds
↓
Auction → SUCCEEDED
Enter fullscreen mode Exit fullscreen mode
The payment service should be idempotent so retries don’t accidentally create duplicate charges.
The auction system only needs to reliably react to the final payment result.
29. What If the Winner Doesn’t Pay?
The winner has:
10 minutes
Enter fullscreen mode Exit fullscreen mode
to complete payment.
The Fulfillment Service periodically checks auctions in:
PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
If:
payment_expire_at < now
Enter fullscreen mode Exit fullscreen mode
and the payment hasn’t succeeded:
PAYMENT_PENDING
↓
FAILED
Enter fullscreen mode Exit fullscreen mode
The item can then be handled according to the broader marketplace policy.
The important point is that the auction system has another timed state transition:
ACTIVE
↓
PAYMENT_PENDING
↓
SUCCEEDED / FAILED
Enter fullscreen mode Exit fullscreen mode
30. Why the Fulfillment Service Reads the Cache
There is a deliberate trade-off here.
The Fulfillment Service could query the database directly:
Find every ACTIVE auction
↓
Check expire_at
↓
Find highest bid
↓
Execute expired auctions
Enter fullscreen mode Exit fullscreen mode
The problem is that this can require expensive queries over the Auction DB.
Instead, the cache already contains:
status
highest_bid
highest_bidder_id
expire_at
Enter fullscreen mode Exit fullscreen mode
So the Fulfillment Service can use the cache to find candidates quickly.
The trade-off is:
Cache approach
↓
Lower latency
Less DB load
But possible stale data
Database approach
↓
More accurate
No cache dependency
But more DB load and more expensive queries
Enter fullscreen mode Exit fullscreen mode
The final design can combine both:
Cache → find candidate
↓
DB → verify
Enter fullscreen mode Exit fullscreen mode
This gives us both performance and correctness.
31. The Reconciliation Service
Distributed systems can end up in abnormal states.
For example:
Payment succeeded
↓
Auction DB was not updated
Enter fullscreen mode Exit fullscreen mode
Now we have:
Payment = SUCCESS
Auction = PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
A Reconciliation Service periodically searches for these inconsistencies.
It can compare:
Auction state
Payment state
Cache state
Enter fullscreen mode Exit fullscreen mode
and repair the auction.
For example:
Payment says SUCCESS
Auction says PAYMENT_PENDING
↓
Reconciliation
↓
Auction → SUCCEEDED
Enter fullscreen mode Exit fullscreen mode
This gives the system a recovery mechanism instead of relying only on the happy path.
32. The Complete Stateless Design
Putting everything together:
USER
|
▼
Load Balancer
|
┌──────────┴──────────┐
↓ ↓
Auction Service Bid Update Service
| |
┌──────┴──────┐ |
↓ ↓ |
Auction DB Cache |
| | |
└──────┬──────┘ |
↓ |
Kafka |
| |
↓ |
Async Consumers |
|
User SSE ←─────────────────────────────┘
Auction Service
|
↓
Dispatcher
|
↓
Bid Update Service
|
↓
SSE → viewers
Cache
|
↓
Fulfillment Service
|
↓
Auction DB
|
↓
Winner
|
↓
Notification
|
↓
Payment Service
|
↓
SUCCEEDED / FAILED
Reconciliation Service
|
└── checks abnormal states
Enter fullscreen mode Exit fullscreen mode
The Auction Service itself remains stateless.
The state lives in:
Auction DB
Cache
Enter fullscreen mode Exit fullscreen mode
and temporary connection state lives in:
Bid Update Service
Dispatcher
Enter fullscreen mode Exit fullscreen mode
33. Stateful Auction Service
So far we’ve used a stateless architecture.
There is another interesting option.
We could make the Auction Service itself stateful.
When an auction is created:
Auction A123
↓
Assigned to Auction Service #5
Enter fullscreen mode Exit fullscreen mode
All bids for that auction are then routed to the same server:
Auction A123
↓
Auction Service #5
↓
All bids
Enter fullscreen mode Exit fullscreen mode
The server could keep the current auction state in memory.
This reduces the need for multiple servers to coordinate on the same auction.
34. Routing Requests to the Correct Server
With a stateful design, the load balancer needs to know:
Auction A123 → Server #5
Auction A456 → Server #8
Enter fullscreen mode Exit fullscreen mode
This requires service discovery or a consistent routing mechanism.
The request:
POST /auctions/A123/bids
Enter fullscreen mode Exit fullscreen mode
must always reach the instance responsible for:
A123
Enter fullscreen mode Exit fullscreen mode
This can make per-auction ordering and consistency easier.
35. The Problem With Stateful Servers
The stateful design introduces a new problem.
Suppose:
Auction A123
↓
Server #5
↓
In-memory state
Enter fullscreen mode Exit fullscreen mode
and Server #5 crashes.
We lose the in-memory state.
Therefore, the server needs recovery mechanisms such as:
Write-ahead log
+
Snapshots
Enter fullscreen mode Exit fullscreen mode
or rebuilding state from the Auction DB.
We may also replicate the state:
Primary
↓
Follower
Enter fullscreen mode Exit fullscreen mode
so that the follower can take over after failure.
This makes the architecture more complicated.
36. Stateless vs Stateful
The two approaches have different strengths.
Area Stateless Stateful Consistency More coordination required Easier per-auction ordering Availability Easier Harder because state must be recovered Scaling Easier to add nodes More difficult due to routing Hot auctions Can be challenging One server can become a hotspot Failure recovery External state survives node loss In-memory state needs recovery Operational complexity Generally simpler More complexThe stateless approach is usually more common.
The stateful approach is still useful when processing a stream of events belonging to the same entity.
37. High Availability
Let’s examine what happens when individual components fail.
The stateless Auction Service is relatively easy to make highly available.
If one node fails:
Client
↓
Retry
↓
Another Auction Service
Enter fullscreen mode Exit fullscreen mode
Because the state is external, the new node can continue processing.
Duplicate requests are possible.
For auction creation, we can use an idempotent request or an upsert-like operation to prevent duplicate auctions.
For bids, the append-only model makes duplicate writes easier to handle, especially when requests have unique IDs.
38. Dispatcher Availability
The Dispatcher is different because it maintains state.
If it fails, the routing table disappears.
Possible solutions include:
WAL + snapshots
External replicated KV store
Active / standby
Enter fullscreen mode Exit fullscreen mode
The important idea is:
Stateful components need a recovery story.
39. Bid Update Service Availability
The Bid Update Service maintains live client connections.
Its state is:
User connection
Auction subscription
Enter fullscreen mode Exit fullscreen mode
But this state is tied to the lifetime of the connection.
If the server crashes:
SSE connection dies
↓
Client reconnects
↓
Another Bid Update Service
Enter fullscreen mode Exit fullscreen mode
We don’t necessarily need to persist every connection in a durable database.
The connection state is temporary.
This is different from:
Auction state
Bid history
Winner
Payment state
Enter fullscreen mode Exit fullscreen mode
which must survive server failures.
40. Cache and Database Replication
The cache and Auction DB are both important infrastructure.
Different replication strategies are possible:
Single leader
Multi-leader
Quorum
Enter fullscreen mode Exit fullscreen mode
A single leader is simpler.
A replicated cache/database can improve availability.
Quorum replication can improve durability and consistency at the cost of more coordination.
The right strategy depends on the underlying technology and the consistency guarantees we need.
For the auction’s authoritative state, correctness during winner selection is more important than squeezing out the last bit of write latency.
41. Scaling the Auction Service
In the stateless architecture, scaling is straightforward:
More traffic
↓
Add more Auction Service instances
Enter fullscreen mode Exit fullscreen mode
A load balancer distributes requests.
In a stateful design, scaling is harder because auctions need to be assigned to specific servers.
We can shard auctions using a key such as:
auction_id
Enter fullscreen mode Exit fullscreen mode
or:
owner_id
Enter fullscreen mode Exit fullscreen mode
The original design notes that auction_id gives good co-location, but can create hot partitions.
Partitioning by owner_id may distribute traffic differently.
The correct partition key depends on the actual workload.
42. Scaling the Dispatcher
The Dispatcher keeps a table such as:
auction → Bid Update Service
Enter fullscreen mode Exit fullscreen mode
The memory footprint can be manageable.
But memory size isn’t the only concern.
The Dispatcher may receive a very high number of requests.
Therefore, we can scale it using:
Read replicas
Sharding
Partitioning by auction_id
Enter fullscreen mode Exit fullscreen mode
Replication can be:
Synchronous
Enter fullscreen mode Exit fullscreen mode
for stronger consistency, or:
Asynchronous
Enter fullscreen mode Exit fullscreen mode
when eventual consistency is acceptable.
43. Scaling the Cache and Auction Database
There are several possible partitioning strategies.
Partition by:
auction_id
Enter fullscreen mode Exit fullscreen mode
This has a useful property:
Auction data
+
Bid data
Enter fullscreen mode Exit fullscreen mode
can be co-located.
But a popular auction can become a hot partition.
Another option is partitioning by:
user_id
Enter fullscreen mode Exit fullscreen mode
This can distribute writes more evenly because an individual user is less likely to become a massive hotspot.
Rate limiting can further protect the system from unusually active users.
There is no universally correct partition key.
We choose based on the workload.
44. Scaling Bid Update Services
Bid Update Services are relatively easy to scale.
Each node maintains its own in-memory connections:
bus1 → users
bus2 → users
bus3 → users
Enter fullscreen mode Exit fullscreen mode
When the number of connections grows:
Add more Bid Update Service instances
Enter fullscreen mode Exit fullscreen mode
The load balancer distributes new SSE connections across them.
The Dispatcher keeps track of which service owns the subscriptions.
45. Scaling Fulfillment
The Fulfillment Service can also be distributed.
Auctions can be partitioned by:
auction_id
Enter fullscreen mode Exit fullscreen mode
and different workers can process different partitions.
For example:
Worker 1 → A–F
Worker 2 → G–M
Worker 3 → N–S
Worker 4 → T–Z
Enter fullscreen mode Exit fullscreen mode
The important requirement is to prevent two workers from closing the same auction simultaneously.
The final winner transition must therefore be protected by an atomic database update or equivalent concurrency control.
46. Cache and Auction DB Consistency
Let’s revisit one of the most subtle problems.
Suppose the system does:
1. Write bid to DB
2. Update cache
Enter fullscreen mode Exit fullscreen mode
The DB write succeeds:
DB = $700
Enter fullscreen mode Exit fullscreen mode
but the cache update fails:
Cache = $650
Enter fullscreen mode Exit fullscreen mode
A retry can fix it.
But retries can also fail.
Therefore, the cache entry includes:
updated_at
Enter fullscreen mode Exit fullscreen mode
When the system detects that the cached state is stale:
Cache
↓
Stale?
↓
Read DB
↓
Repair cache
Enter fullscreen mode Exit fullscreen mode
This can happen when:
Serving a read
Enter fullscreen mode Exit fullscreen mode
or:
Executing an auction
Enter fullscreen mode Exit fullscreen mode
This is why the cache is treated as a fast representation of the state rather than the final authority.
47. Write-Through vs Write-Back
There are two broad ways to synchronize cache and database.
With write-through-style behavior:
Write DB
↓
Update Cache
Enter fullscreen mode Exit fullscreen mode
The database is updated immediately.
With write-back:
Update Cache
↓
Persist to DB later
Enter fullscreen mode Exit fullscreen mode
Write-back can reduce database writes in some workloads.
For example, if we wanted to update the winning bid in the auction table on every bid, write-back could reduce the number of direct database writes.
But it also makes durability and failure recovery more complicated.
For the auction design, keeping the bid history durably in the database and using the cache for the current highest-bid state is a safer model.
48. SSE vs WebSocket
Both technologies can provide real-time communication.
SSE WebSocket Direction Server → Client Bidirectional Protocol style HTTP WebSocket Data Text/event stream Text + binary Reconnection Built in Application typically handles it Best suited for One-way live updates Interactive two-way communicationFor our auction:
Bid placement
→ HTTP
Bid updates
→ SSE
Enter fullscreen mode Exit fullscreen mode
This is simple because the server mainly pushes state to viewers.
WebSocket becomes more attractive if the product eventually needs richer bidirectional interaction.
49. Another Real-Time Design
There is another possible connection strategy.
Instead of opening an SSE connection every time a user navigates to an auction, the application could maintain one long-lived WebSocket connection after login.
For example:
User logs in
↓
WebSocket established
↓
User opens Auction A
↓
Subscribe to A
↓
User opens Auction B
↓
Unsubscribe A
↓
Subscribe B
Enter fullscreen mode Exit fullscreen mode
This may be useful if users frequently move between auctions.
The right choice depends on how users interact with the product.
50. Reliability of Live Updates
We don’t necessarily need exactly-once delivery for every live bid update.
Suppose a client receives:
$700
$700
Enter fullscreen mode Exit fullscreen mode
twice.
The UI can simply keep:
max(currentBid, receivedBid)
Enter fullscreen mode Exit fullscreen mode
and display:
$700
Enter fullscreen mode Exit fullscreen mode
Similarly, if the client receives:
$700
$750
$800
Enter fullscreen mode Exit fullscreen mode
and $750 is duplicated, there is no business impact.
This makes the real-time layer easier to design.
The important part is that the final authoritative winner comes from the database.
51. API Design
The core API surface can be kept simple.
Create an auction:
POST /api/v1/auctions
Enter fullscreen mode Exit fullscreen mode
Get an auction:
GET /api/v1/auctions/{auctionId}
Enter fullscreen mode Exit fullscreen mode
Place a bid:
POST /api/v1/auctions/{auctionId}/bids
Idempotency-Key: bid-123
Enter fullscreen mode Exit fullscreen mode
Example:
{
"amount": 700
}
Enter fullscreen mode Exit fullscreen mode
Get bid history:
GET /api/v1/auctions/{auctionId}/bids
Enter fullscreen mode Exit fullscreen mode
Open live updates:
GET /api/v1/auctions/{auctionId}/events
Enter fullscreen mode Exit fullscreen mode
implemented as an SSE stream.
The API should return conflicts such as:
Auction not found
Auction already closed
Bid is not higher than current bid
Duplicate request
Enter fullscreen mode Exit fullscreen mode
with appropriate HTTP status codes.
52. Data Model
A simplified auction table:
Auction
--------------------------------
auction_id
owner_id
item_id
status
created_at
updated_at
expire_at
winner_id
winner_bid_id
winner_price
payment_expire_at
Enter fullscreen mode Exit fullscreen mode
Possible statuses:
ACTIVE
PAYMENT_PENDING
SUCCEEDED
FAILED
Enter fullscreen mode Exit fullscreen mode
The bid table:
Bid
--------------------------------
bid_id
auction_id
bidder_id
amount
request_id
created_at
Enter fullscreen mode Exit fullscreen mode
Indexes should support common access patterns such as:
auction_id + created_at
auction_id + amount
auction_id + bidder_id
Enter fullscreen mode Exit fullscreen mode
The cache stores:
auction_id
status
highest_bid
highest_bidder_id
updated_at
expire_at
Enter fullscreen mode Exit fullscreen mode
The important distinction is:
Bid table
→ complete history
Cache
→ current hot state
Enter fullscreen mode Exit fullscreen mode
53. Auction State Transitions
The lifecycle can be visualized as:
┌──────────────┐
│ ACTIVE │
└──────┬───────┘
│
no higher bid
for 1 hour
│
▼
┌───────────────────┐
│ PAYMENT_PENDING │
└─────────┬─────────┘
│
┌──────┴───────┐
│ │
payment timeout
success │
│ │
▼ ▼
┌───────────┐ ┌────────┐
│ SUCCEEDED │ │ FAILED │
└───────────┘ └────────┘
Enter fullscreen mode Exit fullscreen mode
Making the states explicit makes recovery much easier.
54. The Hardest Race: Bid vs Auction Expiration
There is a subtle race condition.
Suppose the auction expires at:
11:00:00
Enter fullscreen mode Exit fullscreen mode
At almost exactly the same time:
User A submits a $900 bid
Enter fullscreen mode Exit fullscreen mode
and:
Fulfillment Service tries to close the auction
Enter fullscreen mode Exit fullscreen mode
Which one wins?
We need a clearly defined ordering rule.
A robust approach is to make the final transition conditional in the database.
For example:
Close auction only if:
status = ACTIVE
AND expire_at <= now
Enter fullscreen mode Exit fullscreen mode
A bid should similarly be accepted only if:
status = ACTIVE
AND current time < expire_at
Enter fullscreen mode Exit fullscreen mode
The database transaction / compare-and-set operation determines which state transition wins.
This is an important example of why final winner selection cannot rely only on cache state.
55. Idempotency and Retries
Distributed systems retry requests.
For example:
Client → Bid Service
↓
Request succeeds
↓
Network response lost
↓
Client retries
Enter fullscreen mode Exit fullscreen mode
Without idempotency:
Same logical bid
↓
Two database records
Enter fullscreen mode Exit fullscreen mode
This is why a client request can include:
requestId
Enter fullscreen mode Exit fullscreen mode
or:
Idempotency-Key
Enter fullscreen mode Exit fullscreen mode
The server can detect that the same logical operation has already been processed.
This is especially important for:
Auction creation
Bid placement
Payment
Auction state transitions
Enter fullscreen mode Exit fullscreen mode
56. What Happens If the Auction Service Crashes?
Suppose:
User sends $700 bid
Enter fullscreen mode Exit fullscreen mode
The Auction Service writes the bid successfully:
DB → SUCCESS
Enter fullscreen mode Exit fullscreen mode
but crashes before updating the cache.
After recovery:
DB → $700
Cache → $650
Enter fullscreen mode Exit fullscreen mode
The reconciliation/read-repair mechanism can detect the discrepancy.
The important design principle is:
The durable operation should be recoverable even if the process dies immediately afterward.
57. What Happens If Fulfillment Crashes?
Suppose Fulfillment decides:
Auction A123 should close
Enter fullscreen mode Exit fullscreen mode
but crashes before completing the transition.
Another Fulfillment worker can pick it up.
The final database operation should be conditional:
UPDATE auction
SET status = PAYMENT_PENDING
WHERE auction_id = ?
AND status = ACTIVE
AND expire_at <= now
Enter fullscreen mode Exit fullscreen mode
Only one worker will successfully transition the row.
This makes the operation idempotent and safe to retry.
58. What Happens If Notification Fails?
Suppose:
Auction → PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
but notification delivery fails.
The auction should not remain stuck simply because an email or push notification failed.
Instead:
Auction state
↓
Persisted successfully
Notification
↓
Async retry
Enter fullscreen mode Exit fullscreen mode
The notification system can retry independently.
This is another reason not to put non-critical side effects directly inside the critical transaction.
59. What Happens If Payment Succeeds but Auction Isn’t Updated?
This is one of the most important recovery cases.
Suppose:
Payment
↓
SUCCESS
Enter fullscreen mode Exit fullscreen mode
but:
Auction DB
↓
Still PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
The Reconciliation Service can detect:
Payment = SUCCESS
Auction = PAYMENT_PENDING
Enter fullscreen mode Exit fullscreen mode
and correct the auction:
Auction → SUCCEEDED
Enter fullscreen mode Exit fullscreen mode
This is why reconciliation is not an optional afterthought in distributed systems.
60. Final Stateless Architecture
The complete stateless design can now be summarized as:
USERS
|
▼
Load Balancer
|
┌────────────┴────────────┐
↓ ↓
Auction Service Bid Update Service
| |
┌─────┴─────┐ SSE Connections
↓ ↓ |
Auction DB Cache |
| | |
| └──────┐ |
| ↓ |
| Fulfillment |
| | |
| ↓ |
| Auction DB |
| |
└──────────────┐ |
↓ |
Dispatcher ─────────┘
|
↓
Bid Update Services
|
↓
Viewers
Auction Service
|
↓
Kafka
|
┌───┴──────────────┐
↓ ↓
Notification Reconciliation
|
↓
Winner
Fulfillment
|
↓
Payment Service
|
├── SUCCESS → SUCCEEDED
|
└── TIMEOUT → FAILED
Enter fullscreen mode Exit fullscreen mode
The key property is that the Auction Service itself does not keep auction state in memory.
61. When Would We Choose the Stateful Design?
The stateless architecture is usually the better default.
It is easier to:
Scale
Recover
Deploy
Load balance
Enter fullscreen mode Exit fullscreen mode
The stateful design becomes attractive when we need extremely efficient per-auction processing and want all events for one auction handled by the same process.
For example:
Auction A123
↓
Stateful server #5
↓
All bids for A123
Enter fullscreen mode Exit fullscreen mode
This can make ordering easier.
But now:
Server #5 fails
Enter fullscreen mode Exit fullscreen mode
and we need:
Replica
WAL
Snapshot
Recovery
Routing
Enter fullscreen mode Exit fullscreen mode
Therefore, stateful systems trade simpler per-entity processing for more difficult availability and scaling.
62. The Most Important Trade-offs
There is no single perfect architecture.
The major decisions are:
Stateless vs Stateful
Stateless
→ easier scaling and availability
Stateful
→ easier per-auction ordering
Enter fullscreen mode Exit fullscreen mode
Cache vs Database for Current Bid
Cache
→ fast
Database
→ authoritative
Enter fullscreen mode Exit fullscreen mode
The practical design uses both.
SSE vs WebSocket
SSE
→ simple one-way updates
WebSocket
→ richer bidirectional communication
Enter fullscreen mode Exit fullscreen mode
Dispatcher vs Direct Coordination Store
Dispatcher
→ cleaner separation
Direct coordination
→ fewer components
Enter fullscreen mode Exit fullscreen mode
Cache-driven vs DB-driven Fulfillment
Cache
→ fast, lower DB pressure
DB
→ authoritative, more expensive
Enter fullscreen mode Exit fullscreen mode
The hybrid approach is:
Cache → identify candidate
DB → verify
Enter fullscreen mode Exit fullscreen mode
63. The Design in One Mental Model
The entire system can be remembered through four responsibilities.
First:
Auction Service
Enter fullscreen mode Exit fullscreen mode
handles the business operation:
Create auction
Place bid
Update current bid
Enter fullscreen mode Exit fullscreen mode
Second:
Auction DB
Enter fullscreen mode Exit fullscreen mode
keeps the durable truth:
Auction
Bid history
Winner
Payment state
Enter fullscreen mode Exit fullscreen mode
Third:
Bid Update Service + Dispatcher
Enter fullscreen mode Exit fullscreen mode
handles the real-time experience:
New bid
↓
Dispatcher
↓
Bid Update Service
↓
SSE
↓
Viewers
Enter fullscreen mode Exit fullscreen mode
Fourth:
Fulfillment + Reconciliation
Enter fullscreen mode Exit fullscreen mode
handles time and recovery:
Auction expires
↓
Determine winner
↓
Payment
↓
Success / Failure
Abnormal state
↓
Reconcile
Enter fullscreen mode Exit fullscreen mode
64. A Natural Interview Walkthrough
If you were explaining this system in an interview, the conversation can naturally progress like this.
Start with the requirements:
“Users can create auctions, view active auctions, place bids, and receive real-time updates. An auction closes after one hour without a higher bid. The winner gets ten minutes to pay.”
Then establish the scale:
1B DAU
100K auctions/day
~100M bids/day
10:1 read/write
Enter fullscreen mode Exit fullscreen mode
Then identify the core challenge:
“The most interesting problem is handling real-time updates while maintaining correct winner selection.”
Then introduce the architecture:
Auction Service
Auction DB
Cache
Bid Update Service
Dispatcher
Fulfillment
Enter fullscreen mode Exit fullscreen mode
Explain the live path:
HTTP bid
↓
DB
↓
Cache
↓
Dispatcher
↓
SSE
Enter fullscreen mode Exit fullscreen mode
Then explain expiration:
Cache
↓
Fulfillment
↓
DB verification
↓
Winner
Enter fullscreen mode Exit fullscreen mode
Then discuss failure:
Retry
Idempotency
Read repair
Reconciliation
Enter fullscreen mode Exit fullscreen mode
Finally compare:
Stateless vs Stateful
SSE vs WebSocket
Cache vs DB
Dispatcher vs coordination store
Enter fullscreen mode Exit fullscreen mode
That gives you a coherent system-design discussion rather than a list of technologies.
65. Final Takeaways
The auction system looks simple from the outside:
Seller lists item
↓
Users bid
↓
Highest bidder wins
↓
Winner pays
Enter fullscreen mode Exit fullscreen mode
The distributed system underneath is much more interesting.
The most important ideas are:
1. Keep bid history durable and append-only.
2. Keep the current highest bid in a fast cache.
3. Use SSE for efficient one-way live updates.
4. Use a Dispatcher to route updates to the right
Bid Update Service.
5. Treat the Auction DB as the authoritative source.
6. Allow eventual consistency for live display,
but use strong consistency when selecting the winner.
7. Use a Fulfillment Service to process auction expiration.
8. Verify the winner against the database before closing
the auction.
9. Use idempotency and conditional state transitions
so retries are safe.
10. Use reconciliation to repair abnormal distributed states.
11. Watch for hot auctions and hot cache keys.
12. Stateless architecture is easier to scale and recover;
stateful architecture can simplify per-auction ordering
but makes failure recovery harder.
13. SSE is sufficient when the main real-time requirement
is server-to-client updates; WebSocket is useful when
richer bidirectional communication is needed.
Enter fullscreen mode Exit fullscreen mode
The deepest system-design lesson is this:
The live bidding experience can tolerate some temporary inconsistency, but the final winner cannot.
That single distinction explains why the system combines:
Cache
+
Real-time events
+
SSE
+
Database
+
Scheduler
+
Reconciliation
Enter fullscreen mode Exit fullscreen mode
Each component solves a different part of the problem.
66. Reference
The design and assumptions in this article are based on the free Coding Monkey article:
How to Design Auction System
https://pyemma.github.io/How-to-design-auction-system/
The original article discusses the stateless and stateful designs, live bid routing, SSE, Dispatcher, cache/database consistency, Fulfillment Service, scalability, availability, and SSE/WebSocket trade-offs.
답글 남기기