Hello, I’m Maneshwar. I’m building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.
Every serious API will eventually tell you to sit down and be quiet.
Hammer GitHub, Stripe, or AWS a little too eagerly and your requests start bouncing back with a polite but firm 429.
I always found that fascinating, so let’s build the thing that says no.
By the end of this post we’ll have designed a rate limiter that actually holds up when you put it in front of real traffic, and I promise to only make a reasonable number of bucket puns along the way.
A rate limiter does one job: it decides how many requests a client is allowed to make in a given window of time.
It protects your system from getting flattened, and it keeps one greedy user from eating everyone else’s lunch.
Simple idea. Surprisingly spicy implementation.
Let’s build it up piece by piece, the way you’d actually reason through it in an interview or a design doc.
First, what are we even building?
Before writing a single line, let’s agree on what “good” looks like.
Here’s my wishlist:
- Configurable limits. Something like “100 requests per minute per user.” The rules should not be hardcoded, because free users and premium users deserve different amounts of pain.
-
Honest rejections. When someone goes over, we return HTTP
429 Too Many Requestsand include helpful headers telling them how many requests they have left and when the window resets. No mystery. - Barely-there latency. This check runs on every single request, so it has to be fast. Let’s aim for under 3ms at P95. If your rate limiter is slow, congratulations, you built a second bottleneck.
- Highly available and shared. Multiple servers need to agree on the same counts. More on why that word “shared” is doing a lot of heavy lifting later.
Cool. Now let’s start naive and let reality punch us in the face a few times.
Attempt 1: the fixed window counter
The simplest thing that could possibly work is fixed window counting. Chop time into neat one-minute slices.
Give each user a counter.
Every request bumps the counter by one.
Hit the limit, get rejected, and the counter resets when the next window starts.
# user makes a request
count = get(user_id) # how many so far this minute?
if count >= 100:
reject() # 429, come back later
else:
increment(user_id)
allow()
Enter fullscreen mode Exit fullscreen mode
Clean. Readable. You could explain it to a rubber duck. So where do we keep this counter?
Where do we put the counter? (this trips people up)
Your first instinct might be the database.
Please don’t.
We’d be adding a write to the database on every request, which means the thing we built to protect our system is now quietly overloading it.
That is peak “I have brought peace, freedom, and a full table scan.”
Okay, database is out.
What about keeping counters in memory on the server? Blazing fast. Love it.
Except it only works if you have exactly one server, and nobody runs one server.
The moment you scale out, each box keeps its own private counter.
A sneaky user sends 100 requests to Server A and 100 to Server B and walks away with 200 requests per minute while your limit says 100. Whoops.
What we actually want is somewhere that is memory-fast and shared across every server.
That’s Redis. It’s an in-memory data store, it hands us atomic counter primitives like INCR, and it can expire keys automatically so windows reset on their own.
This is why Redis shows up in basically every rate limiter design ever drawn on a whiteboard.
So far so good. Now let me ruin it.
The fixed window flaw nobody warns you about
Fixed windows have a nasty edge case hiding right at the seams.
Picture a limit of 100 requests per minute.
A user fires 100 requests in the last 10 seconds of one minute, then another 100 in the first 10 seconds of the next minute.
Each window is technically within the limit. Both are 100 or under.
But zoom out and you’ll see 200 requests in a 20 second span, which is very much not the spirit of “100 per minute.”
This happens at every window boundary, and once someone notices the pattern, they will absolutely abuse it.
The counter has no memory across the boundary, so it cannot see the burst spanning two windows.
We need an algorithm that thinks in terms of a smooth rate rather than hard resets.
Attempt 2: the token bucket (our hero)
Enter the token bucket, the algorithm quietly powering the limits at places like AWS and Stripe.
Here’s the mental model, and yes, it is literally a bucket.
Imagine a bucket that holds tokens. Tokens drip in at a steady rate. Every request has to grab one token to pass. No tokens left? Request gets rejected. That’s it.
# refill based on time passed since we last looked
elapsed = now - last_refill
tokens = min(capacity, tokens + elapsed * refill_rate)
last_refill = now
if tokens >= 1:
tokens -= 1
allow()
else:
reject() # 429, and tell them when to retry
Enter fullscreen mode Exit fullscreen mode
Here’s the diagram version of that decision:
Watch how this fixes our boundary nightmare.
Set the bucket capacity to 100 and the refill rate to 100 per minute.
During quiet stretches, tokens pile up toward the cap.
When a burst comes in, the user spends whatever tokens they’ve saved, but they can never outrun the refill rate over the long haul.
No matter how they time things around a boundary, they cannot conjure tokens that were never added.
That’s the whole trick, and it’s genuinely elegant. Two knobs control everything:
- Capacity decides how big a burst you tolerate.
- Refill rate decides your sustained throughput.
Capacity 100 with a refill of 100 per minute means a user can fire up to 100 requests instantly if they’ve been idle, but long term they’re pinned to 100 per minute.
Bursty when it’s calm, strict when it counts. Chef’s kiss.
Are there other algorithms? Sure. Sliding window logs, sliding window counters, leaky buckets, they each solve the boundary problem their own way.
But token bucket hits the sweet spot between “simple enough to actually implement correctly” and “good enough for almost everyone.”
For a deeper rabbit hole, the token bucket writeup on Wikipedia is a decent start.
Okay, but where does the limiter live?
We’ve got the algorithm. Now, architecturally, where do we run it? Three options, each with a personality.
- Client side. Put the logic in the client app. Fast, but hilariously naive. You cannot trust clients to enforce their own limits, because a motivated user will just edit the code or skip the check entirely. This is like asking people to fine themselves for speeding.
- Server side. Bake it into your application code. You get full control over the algorithm and everything lives in one place. The downside is your rate limiting logic gets tangled up with business logic, and every service ends up reinventing the same wheel.
- Middleware. A dedicated layer between clients and your APIs, like an API gateway, a reverse proxy, or a small custom service. Rate limiting stays cleanly separated from business logic, and you get one place to manage all your policies. The cost is a bit more moving parts in your system.
Which one wins? It genuinely depends.
If you already run an API gateway doing auth, tucking rate limiting in there is a no-brainer.
If you need some exotic custom algorithm, server side gives you room to move.
But for most systems, middleware is the sweet spot: control and operational sanity without soldering the limiter to your business code.
Let’s go with middleware and sketch the flow.
The architecture, end to end
We’ll store the rules (“premium gets 1000 per hour, free gets 100 per hour”) in a configuration service.
The middleware reads those rules, keeps token bucket state in Redis, and makes the call on every incoming request.
When a request lands, the middleware figures out who the user is, pulls their bucket from Redis, and checks for a spare token.
Token available? Decrement it and pass the request along.
Bucket empty? Return a 429 with a Retry-After header so the client knows exactly when to come knocking again.
Be a good host. Tell your guests when the kitchen reopens xD
This is lovely for one rate limiting server. Then, as always, scaling shows up to spoil the party.
Scaling breaks it: the race condition
Run multiple rate limiter instances against the same Redis and you can hit a classic race condition.
Here’s the exact sequence that loses a count:
Both servers read 3. Both decide the request is fine.
Both write 4. We just quietly lost a count, and the counter now lies to us.
Do this enough times under load and your “100 per minute” limit turns into “somewhere around 100, we think, on a good day.”
Not exactly the airtight guarantee we promised.
The problem is that read, check, and write are three separate steps, and another server can sneak in between them.
Atomic operations to the rescue
The fix is to make read plus check plus write a single indivisible operation, so nobody can wedge themselves in the middle.
Redis lets us do this with Lua scripts, which run atomically on the server.
The whole check-and-decrement happens as one unit, and the race condition simply cannot occur.
-- KEYS[1] = bucket key
-- ARGV[1] = capacity, ARGV[2] = refill_rate, ARGV[3] = now
local tokens = tonumber(redis.call("GET", KEYS[1]) or ARGV[1])
if tokens >= 1 then
redis.call("DECR", KEYS[1])
return 1 -- allowed
else
return 0 -- rejected, send a 429
end
Enter fullscreen mode Exit fullscreen mode
(That snippet is simplified to keep the point front and center. A production version would compute the refill from elapsed time and store the last-refill timestamp too.)
Atomic operations turn our shaky-under-load counter into something you can actually trust across a fleet of servers.
This is the difference between a rate limiter that works in the demo and one that works on Black Friday.
Stuff I deliberately skipped (so we don’t turn this into a book)
We covered the load-bearing bits, but a real production rate limiter opens up a bunch of fun follow-up questions worth chewing on:
- Geographic latency. How do you handle a multi-region deployment where a global Redis is an ocean away?
- Hot keys. What happens when a handful of users generate most of your traffic and hammer the same Redis key?
- Hot reloading rules. How do you push new limits without restarting servers?
- Fail open or fail closed? If Redis falls over, do you let everything through or block everything? Both answers can ruin your day, just differently.
Each of those is a great whiteboard prompt on its own, and honestly, “fail open vs fail closed” alone has sparked some very heated lunch debates.
Wrapping up
So there’s our bucket list, completed.
We started with a naive counter, watched it leak requests at window boundaries, upgraded to a token bucket, argued about where to run it, moved the state into Redis so servers could agree, then made the whole thing atomic so scaling couldn’t corrupt our counts.
That’s the core of nearly every rate limiter you’ll ever meet in the wild.
Next time an API hits you with a 429, you’ll know there’s a little bucket of tokens somewhere, freshly emptied, quietly telling you to hold your horses.
If you build one, or you have strong opinions on fail open vs fail closed, drop a comment. I’ll try to reply before you hit my rate limit.
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
GenAI today is a race car without brakes. It accelerates fast — you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior — without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…