Rate limiting

Why it matters

Without limits, one misbehaving script, one retry storm or one abusive tenant takes the whole API down for everyone. Rate limiting bounds what any single caller can consume, so the system degrades for the offender instead of for all. It is also a product surface: tiers, quotas and fair use are expressed as limits. It appears in almost every design question because it is the first line of defence and the simplest form of back-pressure.

The idea

A rate limiter decides, per key (user, API key, IP, tenant, endpoint) whether a request may proceed now, and if not, how long to wait. The algorithms differ in how they handle bursts and how much state they keep.

The algorithms

Algorithm	How it works	Bursts	Memory per key	Notes
Token bucket	Bucket holds up to B tokens, refilled at R per second; a request takes one token or is rejected	Allowed up to B, then smoothed to R	Two numbers: tokens, last refill time	The default choice; simple, burst-tolerant, cheap
Leaky bucket	Requests queue in a bucket that drains at a constant rate; overflow is rejected	Smoothed completely; output is steady	Queue length	Shapes traffic rather than just gating it; adds latency
Fixed window	Count requests per calendar window (per minute); reject above N	Up to 2N at a window boundary	One counter	Simplest; boundary burst problem
Sliding window log	Store a timestamp per request; count those within the last window	Exact	One entry per request	Precise but memory-heavy at high rates
Sliding window counter	Weight the previous window's count by its overlap with the current one	Approximately exact	Two counters	Good precision at fixed-window cost; common in practice

Where the limiter lives

At the edge (API gateway, CDN, reverse proxy) to shed abusive traffic before it costs anything. Per service for limits that depend on business context, such as a tenant's plan. Inside clients of downstream dependencies, to stay under a third party's quota. Most systems have all three.

Making it consistent across servers

A counter in one server's memory limits only that server's share. For a global limit, keep state in a shared store, typically Redis, and make the read-modify-write atomic with a Lua script or INCR plus EXPIRE so concurrent servers do not both admit the last request. Keys are limit:{tenant}:{endpoint} with a TTL matching the window. For very high rates, accept approximation: each server limits locally against its share of the global quota, or syncs counts asynchronously, trading exactness for a shared store that is no longer a hot spot.

Responding well

Return 429 Too Many Requests with a Retry-After header saying when to try again, and expose RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers on every response so clients can pace themselves before hitting the wall. Document limits per tier. Distinguish rate limits (requests per second) from quotas (requests per month) in the same headers. Clients should back off exponentially with jitter, which the reliability lessons in later stages revisit.

Keys and fairness

Limit by the identity you can trust: an authenticated user or API key where available, IP as a fallback with care (NAT puts many users behind one address). Separate limits for expensive endpoints (search, export) from cheap ones. Consider cost-based limiting, where each request consumes a weight, for APIs with wildly varying operation costs, including GraphQL.

Trade-offs

Token bucket is the sensible default; fixed windows are trivial but allow double bursts at boundaries; sliding logs are exact and expensive; leaky buckets smooth output at the cost of queueing latency. Centralised state gives exact global limits and adds a dependency and a round trip; local limiting is cheap and approximate. Tight limits protect the system and frustrate legitimate heavy users; the answer is usually tiers, not one number.

In an interview

Give algorithm, key, store and response in one breath: "Token bucket per API key, 100 requests per second with a burst of 200, state in Redis updated atomically with a Lua script, enforced at the gateway. Over limit returns 429 with Retry-After, and every response carries remaining and reset headers." Then the nuance: "Search and export have their own lower buckets because they cost far more than reads."