System Design · Module 1 · Lesson 4 of 4 · 40 min

Rate Limiting Across Multiple Servers

Design one client budget across many app instances, compare fixed windows with token buckets, and specify a limiter-outage policy.

Trace the global-limit bug →

The logical budget must have one coordinated owner even when requests reach different application instances.

01 The idea

The problem this lesson solves

One client may make 3 accepted requests in this window. Five requests arrive; all belong to that same window.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

Fixed windowA counter reset or replaced at a defined time boundary.

Token bucketA budget that refills over time and allows a defined maximum burst.

Fail-openA policy that permits work when the limiter cannot provide a decision.

02 Worked example

Follow one worked example

App A: Same client • App B: Same client • Atomic budget: One decision • Protected API: Allowed calls.

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

Set the budgetOne client may make 3 accepted requests in this window. Five requests arrive; all belong to that same window.

Try local countersWith two app servers and round-robin routing, server A accepts requests 1, 3 and 5. Server B accepts requests 2 and 4. All five pass, exceeding the intended budget.

Share the decisionA shared atomic counter sees 1, 2, 3, 4 and 5. Only the first three are allowed. The last two receive 429; they do not call the protected API.

Name the remaining limitThis fixes cross-server counting, but a fixed window still permits a boundary burst. A requirement about any rolling 60 seconds needs a different algorithm.

The logical budget must have one coordinated owner even when requests reach different application instances.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Choice or concept	Purpose	Boundary or cost
Local counter per instance	No network hop	Global budget multiplies with instances
Shared atomic counter	One logical decision	Network latency and a shared dependency
Fixed window	Small state and easy expiry	Burst near a window boundary
Token bucket	Explicit burst allowance	Needs coordinated token and timestamp updates

Step 3: define data and the API

A conceptual decision API takes a client identifier, a policy and the current window. Its response includes allowed or denied and when a denied client can try again. Authentication supplies the client identity; trusting an arbitrary client_id supplied by the caller would let that caller evade a limit. A shared counter key can contain the authenticated client and the window number. An increment and its first-time expiry must be coordinated atomically so a crash cannot leave a counter without an expiry.

Step 4: trace the high-level design

An application authenticates the request, asks the limiter for a decision, and calls the protected service only if allowed. All application instances consult the same logical counter for this client and window. Saying ‘use Redis’ is not the algorithm: the decision needs an atomic operation or script with specified expiry behavior. Several independent local counters enforce several independent budgets, which violates the global invariant.

Step 5: stress the design

Ask what happens when two requests arrive together, the counter service times out, a process restarts, and a new minute begins. Fail-open permits traffic when the limiter is unavailable; fail-closed denies it. Fail-open may suit a low-risk browsing endpoint, while an expensive endpoint may prefer a bounded emergency allowance. Neither choice is universally correct. Fixed windows also allow a boundary burst: 3 requests immediately before the boundary and 3 immediately after can pass in a short interval. Sliding windows or token buckets address different burst requirements.

Track decisions, not application machines

Suppose a client may make three accepted requests in one fixed minute. If two app servers keep separate counters, alternating traffic can spend two budgets. A process-local counter is correct only for an explicitly process-local policy or a carefully designed partition of ownership.

A shared store can coordinate the decision, but separate GET and SET operations still race. The increment, threshold decision and creation-time expiry must have an atomic contract. A store name alone does not prove atomicity. Authentication must supply the caller identity, otherwise the client can evade the limit by inventing a new key.

Describe bursts precisely

A fixed-minute rule can accept three calls just before the minute changes and three just afterward. A token bucket instead refills tokens continuously up to a capacity. A bucket with capacity three and refill one token per second may admit an immediate burst of three, then approximately one call each second after that.

These are different product policies. “Three per minute” is ambiguous unless you state fixed windows, a rolling window or a token-bucket interpretation. Clocks, request ordering and storage consistency matter when several instances update the same logical bucket. The console focuses on the shared-budget mistake and deliberately does not model all those mechanisms.

A limiter also consumes capacity

Denied requests still need a decision. A malicious caller can overload the counter service even if the protected operation is never reached. Bound request parsing, store deadlines and connections; consider coarse admission controls before fine per-client policy.

When the store times out, choose the documented policy. A low-cost public read may use a conservative local fallback. An expensive generation job may deny the request to bound spend. Monitor fallback decisions separately so the dashboard does not silently count unprotected traffic as healthy enforcement.

Guided tutorial

1. Predict the local result

Route five requests alternately to A and B with a limit of three per client.

2. Use the shared model

Run both presets and compare accepted totals.

3. Test a boundary

Place requests on each side of a fixed-window boundary and state the actual guarantee.

4. Specify outage behavior

Write one policy for public reads and one for a costly job, including a store deadline and alert.

policy: 3 requests per client per fixed 60-second window
identity: authenticated API client
state: counter(client, window), expires after window
decision: atomic_increment(counter) <= 3
on denied: return 429 with retry guidance
on counter timeout: apply the documented failure policy

05 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

Choice or concept	Remember	Check
Local counter per instance	No network hop	Global budget multiplies with instances
Shared atomic counter	One logical decision	Network latency and a shared dependency
Fixed window	Small state and easy expiry	Burst near a window boundary
Token bucket	Explicit burst allowance	Needs coordinated token and timestamp updates

Fixed windowA counter reset or replaced at a defined time boundary.

Token bucketA budget that refills over time and allows a defined maximum burst.

Fail-openA policy that permits work when the limiter cannot provide a decision.

06 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

Primary reference

HTTP 429

Read HTTP 429

Primary reference

Redis atomic scripting

Read Redis atomic scripting

State the implementation version and the workload assumptions when defending your own design.

07 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

What does Fixed window mean?

A counter reset or replaced at a defined time boundary.

What does Token bucket mean?

A budget that refills over time and allows a defined maximum burst.

What does Fail-open mean?

A policy that permits work when the limiter cannot provide a decision.

Why does a shared budget require atomic updates?

Two instances can read the same earlier value and each allow work. The check and state transition must be coordinated for the chosen policy.

Is every three-per-minute limiter equivalent?

No. Fixed windows, rolling windows and token buckets permit different bursts. Define the accepted traffic pattern before selecting an algorithm.

What identity belongs in the counter key?

A trusted authenticated caller or a deliberately chosen network identity, plus the relevant policy scope. An arbitrary untrusted client_id is easily rotated.

Does fail-open always improve availability?

1. Count a budget

All six can pass, while a shared global budget of two should admit only two.

2. Choose identity

The caller can rotate the string to obtain new budgets. Use authenticated identity or another explicitly defended identity policy.

3. Availability budget

30 × 24 × 60 = 43,200 minutes. The unavailable fraction is 0.0005, so the budget is 21.6 minutes.

4. Boundary burst

Send four at t=9.8 seconds and four at t=10.2 seconds. Each fixed window accepts four, yet eight are accepted in 0.4 seconds. A rolling-window requirement would reject this behavior.

5. Failure decision

Article reads could use a bounded local fallback with conservative limits; accepted traffic may exceed the global budget. GPU requests could fail closed with a retryable response to bound spend, at the cost of failed requests during the outage.

6. Design review

Use authenticated identity and an atomic shared decision keyed by client and window. Explain expiry and denied responses. Residual risks include shared-store unavailability, a hot client key, and boundary bursts; specify a failure policy, monitoring and a more suitable algorithm if rolling limits are required.