System Design · Module 5 · Lesson 4 of 4 · 40 min
Health Checks Draining and Failure Capacity
Trace removal of an unhealthy server, distinguish readiness from liveness, and test whether surviving capacity can handle the peak.
Step through a backend removal →01 The idea
The problem this lesson solves
For six equal-cost requests, round robin over A,B,C produces A,B,C,A,B,C. Each receives two.
Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.
02 Worked example
Follow one worked example
Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.
Failure detection, safe request completion and sufficient surviving capacity are three different obligations.
03 The method
Mechanisms decisions and a guided tutorial
Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.
| Choice or concept | Purpose | Boundary or cost |
|---|---|---|
| Readiness removal | Stops new assignments | Detection and propagation take time |
| Draining | Lets in-flight work finish | Needs a deadline and stream policy |
| Safe retry | Recovers an eligible request | Requires operation semantics |
| Failure headroom | Handles load after removal | Health checks do not add capacity |
Health checks have a detection window
A readiness probe determines whether an instance should receive new requests; liveness asks whether a process should be restarted. These are distinct decisions. With checks every five seconds and two consecutive failures required, detection is roughly one to two check intervals plus probe timeout and scheduling effects, depending on when the failure occurs. Do not promise immediate removal. A process can pass a shallow probe while real requests fail, but making every instance fail readiness whenever a shared dependency blips can also turn partial degradation into a total outage.
Removal does not replay an in-flight write safely
If a backend fails after committing a write but before responding, the balancer cannot infer whether the operation happened. Retry only when the operation's semantics and body replayability permit it; the idempotency contract from the API lesson matters here. During planned deployments, stop assigning new requests, drain existing work for a bounded time, and terminate only afterward. Long-lived streams need an explicit drain and reconnect policy.
Budget for the failure domain
Three instances at 100 budgeted requests/s each serve 300 QPS in normal operation. After losing one, they serve only 200 QPS. If the peak is 240 QPS, a green health-check system cannot prevent overload. Place redundant entry points and instances across the failure domains you promise to tolerate, then test the surviving capacity of the cache and database as well. Redundancy without a recovery test is only a diagram.
A probe has a timeline
If checks run every five seconds and require two failures, a real backend can continue receiving selections while detection proceeds. Probe timeout, scheduling and the moment of failure affect the removal time. A green shallow probe can also miss an application-path failure.
Avoid the opposite extreme of making every instance unready during any shared database blip. Removing all app capacity together can prolong an outage. Define local readiness, shared dependency degradation and process restart behavior separately, and test a shared-failure scenario.
Draining addresses planned termination
A rollout should stop new routing to an instance, wait for propagation, allow in-flight work up to a deadline, then terminate it. Long-lived streams need a reconnect policy. Existing writes can finish even when their clients lose the response.
A proxy must not blindly replay every POST after a connection closes. The backend may have committed it. The idempotency contract and request-body replayability determine whether a retry is safe. Health checking does not resolve that unknown outcome.
Health is not spare capacity
Three instances budgeted at 100 QPS each supply 300 QPS normally. Losing one leaves 200. A target of 240 QPS therefore exceeds surviving capacity even after the balancer perfectly removes the dead backend.
Count the failure domain you promise to tolerate. Losing a zone can remove multiple instances, a cache node and a network route at once. Test the combined remaining system rather than validating only the balancer’s list of healthy targets. Define admission or degraded behavior when demand exceeds the tested budget.
Guided tutorial
1. Inspect DNS
Run dig example.com A if dig is available. Find the answer's TTL and address. Repeat and explain why a cached answer might have a different remaining TTL. Record observations rather than assuming a fixed IP.
2. Trace the pool
Run both interactive presets. Record the selected backend for each request. Confirm the failed backend receives no new selection after its removal.
3. Write a readiness contract
List which local checks must pass to accept a redirect request. Separate a local initialization failure from a shared database slowdown and specify degraded behavior for the latter.
4. Rehearse a rollout
Write the sequence: mark one instance unready, wait for routing propagation, drain in-flight work within a deadline, replace it, warm it, then return it to the pool. State how many instances remain and whether they can serve the peak.
# Observational commands; responses vary over time
dig example.com A
curl -I https://example.com
# Simplified routing policy, not a full proxy configuration
eligible = instances whose readiness is healthy
backend = next_round_robin(eligible)
forward request using a bounded upstream deadline05 Cheat sheet
Review the decisions
Use this table to explain the mechanism and its limit without memorizing a product name.
| Choice or concept | Remember | Check |
|---|---|---|
| Readiness removal | Stops new assignments | Detection and propagation take time |
| Draining | Lets in-flight work finish | Needs a deadline and stream policy |
| Safe retry | Recovers an eligible request | Requires operation semantics |
| Failure headroom | Handles load after removal | Health checks do not add capacity |
06 Where & why
Check the primary references
Use official references for protocol and implementation details. The questions here are teaching prompts.
07 Interview questions
Explain the reasoning out loud
Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.
What does Readiness mean?
What does Liveness mean?
What does Draining mean?
Why distinguish readiness from liveness?
Does removing a target recover its in-flight request?
Why can a green probe coexist with user errors?
What should happen before terminating a healthy old instance?
What must be measured during failure testing?
08 Practice problems
Practice a new case
Solve the new cases before revealing hints. Complete worked solutions follow the set.
Readiness decision
EasyShow the hint
In-flight work
EasyShow the hint
Failure capacity
MediumShow the hint
Drain timeout
MediumShow the hint
Shared probe failure
MediumShow the hint
Deployment incident
HardShow the hint
Worked solutions
1. Readiness decision
It should remain unready until the required initialization succeeds, under the documented deployment policy.
2. In-flight work
No. The operation may have committed or failed. Use its retry/recovery contract rather than infer the outcome from target removal.
3. Failure capacity
Three survivors provide 240 QPS, below 260. At least five such instances are needed to retain four survivors and 320 QPS after one failure under this model.
4. Drain timeout
Define a reconnect/resume mechanism or documented termination behavior at the deadline. Coordinate retries and state recovery instead of assuming every stream finishes.
5. Shared probe failure
All frontends may disappear simultaneously and need time to recover readiness. Design degraded behavior and thresholds so a shared transient failure does not automatically remove every useful serving path.
6. Deployment incident
An old instance may commit before losing its response; an automatic retry creates another object. Drain existing work, bound retry behavior, and implement durable scoped idempotency keys in the API. Monitor retries and duplicate-key hits to verify the change.