Health Checks Draining and Failure Capacity

01 The idea

The problem this lesson solves

Backend B stops responding at 10:00:00. The balancer does not instantly know this, requests already in flight do not disappear, and A and C may not have enough spare capacity. Failover is a sequence of detection, routing change and recovery. This lesson makes those stages explicit and explains how readiness, draining and retry rules interact.

Use the peak traffic and failure budgets from module 4 to judge whether an added server actually helps.

02 Worked example

Continue routing after B becomes unavailable

03 The method

Understand the mechanism

Distinguish process life from serving readiness

A liveness check asks whether the process should be considered alive under the platform’s policy. A readiness check asks whether it should receive new work. A process can be alive while still loading configuration, warming critical state or waiting for a required dependency.

Do not make every downstream failure trigger an app restart automatically. If all instances restart when the shared database slows, they may create a synchronized recovery storm without fixing the database. Readiness and liveness should reflect deliberate failure handling, not duplicate the same expensive dependency check by habit.

A useful readiness endpoint is cheap, bounded and meaningful for the work the instance will receive. An endpoint that always returns 200 while the instance cannot serve its main operation is misleading. An endpoint that executes a heavy query on every probe can itself create load.

Define startup, normal serving, degraded mode and shutdown behavior separately. The health-check contract determines which of those states is eligible for traffic.

Detection takes observations and time

An active health check runs periodically and may require several failures before removal. Probe interval, timeout, failure threshold and scheduling determine the observation window. A failure just after a successful probe may remain undetected longer than one just before a probe.

For example, checks every five seconds requiring two failed checks do not imply instantaneous removal at the moment the process fails. The exact delay depends on probe duration and alignment. During that interval, new requests may still be selected for B.

Passive signals can detect failures from real traffic, but they also need policy. A single timeout may reflect a transient network problem rather than a dead instance. Aggressive ejection can remove too much capacity; slow ejection can send too much work to a failing backend.

Recovery also needs hysteresis or readiness evidence. Immediately returning a recovering instance to full traffic can overwhelm it before caches and connections are ready. Balance rapid recovery with avoiding repeated in/out flapping.

Account for in-flight requests

Removing B from the eligible pool affects future selections after the routing change. It does not recover a request already sent to B, undo a committed operation or move an established connection to another server.

The caller or proxy may retry eligible operations, but only within their contract and deadline. A read may be repeatable while a creation needs an idempotency key. A timeout after a write reaches B leaves the familiar ambiguous-outcome problem.

The course simulation removes B before request 3 and continues from the next eligible position. Its selections are A,B,C,A,C,A. Request 2 was selected for B; the model does not claim it succeeded merely because the total selection count is six.

Keep metrics for selected attempts, completed successes, timeouts and retry outcomes. Otherwise a routing dashboard can look healthy while users experience failures during detection and recovery.

Drain an instance before stopping it

Planned shutdown is different from an unexpected crash. First stop sending new work to the instance. Then allow bounded in-flight requests to complete, close or signal persistent connections according to protocol, and terminate after the configured deadline.

The draining deadline must fit the service’s request model. A short HTTP operation can often finish quickly. A streaming connection may need an explicit client reconnection or resume protocol. Waiting forever prevents deployment; terminating immediately can interrupt avoidable work.

Ensure the app stops accepting new local work as it drains. A durable background job should either complete safely or be recoverable by another worker. A task that existed only in process memory can be lost even when HTTP draining looks correct.

Test the order of events in the actual deployment environment. Readiness changes, balancer propagation and process termination are not necessarily simultaneous. A termination grace period is useful only when it covers the observed routing and completion behavior.

Provision survivors and protect them from overload

Suppose three instances each have a safe budget of 100 QPS and the service receives 240 QPS. Normal operation has 300 QPS modeled capacity. Losing one leaves 200 QPS against 240 QPS demand, so successful removal of the failed instance still creates overload.

The remaining instances need reserve capacity, or the service needs a defined degraded/admission policy until capacity returns. Autoscaling has a delay; it cannot retroactively serve requests that timed out while new instances started.

Retries can worsen the shortfall. If failed requests immediately retry against A and C, their offered load rises beyond the original 240 QPS. Bound retries, apply backoff and shed work that cannot finish within useful deadlines.

Health policies must avoid a cascade in which overloaded survivors fail checks and are removed, concentrating traffic further. The response may involve preserving partially useful capacity under controlled admission rather than declaring every slow instance dead.

Include the balancer and configuration in the failure model

A single load-balancer process can be a failure point even when the application pool is redundant. Managed entry services or redundant proxy deployments have their own failure domains, configuration propagation and capacity limits. State what the deployment guarantees rather than drawing one box and assuming it is highly available.

Configuration mistakes can affect all healthy instances at once. A bad route, certificate or health-check path may make the whole pool unreachable. Rollout validation, staged changes and a tested rollback path therefore matter alongside machine redundancy.

Run failure drills that cover one app crash, one slow backend, planned draining, a cold replacement and an entry-point configuration change. Observe user outcomes and recovery time, not merely whether probes eventually turn green.

Failover is complete only when the service resumes acceptable behavior within its promised boundary. Detection, selection, capacity and operation recovery each contribute a different part of that result.

# Observational commands; responses vary over time
dig example.com A
curl -I https://example.com

# Simplified routing policy, not a full proxy configuration
eligible = instances whose readiness is healthy
backend = next_round_robin(eligible)
forward request using a bounded upstream deadline

05 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

06 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

07 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

08 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions