Cache Sizing Server Counts and Failure Headroom

01 The idea

The problem this lesson solves

A system can pass a normal load test and still fail immediately after losing one server or restarting its cache. Capacity planning needs both a working-set model and a failure model. This lesson turns entry counts into memory, cache misses into origin load, and tested per-instance budgets into enough surviving capacity.

You need multiplication, division and unit conversion. Keep reads, writes and bytes in separate calculations.

02 Worked example

Budget for a cold cache and one failed server

03 The method

Understand the mechanism

Estimate the working set rather than all historical data

The working set is the subset of data accessed often enough during the relevant interval to benefit from fast storage. A service may retain millions of links while most traffic repeatedly accesses a much smaller set. Caching every retained record can waste memory on cold data.

Assume 730,000 hot entries at 200 modeled bytes per entry. The payload estimate is 146,000,000 bytes, or 146 decimal MB. This is the starting point, not the final cache process memory requirement.

Keys, object metadata, allocator behavior, hash-table structures, expiration information and replication buffers can add overhead. Measure representative entries and process memory under the intended data structure. A generic “double it” multiplier is less informative than a measured breakdown.

Working sets also change over time. A campaign can introduce many new popular keys; a deployment can change key versions and effectively create a cold cache. Monitor hit ratio and eviction behavior across these transitions rather than assuming yesterday’s hot set remains valid.

Translate hit ratio into origin demand

For 2,000 peak reads/s and 95% hits, misses are 2,000×0.05 = 100/s. At 90% hits, misses are 200/s. A five-percentage-point hit-ratio decline has doubled origin read demand.

At 0% hits, origin reads become 2,000/s, twenty times the normal 100/s. The application’s external traffic can look normal while the database is overwhelmed. This is why cache failure belongs in the capacity plan rather than only in a correctness discussion.

Include the cost per miss. One miss might perform one indexed lookup, while another triggers an expensive aggregation. A single blended hit ratio can hide the expensive misses. Measure origin CPU, I/O and latency by request class.

Also separate request hit ratio from byte hit ratio and local-cache hits from shared-cache hits. A two-level cache may reduce shared-store traffic without changing how much authoritative database work follows the remaining misses.

Derive server count from a tested budget

Assume a representative load test shows that one app instance can sustainably serve 100 QPS while meeting the chosen latency and error objectives with operational headroom. For 450 peak QPS, ceil(450/100) = 5 serving instances.

The 100-QPS figure must be a budget at the required service quality, not the maximum rate before the process crashes. It also assumes the tested request mix and dependency conditions. A CPU-heavy endpoint mixed into the workload can change the budget.

Rounding up matters because fractional servers do not provide the required capacity in this simple model. Five instances provide 500 QPS of modeled serving capacity. Four provide only 400 and cannot satisfy 450 under the same assumptions.

Do not combine this 450-QPS app example with the separate 2,000-QPS cache example as though they are the same workload. They illustrate two calculations. In an actual design packet, every capacity row must refer to a consistent workload or explicitly named scenario.

Add failure capacity without counting the failed instance

To survive one instance loss while retaining five needed serving instances, provision six under the simple equal-capacity model. After one disappears, five remain and still provide the modeled 500-QPS capacity.

If you provision only five, losing one leaves 400 QPS of capacity against 450 QPS demand. Health checking can stop routing to the failed instance, but it cannot create missing capacity. The remaining instances can become overloaded even though the routing system behaves correctly.

Failure domains can be larger than one process. If an availability zone contains half the instances, losing that zone is not the same event as losing one instance. Distribute capacity and calculate survivors for the failure you promise to tolerate.

Autoscaling is not instantaneous spare capacity. Detection, provisioning, startup, warmup and readiness take time. Decide how the service handles the interval before new capacity is useful: existing reserve, bounded queueing, load shedding or a documented degraded mode.

Avoid the recovery feedback loop

Imagine a cold cache sends more reads to the database. Queries slow down, so more app requests remain in flight. Clients time out and retry, creating still more reads. If overloaded instances fail readiness checks, traffic concentrates on fewer survivors. This is a feedback loop, not several unrelated errors.

Protect the origin with bounded concurrency and admission. Coalesce same-key cache fills where appropriate, spread expirations, warm selected hot keys carefully and preserve an acceptable stale response only when the freshness contract permits it.

Do not warm the entire cache at unrestricted speed while simultaneously serving peak user traffic. The warmup itself competes for origin resources. Give it a budget and monitor whether it improves or worsens user-visible recovery.

The capacity model should include ordinary peak, cold cache, one app loss and the combinations required by the service contract. Some combinations may be deliberately out of scope, but they should be excluded explicitly rather than forgotten.

Validate the assumptions in a recovery test

Create a controlled test that starts with warm caches, records baseline origin work, clears or bypasses the relevant cache safely in a practice environment, and observes the change. Measure admitted traffic, origin concurrency, queue time, errors and recovery duration.

Remove one app from the serving pool and verify both routing and surviving capacity. An even distribution of request counts is not enough; check that tail latency and errors remain within the target. Long-lived connections can make the distribution uneven.

Compare predicted and observed load. If misses generate multiple origin operations, update the model. If per-instance performance falls when all instances share one saturated dependency, stop treating capacity as a simple linear multiplication.

A good worksheet ends with conditions: six instances suffice for one loss only under the tested 100-QPS budget, request mix and dependency capacity. Stating those conditions makes the calculation useful rather than fragile.

cache_bytes = hot_entry_count * measured_bytes_per_entry
db_reads = peak_reads * (1 - hit_fraction)
serving_instances = ceil(peak_qps / budgeted_instance_qps)
# For one-instance loss under this simple model:
provisioned_instances = serving_instances + 1

from math import ceil
print(f"cache_payload_MB={730000*200/1_000_000:.0f}")
for hit_fraction in [0.95, 0.90, 0.0]:
    print(f"origin_reads={2000*(1-hit_fraction):.0f}/s")
needed = ceil(450/100)
print(f"serving_instances={needed}, provision_for_one_loss={needed+1}")

cache_payload_MB=146
origin_reads=100/s
origin_reads=200/s
origin_reads=2000/s
serving_instances=5, provision_for_one_loss=6

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions