Cache Sizing Server Counts and Failure Headroom

01 The idea

The problem this lesson solves

730,000 hot entries at 200 modeled bytes each need 146 MB before overhead.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

02 Worked example

Follow one worked example

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

Capacity must cover the workload during the failure you promise to tolerate, not only the warm steady state.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Estimate the hot set, not the whole database

Suppose 20 percent of the year's 3.65 million links should stay cached and each in-memory entry takes 200 bytes including its modeled key and metadata. That is 730,000 entries × 200 = 146 MB before allocator overhead and replication. These values must be measured for the chosen data structure. A hot-set percentage is about which items receive requests; it is not automatically the cache hit rate, which depends on the distribution of requests.

A hit rate changes the backing-store budget

With a 90 percent cache hit rate, database reads are 57.87 × (1 − 0.9) ≈ 5.79 QPS, excluding refreshes and writes. A cold cache drops the hit rate to zero and sends about 57.87 read QPS to the database. That is ten times the warm-cache read demand. The cache does not eliminate the need for overload protection or a recovery plan. Test both steady state and warm-up.

Convert a measured service rate into a starting count

Assume a load test finds one app instance can sustain 100 requests/s at the required latency for this workload. At 60 percent target utilization, budget 60 requests/s per instance. ceil(57.87 ÷ 60) = 1 serves the modeled read peak. Two instances can be a starting point for surviving one instance loss at that load, but this does not establish database, load-balancer or zone resilience. Capacity tests must include writes and realistic connection behavior.

Translate the result into a decision

Our small modeled workload does not justify sharding. A relational database, a modest stateless application pool, and a measured caching plan are reasonable candidates to test. If retention or traffic increases by 1,000×, rerun the arithmetic and locate the first limit. Use estimates to choose what to benchmark and where to investigate; a spreadsheet cannot prove tail latency, replication behavior or failure recovery.

Hot objects and popular requests are different

Caching twenty percent of the records does not imply twenty percent hits. Request popularity may concentrate heavily on a small set, or it may be almost uniform. Measure the access distribution and choose the working-set interval before selecting a cache size.

The yearly model contains 3.65 million links. Twenty percent is 730,000 entries. At an assumed 200 bytes per in-memory entry, that is 146 MB before allocator and replication overhead. Measure actual key, value and metadata costs in the chosen cache; a database row’s byte count is not automatically its in-memory footprint.

Model both the warm and cold paths

At 2,000 read QPS with 95 percent hits, the database receives about 100 read QPS before other work. At zero hits it receives all 2,000. A cache restart therefore creates a twenty-times increase in modeled read demand even though incoming client traffic stays constant.

A recovery plan can combine gradual warming, coalescing, concurrency limits and bounded stale responses where allowed. The cache lesson develops those controls; the sizing lesson makes their need visible in the worksheet. Do not assume unlimited fallback capacity merely because steady-state database charts look quiet.

Reserve the promised surviving capacity

Suppose load testing establishes a budget of 100 QPS per app instance at the required latency, and the target load is 450 QPS. Five instances serve it normally. To keep five after losing one, provision at least six under this simplified one-instance model.

A zone failure can remove several instances and a shared dependency, so the same arithmetic must match the actual failure domain. Capacity measurements should include the expected mix of reads and writes, connection reuse and realistic dependency latency. The worksheet identifies what to test; it does not replace the test.

Guided tutorial

1. Separate two percentages

Write object coverage and request hit fraction in different columns.

2. Size entries

Include modeled keys, values, metadata and copy count.

3. Add a cold row

Compute database demand when every read misses.

4. Remove capacity

Cross out one instance or one zone and recompute whether the remaining budget meets demand.

cache_bytes = hot_entry_count * measured_bytes_per_entry
db_reads = peak_reads * (1 - hit_fraction)
serving_instances = ceil(peak_qps / budgeted_instance_qps)
# For one-instance loss under this simple model:
provisioned_instances = serving_instances + 1

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions