Storage Retention Replicas and Backup Budgets

01 The idea

The problem this lesson solves

A storage estimate often starts correctly and ends too early. Multiplying records by bytes gives logical data volume, but the deployed system also stores indexes, replicas, logs and recovery data. Deletion and retention create another set of questions. This lesson builds the budget in layers so each multiplier corresponds to something the system actually keeps.

You need multiplication, division and unit conversion. Keep reads, writes and bytes in separate calculations.

02 Worked example

Budget one year of retained records

03 The method

Understand the mechanism

Measure the object you intend to retain

Begin with a record definition. A short-link mapping may include a code, owner id, destination, timestamps and metadata. The serialized request size is not necessarily the database row size, and the row size is not the entire physical storage cost.

For the exercise, assume 500 modeled logical bytes per newly created mapping. This is an input, not a claim about every URL or database engine. Real destinations vary in length, and storage formats add headers, alignment and other overhead.

Use a distribution when sizes vary substantially. An average is useful for aggregate growth, while maximum accepted size matters for per-request memory, indexes and abuse resistance. A small number of very large values can dominate bytes even when they are rare by count.

Keep metadata and large object content separate. A database row that points to an uploaded video does not include the video’s storage merely because the application treats them as one resource. Estimate each storage system at its own boundary.

Apply creation rate and retention

At 10,000 new mappings/day and 500 bytes each, daily logical growth is 5,000,000 bytes. Retaining 365 days gives 1,825,000,000 bytes, or 1.825 decimal GB. The formula is records/day × bytes/record × retained days.

Decimal GB means 1,000,000,000 bytes; GiB means 1,073,741,824 bytes. Use one convention consistently and label it. Confusing bytes with bits creates an eightfold error; confusing GB and GiB creates a smaller but still relevant difference.

Retention is not always a simple fixed window. If users can keep links indefinitely, the active dataset grows with cumulative creations minus deletions. If the service retains tombstones, audit history or deleted records for recovery, “deleted by the user” may not mean “all physical copies removed immediately.”

Updates can create additional retained versions, logs or history even when the number of live mappings stays constant. A system with heavy editing can therefore write much more than its net live-data growth suggests.

Count live copies without confusing them with backups

If there are three complete live copies, the logical data component across them is 1.825×3 = 5.475 GB before other costs. Be explicit whether “three replicas” means three copies total or one primary plus three additional copies. Ambiguous naming causes off-by-one capacity plans.

Each full copy may also maintain indexes and local operational data. If an index estimate is for one copy, multiply it consistently with the copies that actually keep that index. Do not multiply everything blindly when different replicas or storage tiers have different structures.

Live replication serves availability, read placement or failover goals. It is not automatically a backup against accidental deletion or bad application writes, because those changes can replicate too. Recovery from a historical mistake requires a retained recovery mechanism with a suitable point in time.

Replication can also consume network bandwidth and apply capacity. A standby with enough disk but insufficient replay throughput can fall behind. Storage capacity is one part of maintaining useful copies, not the whole design.

Budget indexes, logs and free space separately

An index duplicates selected keys and references in an access structure. Its size depends on key widths, row count and implementation details. Estimate it from representative schema/data or measurement rather than a universal “indexes add exactly 20%” rule.

Transaction logs support durability and replication. Retention can grow when a replica or consumer falls behind, or when the recovery policy retains more history. Compaction, vacuum, reindexing and migration can temporarily require additional working space.

Free space is an operational requirement, not wasted capacity. A disk nearly full can turn an ordinary maintenance action into an outage. State a headroom policy and test the operations it is intended to permit.

A useful budget has separate lines: live table data, live indexes, retained logs, temporary maintenance space and safety margin. This makes it possible to revise one assumption without accidentally multiplying unrelated components.

Derive the recovery budget from recovery goals

A backup policy specifies what is captured, how often, how long it is retained and how it is restored. Full snapshots, incremental backups and log archives have different growth patterns. Some systems deduplicate or compress data; the effectiveness must be measured for the workload.

Recovery point objective describes the acceptable data-loss window. Recovery time objective describes the acceptable time to restore usable service. Frequent backup creation alone does not establish a short restore time, and a successful backup job does not prove the restored data is usable.

Estimate restore bandwidth and processing work. Moving a large snapshot to replacement storage, replaying logs, rebuilding indexes and warming caches can all contribute. Include the time needed to verify and redirect service after restoration.

Test with representative data and retain evidence. A recovery drill should check business invariants and recent records, not only whether a database process starts. The backup budget is justified by a recovery behavior the product needs.

Handle retention as a data lifecycle

Define when records expire, how deletion is executed and which dependent data follows them. Removing a mapping may require invalidating cached copies and updating indexes or projections. Historical backups may retain prior versions under a separate policy.

Large deletes can create database work and temporary storage pressure. Partitioning by time can help some retention workloads, but only if the access and ownership model fits. A time partition is not automatically a good sharding key for a hot write path.

Track actual growth against the estimate. Separate logical record count, average payload size, index bytes, log retention and backup bytes. If physical storage grows while live records stay flat, investigate retained versions, logs or index behavior instead of assuming the user forecast was wrong.

Your final budget should answer two questions: how much capacity is needed during normal operation, and how much is needed during the recovery or maintenance operation you promise to survive. A single records×bytes number answers neither completely.

logical_bytes = writes_per_day * bytes_per_record * retained_days
live_data_copies = logical_bytes * copy_count
physical_budget = live_data_copies + index_copies + logs + free_space
backup_budget = separately modeled retained recovery data
# decimal GB = bytes / 1_000_000_000

records_per_day, bytes_per_record = 10000, 500
days, copies = 365, 3
logical = records_per_day * bytes_per_record * days
print(f"logical_GB={logical/1_000_000_000:.3f}")
print(f"live_data_GB={logical*copies/1_000_000_000:.3f}")
print(f"logical_GiB={logical/1_073_741_824:.3f}")

logical_GB=1.825
live_data_GB=5.475
logical_GiB=1.700

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions