Capstone Design a Short Link Service End to End

Replication Sharding and the Capstone · 50 min

System Design · Module 7 · Lesson 4 of 4 · 50 min

Capstone Design a Short Link Service End to End

Assemble requirements APIs schema capacity routing and caching into one defended design, then review retries freshness and failure recovery.

A complete design connects each guarantee to the request path and state transition that enforces it.

01 The idea

The problem this lesson solves

Write features, excluded analytics, mutable-destination policy and measurable targets.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

Design packetThe linked set of requirements, interfaces, schema, estimates, diagram and failure review.
Trade-off logA record of a choice, its benefit, cost and revisit trigger.
Failure domainThe set of resources that can be lost together in a modeled incident.

02 Worked example

Follow one worked example

Capstone Design a Short Link Service End to End worked diagram01ClientsCreate / redirect02Stateless APIAuth + contract03Eligible cacheBounded freshness04Durable databaseIdentity + recoveryCapstone Design a Short Link Service End to End compact diagram01ClientsCreate / redirect02Stateless APIAuth + contract03Eligible cacheBounded freshness04Durable databaseIdentity + recovery
Clients: Create / redirect • Stateless API: Auth + contract • Eligible cache: Bounded freshness • Durable database: Identity + recovery.

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

1
SpecifyWrite features, excluded analytics, mutable-destination policy and measurable targets.
2
BuildDefine management endpoints, redirect path, unique code schema and scoped operation identity.
3
SizeEstimate external traffic, origin misses, storage and capacity during the chosen failure.
4
ReviewWalk a retry, stale fill, cold cache, missing app and lagging follower; record the user-visible outcomes.

A complete design connects each guarantee to the request path and state transition that enforces it.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Choice or conceptPurposeBoundary or cost
CreationAuthenticated idempotent management actionDurable scoped replay and unique code
RedirectLookup plus documented redirectMissing vs unavailable; cache freshness
CapacityWorkload and surviving budgetsWarm, cold and failure tests
GrowthMeasured revisit thresholdSecondary paths and migration cost

Exhaust the simpler explanations first

Inspect slow queries, missing indexes, connection-pool saturation, lock contention and unnecessary work before splitting data. A single well-sized relational database can serve many workloads. If reads dominate and can tolerate lag, replicas may help. If one node cannot hold or serve the write workload despite reasonable tuning, evaluate sharding against its operational cost. Base the decision on measured constraints and growth forecasts.

One primary partitioning choice cannot optimize every query

For a short-link service, sharding by short code makes redirect lookup a single-shard operation. An owner's ‘list all my links’ query then spans many code partitions unless you maintain a separate owner index or materialized view. Sharding by owner solves the listing but requires another route from code to owner for redirects. State which access path matters most and how the secondary path stays correct. A globally unique code also needs a deliberate generation or collision-check strategy.

Cross-shard queries and transactions are design costs

A global top-k query may ask each shard for its local top k and merge the candidates under the same total order. A scattered query's latency is affected by its slowest participant; partial failures need a policy. Global uniqueness, joins and all-or-nothing writes across shards may require extra coordination or a workflow with compensating actions. Do not pretend a transaction that is easy inside one SQL database remains equally simple after partitioning.

Connect the artifacts

Start with create and redirect in one region. Define whether links can change, who can manage them, how long they live and what success means. Use scoped idempotency for management creations and a unique code in durable storage. A public redirect begins with the code; an owner listing begins with owner identity.

Trace both operations through the same architecture. Management requests authenticate and authorize. A redirect may use an eligible cached mapping but must distinguish missing from unavailable. The cache key and freshness policy must match destination mutability. Each diagram arrow should name an operation and the relevant identity.

Let the capacity worksheet constrain complexity

Use the course’s declared workload first: 100,000 DAU, ten redirects each per day, ten thousand creations daily and a five-times peak factor. This gives approximately 57.87 peak read QPS and 0.579 peak creation QPS before additional work. One-year logical mapping data at 500 bytes each is 1.825 GB.

These numbers alone do not justify sharding. A relational source of truth with measured queries and a modest stateless app pool is a reasonable candidate to test. If growth later requires partitioning, explain code routing and the owner-listing secondary path. Keep the revisit threshold measurable instead of declaring the first design infinitely scalable.

Defend the five incidents

A lost create response needs replay of the original result. A stale fill needs the declared freshness contract or a stronger path. A cold cache needs bounded origin work. An app failure needs routing removal, safe retry and surviving capacity. Replica lag needs a client consistency policy.

For each incident, write what the user sees, who owns recovery and what signal reveals the problem. Avoid a diagram that silently assumes every dependency is redundant or every failure is retryable. The review is finished when another reader can execute both the happy path and the incident paths using only your packet.

Guided tutorial

1. Assemble the service

Draw client → DNS/entry → load balancer → stateless app → cache → database. Add create and redirect endpoints, owner authorization, code uniqueness and the conditional behavior of mutable destinations. Separate management APIs from public redirects.

2. Attach a capacity worksheet

Reuse the estimation module's declared workload. Estimate reads, writes, retained data, misses and capacity during one instance loss. Decide whether a single relational database is still enough before drawing shards.

3. Defend failure behavior

Explain a lost create response, stale cache fill, cold cache, app-server failure and replica lag. For each, state the user-visible result, a recovery mechanism and one metric that reveals the failure.

4. Deliver a reviewable packet

Produce one page of requirements, one diagram, an API table, a schema sketch, a capacity worksheet and a trade-off log. If you choose sharding, include the secondary owner-listing path and a migration plan; otherwise document the threshold that would make you reconsider.

requirements → API and authorization table
API → schema keys and atomic transitions
workload → QPS, storage and failure capacity
request flow → routing, cache and authoritative store
incident review → user result, recovery owner and metric
trade-off log → choice, cost, risk and revisit trigger

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

Choice or conceptRememberCheck
CreationAuthenticated idempotent management actionDurable scoped replay and unique code
RedirectLookup plus documented redirectMissing vs unavailable; cache freshness
CapacityWorkload and surviving budgetsWarm, cold and failure tests
GrowthMeasured revisit thresholdSecondary paths and migration cost
Design packetThe linked set of requirements, interfaces, schema, estimates, diagram and failure review.
Trade-off logA record of a choice, its benefit, cost and revisit trigger.
Failure domainThe set of resources that can be lost together in a modeled incident.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

Primary reference
PostgreSQL replication

Read PostgreSQL replication

Primary reference
PostgreSQL synchronous replication

Read PostgreSQL synchronous replication

Primary reference
Dynamo paper

Read Dynamo paper

State the implementation version and the workload assumptions when defending your own design.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

What does Design packet mean?
The linked set of requirements, interfaces, schema, estimates, diagram and failure review.
What does Trade-off log mean?
A record of a choice, its benefit, cost and revisit trigger.
What does Failure domain mean?
The set of resources that can be lost together in a modeled incident.
What makes this more than separate documents?
The identities and guarantees agree across requirements, API, schema, diagram and tests. Each artifact constrains and explains the others.
Why is sharding optional in the capstone?
The declared traffic and data size do not alone require it. A simpler candidate should be tested before accepting partitioning costs.
Where is code uniqueness enforced?
At the authoritative creation boundary using the chosen generation/collision strategy and durable uniqueness rule. A cache lookup is not a sufficient global guard.
What belongs in the failure table?
The trigger, affected request, user-visible result, recovery policy, responsible component and a useful observation or metric.
How do you defend a future scaling change?
Name a measured limit or forecast threshold, the benefit of the change and its new operational and consistency costs.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Identify the primary path

Easy
Name the main lookup key for a public redirect and for an owner listing.
Follow-up
State the assumption or edge case that makes the answer correct.
Show the hint
The entry identities differ.

Name the authority

Easy
Choose where durable code ownership belongs in the initial design.
Follow-up
State the assumption or edge case that makes the answer correct.
Show the hint
The copy may disappear.

Capacity check

Medium
Compute peak reads for 50,000 DAU, eight reads each daily and a four-times peak factor.
Follow-up
State the assumption or edge case that makes the answer correct.
Show the hint
Use a full-day conversion.

Unknown create

Medium
A client times out after creation may have committed; specify the next safe request.
Follow-up
State the assumption or edge case that makes the answer correct.
Show the hint
Reuse one logical action identity.

Mutable destination

Medium
State the cache decision if updates must be visible immediately to new reads after acknowledgment.
Follow-up
State the assumption or edge case that makes the answer correct.
Show the hint
Plain TTL does not eliminate late fills.

Capstone defense

Hard
Submit the six-part short-link design packet from the tutorial and justify whether it needs sharding under the the estimation module workload.
Follow-up
You must handle retries, cache failure and read-after-write behavior as well as peak reads.
Show the hint
Start with a simpler candidate and name measurable expansion triggers.

Worked solutions

1. Identify the primary path

A redirect starts with code; an owner listing starts with authenticated owner identity.

2. Name the authority

The relational source of truth owns the mapping and uniqueness contract; an eligible cache is replaceable serving state.

3. Capacity check

400,000 / 86,400 × 4 ≈ 18.52 peak read QPS before extra internal work.

4. Unknown create

Retry the same scoped idempotency key and payload under the documented retry window, then replay or recover the stored result. A new key can mean a new action.

5. Mutable destination

Use a coordinated stronger freshness path or bypass cache for these mutable reads. Define in-flight behavior and test the stale-fill race.

6. Capstone defense

A defensible initial design uses stateless app instances, an authenticated management API with scoped idempotency, a relational source of truth with unique codes, and cache-aside only under a stated freshness policy. the estimation module's modeled load alone does not justify sharding. Include overload limits, instance-loss capacity and a replica read policy if replicas are used. If growth forces sharding, explain code routing, the owner index, migration and cross-shard costs.