Sharding Strategies and Choosing a Partition Key

01 The idea

The problem this lesson solves

Map short code a7 to a shard owner.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

02 Worked example

Follow one worked example

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

The shard key determines ownership and locality, so evaluate both the main lookup and the important secondary access paths.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Choose a shard key from access patterns

Range sharding keeps nearby keys together and can make range scans efficient, but a growing time range can concentrate all new writes on one shard. Hash sharding spreads keys more evenly under suitable assumptions, but loses range locality. Directory routing maps logical partitions to physical owners and gives migration flexibility at the cost of maintaining reliable routing metadata. A key that places all of one tenant's data together helps tenant queries but can create an oversized tenant shard.

One primary partitioning choice cannot optimize every query

For a short-link service, sharding by short code makes redirect lookup a single-shard operation. An owner's ‘list all my links’ query then spans many code partitions unless you maintain a separate owner index or materialized view. Sharding by owner solves the listing but requires another route from code to owner for redirects. State which access path matters most and how the secondary path stays correct. A globally unique code also needs a deliberate generation or collision-check strategy.

Cross-shard queries and transactions are design costs

A global top-k query may ask each shard for its local top k and merge the candidates under the same total order. A scattered query's latency is affected by its slowest participant; partial failures need a policy. Global uniqueness, joins and all-or-nothing writes across shards may require extra coordination or a workflow with compensating actions. Do not pretend a transaction that is easy inside one SQL database remains equally simple after partitioning.

Sharding solves a particular limit

Read replicas copy data; shards divide ownership. If reads are the only limit and can tolerate lag, copying may help without changing all data routing. If write throughput, size or tenant isolation requires partitioning, sharding may be justified. Diagnose the limit before accepting the added routing and migration costs.

Range placement preserves useful locality but can concentrate a growing newest range. Hash placement can distribute keys under suitable assumptions but does not preserve ranges. Directory placement records an explicit owner mapping and offers flexibility while making reliable routing metadata part of the serving system.

Work both short-link access patterns

Redirects begin with a short code. Sharding by that code makes the primary lookup a single-shard operation. The owner’s “all my links” page begins with owner_id instead, so the owner’s codes may be spread across partitions.

Choose a secondary owner index, a maintained projection or a bounded scatter query with a defined consistency policy. Sharding by owner reverses the trade-off: owner listings become local while redirects need a route from code to owner. One partition key cannot automatically optimize every query. Code uniqueness also needs an explicit generation or collision-handling contract.

Cross-shard work has coordination costs

A global top-k query can request each shard’s local top k under one total order and merge candidates. The fan-out waits on several participants and can fail partially. A global join or all-or-nothing update across partitions needs more coordination than a local transaction.

State what partial results mean and bound the fan-out. If one tenant can grow much larger than others, an owner-based shard may become a hotspot even with many tenants overall. Plan thresholds and migration paths from measured size and demand; uniform key count does not guarantee uniform work.

Guided tutorial

1. List queries

Write lookup keys, expected result size and consistency for redirect and owner listing.

2. Compare three keys

Evaluate code, owner and creation date against those queries.

3. Test skew

Add one owner with half the workload and see where it lands.

4. Plan the secondary path

Define how owner listings stay consistent enough after a code-keyed write.

redirect: code → partition owner → mapping lookup
owner listing: owner_id → secondary index / projection / scatter
range placement: preserves selected ordering, may skew
hash placement: spreads suitable keys, loses range locality
directory placement: explicit logical owner metadata

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions