System Design · Module 2 · Lesson 3 of 4 · 40 min

Idempotency Timeouts and Safe Retries

Make one logical creation survive response loss and concurrent retries without silently creating another object.

Simulate an ambiguous timeout →

A timeout leaves the outcome unknown; a durable operation identity lets a retry recover that outcome.

01 The idea

The problem this lesson solves

The client sends POST /v1/links with key k1. The server saves code a7 and the replay response together.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

Idempotency keyA stable identifier reused for retries of one logical operation.

FingerprintA representation of the original payload used to detect mismatched key reuse.

Replay recordA durable record linking an operation identity to its completed result.

02 Worked example

Follow one worked example

Client: Send operation k1 • Server: Commit a7 • Network: Response lost • Client: Retry k1 → a7.

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

First attemptThe client sends POST /v1/links with key k1. The server saves code a7 and the replay response together.

Lose the responseThe client times out after the commit. No acknowledgment reaches it, but a7 already exists.

Retry the same operationThe client resends the same payload with k1. The server finds the completed record and returns a7 again. One object exists.

Change the payloadThe client sends k1 with a different destination. The server rejects the conflict. A new logical creation needs a new key.

A timeout leaves the outcome unknown; a durable operation identity lets a retry recover that outcome.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Choice or concept	Purpose	Boundary or cost
Same key, same payload	Replay original result	Requires retained durable record
Same key, changed payload	Reject conflict	Do not silently reuse a result
Concurrent same key	One winner	Constraint + transaction
Remote effect	Stable downstream operation	Recovery still needs a workflow

A timeout means unknown outcome

Suppose the server commits a link, but its response is lost. The client does not know whether creation happened. Give one logical creation an idempotency key and reuse that key on retry. Scope the key to the authenticated caller and operation. Persist a fingerprint of the request and the completed response. A reused key with a different body must produce a documented conflict instead of silently returning an unrelated link.

Make concurrent retries safe

Two requests can carry the same key at the same time. A read-then-insert check is not enough: use a unique constraint on the scoped key, claim the operation atomically, and commit the created link and replay record in the same transaction where possible. Define behavior for in-progress work and a key retention period. If the operation includes an external side effect, a local transaction cannot make that remote action atomic; pass a stable key downstream or design a recovery workflow.

The dangerous gap is after commit

A client sends a creation request. The server commits a new link and then loses its connection before the response is delivered. The client observes a timeout, but the system already contains the link. Sending another ordinary POST can create another record.

Give the logical action an identity before the first attempt. Reuse it for every retry of that action. A genuinely new action needs a different identity even if its payload happens to match. Scope the key to a caller and operation so one tenant cannot replay another tenant’s result.

One transaction must decide the winner

Persist a unique scoped key, a request fingerprint, the resource identity and enough response data to replay the promised outcome. Claiming the key, creating the resource and saving the replay result should commit atomically when they share a database. Two separate “check key absent” reads are not a concurrency strategy.

Specify what a competing request sees while the winner is in progress: it may wait within a deadline or receive a documented in-progress response. If the winner rolls back, another attempt can proceed. If it commits, a later attempt receives the committed result. Reusing the key with a different fingerprint must be rejected.

Retention and external effects need separate decisions

A replay record retained for one day protects retries within that window. A retry after expiration may be treated as new, so align retention with the caller’s expected retry behavior and provide an operation lookup when needed. Business uniqueness can supply a stronger independent guard.

A local transaction cannot atomically commit a remote payment. Propagate a stable downstream key when the remote API supports it and design reconciliation for ambiguous outcomes. Bound retry attempts by a deadline and use backoff with jitter for eligible transient failures. Never turn every validation error into repeated load.

Guided tutorial

1. Trace the gap

Separate receive, commit, response send and response receipt on a timeline.

2. Run the presets

Compare object counts with and without a durable key.

3. Design the row

List scope, key, fingerprint, state, resource id, result and expiry.

4. Write a recovery table

Cover same payload, different payload, in-progress request, rollback and key expiry.

UNIQUE(caller_id, operation, idempotency_key)
BEGIN
  claim key and fingerprint
  create resource
  save completed replay result
COMMIT
retry same key + same payload → return saved result

05 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

Choice or concept	Remember	Check
Same key, same payload	Replay original result	Requires retained durable record
Same key, changed payload	Reject conflict	Do not silently reuse a result
Concurrent same key	One winner	Constraint + transaction
Remote effect	Stable downstream operation	Recovery still needs a workflow

Idempotency keyA stable identifier reused for retries of one logical operation.

FingerprintA representation of the original payload used to detect mismatched key reuse.

Replay recordA durable record linking an operation identity to its completed result.

06 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

Primary reference

HTTP method semantics

Read HTTP method semantics

Primary reference

HTTP status codes

Read HTTP status codes

State the implementation version and the workload assumptions when defending your own design.

07 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

What does Idempotency key mean?

A stable identifier reused for retries of one logical operation.

What does Fingerprint mean?

A representation of the original payload used to detect mismatched key reuse.

What does Replay record mean?

A durable record linking an operation identity to its completed result.

Why not deduplicate only by payload?

Two intentional actions can have identical payloads. An explicit operation identity distinguishes a retry from a new user action.

Why persist the replay result?

A process restart must not erase the relationship between an accepted operation and its outcome. Memory-only deduplication loses that relationship.

What does the unique constraint protect?

It prevents concurrent requests from independently claiming the same scoped operation identity. The surrounding transaction must still preserve resource and result consistency.

How should a mismatched payload be handled?

1. Logical operation

No. Use one key per intended creation; reuse it only for retries of that creation.

2. Timeout outcome

The operation may not have committed, or it may have committed with a lost response. The client needs recovery rather than assuming either outcome.

3. Concurrent winner

Use a unique scoped key and a transaction tying key claim, resource creation and replay result together.

4. Expiry of deduplication

The API may treat the request as new and create a second object. State the supported retry window, retain keys long enough for expected retries, and offer an operation lookup or business uniqueness rule if duplicate creation is unacceptable.

5. Different payload

Reject the fingerprint mismatch as a documented conflict rather than returning the old resource or silently creating another.

6. Concurrent retry review

Use a unique key scoped to caller and operation, a payload fingerprint and a stored outcome. In one database transaction, claim the key, create the resource and save the response. A competing request waits or receives a documented in-progress response, then reads the committed result. A retry after commit returns it; a rolled-back transaction leaves no completed operation.