Idempotency Timeouts and Safe Retries

01 The idea

The problem this lesson solves

A client sends “create a link,” the server stores a7, and the response disappears on the network. The client’s screen says timeout, but the database already contains the link. Sending the same request again can create a second link unless the API recognizes that both attempts belong to one logical operation. This lesson builds that recognition and connects it to transaction boundaries, deadlines and retry behavior.

Use the requirements and ownership vocabulary from module 1. This module follows one request across the API boundary.

02 Worked example

Retry after a committed response is lost

03 The method

Understand the mechanism

A timeout describes knowledge, not the committed state

Draw five events: send request, receive request, commit, send response, receive response. A failure can occur between any two. If the client times out before the server receives the request, no work may have happened. If it times out after commit, the effect already exists. The client often cannot distinguish those cases from its own connection state.

This is an ambiguous outcome. Treating it as definite failure causes duplicate work; treating it as definite success can claim work that never occurred. A correct recovery protocol gives the client another way to identify and learn the result of its operation.

Transport reliability does not remove the ambiguity. A TCP acknowledgment concerns bytes at a transport endpoint, not the database commit of a business action. Canceling a browser request also does not guarantee cancellation of remote work already executing.

The design should count logical operations separately from network attempts. One user action may produce two attempts and one committed resource. Monitoring only HTTP request count can make retries look like additional business activity.

Give one logical operation a stable identity

For creation, let the client choose key k1 and reuse it for retries of the same intent. Scope the key to an authenticated caller and an operation type. The same string used by two unrelated clients should not cause one client to receive the other’s result.

Store a fingerprint of the meaningful request parameters with the key. If k1 first meant destination A and later arrives with destination B, the server should reject that reuse rather than quietly return an unrelated creation. Define canonicalization carefully so harmless representation differences do not accidentally change identity, and meaningful differences do not disappear.

A completed idempotency record contains enough result information to reproduce the promised response, such as the resource id and status. Saving only “processed=true” can leave a retry unable to identify the created link. Depending on the API, the replayed representation may be the original result or a documented current representation; choose explicitly.

The client must keep k1 after a timeout. Generating a new key for every attempt defeats deduplication because the server sees different logical operations. A genuinely new creation should use a new key even if its payload happens to match a previous request.

Protect concurrent retries with database state

Imagine A and B both ask whether k1 exists. Both see no record, both create a link, and both insert a completion record afterward. A preliminary lookup has not serialized the decision. A unique constraint on the scoped operation key provides a conflict point that concurrent attempts cannot both win.

When the mapping and replay result live in one database, commit them in the same transaction where possible. The transaction claims k1, verifies the fingerprint, creates the mapping and records the completed outcome. If it rolls back, neither a completed operation nor an orphaned claimed success should remain.

A competing request may wait for the transaction, observe the committed result, or receive a documented in-progress response. Bound that wait. If the first transaction aborts, the retry should be able to attempt the operation again according to the claim protocol.

Think through crash positions. A crash before commit should leave no committed creation from that transaction. A crash after commit but before responding should leave a replayable result. A design that commits the resource and records k1 later has a gap in which recovery may create a duplicate.

Bound the guarantee in time and across systems

Idempotency records consume storage, so many services retain them for a defined retry window. After a record expires, a late request with k1 may be treated as new even though the original resource still exists. The API must document that boundary. A business uniqueness constraint or an operation lookup may be necessary when duplicates remain unacceptable indefinitely.

One local transaction cannot atomically commit an unrelated remote service’s side effect. Suppose creation also charges a payment provider. Committing a database row and calling the provider are two actions with independent failures. A stable downstream idempotency key, a durable workflow and reconciliation can make recovery controlled; a local BEGIN/COMMIT alone cannot.

An outbox is one pattern for durable follow-up work: store an event with the business change in the same database transaction, then deliver it asynchronously. The relay may send more than once, so the consumer still needs a deduplication or idempotent-effect strategy. “Exactly once” must be qualified by the system boundary and failure model.

For the course’s link creation, keep the first implementation within one database. Understanding that simpler guarantee makes it easier to see what changes when an external side effect is later introduced.

Design retries around a total deadline

A timeout limits waiting for an attempt or stage. A deadline limits the total time available for the overall operation. If the caller has 800 ms remaining, a dependency should not start a 2-second attempt and pretend the caller is still waiting for useful work.

Retry only eligible failures and only when repeating the operation is safe under its contract. A malformed URL will not become valid on its third attempt. An overload response may merit a later retry, but immediate retries can increase the very load causing the failure.

Use bounded exponential backoff with jitter to spread retry attempts. For example, a policy might permit at most three attempts within a total deadline, with randomized delays capped by the remaining budget. The exact values should come from latency and recovery requirements rather than being copied as universal defaults.

Avoid independent retries at every layer. If a client makes three attempts, a gateway retries each three times, and an app retries each dependency call three times, one operation can cause up to 27 dependency attempts. Choose a retry owner and propagate budgets. Hedged requests also create duplicate load and require their own safety analysis.

Verify the effect, not just the response count

The test for idempotent creation is not “both responses are 200.” It is that repeated attempts for one scoped key and payload create one intended resource and return a compatible outcome. Count database records and inspect the saved operation result.

Test two simultaneous attempts, different payloads with the same key, a crash before commit, a lost response after commit and a retry near the retention boundary. Also test a slow first attempt whose retry arrives before it finishes. These cases exercise the boundaries a happy-path test misses.

Observe attempt count, deduplication hits, conflicts, in-progress waits and logical creations separately. A rise in retries with stable creation count can reveal network or latency trouble even when business correctness holds. A rise in created resources per operation key reveals a much more serious invariant failure.

The resulting API lets a client say “complete operation k1 or tell me its recorded result,” rather than “try making something and hope it did not already happen.” That is the useful promise to carry into later distributed workflows.

UNIQUE(caller_id, operation, idempotency_key)
BEGIN
  claim key and fingerprint
  create resource
  save completed replay result
COMMIT
retry same key + same payload → return saved result

operations, links = {}, {}
def create(caller, key, destination):
    identity = (caller, "create", key)
    if identity in operations:
        previous, code = operations[identity]
        if previous != destination:
            return "conflict"
        return code
    code = "a" + str(len(links) + 7)
    links[code] = destination
    operations[identity] = (destination, code)
    return code

print(create(7, "k1", "https://example.com/guide"))
# Pretend the first response never reached the client.
print(create(7, "k1", "https://example.com/guide"))
print(create(7, "k1", "https://example.com/other"))
print(f"resources={len(links)}")

a7
a7
conflict
resources=1

05 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

06 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

07 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

08 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions