Replication Lag Read Guarantees and Failover

01 The idea

The problem this lesson solves

Create a7 and acknowledge it to the client.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

02 Worked example

Follow one worked example

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

A copied dataset is useful only when you state how current it is, which writes it acknowledges and who may become the writer.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Exhaust the simpler explanations first

Inspect slow queries, missing indexes, connection-pool saturation, lock contention and unnecessary work before splitting data. A single well-sized relational database can serve many workloads. If reads dominate and can tolerate lag, replicas may help. If one node cannot hold or serve the write workload despite reasonable tuning, evaluate sharding against its operational cost. Base the decision on measured constraints and growth forecasts.

Follow leader-follower replication

A leader accepts writes and records changes; followers receive and apply those changes. In asynchronous replication, the leader can acknowledge a commit before a follower has applied it. That follower may return older data. Reading a link immediately after creating it can therefore report ‘not found’ if the read goes to a lagging replica. A replica is a copy, not a separate shard: it normally holds the same dataset within its replication group.

Define the user's consistency promise

Read-your-writes means a user sees their own completed changes. Routing that user's immediate reads to the leader can provide it within a stated failure model. A more explicit mechanism waits until a chosen replica reaches a write position, with a deadline and fallback. A fixed ‘read leader for five seconds’ rule is a heuristic unless lag is guaranteed below that bound. Monotonic reads also matter: switching between followers with different lag can make an observed value appear to move backward.

Pay for durability deliberately

Asynchronous replication usually keeps write latency lower but can lose acknowledged writes if a leader fails before replication and an incomplete follower is promoted. Synchronous configurations wait for specified remote acknowledgments, increasing latency and sometimes reducing write availability when replicas are unavailable. The exact acknowledgment matters: receipt, durable flush and application are different milestones. Choose recovery-point and recovery-time objectives, then verify the configured system against them.

Failover requires authority, not just a second copy

Promoting a follower while the old leader still accepts writes can create competing leaders. A real failover mechanism needs a way to establish authority and fence the old writer. Backups remain necessary because replicas can faithfully copy accidental deletes and corruption. Multi-leader writes add conflict resolution: two writers can change the same logical record independently, and ‘last write wins’ may discard a valid business update.

Copying does not mean simultaneous visibility

The leader commits code a7 and returns success. A follower has not replayed that write yet. A read routed to that follower may return missing even though creation succeeded. This is a normal possible outcome of asynchronous replication, not proof that the create response was fabricated.

Choose the client guarantee. Route immediate owner reads to the leader, or wait for a selected follower to reach a known write position with a deadline and fallback. A five-second leader-pinning rule is a heuristic unless replication progress is guaranteed within that bound. Also prevent a caller from moving backward between followers if monotonic reads are required.

Name the acknowledgment milestone

A remote node can receive a change, durably flush it and later apply it for queries. These are distinct milestones. Synchronous replication settings determine which acknowledgment the leader waits for. Do not promise immediate query visibility merely because a remote copy confirmed receipt or flush.

Asynchronous failover may lose acknowledged changes that never reached the promoted follower. Stronger acknowledgment can reduce the permitted loss window while adding write latency or unavailability when required peers cannot respond. State the recovery-point and recovery-time objectives, then verify the actual configuration.

Authority is a separate mechanism

If the old leader still accepts writes while a follower is promoted, two writers can diverge. Safe failover needs a mechanism to establish authority and fence the old writer under the system’s failure model. A spare database icon does not define that mechanism.

Replicas also do not replace retained recovery data: accidental deletion may be copied faithfully. Test failover and restore separately. Multi-leader designs introduce conflict rules when regions can write independently; “last write wins” can discard a business update, so a conflict policy must match the operation’s meaning.

Guided tutorial

1. Draw a progress timeline

Separate leader commit, remote receipt, remote flush and query replay.

2. Write a read guarantee

Define the immediate owner read and the public eventual read separately.

3. Cross out the leader

State who can become writer and how the old one is fenced.

4. Review recovery

Distinguish failover testing from restoring an earlier logical state.

# PostgreSQL observation in an appropriately configured practice cluster:
SELECT application_name, state, sync_state,
       write_lag, flush_lag, replay_lag
FROM pg_stat_replication;
# Empty output can mean this server currently has no connected replication clients.

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions