Replication Lag Read Guarantees and Failover

01 The idea

The problem this lesson solves

A user creates a7 and immediately opens it. Creation succeeds, but the redirect says the code is missing. The write reached the leader while the read reached a follower that has not applied it yet. Replication creates useful copies, but it also creates a question every read must answer: which copy is allowed to speak for this operation?

Use the API, database, capacity and cache concepts from earlier modules. These lessons add distributed ownership and failure trade-offs.

02 Worked example

Read a newly created link from a lagging follower

03 The method

Understand the mechanism

Explain what a replica copies

In a leader-follower design, writes are accepted through the designated leader and changes are transferred to followers. A follower may serve suitable reads or become a failover candidate under the deployment’s protocol.

A full replica stores another copy of the same logical dataset. It does not divide the write workload the way sharding does. Each full follower must receive and apply the relevant changes. Adding followers can increase read-serving capacity but also adds replication work and operational state.

Replication can be physical or logical depending on the database and configuration. The designer’s immediate questions are which changes are copied, how far each copy has progressed, what acknowledgment means and how a new authority is chosen after failure.

Keep backups separate. A mistaken deletion can replicate to every follower. Historical recovery needs retained backup or log history under a tested restore policy, not merely another live copy of the current mistake.

Separate receipt, durable storage and replay

A follower can receive a log record before it has durably stored it, and can store it before applying it to query-visible state. These stages matter for both durability and read freshness.

For a7, the leader may have committed and acknowledged the creation while the follower has not replayed the change. A query on that follower can legitimately see no row under its current state. That missing observation is not evidence that the original creation failed.

Waiting for a follower to receive or flush a record is not automatically the same as waiting for it to apply the record for queries. Database-specific synchronous modes define different acknowledgment boundaries. Name the actual condition instead of using “sync” as a complete explanation.

Lag can be described in time, bytes or log positions, and each measurement has interpretation limits. A single lag field may be unavailable or uninformative during idle periods. Combine replication state with workload-aware observations and the guarantee the read path needs.

Identify three user-visible lag problems

Read-your-writes can fail when a user writes to the leader and then reads a lagging follower. The interface appears to forget the user’s own action. A retry can also create duplicate work if the application mistakes follower absence for proof that a prior operation did not exist.

Monotonic reads can fail when a user first reads a more advanced copy and later reads a less advanced copy. The displayed state goes backward. Pinning a user to one copy may reduce some movement, but failover and routing changes still need a consistency policy.

Related facts can be observed inconsistently if they are read from copies at different progress points. A client might see a newly referenced object but not an earlier prerequisite from another path. The exact anomaly depends on the data and replication model, but arbitrary read routing should not be assumed to preserve causal expectations.

These are product behaviors, not only database metrics. Decide which operations tolerate them. A public historical report and a post-purchase confirmation may need different read policies.

Choose an explicit fresh-read policy

One simple policy routes consistency-sensitive reads to the authoritative leader, assuming its read semantics meet the contract. This concentrates some load but makes the reasoning straightforward.

Another approach carries a write progress token and waits until a selected follower has replayed at least that position before reading. The wait must be bounded and the token interpreted correctly across the database’s timeline or epoch changes. On timeout, the service can fall back or return a documented unavailable outcome.

Temporarily routing a user to the leader for a fixed interval after a write is a heuristic unless the system can prove lag stays below that interval under all relevant conditions. It may improve experience, but should not be advertised as an unconditional guarantee.

Caches can defeat any of these policies if they return an older value first. The complete read path must honor the requested consistency, including local caches, shared caches and HTTP caching.

Compare acknowledgment and availability trade-offs

Asynchronous replication allows the leader to acknowledge without waiting for a follower under that mode’s contract. It can reduce write-path latency but leaves a window in which a leader loss may lose acknowledged changes if the surviving copy has not received the needed data.

Waiting for specified remote durability can reduce that loss window for the failures covered by the configuration. It adds network and remote processing to the write path, and an unavailable required acknowledgment partner can affect write availability.

Do not equate every product’s synchronous or semisynchronous terminology. Verify whether acknowledgment concerns receipt, flush, application, a quorum or a fallback mode. The operational configuration is part of the guarantee.

For a concrete product example, MySQL 8.4 semisynchronous replication waits for the configured replica acknowledgments of received and logged events, rather than waiting for those transactions to be applied for queries. A configured acknowledgment timeout can switch the source back to asynchronous behavior. Consequently, “semisynchronous” is not a blanket claim of immediate follower visibility or identical durability through every outage.

Choose from recovery objectives and failure domains. A second copy on the same vulnerable host or storage system does not protect against losing that shared dependency. Place copies and acknowledgment rules to match the failures the service promises to survive.

Fail over without creating competing authorities

Failover must select a suitable new leader, establish that it is allowed to accept writes, redirect clients and prevent the old leader from continuing conflicting writes if it reappears. This last requirement is often called fencing.

Simply promoting a follower because the leader is unreachable can be unsafe if the old leader is still serving another part of the network. The control plane needs an authority protocol appropriate to the database. Consensus and election algorithms are deeper topics; this lesson establishes the problem they must solve rather than pretending a health check alone solves it.

After failover, clients can have old connections, cached addresses and uncertain write outcomes. Recovery still needs idempotency and read policies. A new leader also may not contain every asynchronously acknowledged write, so the data-loss boundary must be honest.

Test failover with ongoing writes and reads. Verify both availability and the survival of acknowledged records. A green process check after promotion does not prove the service preserved its data contract.

Recognize the extra problem introduced by multiple leaders

Multi-leader designs accept writes at more than one location. They can support local writes or disconnected operation, but concurrent changes to the same logical object need a conflict policy.

If two regions assign different destinations to a7 while isolated, both cannot silently remain the single latest value without reconciliation. Last-write-wins chooses according to an ordering rule but can discard a legitimate update; clock assumptions and user expectations matter.

Some data has mergeable operations, such as certain counters or sets under a carefully designed model. Other data, such as exclusive seat ownership, needs stronger coordination or ownership partitioning. “Merge later” is not a universal solution.

Prefer one clear authority when it meets the workload and availability requirements. Introduce multiple writers only with a defined conflict, ordering, recovery and user-visible contract.

# PostgreSQL observation in an appropriately configured practice cluster:
SELECT application_name, state, sync_state,
       write_lag, flush_lag, replay_lag
FROM pg_stat_replication;
# Empty output can mean this server currently has no connected replication clients.

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions