Load Balancing Algorithms and Layer 4 versus Layer 7

01 The idea

The problem this lesson solves

Distributing six requests as A, B, C, A, B, C looks fair, but equal request counts do not imply equal work. One request might take 5 ms and another several seconds. A load balancer chooses among eligible destinations using the information it can observe. Its algorithm, protocol layer and connection model determine what “balanced” actually means.

Use the peak traffic and failure budgets from module 4 to judge whether an added server actually helps.

02 Worked example

Compare equal and weighted routing

03 The method

Understand the mechanism

Separate routing eligibility from selection policy

Eligibility answers which backends are currently allowed to receive new work. Health checks, readiness state, draining and administrative configuration influence that set. Selection policy chooses a destination from the eligible set.

Keeping the two concepts separate prevents a common mistake: an excellent selection algorithm still sends bad traffic if the pool includes failed or unready instances. Conversely, removing unhealthy instances cannot compensate for insufficient capacity among survivors.

Define the unit being selected. A layer-4 balancer commonly selects a destination for a transport connection. A layer-7 proxy can make decisions using parsed application requests. One long-lived connection can carry many operations, so connection counts and request counts are not interchangeable.

The balancer also needs limits and deadlines. Accepting unlimited incoming work while upstreams are slow can create queues at the entry point. Routing is part of admission and failure behavior, not just an array index calculation.

Round robin works when the work is comparable

Round robin cycles through the eligible destinations. With A, B and C and six equal-cost requests, the sequence A,B,C,A,B,C gives each two selections. It is simple and requires little per-request information.

If C has twice the tested capacity, weighted round robin can target relative shares 1:1:2. Over an appropriate interval, ideal shares are 25%,25%,50%. Exact short sequences depend on the implementation; do not claim every group of four must follow one universal ordering.

Weights should reflect useful capacity under the relevant workload and latency target. Twice the CPU does not always mean twice the serving budget if all instances wait on the same dependency.

Round robin can perform poorly when request costs differ significantly or long-lived connections carry unequal amounts of work. It balances selection counts under its unit, not CPU time, memory use or response latency.

Use dynamic signals with their limitations

Least-connections routing prefers a backend with fewer active connections, possibly adjusted by weight. This can help when connections remain busy for different durations. It is still a proxy for load: one connection may be idle while another carries many multiplexed requests.

Least outstanding requests or latency-aware policies can use different signals. Their usefulness depends on measurement freshness and the ability to distinguish overload from other causes of latency. A temporarily fast backend can become overloaded if too much traffic shifts to it at once.

Distributed balancers may not share an instantaneous global view. Each can make locally reasonable decisions based on partial observations. Policies need damping, limits and careful behavior when a backend first enters the pool with little history.

Choose a signal that relates to the constrained resource. If work is CPU-bound and request cost varies widely, raw connection count may be weak evidence. Measure per-backend saturation and user-visible latency to evaluate whether the routing policy improves the outcome.

Hashing provides affinity, not universal balance

A hash of a client or request key can choose a stable backend. This can improve local cache reuse or keep related requests together. IP hashing can be skewed when many users share an address, and a popular key can overload its selected owner.

With hash(key) modulo N, changing N remaps many keys. Consistent hashing or a stable logical-slot map can reduce broad remapping, but neither removes the need to handle backends joining, leaving or failing.

Do not use application-server affinity as the only protection for durable state. If correctness requires a private session or file on one backend, remapping after failure breaks it. Externalize required authority or define a recovery protocol.

Compare affinity benefit with imbalance and failure costs. A local cache miss is often acceptable after rerouting. Losing the only record of an accepted operation is not. The state classification from the preceding lesson tells you which case you have.

L4 and L7 offer different visibility

A layer-4 decision uses transport/network information such as addresses, ports and connection state. It can forward traffic without interpreting the full application request. A layer-7 proxy parses application semantics and can route by HTTP host, path or other fields under its configuration.

To route by an encrypted HTTP path, a component generally needs access to the decrypted application request at the relevant TLS termination point. TLS passthrough and TLS termination are different architectures with different visibility and trust boundaries.

Layer-7 features can include request-level routing, header policies and application-aware retries. They also add parsing, buffering and configuration responsibilities. Layer-4 routing can be simpler for protocols that do not require those decisions.

Neither layer is automatically faster or more reliable in every deployment. Evaluate the required features, protocol support, connection behavior, operational implementation and measured cost. The design should state what the balancer must observe and why.

Evaluate balance by completed useful work

Record per-backend selection count, active work, latency, errors and saturation. An evenly distributed request count with one slow backend is not a balanced user experience. A routing sequence also does not prove each selected request completed successfully.

Test heterogeneous request durations, one slower instance, a backend entering the pool and a backend being drained. Include persistent connections, because a policy that looks balanced with one request per connection can behave differently with connection reuse.

Control retries during the test. A proxy that silently retries failed upstream work can make final responses look healthy while doubling load and hiding a degraded backend. Report both attempts and logical requests.

The outcome you want is efficient use of eligible capacity while respecting deadlines and correctness. The algorithm is a means to that outcome; its name is not the evidence.

# Equal-cost baseline: A B C A B C
# Weighted 1:1:2 ideal shares: 25%, 25%, 50%
# L4: addresses / ports / connection policy
# L7: parsed application fields such as host and path
# Inspect saturation and latency, not just selected-request totals.

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions