Bandwidth Latency and Request Concurrency

01 The idea

The problem this lesson solves

Two services both handle 200 requests per second. One returns a tiny redirect; the other streams large files. Their network needs are completely different. Two dependencies can also have the same throughput but very different numbers of waiting requests. Bandwidth, latency and concurrency measure different dimensions of work, and confusing them leads to the wrong scaling decision.

You need multiplication, division and unit conversion. Keep reads, writes and bytes in separate calculations.

02 Worked example

Separate bandwidth from concurrent work

03 The method

Understand the mechanism

Convert payload rate into network demand

Bandwidth demand is bytes transferred per unit time. If 200 responses/s each contain 2,000 payload bytes, the payload rate is 400,000 bytes/s. Multiply by eight to obtain 3,200,000 bits/s, or 3.2 decimal Mbps.

This is payload only. Headers, protocol framing, encryption overhead, retransmissions and other traffic use additional capacity. Request and response directions may be very different, so estimate ingress and egress separately. Replication and backup traffic can compete for the same interfaces.

Response-size distributions matter. Averages can hide a few very large transfers that occupy connections and dominate bytes. Count requests and bytes as separate metrics. A 90% request cache-hit ratio may save far less than 90% of origin bandwidth if the misses are the largest objects.

Check the units at every conversion. Mbps is megabits per second, while MB/s is megabytes per second. Dividing a byte count by a bit-rate without converting creates an eightfold timing error.

Separate transmission time from network distance

Transmission time is the time required to put a quantity of data onto a link at the effective throughput available to that transfer. A 10 MB payload contains 80 Mb; at 20 Mbps, the idealized transmission time is four seconds before other delays.

Propagation and network round trips add delay even for small messages. A tiny database request across a long-distance path can take longer than a larger request to a nearby service. Increasing link bandwidth does not eliminate the time needed for signals and protocol exchanges to travel.

For a redirect response, payload transfer may be negligible compared with connection setup and round trips. For a large download, sustained throughput can dominate completion time. Identify which term matters before recommending compression, caching, connection reuse or geographic placement.

Actual transfer throughput can be below a link’s advertised capacity because of contention, congestion control, packet loss, server limits or client limits. Use the simple calculation as a lower-bound-style model under stated assumptions, then measure the real path.

Build a latency budget from the request path

For sequential work, durations add along the path. If application work takes 10 ms, a database call 30 ms and response preparation 5 ms in a particular trace, that trace contributes 45 ms before uncounted network and queueing stages.

Parallel work behaves differently. If two independent calls of 30 ms and 50 ms start together and both are required, the dependency portion is roughly the slower completion plus coordination overhead, not 80 ms. If one call depends on the other’s output, they cannot be parallelized that way.

Queueing time is often the missing term. A server may need only 5 ms of CPU but wait 100 ms for a connection or worker. Under rising utilization, that waiting can grow sharply. Measuring only the handler’s active compute time can hide the user’s dominant delay.

Percentile arithmetic requires care. You can add durations within one measured trace. You cannot generally add separately reported p99 values and claim the sum is the system’s p99. Preserve end-to-end measurements and use component observations to explain them.

Relate mean latency to in-flight work

For a stable system with consistent boundaries, Little’s law relates average in-system work L, arrival/completion rate λ and average time W: L = λW. Use seconds consistently. At 200 requests/s and 50 ms mean time, L = 200×0.05 = 10 average in-flight requests.

If the same admitted rate experiences 200 ms mean time, average in-flight work becomes 40. Nothing about the incoming QPS changed. A slower dependency caused more requests to overlap, consuming connections, memory or worker slots.

This is an average relationship, not a promise about maximum concurrency or p99 latency. Bursts and variability can create much larger instantaneous values. The model also needs care when the system is unstable and queues grow without bound; a finite steady-state average is then not established.

Define the boundary. If W measures only database time, L concerns work in that database boundary, not all requests in the browser-to-server path. Mixing an end-to-end latency with an internal arrival rate produces a misleading result.

Understand pools, queues and overload

A connection pool limits simultaneous use of a dependency. If ten database connections are available and each is occupied for 100 ms on average by serial work, a rough maximum service rate is 100 operations/s before other limits. Real variation and latency targets generally require headroom below a simplistic maximum.

When arrivals exceed the completion rate, queued work grows. An unbounded queue postpones failure while consuming memory and increasing latency. Clients may time out and retry while their original work remains queued, amplifying load.

Bound queues and admission, propagate deadlines, and reject or degrade work when it can no longer finish usefully. Increasing the pool can help if the pool itself is unnecessarily restrictive, but it can also overload the database by allowing more concurrent expensive work.

The correct intervention follows the bottleneck. Reduce work per request, improve the dependency, cap concurrency, serve an acceptable cached response or shed load. Adding app instances without controlling their combined dependency concurrency can worsen the incident.

Compare designs using the dominant cost

Suppose option A makes three sequential 20-ms remote calls and option B makes one 25-ms call returning slightly more metadata. Ignoring other costs, B reduces the dependency path from 60 ms to 25 ms. The trade-off may include larger payloads or tighter coupling; name it rather than optimizing only request count.

For large media, an edge cache can reduce origin bytes and geographic distance. For tiny internal calls, connection reuse or colocation may matter more. Compression saves transfer bytes but costs CPU and may not help already-compressed content.

Measure latency distributions, transferred bytes, in-flight work, pool waits and error rate together. A throughput increase accompanied by unacceptable tail latency is not necessarily a successful optimization. A lower mean with worse failures may also miss the service objective.

Your final explanation should say which resource each number describes. “3.2 Mbps payload,” “50 ms mean request time” and “10 average in-flight requests” are connected calculations, but they are not interchangeable capacity limits.

payload_bits_per_second = qps * response_bytes * 8
sequential_latency = app_time + dependency_time
average_in_flight = arrival_rate * average_time_seconds
# These are models with explicit boundaries, not p99 guarantees.

qps, response_bytes = 200, 2000
print(f"payload_Mbps={qps*response_bytes*8/1_000_000:.1f}")
for mean_seconds in [0.05, 0.20]:
    print(f"mean_ms={mean_seconds*1000:.0f}, in_flight={qps*mean_seconds:.0f}")

payload_Mbps=3.2
mean_ms=50, in_flight=10
mean_ms=200, in_flight=40

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions