Bandwidth Latency and Request Concurrency

01 The idea

The problem this lesson solves

200 responses/s × 2,000 bytes gives 400,000 bytes/s.

Prerequisite: earlier lessons introduce the vocabulary and shared examples used here. Follow the module order.

02 Worked example

Follow one worked example

Trace the example in order. Follow the same identities, units and state transitions through the diagram and the events below.

Bandwidth is a rate, latency is a duration, and concurrency relates work arriving to work still in progress.

03 The method

Mechanisms decisions and a guided tutorial

Work through the explanation in order, then apply it in the tutorial. Each comparison names a cost as well as a benefit.

Size network traffic in bits when comparing link rates

If each redirect response averages 1,000 bytes, the peak response payload is 57.87 × 1,000 ≈ 57,870 bytes/s. Multiply by eight to compare with a bits-per-second network rating: about 463,000 bits/s, or 0.463 Mbps. This is only modeled response payload. Request bytes, headers, TLS, retransmissions and replication add traffic. An image or video service can be bandwidth-bound long before its API becomes CPU-bound.

Use Little's Law with matching boundaries

For a stable system, average in-flight work is approximately arrival rate × average time in the system. At 200 requests/s and 0.05 seconds average service time, about 10 requests are in flight on average. This can help reason about concurrency, but it is not a p99 bound or a safe connection-pool size by itself. A slower dependency increases in-flight work even if incoming QPS is unchanged.

Convert payload to a network rate

At 200 responses per second and 2,000 bytes per response, payload throughput is 400,000 bytes/s. Multiply by eight to obtain 3.2 million bits/s, or 3.2 Mbps. Headers, request traffic, TLS overhead and retransmissions add to that payload model.

A large-file service can be network-bound at low request QPS. A tiny response can have low bandwidth demand but high latency because it waits on a slow dependency. Choose the resource model for the actual path instead of using QPS as a universal capacity measure.

Add only the costs on the path

A serialized request with 10 ms application work and a 40 ms database call has a modeled 50 ms path before other costs. Two independent calls performed concurrently have a different model from two calls performed in sequence. State dependency order before adding latencies.

A weighted average of cache-hit and cache-miss times estimates a mean under the assumed costs. It does not predict p99. During overload, queueing can make the miss path much slower than its isolated benchmark, so the distribution changes along with the traffic.

Use Little’s Law with matching boundaries

For stable long-run averages, in-flight work is approximately arrival rate multiplied by average time in the measured system. At 200 requests/s and 0.05 seconds average time, the modeled average concurrency is ten. If the dependency slows to 0.2 seconds at the same stable rate, it becomes forty.

This is not a safe thread-pool size or a p99 guarantee. Retries, burstiness and waiting policies affect the actual distribution. Bound concurrency and deadlines so a slow dependency cannot accumulate unlimited in-flight work. Measure throughput and queueing together when testing capacity.

Guided tutorial

1. Make a unit sheet

Write bytes, bits, milliseconds and seconds conversions.

2. Draw two paths

Compare sequential and independent parallel dependencies.

3. Calculate concurrency

Use the same arrival rate with two mean service times.

4. Set boundaries

Specify a timeout and maximum in-flight limit for the slow path.

payload_bits_per_second = qps * response_bytes * 8
sequential_latency = app_time + dependency_time
average_in_flight = arrival_rate * average_time_seconds
# These are models with explicit boundaries, not p99 guarantees.

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions