Vertical Horizontal and Stateless Application Scaling

01 The idea

The problem this lesson solves

You add a second application server to increase capacity. CPU usage falls, but users are randomly logged out and recently uploaded files disappear depending on which server answers. The application was scalable in deployment count but not in state ownership. Before discussing load-balancing algorithms, you need to make instances replaceable and understand which resource adding them can actually improve.

Use the peak traffic and failure budgets from module 4 to judge whether an added server actually helps.

02 Worked example

Remove an app without losing the session

03 The method

Understand the mechanism

Vertical and horizontal scaling solve different constraints

Vertical scaling gives one machine more CPU, memory or faster storage. It can be operationally simple and avoids distributing some state. It has hardware limits, may require disruptive changes and does not by itself remove the machine as a failure domain.

Horizontal scaling adds instances. It can increase aggregate capacity and permit replacement, but only if work can be divided and required state remains accessible. Adding ten copies of an application does not help when every request waits on the same saturated database row.

Suppose one app spends most of its time compressing independent responses on CPU while the database is lightly loaded. More app capacity may help. If the app spends most of its time waiting for a database connection, adding instances may create more competing connections and worsen the bottleneck.

Start with CPU, memory, dependency latency, pool waits and throughput measurements. The scaling direction should follow the limiting resource and failure objective rather than a slogan that horizontal scaling is always superior.

Stateless means no required private history between requests

A stateless app instance can process a request using the request’s information and explicitly accessible state, without depending on a previous request having reached that same instance. It can still have local variables, connection pools, compiled code and replaceable caches.

The problem is required private history. If A stores session s1 only in memory, B cannot authenticate the next request carrying s1. If A stores the only copy of an uploaded file on its local disk, B cannot serve it and replacing A can lose it.

Classify state by its role. Request-local data disappears when the operation ends. A local cache can disappear if misses recover from an authority. Session records and accepted business work need an explicit shared or durable ownership design. Upload content may belong in object storage with a stable reference in the database.

This classification is more useful than saying “move everything to Redis.” A shared store also needs persistence, expiry, authorization and failure behavior appropriate to the state it holds. A cache configured to evict arbitrary keys may be unsuitable as the only authority for important sessions or accepted work.

Understand sessions and signed tokens

With server-side sessions, a cookie can carry a session identifier while a trusted store holds the session state. Any app instance can look it up, provided the store is reachable and the session remains valid. The lookup adds a dependency and requires expiry and revocation behavior.

A signed token can carry claims that instances verify without a session lookup on every request. Signature verification establishes integrity under the key model; it does not make claims current forever. Expiry, key rotation, revoked privileges and logout requirements still need a design.

Neither approach means trusting arbitrary user-supplied owner_id. Authentication establishes a trusted identity context, and authorization uses that context for each operation. Sharing the mechanism across instances prevents routing from changing the caller’s effective permissions.

Choose based on revocation needs, token size, dependency cost and operational control. State where current authority lives and what happens during outages. A stateless request-processing layer can rely on stateful identity infrastructure; the state has been assigned an owner rather than abolished.

Sticky routing is an optimization with a failure cost

Session affinity tries to send related requests to the same instance. It can improve local cache reuse or support a workload that has not yet externalized state. It does not make private state durable or available after the chosen instance disappears.

Affinity can also create imbalance. One user or connection may generate much more work than another. IP-based affinity can group many users behind the same shared address. A newly added instance may receive little traffic if existing sessions remain attached elsewhere.

Long-lived connections need explicit handling. A WebSocket connection normally stays with the instance that accepted it. If that instance fails, the client reconnects elsewhere and needs a protocol to resume, reauthenticate or replay missed information. “Stateless apps” does not mean an established connection can teleport between machines.

Use affinity when its benefit is understood, but test instance loss and routing changes. If correctness depends on never changing the destination, the design has not met a replaceability goal.

Scale the shared dependencies too

Each app instance may open a connection pool. With ten instances and fifty connections each, the database can see up to five hundred connections from that layer alone. A pool size that was harmless on one instance may exceed the database’s useful concurrency after scaling out.

Budgets should be coordinated. Estimate total possible dependency concurrency, actual active work and the cost of queued requests. Apply bounded queues and deadlines so one slow dependency does not retain unbounded application memory.

Caches can reduce shared work but introduce their own connection, memory and freshness costs. Background workers can also compete with interactive requests. Scaling the front end while leaving a bulk importer unrestricted can produce little user-visible improvement.

Measure the entire path during scale tests. If doubling app instances does not increase useful throughput, inspect the resource that stopped scaling. This prevents repeated infrastructure changes that only move waiting work around.

Design replacement as an ordinary operation

An instance should start, load configuration, establish required dependencies and become ready only when it can serve its intended work. During shutdown, it should stop accepting new work, finish bounded in-flight work where possible and exit within the deployment policy.

Externalize accepted durable jobs before acknowledging them. A background task living only in an app process can vanish during deployment. If an HTTP response means “accepted for later processing,” a durable queue or job record must support that promise.

Practice replacing an instance under load. Verify session continuity, accessible uploads, duplicate-operation behavior and the recovery of in-flight requests. This is stronger evidence of statelessness than inspecting whether a codebase uses a global variable.

The goal is predictable replacement: losing A should not erase unique knowledge required by B. Once that property holds, load balancing and horizontal scaling become much easier to reason about.

request state: private, short-lived
local cache: replaceable, misses are recoverable
session / accepted work: explicit shared or durable authority
upload: durable object storage or another shared contract
scale decision: measured bottleneck + surviving capacity

04 Cheat sheet

Review the decisions

Use this table to explain the mechanism and its limit without memorizing a product name.

05 Where & why

Check the primary references

Use official references for protocol and implementation details. The questions here are teaching prompts.

06 Interview questions

Explain the reasoning out loud

Answer before opening the explanation. The questions progress from vocabulary to decisions and failure behavior.

07 Practice problems

Practice a new case

Solve the new cases before revealing hints. Complete worked solutions follow the set.

Worked solutions