N
navaprasad maram
Guest
Horizontal scaling is a credit card, not an architecture.
When an API slows down, adding instances works—which is exactly why it is so tempting. But it can also hide inefficiencies and leave you paying for far more capacity than the workload actually requires.
My goal on a real-time inventory API was the opposite. A single request could ask for hundreds of SKU-location records, and peak-sale traffic made every unnecessary allocation, network round trip, and context switch matter.
Before adding more instances, I wanted to find out how much useful work one instance could actually do.
What followed was a series of experiments involving payload design, Redis access patterns, response streaming, servlet containers, HTTP/2, load balancers, and one optimization we deliberately chose not to take.
The exact throughput numbers will vary with hardware, JVM configuration, payload size, Redis topology, concurrency, and client behavior. The useful part is the sequence: each experiment attacked a different bottleneck.
Profile a typical JSON API under load and a shocking share of CPU goes to object mapping — deserializing from the datastore into objects, then serializing those objects right back into JSON. My first and biggest win was refusing to pay that tax: Redis stores each SKU-location record as the exact JSON bytes the API will emit. The service never parses them. Its entire job is to fetch byte arrays and concatenate them with [, ,, and ].
There is no JSON library on the request path — not a faster one, none. Bytes go socket to socket. The side effect is as valuable as the CPU saving: a request allocates almost no objects, so the garbage collector has nearly nothing to do. The best GC tuning I found was not allocating.
The cost is paid where it's cheap: the ingestion pipeline pre-renders payloads at write time — and where two consumers need two shapes, we store two denormalized copies. Memory is cheaper than per-request CPU, and reads outnumber writes by orders of magnitude.
Fewer, fatter calls beat many thin ones. Three tunings, each independent:
Group by access pattern. A request for one SKU across twenty locations shouldn't be twenty lookups. Storing all of a SKU's per-location values as fields of one Redis hash — keyed with a cluster hash tag like {sku:12345} so every field lands on the same cluster slot — turns it into a single HMGET, or one HVALS when no location filter is given. N round trips become one before any concurrency code exists.
One connection, not a pool. With a thread-safe async client (Lettuce), a single multiplexed connection per node replaces the whole connection-pool ritual — no checkout contention, and concurrent requests' commands pipeline naturally onto one socket. In the most aggressive build, I turned auto-flush off and coalesced flushes by hand — a floor of a few microseconds between flushes, a scheduled sweep as a ceiling — because at high load the flush() syscall itself was measurable.
Tune for failure speed. The client's default command timeout was 60 seconds; I cut it to 500 milliseconds. A call that takes a minute to fail holds a thread hostage for a minute. I also enabled replica reads (ReadFrom.NEAREST) to spread load across the cluster, and adaptive topology refresh so failovers heal without a restart.
With N async Redis calls in flight per request, the tempting move is "go fully reactive." I used a plain BlockingQueue instead. Each Redis callback pushes its result bytes onto the queue; the request thread loops — take, write to the socket, repeat — until a sentinel says every SKU reported in:
No Content-Length is set, so the container falls back to chunked transfer encoding: the first byte reaches the client before the last Redis reply arrives. A tiny lock guards the JSON bookkeeping (first item? comma? closing bracket?) across racing callbacks. The consuming code is a while loop anyone can debug at 2 a.m.
The streaming design blocks a thread per request, so threads become the scarce resource. I ran the same workload through three servers:
Netty won the benchmark. We shipped Tomcat.
That decision is the part of this story I'd defend hardest. The Netty build was beautiful — but every future feature, every on-call page, every new hire now depends on hand-rolled HTTP handling that three people understood. Tomcat with maxThreads raised to 10,000 got within a margin we could buy back with one or two extra instances, and everyone on the team could operate it. Per-instance capacity was the goal, but it's still a means; the end is a system the team can run. Take the 80% that's boring before the 100% that's clever.
HTTP/2 was configured and benchmarked (multiplexed streams, header compression, per-stream flow control — which composes beautifully with a streaming response). But it only helps callers that negotiate it via ALPN, so it shipped as a ready lever rather than a forced migration: cheap to have, valuable on a schedule the clients control.
NLB vs. classic ELB. We tested both with TCP listeners. A network load balancer does Layer-4 passthrough — no HTTP parsing, connections held open end-to-end — which suits this design: long-lived keep-alive connections and TLS terminating at the instance, per AWS's own positioning of NLB for latency-sensitive TCP workloads.
The optimization we refused. Terminating TLS twice — at the load balancer and again at the server — costs two handshakes and double the crypto. We measured skipping one, and the numbers were better. We kept both anyway. Encryption in transit on every hop, including inside your own network, is table stakes; a scaling win that weakens your security posture isn't a win. I'm including this because a playbook is defined as much by what you decline as what you adopt.
None of these techniques fixed the same bottleneck twice, and the order mattered: eliminate work (pre-rendered payloads), collapse round trips (hashes, pipelining), overlap what remains (async fan-out, streaming), then — and only then — tune the machinery underneath (threads, container, protocol, load balancer). Run that sequence and the horizontal-scaling conversation changes: you're no longer multiplying an inefficient instance, you're multiplying an optimized one. That difference compounds on every invoice.
Measure with two clocks while you do it — time to compute the answer and time to get bytes to the caller are different diseases with different cures — and keep every log line and metric off the request path (async loggers, UDP fire-and-forget metrics). You can't tune what your own instrumentation is quietly slowing down.
Scale horizontally, absolutely. Just make sure each box has earned it first.
When an API slows down, adding instances works—which is exactly why it is so tempting. But it can also hide inefficiencies and leave you paying for far more capacity than the workload actually requires.
My goal on a real-time inventory API was the opposite. A single request could ask for hundreds of SKU-location records, and peak-sale traffic made every unnecessary allocation, network round trip, and context switch matter.
Before adding more instances, I wanted to find out how much useful work one instance could actually do.
What followed was a series of experiments involving payload design, Redis access patterns, response streaming, servlet containers, HTTP/2, load balancers, and one optimization we deliberately chose not to take.
The exact throughput numbers will vary with hardware, JVM configuration, payload size, Redis topology, concurrency, and client behavior. The useful part is the sequence: each experiment attacked a different bottleneck.
Experiment 1: Stop serializing. At all.
Profile a typical JSON API under load and a shocking share of CPU goes to object mapping — deserializing from the datastore into objects, then serializing those objects right back into JSON. My first and biggest win was refusing to pay that tax: Redis stores each SKU-location record as the exact JSON bytes the API will emit. The service never parses them. Its entire job is to fetch byte arrays and concatenate them with [, ,, and ].
There is no JSON library on the request path — not a faster one, none. Bytes go socket to socket. The side effect is as valuable as the CPU saving: a request allocates almost no objects, so the garbage collector has nearly nothing to do. The best GC tuning I found was not allocating.
The cost is paid where it's cheap: the ingestion pipeline pre-renders payloads at write time — and where two consumers need two shapes, we store two denormalized copies. Memory is cheaper than per-request CPU, and reads outnumber writes by orders of magnitude.
Experiment 2: Make Redis round trips collapse
Fewer, fatter calls beat many thin ones. Three tunings, each independent:
Group by access pattern. A request for one SKU across twenty locations shouldn't be twenty lookups. Storing all of a SKU's per-location values as fields of one Redis hash — keyed with a cluster hash tag like {sku:12345} so every field lands on the same cluster slot — turns it into a single HMGET, or one HVALS when no location filter is given. N round trips become one before any concurrency code exists.
One connection, not a pool. With a thread-safe async client (Lettuce), a single multiplexed connection per node replaces the whole connection-pool ritual — no checkout contention, and concurrent requests' commands pipeline naturally onto one socket. In the most aggressive build, I turned auto-flush off and coalesced flushes by hand — a floor of a few microseconds between flushes, a scheduled sweep as a ceiling — because at high load the flush() syscall itself was measurable.
Tune for failure speed. The client's default command timeout was 60 seconds; I cut it to 500 milliseconds. A call that takes a minute to fail holds a thread hostage for a minute. I also enabled replica reads (ReadFrom.NEAREST) to spread load across the cluster, and adaptive topology refresh so failovers heal without a restart.
Experiment 3: Stream the response as results land
With N async Redis calls in flight per request, the tempting move is "go fully reactive." I used a plain BlockingQueue instead. Each Redis callback pushes its result bytes onto the queue; the request thread loops — take, write to the socket, repeat — until a sentinel says every SKU reported in:
Code:
try (ServletOutputStream out = response.getOutputStream()) {
while (true) {
byte[] bytes = queue.take();
if (bytes.length == 0) break;
out.write(bytes);
}
}No Content-Length is set, so the container falls back to chunked transfer encoding: the first byte reaches the client before the last Redis reply arrives. A tiny lock guards the JSON bookkeeping (first item? comma? closing bracket?) across racing callbacks. The consuming code is a while loop anyone can debug at 2 a.m.
Experiment 4: The container bake-off — and the choice that surprised me
The streaming design blocks a thread per request, so threads become the scarce resource. I ran the same workload through three servers:
| | Threads | Model | Result |
|---|---|---|---|
| Tomcat (NIO) | maxThreads=10000 | thread-per-request, blocking | Solid; threads are cheap when they're parked on waits, not burning CPU |
| Jetty | 500 + HTTP/2 (1,280 streams/connection) | async connector | Comparable capacity at 1/20th the threads |
| Raw Netty (epoll) | 2 per core | event loop, no servlet container at all | Won. Highest per-instance capacity, watermarked write buffers, 20ms GC pause target |
Netty won the benchmark. We shipped Tomcat.
That decision is the part of this story I'd defend hardest. The Netty build was beautiful — but every future feature, every on-call page, every new hire now depends on hand-rolled HTTP handling that three people understood. Tomcat with maxThreads raised to 10,000 got within a margin we could buy back with one or two extra instances, and everyone on the team could operate it. Per-instance capacity was the goal, but it's still a means; the end is a system the team can run. Take the 80% that's boring before the 100% that's clever.
Experiment 5: Protocol and the load balancer
HTTP/2 was configured and benchmarked (multiplexed streams, header compression, per-stream flow control — which composes beautifully with a streaming response). But it only helps callers that negotiate it via ALPN, so it shipped as a ready lever rather than a forced migration: cheap to have, valuable on a schedule the clients control.
NLB vs. classic ELB. We tested both with TCP listeners. A network load balancer does Layer-4 passthrough — no HTTP parsing, connections held open end-to-end — which suits this design: long-lived keep-alive connections and TLS terminating at the instance, per AWS's own positioning of NLB for latency-sensitive TCP workloads.
The optimization we refused. Terminating TLS twice — at the load balancer and again at the server — costs two handshakes and double the crypto. We measured skipping one, and the numbers were better. We kept both anyway. Encryption in transit on every hop, including inside your own network, is table stakes; a scaling win that weakens your security posture isn't a win. I'm including this because a playbook is defined as much by what you decline as what you adopt.
The one thing to take away
None of these techniques fixed the same bottleneck twice, and the order mattered: eliminate work (pre-rendered payloads), collapse round trips (hashes, pipelining), overlap what remains (async fan-out, streaming), then — and only then — tune the machinery underneath (threads, container, protocol, load balancer). Run that sequence and the horizontal-scaling conversation changes: you're no longer multiplying an inefficient instance, you're multiplying an optimized one. That difference compounds on every invoice.
Measure with two clocks while you do it — time to compute the answer and time to get bytes to the caller are different diseases with different cures — and keep every log line and metric off the request path (async loggers, UDP fire-and-forget metrics). You can't tune what your own instrumentation is quietly slowing down.
Scale horizontally, absolutely. Just make sure each box has earned it first.