Continuous batching and the serving scheduler

LLM serving is not just model execution. It is queueing.

A static batch is easy to picture: collect requests, run them together, wait until all are done, then collect the next group. That works poorly for generation because outputs have different lengths. Short requests finish early, long requests keep running, and batch slots sit unused.

Continuous batching changes the unit of scheduling. Instead of waiting for the entire batch to finish, the scheduler can add new requests between decode iterations as old requests complete.

Decode as a repeated step

Generation is a loop:

choose active requests
run one forward pass
sample one token per active request
update KV cache
remove finished requests
admit new requests if capacity allows
repeat

That "admit new requests" line is the heart of continuous batching. The batch is not a fixed group. It is a living set.

The scheduler must respect memory, max sequence length, priority, fairness, and latency targets. This is why serving engines look like systems software even when the model code is only a few lines.

Why it helps

Imagine four requests:

A: 4 decode tokens
B: 4 decode tokens
C: 40 decode tokens
D: 40 decode tokens

With static batching, A and B finish early, but their slots remain effectively occupied until C and D finish. With continuous batching, the scheduler can insert E and F into the next decode steps after A and B are done.

The GPU sees more useful work per step. The queue drains faster. Throughput rises, especially at high concurrency and varied output lengths.

The catch is that better throughput does not automatically mean better user experience. A scheduler can improve aggregate tokens per second while making tail latency ugly. That is why every serving experiment needs both throughput and latency percentiles.

The metrics

Track at least:

1. Time to first token.
2. Inter-token latency.
3. End-to-end latency.
4. Output tokens per second.
5. Request throughput.
6. Queue wait time.
7. GPU memory used by KV cache.
8. Active live tokens.

Those metrics separate three different experiences:

Fast first token, slow completion. Good for chat responsiveness, bad for long answers.

High throughput, high queue wait. Good benchmark headline, bad product.

Low latency, low utilisation. Pleasant at small scale, expensive at real traffic.

Inference engineering is choosing which tradeoff the product actually wants.

The scheduler knobs

Common knobs include:

1. Maximum number of concurrent sequences.
2. Maximum number of batched tokens.
3. Prefill chunk size.
4. Admission policy for long prompts.
5. Priority classes.
6. KV-cache memory fraction.

The dangerous knob is the one that sounds purely technical but changes product behaviour. For example, shrinking max batched tokens may improve latency for short requests while starving long-context jobs. Increasing concurrency may raise throughput while making time-to-first-token worse.

Treat scheduler changes like product changes. Measure the distribution, not just the mean.

Prefix caching

Prefix caching reuses KV blocks when requests share a prompt prefix. It is common in workloads with repeated system prompts, retrieval templates, agent scaffolds, or batch evaluation. It can be close to free when it applies, but it does not change the model's outputs; it skips repeated prefill work for shared prefixes.

The reason it belongs beside continuous batching is that both are scheduler-level wins. Neither changes the model. Both change how much useful work the serving system extracts from the same GPU.

References

• vLLM documentation
• vLLM automatic prefix caching design
• Efficient Memory Management for Large Language Model Serving with PagedAttention

Checkpoint

1. Why does static batching waste capacity during decode?
2. Name two metrics that can get worse while throughput improves.
3. Why is prefix caching a scheduler/system optimisation rather than a model-quality optimisation?