Prefill or decode: the incident

"It's slow" is not a ticket. TTFT, ITL, queue wait, and OOM are tickets. This drill is the habit of splitting them before anyone opens a profiler.

The product

A 13B bf16 chat model on one A100 80GB. Serving stack does continuous batching. Advertised:

P95 TTFT  <  800 ms   (2K prompt)
P95 ITL   <  40 ms

Peak bandwidth 2.0 TB/s. Weights ≈ 26 GB. KV ≈ 160 KiB/token (assume that number; you will compute a real one in module 5).

Part A — First arithmetic

1. Batch-1 decode floor in ms/token and tok/s.
2. Is a 40 ms ITL SLO above or below that floor?
3. A 2K prefill is roughly 2 × 13e9 × 2048 FLOPs for the matmuls. Compute a compute-only floor in ms on 312 TFLOP/s. This ignores attention and is therefore optimistic.

Write whether the SLOs are physically possible at batch 1 before any software is blamed.

Part B — Five tickets

For each ticket: phase, metric to plot, one plausible cause, one fix that could work, one fix that cannot.

B1. Users say "it takes forever to start typing." P95 TTFT is 2.4 s. P95 ITL is 18 ms. GPU util during the wait is 90%+.

B2. Users say "it types slow." P95 TTFT is 220 ms. P95 ITL is 95 ms. Concurrency is 1.

B3. Same as B2 but concurrency is 48. KV memory is 54 GB. ITL is 95 ms. TTFT is 1.8 s. Queue wait is 1.1 s of that TTFT.

B4. Overnight batch jobs (8K prompt, 2K output) share the GPU with chat. Chat TTFT explodes every time a batch job lands. Decode ITL for chat is fine when no batch job is running.

B5. Error rate 2%, mostly CUDA OOM at 8K context. Chat at 2K is fine. GPU memory shows weights 26 GB, KV climbing to 50+ GB, then crash.

Part C — The sentence for standup

Write a six-line status that an on-call engineer could paste:

Symptom:
Phase:
Floor we compared against:
What is actually happening:
Change we will try:
How we will know in 30 minutes:

Pick B3 as the active incident.

Rules

1. If ITL is near the bandwidth floor at batch 1, you do not get to "optimise the kernel" as the first move.
2. Queue wait is not model time. Splitting them is mandatory in B3.
3. OOM at long context is a KV problem until proven otherwise.

Acceptance

1. Batch-1 floor ≈ 26e9 / 2.0e12 = 13 ms/tok (77 tok/s). 40 ms SLO is above the floor — possible.
2. Optimistic 2K prefill floor is tens of ms, not seconds — so a 2.4 s TTFT is not "the matmul is slow," it is queueing, prefill contention, or a huge prompt.
3. B1 is prefill/TTFT. B2 at concurrency 1 is decode. B3 is admission/concurrency. B4 is noisy-neighbour prefill. B5 is KV capacity.
4. The B3 standup names queue wait and live-token limits, not FlashAttention.

Stretch

Sketch an admission rule that protects chat TTFT from B4 without banning batch jobs: two queues, a max batched-token cap for prefill, or a reserved KV fraction. Pick one and state the metric that would prove it worked.

Check your work

Floor. 13 ms/tok. SLO 40 ms has headroom at batch 1. If B2's 95 ms is at concurrency 1, something else is wrong (sync, sampling on host, non-overlap, or a measurement that included prefill).

B1. Prefill. Plot TTFT vs prompt length. Cause: long prompts or unchunked prefill blocking. Fix: chunked prefill, prompt limits, more prefill compute. Cannot-fix: decode kernel.

B2. Decode. Plot ITL at concurrency 1 against the 13 ms floor. 95 ms is ~7× the floor — look for host stalls, tiny kernels, or copying, not "we need H100." Cannot-fix: shortening the system prompt (that is TTFT).

B3. Mixed. Queue wait 1.1 s is the TTFT story. High concurrency is the ITL story. Fix: cap max_num_seqs, reserve KV for interactive, separate pools. Cannot-fix: a fused GELU.

B4. Prefill noisy neighbour. Fix: isolate batch prefill, priority class, chunked prefill with a token budget. Cannot-fix: quantising KV only (might help capacity, not the prefill collision).

B5. KV OOM. Fix: GQA (already in some 13B), KV quant, paging, hard cap on context, admission on live tokens. Cannot-fix: torch.compile on the decode step.

Debrief

Prefill and decode are different machines that happen to share weights. A serving incident that does not say which one is being slow is not ready for engineering yet.