KV Cache Capacity Planner

Prompt

You are deploying a model with:

layers: 32
kv_heads: 8
head_dim: 128
KV dtype: bf16
GPU memory: 48 GiB
weights and runtime overhead: 22 GiB
reserve for fragmentation/workspace: 4 GiB

Product wants to support:

1. Chat: 2K prompt, 512 output.
2. Long document QA: 16K prompt, 1K output.
3. Agent traces: 8K prompt, 4K output.

Deliverable

Create a capacity plan:

workload	tokens/request	KV GiB/request	rough max concurrent requests

Then answer:

1. Which workload is most dangerous for concurrency?
2. What is the max live-token budget after weights, runtime overhead, and reserve?
3. What admission control rule would you add before launch?
4. What metric should alert before OOM?

Rules

1. Use the KV formula from the lesson.
2. Treat 1 GiB as 1024 ** 3 bytes.
3. Do not use all theoretical capacity. Leave the stated reserve untouched.

Acceptance criteria

A good answer turns memory into product behavior:

1. It does not say "the model fits" and stop.
2. It explains why long document QA and agent traces need separate limits.
3. It proposes a live-token or KV-memory admission rule.

Stretch

Recompute the table for a model with 4 KV heads instead of 8. Explain the product impact without using the phrase "it is faster" unless you can justify the mechanism.

Debrief

Serving capacity is live tokens, not vibes. Once you can compute the cache, you can argue about context windows like an engineer instead of a fortune teller.

Part B — Show the arithmetic

Bytes/token = 32 × 2 × 8 × 128 × 2 = 131072 = 128 KiB/token.

Free pool = 48 − 22 − 4 = 22 GiB = 22 × 2³⁰ bytes.

Live-token budget ≈ 22 × 2³⁰ / 131072 ≈ 180,224 tokens.

Fill the table with that budget (not 22 GiB / request_tokens guessed):

workload	live tokens	KV/request	max seqs
Chat 2560	2560	2560 × 128 KiB ≈ 0.31 GiB	floor(180224/2560) ≈ 70
Doc QA 17408	17408	≈ 2.12 GiB	≈ 10
Agent 12288	12288	≈ 1.50 GiB	≈ 14

If your Part A numbers are not within ~10% of this, redo the bytes/token line before you invent admission rules.

Part C — Mixed load

The GPU does not run one product. It runs 40 chats (2560 tok) plus some Doc QA. How many Doc QA jobs fit in the remaining tokens? Write the inequality the scheduler should evaluate on every admission.