Decode floor calculator

The lesson's punch line is one formula:

min_seconds_per_token  =  weight_bytes  /  bandwidth
tokens_per_second_ceiling  =  1  /  min_seconds_per_token

This drill is that formula, used until it is muscle memory, including the cases where people "beat" it and are therefore wrong.

Part A — Weight bytes

Compute weight storage for:

model	params	dtype	bytes
7B	7e9	bf16
7B	7e9	int8
13B	13e9	bf16
70B	70e9	fp16
70B	70e9	int8
70B	70e9	int4 (packed)

Use params × bytes_per_param. Ignore embedding double-counting and MoE routing. This is a floor on weights, not a full VRAM model.

Part B — Floors

Cards:

card	bandwidth
L4	300 GB/s
A100	2.0 TB/s
H100 SXM	3.35 TB/s
H200	4.8 TB/s

Fill tokens/s ceilings at batch 1, ignoring KV:

	L4	A100	H100	H200
7B bf16
7B int8
70B int8
70B int4

Mark cells where the weights themselves do not fit in 24 GB (L4) or 80 GB (A100/H100) or 141 GB (H200). A ceiling that requires the model to magically fit is not a ceiling. It is a fantasy.

Part C — Lie detection

An intern reports these measurements. For each, say plausible, optimistic but possible (say why), or impossible (name the floor it crossed).

1. 7B bf16, batch 1, L4, 19 tok/s.
2. 7B bf16, batch 1, L4, 80 tok/s.
3. 7B bf16, batch 1, A100, 120 tok/s.
4. 7B bf16, batch 32, A100, 2,400 tok/s total.
5. 70B fp16, batch 1, A100 80GB, 40 tok/s.
6. 70B int8, batch 1, H100, 55 tok/s.

Assume "tok/s" means output tokens per second for that configuration, and "total" means summed across the batch.

Part D — KV is not free

7B, 32 layers, 32 heads, head_dim 128, GQA with 8 kv heads, bf16.

1. Bytes of KV per token. (Use the lesson formula.)
2. At batch 1, 512 context, is KV traffic large compared to 14 GB of weights?
3. At batch 32, 8K context, is it still a rounding error?
4. Rewrite the decode-step traffic model:

bytes_per_step ≈ weight_bytes + kv_bytes_per_token × live_tokens

and compute both ends of (2) and (3).

Rules

1. 1 GB = 1e9 bytes for this drill (decimal). Do not mix GiB unless you convert both sides.
2. Batch-1 decode ceilings assume you read every weight once per token.
3. Fitting is a boolean. If it does not fit, the cell is "n/a," not a small number.

Acceptance

1. 7B bf16 is 14 GB. 70B int8 is 70 GB. 70B int4 is 35 GB.
2. L4 7B bf16 ceiling ≈ 21 tok/s. A100 ≈ 143. H100 ≈ 239.
3. Item C2 is impossible on L4 at batch 1. Item C5 is impossible because 140 GB does not fit on 80 GB.
4. Part D shows KV is noise at 512×1 and not noise at 8K×32.

Stretch

Recompute Part B for 7B bf16 at batch 8, still ignoring KV. The weight traffic does not grow. What happens to the per-sequence tok/s ceiling vs the aggregate tok/s ceiling? That distinction is the whole serving business.

Check your work

Weights. 7B bf16: 14e9. 7B int8: 7e9. 13B bf16: 26e9. 70B fp16: 140e9. 70B int8: 70e9. 70B int4: 35e9.

L4 7B bf16. 14e9 / 300e9 = 0.0467 s → 21.4 tok/s.

A100 7B bf16. 14 / 2000 = 0.007 → 143 tok/s.

H100 7B bf16. 14 / 3350 ≈ 0.0042 → 239 tok/s.

H200 7B bf16. 14 / 4800 ≈ 0.0029 → 343 tok/s.

Int8 halves the bytes, doubles the ceiling, on a bandwidth-bound step. That is the entire decode-quantisation argument.

C1 plausible (under the 21 floor). C2 impossible at batch 1. C3 plausible (under 143). C4 aggregate 2400 / 32 = 75 per sequence, under 143, and batching is supposed to raise aggregate — plausible order of magnitude if KV has not exploded. C5 impossible: weights do not fit. C6 H100 70B int8 floor 70e9 / 3.35e12 ≈ 21 ms → 48 tok/s. 55 is slightly over peak; call it impossible-or-measurement-includes-something-else (speculative, cached prefixes, or they were not at batch 1).

KV. 32 * 2 * 8 * 128 * 2 = 131072 bytes/token = 128 KiB/token.

512 tokens: 64 MiB. Versus 14 GB of weights: ~0.5%. Noise.

32 × 8192 = 262144 live tokens × 128 KiB ≈ 32 GiB of KV, and the traffic each step is no longer "just the weights." Batching stopped being free.

Debrief

A tokens/s number without batch, context, dtype, and phase is atmosphere. A tokens/s number that beats bandwidth / weights at batch 1 is a bug. Keep that sentence until module 5, where you will use it in a capacity plan.