Regime from the symptoms

The lesson is eight hundred words. This drill is the hours. You do not need a GPU. You need a pencil, and you need to refuse the instinct that says "write a kernel."

Why this drill exists

Almost every bad optimisation week starts with a true statement attached to the wrong regime. "Attention is expensive." Yes. So is the copy that precedes it. So is the Python loop that launches it 4,096 times. The regime test is how you stop spending a week on the true statement that is not the bottleneck.

Part A — Classify

For each incident, write four lines:

Regime:
Evidence:
Lever that helps:
Lever that wastes the week:

A1. A 4096×4096 float16 matmul reports 70% of the card's peak TFLOP/s. An engineer wants to fuse the preceding relu.

A2. An elementwise chain relu(x * 2 + 1) * 0.5 on a (8, 4096, 4096) tensor is 3.8× slower than the same math under torch.compile. Achieved bandwidth of the eager chain is 1.7 TB/s on a 2.0 TB/s card.

A3. The same chain on a (8, 32, 32) tensor. Eager and compiled both take ~12 µs. The compiled version is 1.08× faster.

A4. Decode of a 7B bf16 model at batch 1, context 512, reports 38 tokens/s on an L4 (300 GB/s). An engineer proposes switching the attention kernel to a "faster" MMA-heavy implementation.

A5. Prefill of a 4,096-token prompt on H100 reports 62% of peak tensor TFLOP/s. TTFT is 180 ms. Product wants 80 ms.

A6. A custom CUDA extension launches 40,000 pointwise kernels per request. Nsight shows a picket fence: 2 µs of kernel, 8 µs of gap, repeat. The kernels themselves are well written.

A7. After tensor.item() in a decode loop, ITL jumps from 8 ms to 14 ms. The matmul and attention kernels are unchanged.

A8. A 70B int8 model at batch 8, 2K context, saturates HBM and holds 55% occupancy. Someone wants to raise occupancy by cutting registers.

Part B — The ridge, by hand

A card claims 312 TFLOP/s and 2.0 TB/s.

1. Compute the ridge point in FLOP/byte.
2. Place these kernels relative to the ridge. Show the arithmetic.

kernel	FLOPs	bytes
elementwise add, n=4096, fp16	1.68e7	5.03e7
matmul, n=4096, fp16	1.37e11	1.01e8
decode step, 7B, fp16, batch 1	~1.4e10	~1.4e10

3. For each kernel, state the only class of optimisation allowed.

Part C — Write the argument

In one paragraph, explain why "we should write a fused kernel" is sometimes the right sentence and sometimes a career-limiting one. Use at least one number from Part A or B.

Rules

1. Do not reach for a profiler. The point is to decide before the tool.
2. Intensity is FLOPs / bytes. Ridge is peak FLOP/s / peak bytes/s.
3. If shrinking the tensor 4× barely changes time, it is not memory-bound or compute-bound yet.
4. Occupancy is not a regime.

Acceptance

A passing write-up:

1. Puts A1 and A5 in compute, A2 and A4 and A8 in memory, A3 and A6 and A7 in overhead / host.
2. Refuses fusion as the fix for A1.
3. Refuses a "faster MMA kernel" as the fix for A4.
4. Computes a ridge near 156 FLOP/byte.
5. Places decode far below the ridge and the 4096 matmul far above it.

Stretch

Take a real kernel from your machine — or from a blog benchmark that quotes a shape. Compute intensity, pick a ridge for that card, classify it. If the blog does not quote a shape, write one sentence about why the benchmark is unusable.

Check your work

Try first.

A1. Compute-bound. Fusion of an adjacent relu does not change the matmul's arithmetic or its traffic in any way that matters. Waste: writing a fused relu-matmul for a kernel already on tensor cores.

A2. Memory-bound. Achieved GB/s near peak. Fusion (or compile) is the lever because it deletes intermediate round trips.

A3. Overhead-bound. 12 µs is launch-scale. Fusion still helps a little (fewer launches) but the 3.8× traffic win is gone because there was never much traffic.

A4. Memory-bound decode. Floor is 14 GB / 300 GB/s ≈ 47 ms/tok ≈ 21 tok/s. 38 tok/s is already in the right neighbourhood (or the measurement includes some batching / quant — either way, MMA will not 2× it). Waste: a compute kernel rewrite.

A5. Compute-bound prefill. Faster math, more GPUs, shorter prompts, or chunked prefill with more parallel compute. Not KV paging.

A6. Overhead-bound. Fuse, graph-capture, or stop launching 40k grids. Do not micro-optimise the 2 µs kernel.

A7. Host / sync. .item() stalls the CPU on the GPU. Remove the per-token host read. The kernels were never the problem.

A8. Memory-bound. Occupancy theatre. Cut traffic (quant, GQA, paging), not registers.

Ridge. 312e12 / 2.0e12 = 156 FLOP/byte. Add intensity ≈ 0.33, below. Matmul intensity ≈ 1360, above. Decode intensity ≈ 1, below.

Debrief

If you can do this without a GPU, you have the only prior that later modules need. The notebooks exist to stop you lying to yourself about the numbers. The regime test exists to stop you lying to yourself about the kind of number.