Roofline Classifier
Classify toy kernels as memory-bound, compute-bound, or overhead-bound from flops, bytes, time, and hardware ceilings.
- Module
- PyTorch and the memory wall
- Objective
- Use arithmetic intensity and measured rates to choose the right optimization lever.
Prompt
You are given measurements from three kernels on a GPU with:
peak compute: 120 TFLOP/s
peak memory bandwidth: 3,000 GB/s
typical launch overhead: 8 us
Measurements:
| kernel | flops | bytes moved | measured time |
|---|---|---|---|
| A | 2.1e9 | 8.6e9 | 3.2 ms |
| B | 9.0e12 | 1.2e11 | 82 ms |
| C | 1.0e6 | 1.0e6 | 12 us |
Deliverable
For each kernel:
- Compute arithmetic intensity.
- Compute achieved TFLOP/s.
- Compute achieved GB/s.
- Classify the limiting regime: memory-bound, compute-bound, or overhead-bound.
- Pick one optimization lever and one lever that would probably waste time.
Hints
Arithmetic intensity:
flops / bytes_moved
Rough roofline threshold:
peak_flops / peak_bandwidth
If a kernel runs in roughly launch-overhead time and barely does work, it is not meaningfully memory-bound or compute-bound yet. It is too small.
Acceptance criteria
The answer should not just label kernels. It should connect each label to action:
- Memory-bound: reduce bytes, improve locality, fuse away intermediates.
- Compute-bound: use faster math paths, better tiling, tensor cores, lower precision when valid.
- Overhead-bound: batch more work, fuse, compile, use CUDA graphs, or move the loop.
Stretch
Add a fourth row for a real measurement from your machine and classify it with the same method.
Debrief
The roofline model is not a perfect oracle. It is a guardrail against optimizing the wrong scarce resource.
Part B — Same kernels, your ridge
Use the A100 ridge ~156 FLOP/byte and the L4 ridge ~807 FLOP/byte. Reclassify A, B, C on both cards. Which kernel changes class? Which never will?
Part C — Decode as a fourth row
Add kernel D: 7B decode step, flops = 1.4e10, bytes = 1.4e10, time = 8.0 ms. Classify on A100 (2.0 TB/s, 312 TFLOP/s). Compute achieved GB/s. Is 8 ms above or below the bandwidth floor 14e9 / 2.0e12?
Check
A: intensity ~0.24, ~2.7 TB/s, memory-bound. B: intensity 75, ~110 TFLOP/s, near-compute on this 120 TFLOP card. C: 12 µs vs 8 µs launch — overhead. D: intensity 1, 1.75 TB/s of 2.0, memory-bound; floor is 7 ms so 8 ms is a plausible engine. D never becomes compute-bound on L4 either.