Fusion, and the traffic you did not need

You now have the two halves — traffic and honest timing. This lesson uses them on the optimisation that matters most at this layer, and closes module 1 by putting all three regimes on one table.

The waste

This looks like one operation:

torch.relu(x * 2.0 + 1.0) * 0.5

Executed eagerly it is several kernels. Each one reads its input from memory, does a trivial amount of arithmetic, and writes its output back. Those intermediates are used exactly once, immediately, by the next kernel — and they still make a full round trip to HBM and back, because a kernel's output has nowhere else to live.

The arithmetic here is nearly free. The traffic is the entire cost, and most of the traffic is unnecessary.

Fused, the floor is one read and one write. Everything between stays in registers and never leaves the chip. So the available speedup is roughly the ratio of unfused traffic to fused traffic — a number you can compute before running anything, which is exactly what the notebook asks you to do.

Removing it

compiled = torch.compile(chain)
_ = compiled(x)          # compile now, so you do not time the compiler

torch.compile captures the graph and generates a single fused kernel. Note the throwaway call: the first invocation pays for compilation, and including it in a benchmark is the most common way to produce a nonsense result with this API. This is the warm-up rule from lesson two, in its most expensive form.

Then measure, and check the speedup against your predicted traffic ratio. If they roughly agree, your model of the kernel is correct. If they do not, something else is the binding constraint — and at small sizes, it usually is.

Explaining it in bytes

The discipline worth building here is refusing to describe a result as "optimised" or "faster". Say what changed in bytes:

Unfused, this moved about 3× the input size per step across three steps. Fused, it moves 2× the input size total. The predicted speedup was ~4×, I measured 3.6×, and the gap is the launch overhead the fused version still pays once.

That sentence is worth more in a design review than any amount of profiler output, because it demonstrates a model rather than a measurement. It is also the exact form of argument behind the famous attention rewrite: same mathematics, same result, dramatically fewer round trips to memory.

Where fusion stops helping

Shrink the tensor and re-run the comparison. The speedup collapses.

At a few thousand elements there is not enough work for memory traffic to be the constraint. The time is going to launching kernels, dispatching through Python, and synchronising — the overhead-bound regime. Fusing kernels still removes launches, so it helps a little, but the enormous traffic win has evaporated because there was never much traffic.

This completes the picture module 0 set up, with all three regimes now measured rather than asserted:

Regime	Where you saw it	The lever
Compute-bound	Large matmul, near peak TFLOP/s	Less or cheaper arithmetic
Memory-bound	Elementwise chains on large tensors	Move fewer bytes — fuse, quantize, reuse
Overhead-bound	Any operation on tiny tensors	Fewer, bigger launches — batch, fuse, capture

The exercise that matters

The notebook ends by asking you to explain the shape of the size sweep — small speedup at 4096 elements, largest in the middle, flat at the top — in bytes and launches. Write the paragraph. If you can produce it without hedging, module 1 has done its job.

Then predict what changes in float32 and commit the prediction before running it. The answer follows directly from everything above, which is the point.

Checkpoint

1. Why does an unfused chain of three elementwise ops move so much more memory than a fused one?
2. You fuse a chain and get 1.05× instead of the predicted 4×. Name the two most likely explanations.
3. Which regime is a kernel in if torch.compile barely helps and the tensors are tiny?

---

Module 1 complete. You can measure traffic, time a GPU honestly, place a kernel in its regime from your own numbers, and explain a speedup in bytes. Module 2 stops inferring the regime from wall-clock time and reads it directly off a profile.