Timing a GPU without lying to yourself
CUDA is asynchronous, so the obvious way to time a kernel measures how long it took to queue the work. Build the harness that measures the work instead.
By the end
Benchmark a GPU operation correctly, and explain why the same kernel has no single speed.
Run it
This lesson has a notebook. Read the page first, then work through the cells — every # PREDICT: marker is a place to commit a number before you run it.
In Colab: File ▸ Upload notebook, then set the runtime to GPU. Locally: jupyter lab.
Every measurement in the rest of this course depends on this lesson, and it is the one people skip.
The failure
CUDA kernel launches are asynchronous. When Python returns from x @ x, the multiplication has been queued, not performed. So this:
t0 = time.perf_counter()
y = x @ x
print(time.perf_counter() - t0)
measures how long it took to put an item on a queue. On a large matmul the honest number can be a hundred times larger. The reading is not noisy or approximate — it is measuring a different thing entirely, and it is confidently, reproducibly wrong.
The notebook shows the two numbers side by side. It is worth seeing once, because the wrong version looks perfectly reasonable in isolation.
The three rules
Warm up. The first call to any operation pays costs you do not want in your measurement: lazy CUDA initialisation, memory pool growth, algorithm autotuning, and — if you are using torch.compile — an entire compilation. Run it several times before you start timing.
Synchronize, or use events. Either call torch.cuda.synchronize() so the CPU waits for the GPU to finish, or record CUDA events on the stream and ask the device how much time passed between them. Events are better: they measure on the GPU's own clock and do not include the cost of the sync itself.
Repeat and take the median. One sample is noise. Take tens of samples and use the median rather than the mean — the mean is dragged around by a single scheduling hiccup, and you care about typical behaviour.
Those three rules become a nine-line function in the notebook. Retype it in each later notebook rather than importing it; typing it a few times is how the rules stop being a checklist and become reflex.
Turning time into a rate
A raw millisecond figure is hard to sanity-check. A rate is not, because you can compare it against what the hardware claims.
For a square matmul, the operation count is 2n³ — one multiply and one add per element of the inner product. So:
TFLOP/s = 2 * n**3 / (seconds * 1e12)
Sweep n and something important shows up:
| n | ms | TFLOP/s |
|---|---|---|
| 512 | small | poor |
| 1024 | better | |
| 2048 | better still | |
| 4096 | large | close to peak |
The exact figures depend on your card. The shape is the lesson. Small matrices achieve terrible throughput on hardware that is perfectly capable, because the work is too small to fill the machine and the time is dominated by launching the kernel at all. That is the overhead-bound regime from module 0, showing up in a measurement for the first time.
Which gives you the rule that governs everything that follows:
A kernel does not have a speed. It has a speed at a size.
Any benchmark quoted without the shape it was measured at is close to meaningless, and any optimisation validated at one size should be re-checked at the size you actually run.
Predict first
Before running the sweep, write down the n at which you expect throughput to flatten out, and why. Then run it. Most people are surprised by how large the matrices have to get, which is exactly the kind of calibration error worth finding now rather than during a production incident.
Checkpoint
- Why does timing a CUDA op without synchronising give a small number?
- Why take the median rather than the mean?
- Your matmul achieves 8 TFLOP/s at n=256 and 60 TFLOP/s at n=4096 on the same card. What is limiting the small case, and is it worth optimising the kernel?
Practice this lesson
The reading is the model. These drills are the hours — 2 problems that force the numbers onto paper before the next lesson.