Honest GPU Timer

Prompt

Write a function:

def bench_cuda(fn, *, warmup=20, repeat=100):
    ...

It should benchmark a zero-argument callable fn that launches CUDA work.

Requirements

1. Run warmup iterations before measuring.
2. Use torch.cuda.Event(enable_timing=True) for timing.
3. Synchronize before the measurement loop begins and after each measured event pair.
4. Return a dictionary with median_ms, p10_ms, p90_ms, and samples.
5. Raise a useful error if CUDA is unavailable.
6. Do not include tensor allocation in the timed function unless the caller put it inside fn intentionally.

Test workload

Use the harness on:

x = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
w = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)

def matmul():
    return x @ w

Then test a deliberately bad version:

def alloc_and_matmul():
    x = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
    w = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
    return x @ w

Deliverable

Submit:

1. The benchmark function.
2. The two result dictionaries.
3. A paragraph explaining why the second benchmark answers a different question.
4. A sentence describing when including allocation would be the correct thing to do.

Acceptance criteria

The solution is correct if:

1. Removing synchronization makes the reported time suspiciously tiny.
2. The function produces stable-ish medians across repeated calls.
3. The bad workload is called bad because its timed region changed, not because allocation is morally wrong.

Stretch

Add an optional bytes_moved argument and report effective bandwidth in GB/s when it is provided.

Debrief

The harness is a ritual. Warm up, measure on the device clock, repeat, summarize. Everything later in the track assumes this reflex.

Part B — Catch three lies

Extend the harness (or wrap it) so each of these fails loudly or is documented as a different question. For each, write the median you would trust, and one sentence on what the naive timer is actually measuring.

Lie 1. time.perf_counter() around y = x @ w with no synchronize.

Lie 2. Include the first call after torch.compile(fn) in the timed loop.

Lie 3. repeat=1 on a noisy Colab GPU.

Then run a size sweep for n in [256, 512, 1024, 2048, 4096] on the honest timer. Plot or table: median ms, TFLOP/s. Write three sentences:

1. Where is the kernel overhead-bound?
2. Where does TFLOP/s flatten?
3. Why "the matmul takes X ms" is not a sentence you are allowed to say without n.

Part C — Bandwidth from the same harness

Time y = x + x for x of shape (4096, 4096) fp16. Traffic is 3 × n² × 2 bytes (two reads, one write). Report GB/s against a spec-sheet peak for the card you used (or L4 300 GB/s if you are on paper). Classify: memory, compute, or overhead.

Check

Lie 1 measures launch. Lie 2 measures compilation. Lie 3 measures noise. Small n is overhead. Large n GEMM approaches a TFLOP/s plateau. The add should land near a large fraction of HBM, not near tensor peak.