Inference practice
M02.03·Profiling·Hard·90 minutes·2 min read

Timeline Idle Gap Detective

Given a toy timeline, identify whether the GPU is waiting on launch overhead, host work, synchronization, or data transfer.

Module
See inside the box
Objective
Explain GPU idle gaps from timeline evidence and propose the smallest confirming experiment.

Prompt

You are given a simplified timeline for one decode request:

0.000 ms  CPU decode_step begin
0.030 ms  CPU sample previous token
0.090 ms  CPU tensor.item()
0.250 ms  CPU launch matmul
0.258 ms  GPU matmul begin
0.410 ms  GPU matmul end
0.422 ms  CPU launch attention
0.430 ms  GPU attention begin
0.590 ms  GPU attention end
0.760 ms  CPU logging callback
1.050 ms  CPU decode_step end

This pattern repeats for every generated token.

Deliverable

Write a diagnosis with:

  1. The largest idle gap.
  2. The likely cause.
  3. Whether this is a kernel problem, host problem, transfer problem, or synchronization problem.
  4. The first code change you would try.
  5. The measurement that would prove the change helped.

Constraints

  1. You are not allowed to change the model weights.
  2. You are not allowed to change output quality.
  3. You may change logging, sampling implementation, batching, and where scalar values are read.

Acceptance criteria

A good answer should notice that:

  1. tensor.item() can force the host to wait for GPU work.
  2. Logging inside the per-token loop can create host-side gaps.
  3. The matmul and attention kernels are not obviously the first target.
  4. The repeated pattern matters more than any single token's timeline.

Stretch

Rewrite the decode loop as pseudocode where all per-token host reads are delayed until after generation, unless they are required for sampling correctness.

Debrief

Timelines teach humility. The GPU may be doing exactly what you asked and still spend most of the request waiting for the host to ask again.

Part B — Budget the step

From the timestamps:

  1. GPU busy time in the 1.050 ms step (matmul + attention).
  2. Fraction of the step the GPU is idle.
  3. Tokens/s if this step is the steady-state ITL (ignore prefill).
  4. Tokens/s if you deleted the .item() stall and the logging gap (assume GPU work stays 0.322 ms and launch tax stays ~0.03 ms — state your other assumptions).

Part C — Second trace

0.000  CPU launch fused decode graph
0.012  GPU graph begin
0.340  GPU graph end
0.348  CPU step end

Same model, now graph-captured. Write five lines: what smell died, what smell might remain at batch 1, and what you still cannot claim (kernel math got faster).