Named ranges and the five-line report

A trace without names is a map with the cities scraped off. This drill is the naming habit, then the report habit.

Part A — Name the program

You will profile one chat request: tokenize, 2K prefill, 64 decode tokens, detokenize, JSON response.

Write the exact record_function / NVTX strings you will wrap, nested. At least:

request
  tokenize
  prefill
  decode
    decode_step   (loop)
  detokenize

Add two more ranges that would save you an hour next week. Name them.

Then write the PyTorch profile(...) call: activities, record_shapes, profile_memory, and what you sort the table by first.

Part B — Report from a named table

range / op	CUDA total	CPU total	calls
prefill	140 ms	8 ms	1
decode	96 ms	88 ms	1
decode_step	1.4 ms avg	1.3 ms avg	64
aten::mm	110 ms	6 ms	48
aten::copy_	22 ms	24 ms	64
tokenize	0 ms	31 ms	1

Workload: 7B bf16, batch 1, L4, 2K→64.

Write:

Workload:
Hardware:
Top CUDA time:
Top CPU / orchestration cost:
Next experiment:

Then: one sentence on whether decode is kernel-bound or host-bound, using the decode vs decode_step rows.

Part C — Without looking

Write the five field names from memory. Fill them for a run where tokenize is 400 ms and GPU decode is 40 ms. Next experiment must not mention FlashAttention.

Acceptance

1. Ranges nest prefill vs decode. copy_ inside decode_step is visible as a hypothesis (dtype/device).
2. Prefill owns CUDA; decode's CPU ≈ CUDA suggests host/sync per token.
3. 400 ms tokenize is a TTFT bug on the host.

Check

Sort CUDA first. Prefill mm is expected. 64 copies in 64 steps is a smell. decode CPU 88 ms vs GPU 96 ms: the loop is not "free." Tokenize is TTFT, not a kernel.

Debrief

The first profiler pass is triage with names. If you cannot write the five lines, you are not done collecting — you are decorating a JSON file.