Profiler Triage Report

Prompt

You receive this simplified profiler table for one request:

op	CUDA total	CPU total	calls	memory
aten::matmul	38.4 ms	4.1 ms	18	0 B
aten::_scaled_dot_product_flash_attention	31.7 ms	2.9 ms	18	0 B
aten::copy_	19.6 ms	20.4 ms	54	2.8 GiB
aten::to	15.1 ms	16.8 ms	54	2.8 GiB
aten::slice	1.4 ms	7.9 ms	436	0 B
tokenizer.encode	0 ms	22.0 ms	1	0 B

Workload:

model: decoder-only LLM
prompt: 4096 tokens
output: 128 tokens
dtype: bf16 weights
batch: 1
hardware: L4

Deliverable

Write a five-line report:

Workload:
Hardware:
Top CUDA time:
Top CPU / orchestration cost:
Next experiment:

Then add:

1. Two hypotheses for why copy_ and to are present.
2. One experiment that would confirm or reject each hypothesis.
3. One thing you would not optimize yet, and why.

Acceptance criteria

A good answer:

1. Separates model math from avoidable movement.
2. Mentions that tokenizer CPU time is outside GPU execution but still affects request latency.
3. Does not propose writing a custom attention kernel while copies and casts are unexplained.
4. Chooses one next experiment, not five.

Stretch

Add NVTX or record_function ranges you would insert before the next run. Name the ranges exactly.

Debrief

The profiler is a discipline tool. It keeps you from choosing glamorous work before the boring evidence has been handled.

Part B — A second table, opposite smell

op	CUDA total	CPU total	calls	memory
aten::matmul	6.1 ms	3.4 ms	240	0 B
aten::add	4.8 ms	5.2 ms	240	0 B
aten::relu	4.1 ms	4.9 ms	240	0 B
aten::copy_	0.4 ms	0.6 ms	8	12 MiB
Python decode loop	0 ms	38.0 ms	128	0 B

Workload: batch 1, prompt 32, decode 128, tiny hidden size, L4.

Write the same five-line report. Then contrast it with Part A in four bullets: what the top CUDA name means in each case, and why a fused attention kernel is the wrong next experiment here.

Part C — The five-line template, memorised

Without looking, write the five field names. Fill them for a fake run where aten::copy_ is 61% of CUDA time on a 7B decode. Next experiment must mention dtype or device movement, not FlashAttention.

Check

Part A: copies/casts unexplained → experiment is "who is calling to() / .cpu()", not a new attention kernel. Tokenizer is CPU-side TTFT. Part B: many tiny launches + Python loop → overhead-bound decode; compile/graphs/batch, not MMA. Five fields: Workload, Hardware, Top CUDA time, Top CPU / orchestration, Next experiment.