Inference practice
M03.02·Kernels·Core·90 minutes·3 min read

Coalescing and the transpose tax

A free transpose, a slow consumer, and a contiguous copy that may or may not be worth it. Account for the extra segments.

Module
The GPU execution model
Objective
Explain when a view is free, when the next kernel pays in uncoalesced loads, and when .contiguous() is a measured trade — not a reflex.

Warps load together. Neighbours in the warp should be neighbours in HBM. That is coalescing. A transpose that only rewrites strides does not move bytes at creation time. It can force the next kernel to ask HBM for a strided mess.

Part A — Picture the warp

A row-major float16 matrix M of shape (4096, 4096). Warp of 32 threads.

  1. Threads read M[0, 0:32]. How many 32-byte sectors is a best case 64-byte transaction story (order of magnitude: 1–2 transactions vs 32)?
  2. Threads read M[0:32, 0] (a column). Why is that the bad case?
  3. T = M.t() as a view. Who pays, t() or the kernel that streams T as if it were row-major?

Part B — The consumer

a = torch.randn(4096, 4096, device="cuda", dtype=torch.float16)
b = a.t()                 # view
c = b.contiguous()        # copy
y1 = b * 2                # strided
y2 = c * 2                # dense

Predict, then (if you have a GPU) measure with the honest timer:

  1. Time of t() vs contiguous() vs each multiply.
  2. Which multiply should be closer to peak GB/s?
  3. A sentence for the review: ".contiguous() is not free; it is a prepayment." When is the prepayment worth it?

Part C — Gather / embedding

A kernel reads random rows of a (100_000, 4096) fp16 table, batch 8.

  1. Coalescing: good or bad? Why "random rows" is the whole story.
  2. Occupancy vs coalescing: which do you suspect first?
  3. One systems-level mitigation (cache, padding, grouping) in one sentence — no custom CUDA required.

Acceptance

  1. Row read: coalesced. Column read: many segments. Transpose view defers the tax.
  2. y1 slower / lower GB/s than y2; contiguous() cost ≈ one full 32 MiB×2 traffic (read+write of 32 MiB? 4096²×2 = 32 MiB storage; copy reads 32 writes 32).
  3. You do not sprinkle .contiguous() on every view; you measure the consumer.
  4. Embedding gather: uncoalesced first, occupancy second.

Stretch

Bank conflicts: 32 threads hit the same smem bank. In four sentences, why tiled softmax in shared memory cares, and why this is not the same as HBM coalescing.

Check

a.t() ~0 µs of traffic. contiguous() ~32+32 MiB. Pointwise on strided b can drop to a fraction of peak GB/s. If many consumers follow, pay once. If one tiny consumer follows, maybe live with strides. Gather: each thread's address is a lottery; the bus fetches fat segments for skinny useful bytes.

Debrief

Module 3 is the reason module 1's "transpose is free" came with a warning. The view is free. The access pattern is not.