Tensors and the bytes they move

The previous module argued that performance is mostly an accounting problem: bytes moved, divided by how fast they can move. This lesson gets you the numerator.

Bytes, not shapes

A tensor's shape is a logical description. What the memory system cares about is numel × element_size, and PyTorch will tell you both:

x = torch.zeros(1024, 1024, dtype=torch.float32)
x.numel()         # 1_048_576
x.element_size()  # 4
x.nbytes          # 4_194_304

Change nothing but the dtype and the traffic changes proportionally:

dtype	bytes/element	1024×1024 tensor
float32	4	4.19 MB
bfloat16 / float16	2	2.10 MB
int8	1	1.05 MB

That table is the whole quantization argument in miniature. The shape did not change, the arithmetic did not change, but a memory-bound kernel over that tensor just got twice as fast because there is half as much to read.

Apply it to something real. A 7-billion-parameter model in bfloat16 holds roughly 14 GB of weights. Hold onto that number — in lesson three it becomes the reason token generation has a hard speed limit.

Views and copies

Some operations hand back a new view over the same underlying storage and move nothing at all. Others allocate fresh memory and copy. The difference is invisible in the source and enormous in the profile.

a = torch.arange(12).reshape(3, 4)
b = a.t()               # a view — no data moved
d = a.t().contiguous()  # a copy — a full pass over memory

You can prove which is which by comparing storage pointers, and the notebook does exactly that for a list of common operations. Predict each one before you run it; the ones people get wrong are usually reshape (sometimes a view, sometimes not) and dtype casts (always a copy).

Strides are the mechanism

A tensor is a flat buffer plus a set of strides — how far to step in memory to move one position along each axis. A transpose does not touch the buffer; it swaps the strides.

a.stride()      # (4, 1)  — contiguous
a.t().stride()  # (1, 4)  — same buffer, walked differently

This is why a transpose is free. It is also why the next operation may be slower than you expect: it now walks memory in a pattern the hardware likes less. A GPU reads memory in wide contiguous chunks, so a kernel striding across rows can end up fetching far more than it uses.

That effect has a name — coalescing — and a full treatment waiting in module 3. For now the useful takeaway is narrower and still valuable: .contiguous() is never free, and a "free" transpose sometimes just moves the cost into whatever runs next.

What this unlocks

You can now look at any tensor operation and answer how many bytes does this touch. Combined with honest timing — the next lesson — that gives you a measured bandwidth figure, which you can compare against what the card can actually sustain. When those two numbers are close, you have found a memory-bound kernel and you know precisely which lever applies.

Checkpoint

1. How many MB is a float16 tensor of shape (4096, 4096)?
2. Which of view, reshape, t(), slicing, and .to(torch.float16) can return a view?
3. A transpose is free. Give a concrete reason the operation after it might not be.