Library

Inference Engineering

A course and practice ladder on why models are slow and what the levers actually are — from counting the bytes a tensor moves, up through profiling, GPU execution, KV-cache math, batching, and serving. It ramps deliberately: nothing in a later module is assumed earlier.

Practical modules are notebooks or drills. The pages are short on purpose; the hours live in the problems under each lesson. Run notebooks in Google Colab on the free GPU tier, work the drills on paper or in a local file, and keep a log of prediction against measurement. The holy grail here is not reading — it is calibration.

Start at the beginningOpen the practice ladder
Setup

In Colab

Download a lesson's notebook, then in Colab choose File ▸ Upload notebook. Set Runtime ▸ Change runtime type ▸ GPU before you run anything, or every measurement will be a CPU measurement.

PyTorch is already installed there. The free tier is enough for every module in this course.

Locally in Jupyter

If you have an NVIDIA card, run it at home and the numbers get more stable — no shared tenancy, no session limits.

pip install jupyterlab
pip install torch --index-url \
  https://download.pytorch.org/whl/cu124
jupyter lab
The ramp

5 of 8 modules are open, with 11 lessons and 22 drills — two or more on every open lesson. The pages are short; the work is the ladder. Planned modules stay listed so you can see the road.

00

Orientation

The one mental model the rest of the course hangs off: every workload is waiting on arithmetic, on bytes, or on the launch queue, and optimising the wrong one buys nothing. The reading is short; the drills are the hours.

  1. 01The three regimes8 min · 2 drills
  2. 02Why inference is asymmetric10 min · 2 drills
~5 hours, reading and drills
01

PyTorch and the memory wall

Make the model measurable. Count the bytes a tensor really moves, time a GPU honestly, find the line where a kernel stops being limited by maths and starts being limited by memory, and watch fusion move it. Four notebooks plus a drill on every lesson.

  1. 01Tensors and the bytes they move3 min · 2 drills
  2. 02Timing a GPU without lying to yourself4 min · 2 drills
  3. 03Arithmetic intensity, measured4 min · 2 drills
  4. 04Fusion, and the traffic you did not need4 min · 2 drills
~8 hours, notebooks and drills
02

See inside the box

Stop guessing where the time goes. torch.profiler for operator-level timing, Nsight Systems for the timeline, and learning to read the idle gaps — launch overhead, sync points, transfers. The hours are in the traces you write up, not the pages you read.

  1. 01The first profiler pass4 min · 2 drills
  2. 02Reading the GPU timeline4 min · 2 drills
~8 hours, drills
03

The GPU execution model

Why kernels are fast or slow, before writing any. SMs, warps and occupancy; the latency and bandwidth gaps between registers, shared memory, L2 and HBM; coalescing and bank conflicts. One reading, then drills that force the model to do work.

  1. 01The GPU execution model without mysticism4 min · 2 drills
~6 hours, reading and drills
04

Writing kernels in Triton

Planned

The first kernels, without the C++ tax. Vector add, fused softmax — the first kernel that beats PyTorch and where you can see why in the memory traffic — then tiled matmul.

~20 hours
05

Serving an LLM

Where the whole model pays off: prefill against decode, what the KV cache actually caps, continuous batching, and finding the knee of the throughput-versus-latency curve. Capacity math and a scheduler simulation, not just the argument.

  1. 01KV cache math3 min · 2 drills
  2. 02Continuous batching and the serving scheduler4 min · 2 drills
~8 hours, reading and drills
06

Compilers and graphs

Planned

torch.compile and reading the Triton it generates, CUDA graphs and launch overhead, then C++ and CUDA proper for anyone who wants the specialist path.

months, not evenings
07

Production inference

Planned

The job after the kernel is fast: SLOs and percentiles, dollars per million tokens, tensor / pipeline / expert parallel, and disaggregated prefill-decode.

~10 hours
Practice

The ladder that makes it stick

Each problem produces an artifact, and every open lesson has at least two: a byte ledger, a benchmark, a profiler report, a scheduler simulation, or an operations playbook. That is the difference between a 12-minute page and an 8-hour module.

View every practice problem

Current set

22

drills across accounting, benchmarking, profiling, kernels, serving, and operations.

M00.01Regime from the symptomsAccounting
M00.02Ridge-point workbookAccounting
M00.03Decode floor calculatorAccounting
M00.04Prefill or decode: the incidentServing
Method

Predict before you measure

Every notebook marks the places to commit a number before running the cell. Write the number and one sentence of reasoning, then measure. The gap between the two is the actual lesson, and it is invisible if you skip the prediction — you will read the result, nod, and learn nothing.

Keep a running log of predicted against actual. When the gaps start closing, that is direct evidence the model in your head matches the machine.

Explain it in bytes

“It got faster” is not an explanation. “It moves a third of the memory it used to, and the measured speedup matches that ratio to within the launch overhead” is. The second one is checkable, and it generalises to the next problem.

Every module ends with a checkpoint you should be able to answer out loud, without notes, before moving on.