Inference practice
M05.03·Systems·Hard·2 hours·2 min read

Continuous Batching Simulator

Write a small scheduler simulation that admits new requests between decode steps and compares it to static batching.

Module
Serving an LLM
Objective
Model iteration-level scheduling and measure throughput, time to first token, and tail latency.

Prompt

Build a Python simulation for decode scheduling. You do not need a model. Each request has:

Request(
    id: str,
    arrival_ms: int,
    prompt_tokens: int,
    output_tokens: int,
)

Assume:

prefill time = 0.04 ms per prompt token
decode step time = 12 ms for the active batch
max active requests = 8
max live tokens = 32768

Compare two policies:

  1. Static batching: collect up to 8 requests, run them until all finish, then admit the next batch.
  2. Continuous batching: after every decode step, remove finished requests and admit new ones if request and live-token limits allow.

Starter workload

requests = [
    Request("a", 0, 512, 20),
    Request("b", 0, 1024, 80),
    Request("c", 20, 512, 10),
    Request("d", 40, 4096, 120),
    Request("e", 60, 256, 30),
    Request("f", 80, 2048, 60),
    Request("g", 100, 512, 12),
    Request("h", 120, 8192, 160),
    Request("i", 140, 256, 25),
    Request("j", 160, 512, 25),
]

Deliverable

Report for each policy:

  1. Request throughput.
  2. Output tokens per second.
  3. Median end-to-end latency.
  4. P95 end-to-end latency.
  5. Median time to first token.
  6. P95 time to first token.

Then explain why the policies differ.

Acceptance criteria

A good simulation:

  1. Tracks request arrival times.
  2. Does not admit a request before its arrival.
  3. Enforces both max active requests and max live tokens.
  4. Separates time to first token from full completion latency.
  5. Shows at least one case where throughput and latency pull against each other.

Stretch

Add a priority class for interactive chat requests. Decide whether priority can starve long jobs, then add a fairness rule.

Debrief

The serving scheduler is where model performance becomes product performance. A simulator forces you to say what policy you are actually testing.