GPU Inference Platform

The router dispatches on the class, so a new model inherits a lane instead of needing new plumbing.

If routing keys on the model name, every model launch is a deploy of the router, and the day someone ships a fourth model it lands in whichever lane the config happened to default to. Keying on the class means a new video model is a registry entry and a capacity number, not a code change. The class also carries the admission policy: a per-tenant concurrency ceiling on video, which is cheap to enforce here and impossible to enforce once the work is on a card.

Video gets a durable queue and a job id. Image gets a short one. Speech gets neither.

A queue in front of the synchronous lane is latency you cannot get back, so speech does not get one — it gets headroom instead, which is the same insurance bought in capacity rather than in waiting. Video gets a durable queue because a five-second clip is 45 GPU-seconds of work and losing it after 40 of them is unacceptable; the caller gets a job id immediately and fetches the clip from object storage later. Image sits between: a short queue whose depth is a latency budget, not a buffer. Separate queues per lane, always — one shared queue means the head of the line is whatever is slowest, and the slowest thing here is 45 seconds long.

Three node pools — L4, L40S, H100 — labelled and tainted, with pods requesting a GPU by product, not just by count.

Every node carries a label naming its accelerator and a taint that keeps non-GPU work off it, and every serving pod carries the matching toleration plus a node selector on the product. A device plugin advertises the cards as a schedulable resource so a pod asks for one GPU and gets one GPU. Requesting a count alone is the classic mistake: a speech replica that lands on an H100 because the scheduler only counted cards is burning $6.88 an hour to do $0.80 of work, and nothing in the system will tell you. Naming the product in the request is what makes the fleet legible.

Cold start is minutes, so I keep N warm replicas per lane and I am going to tell you exactly how I picked N.

Instance boot, then a multi-gigabyte image pull, then weights off disk into HBM: minutes, not seconds. Nothing about autoscaling shortens that, which is why warm capacity is the design and autoscaling is the follow-up. N is a floor, not a forecast — the share of the steady lane I refuse to scale below. Speech is three cards at steady state and all three stay warm, because the whole lane is $2.41 an hour and scaling it to zero saves nothing worth a cold start. Video is 54 cards at steady state and four stay warm: $27.52 an hour, roughly $20K a month, held under the quietest part of the day. Then be honest about what four buys. One warm H100 finishes a clip every 22.5 seconds, so across a two-minute cold start four cards absorb about 21 of the 200 video requests that arrive at 1.67 rps. The floor makes the first minutes of a cold ramp degrade rather than fail; it does not make them invisible, and anyone who says a warm pool eliminates cold start has not multiplied it out. Pre-pulling the image onto the node and caching weights on local NVMe cuts the cold start itself, which is worth more than buying idle cards and is the thing to do first.

Pods scale on queue depth and in-flight requests; nodes scale behind them, and the node loop is the one that bounds recovery.

GPU utilization is a terrible scaling signal — a card can sit at 100% busy on a batch of one and at 100% busy on a batch of eight, and only one of those is a problem. Queue depth and in-flight count are the honest signals, per lane. The fast loop adds pods against warm nodes and settles in seconds. The slow loop notices unschedulable pods and provisions nodes, and that one is minutes because it is bounded by the cold start above. Recovery time is the slow loop, not the fast one, and quoting the fast one as your recovery time is how a burst gets away from you. Scale down deliberately slower than up, with a floor per lane that is the warm pool.

Batching and partitioning are free capacity, and both come before the purchase order.

Batching is the cheapest throughput on the board: speech at batch 8 sustains 10 rps on one L4, and dropping to batch 1 costs eight times the fleet for the same traffic. The batch window is the knob — a few tens of milliseconds of waiting buys a full batch and is invisible next to generation time. In the other direction, partitioning is how one card serves more than one thing at once. The video lane is the only place that pays: an H100 cut into MIG slices runs a reduced-resolution variant alongside the full-size model, each with its own memory and its own SMs, instead of standing a second $6.88 card up for the cheaper variant. Two constraints go with it. MIG geometry is static and reconfiguring it drains the node, so partitioning is a capacity plan and not an autoscaling lever. And only the P family has MIG at all — the L40S and L4 lanes are Ada, so what they get is time-slicing, which shares a card without isolating it and is a worse trade under a latency SLO. What partitioning is never for is rescuing a small model that landed on a large card: a speech replica on an H100 is a scheduling bug, and the fix is labels and taints, not a slicer. The rule is the same both ways — fill the card you already own before you rent another one, because the next card costs $6.88 an hour forever and the batch window costs 30 milliseconds once.

Reservations for the floor, on-demand for the working range, spot for the batch lane only.

The floor — the warm pool and the steady load under it — goes on reservations or capacity blocks, because it is running anyway and committed pricing is the only real discount available. The working range above that is on-demand at list. Spot, at roughly 60-70% off, goes only where preemption is survivable, which here means video: it is already async, already checkpointed, already behind a durable queue. Spot for speech would be trading a two-minute recovery against a sub-second SLO, which is not a trade. And the honest failure: P5 capacity is frequently just unavailable in region. The fallback is written down before it is needed — degrade video to A100 at longer service time and recompute the fleet, or move the lane to another region and eat the transfer, but do not let the answer be an unattended autoscaler retrying a launch that will never succeed.

Every request carries its lane, and the meter multiplies GPU-seconds by the per-GPU-hour rate for the card it ran on.

The numbers only argue if somebody sees them: $61.92 per thousand video requests against $0.0402 per thousand speech requests, a 1,539x spread, on lanes whose traffic share is 5% and 50%. Attribute at the request, not at the account — an invoice split by instance family tells you what you bought, never who spent it. The ledger puts cost share and traffic share on the same row on purpose, because the gap between those two columns is the whole design. This is also the guardrail on model updates: cost per request is the metric that catches a new checkpoint doubling GPU-seconds, and no latency dashboard will.

GPU utilization, queue depth, and p99 — and when they disagree, the policy already says which lane I protect.

The three curves are deliberately different questions: utilization says whether the cards are busy, depth says whether the fleet is losing, p99 says whether anyone noticed. Utilization high with depth flat is a healthy fleet. Depth climbing with utilization flat is a scheduling failure, not a capacity one. At 3x traffic the fleet does not exist and cannot be conjured in cold-start time, so the policy is written in advance: protect speech, because it is 50% of requests, 0.6% of cost, and the only lane where late means broken. Image degrades to a longer queue and a smaller resolution. Video sheds first — it stops accepting new jobs, tells the caller a real number instead of accepting work it will not finish, and drains what it already owns. Shedding the lane that is 93% of the bill is also the only lever that moves the bill.