Learning path · Transformers & Attention · 19
Pretraining vs Inference
Pretraining learns general language patterns offline at massive scale; inference applies those weights to new prompts at serving time.
Why it matters
- Clarifies what you can change quickly (prompts, retrieval) vs slowly (weights).
- Inference economics drive batching, caching, and model routing.
- Explains emergent abilities that appear only at certain scales.
Key ideas
- Offline training
- Online serving
- Emergence at scale
Pretraining optimizes next-token prediction on corpora once; inference is repeated forward passes per user request. Product teams live entirely in inference unless they fine-tune or distil. Serving introduces constraints pretraining ignores: p95 latency, KV-cache memory, concurrent tenants, and safety filters. Design features assuming inference is the billable moment—every retrieved token and tool round-trip adds to it. Capacity planning belongs on the inference path: batch sizes, KV cache, concurrent sessions, and autoscaling policies drive real bills. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path