Learning path · Transformers & Attention · 20
Scaling Laws
Empirical relationships showing predictable quality gains as models, data, and compute grow—guiding frontier vs efficient model choices.
Why it matters
- Explains why smaller models plus RAG often beat naive huge-model calls.
- Sets expectations for diminishing returns on prompt tweaks alone.
- Informs build-vs-buy when capabilities jump each generation.
Key ideas
- Compute-optimal training
- Chinchilla insights
- Capability jumps
Scaling laws describe smooth improvements in loss—and often downstream capability—as parameters and training tokens increase. They explain vendor roadmaps and why mid-size models suddenly match last year's flagship. For practitioners, scaling laws argue for matching model size to task difficulty: don't pay frontier prices for extraction you can route to a small model with good retrieval. Re-evaluate that boundary each release cycle. Re-benchmark small-model-plus-RAG baselines whenever frontier models launch; the cost-performance frontier moves faster than procurement cycles. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path