Learning path · Models & Tokens · 09
Tokens and Cost
Provider pricing and latency scale with tokens processed—input, output, and often cached-input discounts.
Why it matters
- Unit economics determine which features ship at scale.
- Long contexts and verbose tool JSON silently inflate bills.
- Cache-aware design is a cost optimization lever reported in industry case studies.
Key ideas
- Input vs output pricing
- Cached tokens
- Budget caps
Every API call has a token ledger: system prompt, retrieved chunks, chat history, tool definitions, model completion. Output tokens are usually pricier than input. Cached prefix tokens—where providers support them—can dramatically cut spend on stable instructions and static knowledge snippets. Instrument token usage per feature, per tenant, and per model route. Teams that treat tokens as a first-class metric catch runaway agent loops before finance does. Set per-tenant budgets and anomaly alerts before launching agent features, because unbounded loops have caused surprise invoices in reported industry case studies.
Updated 2026-08-09 · Full learning path