Learning path · Evaluation & Quality · 75
Observability for LLM Apps
Tracing prompts, retrievals, tool calls, latencies, token costs, and scores across production requests.
Why it matters
- Debuggability separates demo agents from operable systems.
- Surfaces cache misses, retrieval gaps, and tool failures.
- Feeds continuous eval from live traffic samples.
Key ideas
- Distributed traces
- Prompt versioning
- Cost dashboards
Observability captures each span: retrieval candidates, rerank scores, assembled prompt hash, model route, tool arguments, completion tokens, and user feedback. Correlate traces with eval metrics when incidents spike. Redact PII at ingest. Platforms like LangSmith, Phoenix, and custom OpenTelemetry pipelines are common—pick one and enforce instrumentation standards across teams. Sample production traces into weekly human review queues; live traffic finds edge cases synthetic evals miss entirely. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path