Learning path · Evaluation & Quality · 73
LLM as Judge
Using a strong model to score another model's outputs against rubrics—relevance, safety, coherence.
Why it matters
- Scales eval beyond manual review for fast iteration.
- Biases exist—judges favor verbose or self-similar styles.
- Calibrate against human labels regularly.
Key ideas
- Rubric prompts
- Pairwise comparison
- Judge bias
LLM judges apply structured rubrics: "Score 1–5 whether each claim is supported by the passage." They enable overnight eval sweeps across prompt variants. Mitigate bias by rotating judge models, using chain-of-thought scoring templates, and anchoring with human-labeled gold items. Never deploy judge-only metrics without periodic human reconciliation. Rotate judge models and compare scores quarterly; judge drift has invalidated promotion decisions in mature teams. Ship only after eval gates pass on representative production failures.
Updated 2026-08-09 · Full learning path