LLM-as-judge
LLM-as-judge is an evaluation technique where one language model scores another model’s output against a written rubric, replacing human graders for tasks where output quality cannot be checked by exact match.
It is cheap and scales, which is why it is popular, and it is untrustworthy until calibrated. Calibration means measuring how often the judge agrees with human labels on a sample of the same data. A judge with poor agreement is not producing a quality score; it is producing a number that looks like one. Report the agreement figure alongside any judge-derived metric.
Related terms
- Eval harness
An eval harness is the test infrastructure for a non-deterministic AI system: a dataset of representative inputs, a method for scoring each output, and a threshold that determines whether a run passes or fails.
- Inter-rater agreement
Inter-rater agreement measures how consistently two or more independent graders assign the same score to the same content, and is the standard check on whether a quality rubric is applied reliably.
- Rubric
A rubric is a written, testable definition of output quality — the dimensions being judged, the scale for each, and worked examples at each level — used so that two different graders reach the same score.
Tell us what you are building
Send us the problem in a paragraph. You will get a straight answer on whether we can help, and what we would do first.
Book a call