Eval harness
An eval harness is the test infrastructure for a non-deterministic AI system: a dataset of representative inputs, a method for scoring each output, and a threshold that determines whether a run passes or fails.
Unlike a traditional test suite, an eval harness cannot rely on exact output matching, because the same input produces different text on each run. It therefore scores outputs against a rubric — structurally, by exact match on constrained fields, by a model judge, or by human review — and reports an aggregate pass rate. The harness becomes useful the moment it runs in CI and can block a merge.
Related terms
- Golden dataset
A golden dataset is a curated set of inputs paired with known-correct outputs, used as the fixed reference against which an AI system’s quality is measured over time.
- LLM-as-judge
LLM-as-judge is an evaluation technique where one language model scores another model’s output against a written rubric, replacing human graders for tasks where output quality cannot be checked by exact match.
- Regression gate
A regression gate is a CI check that fails a build when an evaluation suite’s pass rate drops below a defined threshold, preventing a prompt or model change from shipping a quality regression.
Tell us what you are building
Send us the problem in a paragraph. You will get a straight answer on whether we can help, and what we would do first.
Book a call