What we have learned doing this
Writing from the DevMations team on testing AI systems, building assistants that hold up with real customers, connecting AI to internal tools, and doing quality assurance on software that answers differently every time.
- Evals
- Quality
- RAG
- Retrieval
If a term above was unfamiliar
The glossary defines everything we use, in plain language, without assuming you already know it.
Eval harness
An eval harness is the test infrastructure for a non-deterministic AI system: a dataset of representative inputs, a method for scoring each output, and a threshold that determines whether a run passes or fails.
LLM-as-judge
LLM-as-judge is an evaluation technique where one language model scores another model’s output against a written rubric, replacing human graders for tasks where output quality cannot be checked by exact match.
MCP (Model Context Protocol)
The Model Context Protocol is an open standard that defines how AI applications connect to external tools and data sources through a server interface, so one integration works across any MCP-compatible client.
RAG (retrieval-augmented generation)
Retrieval-augmented generation is a technique where relevant documents are fetched from an external store and inserted into a language model’s prompt, so the model answers from that specific source material rather than from its training data alone.
Tell us what you are building
Send us the problem in a paragraph. You will get a straight answer on whether we can help, and what we would do first.
Book a call