The vocabulary, defined plainly
This glossary defines the terms used in AI evaluation, agent engineering, the Model Context Protocol and QA automation. Each entry states what the term means and why it matters in practice, without assuming prior knowledge.
Evaluation
6 termsEval harness
An eval harness is the test infrastructure for a non-deterministic AI system: a dataset of representative inputs, a method for scoring each output, and a threshold that determines whether a run passes or fails.
Read moreLLM-as-judge
LLM-as-judge is an evaluation technique where one language model scores another model’s output against a written rubric, replacing human graders for tasks where output quality cannot be checked by exact match.
Read moreGolden dataset
A golden dataset is a curated set of inputs paired with known-correct outputs, used as the fixed reference against which an AI system’s quality is measured over time.
Read moreRegression gate
A regression gate is a CI check that fails a build when an evaluation suite’s pass rate drops below a defined threshold, preventing a prompt or model change from shipping a quality regression.
Read moreRubric
A rubric is a written, testable definition of output quality — the dimensions being judged, the scale for each, and worked examples at each level — used so that two different graders reach the same score.
Read moreInter-rater agreement
Inter-rater agreement measures how consistently two or more independent graders assign the same score to the same content, and is the standard check on whether a quality rubric is applied reliably.
Read more
Agents
8 termsAgentic RAG
Agentic RAG is a retrieval-augmented generation architecture in which the model decides when and what to retrieve — issuing its own queries, evaluating results, and retrieving again — rather than receiving a single fixed retrieval before generating.
Read moreRAG (retrieval-augmented generation)
Retrieval-augmented generation is a technique where relevant documents are fetched from an external store and inserted into a language model’s prompt, so the model answers from that specific source material rather than from its training data alone.
Read moreChunking
Chunking is the process of splitting source documents into smaller passages for embedding and retrieval, and the chunk size and boundary strategy largely determine how well a RAG system can find relevant information.
Read moreHybrid search
Hybrid search combines semantic vector similarity with traditional keyword search, then merges the two result sets, so that a retrieval system handles both conceptual questions and exact terms like product codes or error strings.
Read moreTool use
Tool use is a language model’s ability to call defined external functions — querying a database, sending a request, performing a calculation — by emitting a structured call that an application executes and returns results for.
Read moreFunction calling
Function calling is the mechanism by which a language model returns a structured, schema-conforming request to invoke a named function, instead of returning free text.
Read moreContext window
A context window is the maximum amount of text, measured in tokens, that a language model can consider in a single request, covering the prompt, any retrieved documents, the conversation history and the generated response together.
Read morePrompt injection
Prompt injection is an attack in which instructions embedded in content the model reads — a retrieved document, a web page, a user message — cause it to ignore its original instructions and act on the injected ones instead.
Read more
Testing
3 termsQA agent
A QA agent is an LLM-driven tester that explores an application by reading its interface, deciding what to try next, and reporting what appears broken — covering exploratory testing that fixed scripts cannot.
Read moreFlake rate
Flake rate is the proportion of test runs that fail intermittently without any change to the code under test, and it is the primary measure of whether a test suite can be trusted.
Read moreSelf-healing tests
Self-healing tests use resilient element location strategies — and often a model to resolve ambiguity — so that an automated test continues to work when the underlying markup changes but the user-facing behaviour does not.
Read more
Infrastructure
3 termsVector database
A vector database stores text as numerical embeddings and retrieves entries by semantic similarity rather than keyword match, forming the retrieval layer of most RAG systems.
Read moreMCP (Model Context Protocol)
The Model Context Protocol is an open standard that defines how AI applications connect to external tools and data sources through a server interface, so one integration works across any MCP-compatible client.
Read moreMCP server
An MCP server is a program that exposes a specific system — a database, an API, a document store — to AI clients as a set of typed tools and resources defined by the Model Context Protocol.
Read more
Need someone who knows this vocabulary?
If you are hiring for it or buying it, we can help with either. Tell us what you are trying to do.
Book a call