# DevMations

> DevMations is an AI engineering agency that builds agent and RAG systems, evaluation harnesses, QA agents and MCP integrations — and ships the web and mobile products around them.

DevMations is a software engineering agency working with clients worldwide. The practice
is organised into two groups: AI & Automation (the lead), and Product Engineering
(web and mobile builds).

## Services — AI & Automation

- [AI Assistants & Agents](https://www.devmations.com/services/ai-agents-rag) — also called AI Agents & RAG Systems: AI assistants that work with your own data — and hold up with real customers.
- [Testing Your AI](https://www.devmations.com/services/ai-evals) — also called AI Evals & LLM Quality: Prove your AI works — before your customers find out it does not.
- [AI That Tests Your Software](https://www.devmations.com/services/qa-agents) — also called QA Agents & Autonomous Testing: AI testers that use your product like a customer would, and report what breaks.
- [Connecting AI to Your Systems](https://www.devmations.com/services/mcp-servers) — also called MCP Servers & Integrations: Let AI tools use your internal systems, safely and with a full audit trail.
- [Checking AI-Written Content](https://www.devmations.com/services/ai-content-evaluation) — also called AI Content Evaluation: Check AI-written content at scale, against a standard that holds up to scrutiny.
- [Automated Software Testing](https://www.devmations.com/services/automation-testing) — also called Automation Testing & QA Engineering: Automated checks that catch bugs before your customers do — and that your team trusts.

## Services — Product Engineering

- [Web App Development](https://www.devmations.com/services/web-app-development): Web products built properly, with AI built in rather than bolted on.
- [Mobile App Development](https://www.devmations.com/services/mobile-app-development): iPhone and Android apps from one build, through app store review and out to customers.

## Glossary

- [Eval harness](https://www.devmations.com/glossary/eval-harness): An eval harness is the test infrastructure for a non-deterministic AI system: a dataset of representative inputs, a method for scoring each output, and a threshold that determines whether a run passes or fails.
- [LLM-as-judge](https://www.devmations.com/glossary/llm-as-judge): LLM-as-judge is an evaluation technique where one language model scores another model’s output against a written rubric, replacing human graders for tasks where output quality cannot be checked by exact match.
- [Golden dataset](https://www.devmations.com/glossary/golden-dataset): A golden dataset is a curated set of inputs paired with known-correct outputs, used as the fixed reference against which an AI system’s quality is measured over time.
- [Regression gate](https://www.devmations.com/glossary/regression-gate): A regression gate is a CI check that fails a build when an evaluation suite’s pass rate drops below a defined threshold, preventing a prompt or model change from shipping a quality regression.
- [Rubric](https://www.devmations.com/glossary/rubric): A rubric is a written, testable definition of output quality — the dimensions being judged, the scale for each, and worked examples at each level — used so that two different graders reach the same score.
- [Inter-rater agreement](https://www.devmations.com/glossary/inter-rater-agreement): Inter-rater agreement measures how consistently two or more independent graders assign the same score to the same content, and is the standard check on whether a quality rubric is applied reliably.
- [Agentic RAG](https://www.devmations.com/glossary/agentic-rag): Agentic RAG is a retrieval-augmented generation architecture in which the model decides when and what to retrieve — issuing its own queries, evaluating results, and retrieving again — rather than receiving a single fixed retrieval before generating.
- [RAG (retrieval-augmented generation)](https://www.devmations.com/glossary/rag): Retrieval-augmented generation is a technique where relevant documents are fetched from an external store and inserted into a language model’s prompt, so the model answers from that specific source material rather than from its training data alone.
- [Chunking](https://www.devmations.com/glossary/chunking): Chunking is the process of splitting source documents into smaller passages for embedding and retrieval, and the chunk size and boundary strategy largely determine how well a RAG system can find relevant information.
- [Vector database](https://www.devmations.com/glossary/vector-database): A vector database stores text as numerical embeddings and retrieves entries by semantic similarity rather than keyword match, forming the retrieval layer of most RAG systems.
- [Hybrid search](https://www.devmations.com/glossary/hybrid-search): Hybrid search combines semantic vector similarity with traditional keyword search, then merges the two result sets, so that a retrieval system handles both conceptual questions and exact terms like product codes or error strings.
- [Tool use](https://www.devmations.com/glossary/tool-use): Tool use is a language model’s ability to call defined external functions — querying a database, sending a request, performing a calculation — by emitting a structured call that an application executes and returns results for.
- [Function calling](https://www.devmations.com/glossary/function-calling): Function calling is the mechanism by which a language model returns a structured, schema-conforming request to invoke a named function, instead of returning free text.
- [MCP (Model Context Protocol)](https://www.devmations.com/glossary/mcp): The Model Context Protocol is an open standard that defines how AI applications connect to external tools and data sources through a server interface, so one integration works across any MCP-compatible client.
- [MCP server](https://www.devmations.com/glossary/mcp-server): An MCP server is a program that exposes a specific system — a database, an API, a document store — to AI clients as a set of typed tools and resources defined by the Model Context Protocol.
- [QA agent](https://www.devmations.com/glossary/qa-agent): A QA agent is an LLM-driven tester that explores an application by reading its interface, deciding what to try next, and reporting what appears broken — covering exploratory testing that fixed scripts cannot.
- [Flake rate](https://www.devmations.com/glossary/flake-rate): Flake rate is the proportion of test runs that fail intermittently without any change to the code under test, and it is the primary measure of whether a test suite can be trusted.
- [Self-healing tests](https://www.devmations.com/glossary/self-healing-tests): Self-healing tests use resilient element location strategies — and often a model to resolve ambiguity — so that an automated test continues to work when the underlying markup changes but the user-facing behaviour does not.
- [Context window](https://www.devmations.com/glossary/context-window): A context window is the maximum amount of text, measured in tokens, that a language model can consider in a single request, covering the prompt, any retrieved documents, the conversation history and the generated response together.
- [Prompt injection](https://www.devmations.com/glossary/prompt-injection): Prompt injection is an attack in which instructions embedded in content the model reads — a retrieved document, a web page, a user message — cause it to ignore its original instructions and act on the injected ones instead.

## Case studies

- [Turning an unpredictable support assistant into one the team trusts](https://www.devmations.com/work/support-assistant-evals): A support assistant was giving confidently wrong answers with no way to tell how often. We built the measurement first, then fixed what it revealed.
- [A test suite nobody trusted, made trustworthy again](https://www.devmations.com/work/flaky-suite-rescue): Forty minutes to run, failing at random one time in five, and universally ignored. We fixed the cause rather than the symptom.

## Writing

- [Fifty test cases is enough to start](https://www.devmations.com/insights/fifty-test-cases): Teams delay testing their AI because they think they need thousands of examples. They need about fifty, and they already have them.
- [Your AI is not lying to you. It is reading the wrong page.](https://www.devmations.com/insights/why-your-rag-system-answers-wrong): Most wrong answers from an AI built on your own documents are not a model problem. They are a search problem, and the fix is usually unglamorous.

## Work

Shipped products, each linking to a live deployment: https://www.devmations.com/work

## Company

- [About](https://www.devmations.com/about)
- [Careers](https://www.devmations.com/careers) — 3 open roles
- [Contact](https://www.devmations.com/contact) — hello@devmations.com

## Optional

- [Full site text](https://www.devmations.com/llms-full.txt): every page as one Markdown document