Testing Your AI
AI evaluation means regularly testing an AI system against a fixed set of real questions, scoring the answers, and blocking any release that scores worse than the last one. DevMations builds that testing for teams who need to know their AI is working rather than hope it is.
Also called evals · LLM evaluation · AI quality assurance · model regression testing
Why teams call us about this
Someone changed the wording to fix a complaint. Did that break anything else? Nobody knows. So either you ship nervously and find out from a customer, or you stop shipping. Normal software testing cannot help here, because the AI words its answer differently every time and there is no simple right-or-wrong to check against.
What you get out of it
- A single number for whether your AI is working, updated automatically
- Changes stop being a gamble — a bad one is caught before customers see it
- Your team can improve the AI quickly, because they can tell if they made it worse
What you actually receive
Working software and documentation — not a report recommending that somebody else build it.
A set of real test questions
Built from your actual customer traffic and the failures already reported to you — not invented examples. The problems you have already had become the things that can never happen again.
A scoring method you agree with
We write down what a good answer looks like for your business, in enough detail that two different people grade the same answer the same way.
Automatic grading, checked against people
Scoring at volume costs almost nothing if a machine does it — but only if we first prove the machine agrees with your team. We measure that agreement and show you the figure.
A safety catch on releases
The tests run every time anything changes, and a change that lowers the score cannot go live. This is the part that actually changes how a team behaves.
A quality report
Score over time, broken down by category, so a slow decline is visible to you long before a customer complains.
The engagement
- 01
We collect what has gone wrong
Support tickets, bad answers, the screenshots in your team chat. Those become the first tests, because those are the failures you already know cost you something.
- 02
We agree what good looks like
Written down properly. This is usually the first time a team has had that conversation, and it is often the most valuable part of the whole engagement.
- 03
We build and check the scoring
Automatic grading, validated against how your own people score the same answers. If the two disagree, we fix that before trusting any number.
- 04
We connect it to your release process
So it runs by itself, forever, without anyone remembering to. Handed over with documentation your team can extend.
Under the hood
The specifics, for whoever on your side will be reviewing this. If that is not you, nothing below changes what you get — skip to the work.
- Golden dataset construction
- Stratified sampling from production traffic plus a curated failure corpus. Versioned alongside the code so a change to the dataset is reviewable.
- Grader selection per task
- Deterministic and structural assertions where output is constrained, LLM-as-judge where it is not, human review sampled where the stakes justify it. Mixing these correctly is most of the work.
- Judge calibration
- Agreement between the model judge and human labels measured with Cohen’s kappa on a held-out sample, and reported alongside every judge-derived metric. An uncalibrated judge is not a score.
- CI regression gates
- Suites wired into GitHub Actions or your existing pipeline, with thresholds tuned to catch real regressions without blocking on noise. Per-category thresholds where the aggregate would hide a problem.
- Observability
- Trace-level logging via Langfuse or equivalent, so a failing case can be opened and inspected rather than guessed at.
What we use
- Promptfoo
- Braintrust
- DeepEval
- Ragas
- Langfuse
- Python
- GitHub Actions
Where we have done this
Live deployments you can open.
DevMations AI
A retrieval-backed assistant that answers questions from a business knowledge base in real time, built as our own reference implementation of a production RAG stack.
Open live site
Mock Interview AI
An interview practice platform that transcribes spoken answers, scores them against a rubric and returns structured feedback — a graded generation pipeline in production.
Open live site
IntelliSummarizer
A document summarisation tool that condenses long documents and books into structured summaries, handling inputs well past a single model context window.
Open live siteTesting Your AI — questions
- How many test questions do we need?
- Fewer than most people expect. Fifty well-chosen questions covering the ways your AI actually fails catch more problems than a thousand generic ones, and can be put together in about a week.
- Can a machine really judge the answers?
- Only once we have proved it agrees with your people. We have your team grade a sample by hand, compare that to the machine, and report how closely they match. If they do not match well enough, we fix the scoring rules before relying on any number from it.
- We already built our AI. Is it too late?
- No — that is the most common way this starts. We work with what you already have. Nothing needs rebuilding for us to begin measuring it.
- How long before we see something?
- A first working set of tests and an initial score typically inside two to three weeks. You get a real number early, because that number is what makes every later decision cheaper.
Thinking about Testing Your AI?
Describe what you have and what is going wrong with it. We will tell you what we would do first, and whether it is worth doing.
Book a call