Checking AI-Written Content
AI content evaluation is the process of checking machine-generated writing against an agreed standard, using a mix of people and automated scoring, so quality can be tracked at a volume no team could read. DevMations designs the standard, builds the checking process, and proves the scores are consistent.
Also called content evals · LLM output grading · rubric scoring · annotation pipelines
Why teams call us about this
You are producing more content than anyone can read, and "it looks fine" is not a quality process. Two people on your team disagree about the same paragraph. You could have a machine check it, but you have no idea whether the machine shares your standards. And nobody can tell you whether this month was better than last.
What you get out of it
- Content quality becomes a number you can track, not an argument
- You catch a drop in quality in days rather than after a customer mentions it
- Your team agrees on what good looks like — often for the first time
What you actually receive
Working software and documentation — not a report recommending that somebody else build it.
A written standard
What good content means for you specifically — the things being judged, the levels, and real examples of each. Detailed enough that two reviewers reach the same verdict.
Checking at full volume
Automated scoring across everything you produce, with people reviewing a sample rather than trying to read all of it.
Proof the scores mean something
We measure how often your reviewers agree with each other and with the automated scoring. A standard people cannot apply consistently is a broken standard, and this is how you find out.
A review tool people will use
Fast, keyboard-driven, and designed so disagreements surface rather than hide.
Quality trends over time
Split by content type and how it was produced, so when something changes you can tell what caused it.
The engagement
- 01
We define good, in writing
With the people who own the content. This takes longer than it sounds and it is where most of the value is.
- 02
We check your reviewers agree
Several people grade the same sample separately. Where they disagree, the standard gets sharpened until they do not.
- 03
We automate the scoring
Built against the agreed standard and validated against how your own people scored the same material.
- 04
We run it continuously
With a sampled human check to keep it honest, and alerts when quality starts to slip.
Under the hood
The specifics, for whoever on your side will be reviewing this. If that is not you, nothing below changes what you get — skip to the work.
- Rubric engineering
- Multi-dimensional rubrics with anchored examples per level. Dimensions kept independent so scores remain interpretable rather than collapsing into a single vague impression.
- Inter-rater agreement
- Cohen’s kappa or Krippendorff’s alpha depending on the scale, computed between human raters and between humans and the model grader. Reported, not assumed.
- Sampling strategy
- Human audit sample sized from the observed agreement and your risk tolerance, rather than a round number picked because it felt right.
- Annotation infrastructure
- Label Studio or Argilla, configured with your rubric and wired to a queryable store for trend analysis.
- Per-language validation
- Grader quality validated separately for each language rather than extrapolated from English performance.
What we use
- Anthropic Claude
- OpenAI
- Python
- Label Studio
- Argilla
- PostgreSQL
- Metabase
Where we have done this
Live deployments you can open.
IntelliSummarizer
A document summarisation tool that condenses long documents and books into structured summaries, handling inputs well past a single model context window.
Open live site
Podcast AI
A podcast platform with AI text-to-audio generation, streaming playback and discovery, letting creators produce episodes without recording.
Open live site
DevMations AI
A retrieval-backed assistant that answers questions from a business knowledge base in real time, built as our own reference implementation of a production RAG stack.
Open live siteAI Content Quality — questions
- Do we still need people reviewing content?
- Some, but far fewer — enough to spot-check that the automated scoring is still behaving. We work out the right sample size from the numbers rather than guessing.
- Our reviewers disagree with each other constantly.
- That is a useful finding rather than a problem with your people. It almost always means the standard is vague about one particular thing. Making that specific is usually the single most valuable result of the work.
- Does this work for languages other than English?
- Yes, with one caveat we take seriously: automated scoring quality varies by language, so we validate each one separately rather than assuming it works because it worked in English.
Thinking about AI Content Quality?
Describe what you have and what is going wrong with it. We will tell you what we would do first, and whether it is worth doing.
Book a call