Building an AI content QA pipeline that actually catches the right things

Most teams handle AI-assisted content the same way: a vague policy, an occasional detector check, and a general sense of unease. That is not a process, and it fails in the specific way ad-hoc processes always fail: inconsistently, and worst on the things that matter most. This is the pipeline we build for clients, stage by stage, with the thresholds and the reasons.

It assumes something specific: your team uses AI, that is fine, and the job is to make the output reliably good rather than to catch anyone. If your goal is enforcement against your own staff, this is the wrong document and probably the wrong strategy.

Stage zero: decide what you are actually protecting

Before any tooling, name the risk. Teams that skip this buy a detector and then discover it was never the thing at stake.

Note how far down the list detection appears. For five of six risks it is the wrong instrument. For the sixth it is the right one and there is no substitute.

Stage one: provenance at the point of creation

The cheapest control in the entire pipeline, and the one almost nobody implements, because it happens before anything is written.

Every piece of work carries a one-line declaration from its author: which tools were used, for what, and what the human contributed. Not a confession, just a metadata field, filled in the same spirit as noting which photographer took an image.

This does more work than it appears to. It removes the incentive to conceal, which is what actually produces evasion behaviour. It gives your reviewers context that changes how they read. It satisfies the disclosure obligations that publishers and, in the EU, Article 50 of the AI Act increasingly require, with that regulation’s own exemption for content that underwent human review with clear editorial responsibility. And it makes your later detection results interpretable, because a flag on a piece declared as AI-drafted is information, while a flag on a piece declared as hand-written is a question.

Stage two: automated screening

This is where detection sits, and it sits here on specific terms.

Screen for triage, not for verdicts. The output of this stage is a routing decision: which items get five minutes of review and which get thirty. Never a rejection.

Set the threshold from your prevalence, not the vendor default. The arithmetic matters more than the accuracy figure. A detector at 95% true positive and 1% false positive gives you flags that are 96% reliable when a fifth of your volume is machine-generated, and roughly a coin flip when 1% is. We worked that through in the piece on detection economics; the practical instruction is to sample two hundred real items, get a prevalence estimate, and set your threshold from that.

Apply a hard length floor. Below roughly 300 words there is not enough statistical signal, and any tool that scores a paragraph is reporting noise. Turnitin requires 300 words and has declined since July 2024 to display scores below 20% at all, and a vendor deciding not to show a number is the most honest disclosure in the category.

Screen the factual layer in the same pass. This is the highest-yield automated check available and it is not detection at all. Extract every citation, statistic, date and proper noun, and check them. Fabricated references remain the single most reliable indicator of unedited model output, they require no classifier, and they are what reviewers and editors actually catch people on.

Normalise Unicode before anything else. In the RAID benchmark at ACL 2024, homoglyph substitution took Originality.ai from 85% accuracy to 9.3%, while GPTZero, which normalises, lost three tenths of a point. Do it in your own pipeline regardless of what your vendor does, and log when the normalisation changes anything, because in a normal workflow it should not.

Stage three: human review, scoped

Reviewers waste time when their task is “check this.” Give them four questions with binary answers.

1. Is every specific claim verifiable? Numbers, dates, names, citations, product behaviour. Not plausible. Verifiable, against a primary source.
2. Does this contain anything that exists nowhere else? First-hand observation, original data, a defended position. If the answer is no, the piece is a summary, and summaries are precisely what an AI Overview replaces.
3. Is the terminology correct for the domain? Models paraphrase fixed technical terms. That is the failure that produced “tortured phrases” in the scientific literature and now drives real retractions, and it is instantly visible to anyone who knows the field.
4. Would we publish this under a named human’s byline? The question that captures everything the other three miss.

Question two is the one that determines commercial outcomes. In a market where two-thirds of searches end without a click and AI summaries halve position-one click-through, content whose only property is competent summarisation has no economic function. That is the real quality bar, and no detector measures it.

Stage four: measure your own error rates

The stage everyone skips, and the one that determines whether the pipeline is worth running in six months.

Log, for every flagged item: the score, the reviewer’s conclusion, and the time spent. After a hundred items you know your actual false positive rate on your actual content, which is the only number that governs your risk. The vendor’s rate was measured on their data.

Watch for three failure modes.

• Silent decay. Detector accuracy falls as new models ship, and nothing tells you. If your flag rate drops steadily while your content mix has not changed, you are probably missing more, not producing less.
• Patterned false positives. If your flags concentrate on particular authors, check whether those authors are second-language writers. The Stanford study by Liang and colleagues found detectors misclassifying human-written TOEFL essays at an average rate above 60% while performing near-perfectly on native-speaker text. That bias, running through an internal QA process, becomes an employment-law problem.
• Review theatre. If reviewers approve everything, the stage is not functioning and you should either fix the brief or remove the stage.

Stage five: feedback, not enforcement

The output of the pipeline should be a better brief, not a disciplinary record.

When a piece fails, the useful question is what was missing from the instruction. Almost every failure traces to a brief that did not specify what original material the piece required, and “write 1,500 words about X” will produce a summary every time, from a model or from a person.

Briefs that produce good work name the specific thing the piece must contain that nothing else does: this dataset, this customer conversation, this position we are willing to defend. That constraint is what a model cannot satisfy on its own, which is exactly why it works as a brief.

What to build versus what to buy

Buy: the detection model. Training and maintaining one is a continuous data-curation commitment against a moving target, and it is not your business unless it is.

Build: everything around it. The provenance field, the routing logic, the threshold, the reviewer checklist, the outcome log. This is a few hundred lines of glue and it is where all the value is. A detector API with no pipeline around it is a dashboard, and dashboards without decisions are how these budgets die.

Do not build: your own factual verification stack, at least initially. Citation and claim checking against primary sources is a well-served problem and there is no advantage in reimplementing it.

A realistic starting configuration

For a team producing thirty to a hundred pieces a month:

• A required one-line tool declaration on every submission.
• Detection screening on everything over 300 words, threshold set from a measured prevalence sample, output routing review depth only.
• Automated extraction and verification of all citations, statistics and dates.
• Unicode normalisation with an alert on any change.
• The four-question reviewer checklist, with question two as a hard gate.
• An outcome log reviewed monthly, tracking flag volume, confirmation rate and false positives by author.
• A quarterly re-evaluation of the detector against a fresh sample, because its accuracy is a wasting asset.

That is a week of engineering and an hour a month of governance. It catches the failures that actually cost money (fabricated facts, unbriefed substitution, thin output shipped at volume) and it does not create the failure that costs the most, which is a confident accusation against someone who did nothing wrong.

The principle underneath all of it

Every stage here is built on one idea: detection produces a prior, not a verdict. It tells you where to look. Verification tells you what is true. Human judgement decides what to publish. Confuse those three and the pipeline either becomes an accusation machine or a rubber stamp, and both fail in ways you will only discover later.

If you want this built and wired into how your team already works, that is a conversation we are happy to have.