Where AI detection actually pays for itself, and where it will bury you

Most organisations buying AI detection have not done the arithmetic that determines whether it will help them or bury them in false accusations. The arithmetic is not hard, it takes about ten minutes, and it will tell you more than any vendor benchmark.

Here it is, along with the deployments where it comes out positive and the ones where it does not.

The number that decides everything

Detector accuracy is not what determines whether your flags are trustworthy. Prevalence is: the proportion of your incoming volume that is actually machine-generated. This is Bayes’ theorem doing what it always does to screening programmes, and it catches people out in medicine, in fraud, and here.

Take a good detector. Call it 95% true positive rate at a 1% false positive rate, which is roughly the operating point a serious commercial tool will give you on long-form text. Run 10,000 documents through it.

Same detector, same accuracy, three completely different products. At 20% prevalence you have a screening tool worth acting on. At 1% prevalence, a coin flip decides whether the person you just flagged did anything wrong.

Two consequences follow immediately.

Detection works best where the problem is already common. This feels backwards. Surely you want it most where the problem is rare and hard to spot. But the mathematics is unsentimental. In a low-prevalence channel, detection generates mostly noise.

You must estimate your own prevalence before you buy. Not the vendor’s, not an industry survey. Sample two hundred items from your actual pipeline, have a human assess them properly, and get a number. If you cannot be bothered to do that, you cannot interpret anything the tool tells you.

The second calculation: what does each error cost

Accuracy is symmetric. Consequences never are.

Write down two numbers before deployment. What does it cost you when a machine-generated item passes undetected? And what does it cost you when a human is wrongly flagged?

In content moderation on a review platform, a missed fake review costs a small amount of trust and a false flag costs one annoyed user. Roughly symmetric, both cheap. Automate freely.

In academic misconduct, a missed case costs one unearned grade. A false accusation costs a student a year, sometimes a visa, and costs the institution a lawsuit. Wildly asymmetric. Never automate.

Where the asymmetry is severe, your threshold should be set far above the vendor default, because vendor defaults are tuned to look good in evaluations, not to protect your downside.

Where it pays

Content marketplaces and freelance platforms

High prevalence, symmetric costs, and, crucially, a natural remedy short of accusation. A flagged deliverable triggers a revision request, not a fraud allegation. The client asked for original work and did not receive it; that is a contract question, not a moral one.

This is close to the ideal deployment. Prevalence is high enough that flags are informative, the sanction is proportionate and reversible, and the accused party has an obvious route to demonstrate their process.

Review and user-generated content platforms

Fake reviews at scale are the archetype: high volume, high prevalence in the abusive segment, and a low cost per individual error. Detection here is one signal in a fraud stack that already includes account age, posting velocity, device fingerprint and network graph, and the value comes from the combination, not the text score.

Text detection alone is weak here for a reason worth understanding: reviews are short. Every statistical method needs length to accumulate signal, and a forty-word review does not provide it. Use detection to score reviewers across their whole corpus, not individual reviews.

Publishing and editorial intake

Submission volumes to journals, magazines and publishers have risen sharply, and a large fraction of the increase is machine-generated. Prevalence is high, the decision is a rejection rather than an accusation, and editors already reject the overwhelming majority of submissions for other reasons.

The signal that works best here is not the detector at all. It is the factual layer: fabricated citations, references to papers that do not exist, confidently wrong specifics. Detection triages; verification decides.

Hiring, with one specific caveat

Written screens and take-home assignments have very high prevalence now. But this is the deployment where the false-positive research should stop you cold.

The Stanford work by Liang and colleagues found detectors misclassifying human-written TOEFL essays at an average rate above 60%, while performing near-perfectly on native-speaker essays. If you run detection over cover letters and written assessments, you are running a system with a documented bias against second-language applicants over a process that is already legally exposed on disparate impact.

The defensible version: use detection to select which candidates get a live, unassisted follow-up exercise. Never to reject. A candidate who wrote the submission can discuss it; that conversation is better evidence than any classifier and it does not encode the bias.

Internal content operations

Quietly the highest-value use and nobody markets it this way. If your organisation produces content at volume through agencies, contractors or a distributed team, detection tells you what proportion of what you are paying for is unedited model output.

Not to punish anyone. To know. The answer is often surprising, and it changes what you brief, what you pay for and where you put editorial time. We wrote about wiring this into a repeatable process in building an AI content QA pipeline.

Where it does not pay

Anywhere the output is a sanction on an individual

Covered at length in our piece on false positives, and the short version is that the error distribution is not random. It concentrates on second-language writers, neurodivergent writers, and anyone writing in a constrained register. If your process converts a score into a penalty without a human investigation in between, you have built a machine for producing specific, patterned injustices.

Short text

Below roughly 300 words, the statistical signal is not there. Turnitin requires 300 words of qualifying prose and has, since July 2024, declined to display scores below 20% at all. Tools that will happily score a paragraph are giving you a number with no information in it.

Highly formulaic professional writing

Technical documentation, regulatory filings, legal drafting, clinical notes, structured reports. These genres have low natural entropy because deviation is a defect. Detectors flag them heavily and are not wrong about the statistics, only about the conclusion.

Anywhere you cannot act on a flag

If a flag does not connect to a process, to someone who reviews it and a next step and a remedy, you have bought a dashboard. Dashboards without decisions are how detection budgets die.

Six questions for a vendor

These separate serious products from marketing pages, and none of them can be answered with a single accuracy number.

1. What is your false positive rate on non-native English writing, specifically? If they have never tested it, that is the answer. The relevant public corpora are well known and testing against them is a weekend of work.
2. How does the score distribution behave between 100 and 300 words? If they will score short text without a caveat, they are selling noise.
3. Do you normalise Unicode before tokenisation? The RAID benchmark at ACL 2024 found that homoglyph substitution, swapping Latin characters for visually identical ones from other scripts, dropped Originality.ai from 85% accuracy to 9.3%. GPTZero, which normalises, lost three tenths of a point. This is three lines of preprocessing and it separated the field.
4. When was the model last retrained, and against which generators? Detector performance decays silently as new models ship. Ask for a retraining cadence, not a release date.
5. Can I set my own threshold, and do you expose calibrated probabilities? A tool that only gives you a verdict has made your risk decision for you, using its priors rather than yours.
6. What happens to my data? If you are uploading student work, employee writing or customer content to a third party, you have a data protection question before you have a detection question.

A deployment that holds up

Assembling all of the above:

• Measure your prevalence on a real sample before buying anything.
• Set the threshold from your cost asymmetry, not the vendor default.
• Never let a score be the last step where a person is affected.
• Apply a hard length floor and simply do not score below it.
• Corroborate with process evidence (drafts, version history, a conversation) and with the factual layer, which remains the most reliable signal available and requires no tooling.
• Log your own outcomes. How many flags, how many survived review? You are measuring your real false positive rate on your real data, which is the only one that governs your risk.
• Re-evaluate on a schedule. Every detector’s accuracy is a wasting asset.

The organisations getting value from detection in 2026 are not the ones with the best detector. They are the ones who understood that detection produces a prior, not a verdict, and built a process that treats it that way.