Watermarking and provenance: the case for marking AI content at source

Every argument about AI detection eventually arrives at the same place: we are trying to reconstruct a fact after the fact. The content already exists. The origin has already been lost. Detection is forensics on a scene that was never photographed. Marking at source solves that, and it has problems of its own that almost nobody discusses honestly.

We build detectors, so this article argues partly against our own product category. That seems worth doing, because the regulatory direction is now unambiguous and anyone planning past next year should understand where it goes.

The structural case against detection

Sadasivan and colleagues at the University of Maryland put the limit formally in work later published in TMLR: the accuracy achievable by any detector is bounded by the total variation distance between the distribution of human text and the distribution of machine text. As models improve at producing human-like text, that distance shrinks, and the bound on every detector, current and future, tightens toward chance.

The uncomfortable part is that shrinking that distance is the explicit training objective of every frontier model. Detection is not fighting a technical problem that better engineering will solve. It is fighting the industry’s central optimisation target, with a budget several orders of magnitude smaller.

That is the case for marking, and it is a strong one. Now the complications.

How the two approaches actually differ

SynthID, and what it can and cannot carry

SynthID Text works by intervening in sampling. At each step the model has a distribution over candidate next tokens; SynthID runs a tournament over those candidates that biases selection according to a key, embedding a statistical signature across the sequence. With enough tokens, the signature is recoverable. Without the key, it is not detectable, and output quality is not meaningfully degraded.

DeepMind open-sourced the text implementation through Hugging Face Transformers in October 2024, the first time a major text watermarking scheme became available to third parties, which matters more than the technique itself. It also covers images, video and audio from Google’s models, with robustness to cropping, filtering, frame-rate changes, compression and, for audio, noise and speed changes.

Three limits, stated plainly.

It only covers marked models. A negative result means “no watermark from a participating model was found.” It does not mean a human wrote it. Every open-weights model anyone can run on their own hardware produces unmarked output, and there are a great many of those. Any workflow that treats absence of a watermark as evidence of human authorship is broken by construction, and this is the single most common misunderstanding we encounter.

Text watermarks do not survive rewriting. This is the finding people underrate. Work on adversarially modified AI text has reported watermark detection on SynthID-marked text collapsing from around 87% to around 5% after a single pass through a strong paraphraser. The signature lives in token choices; replace the tokens and the signature goes with them. For images and audio, robustness to transformation is genuinely good. For text, it is fragile in exactly the way that matters.

Spoofing runs the other direction. The Maryland work also demonstrated the reverse attack: an adversary who can infer a watermark’s signature can craft human text that classifies as machine-generated. That is a reputational attack vector, a way to make someone’s genuine writing look fabricated, and a category of harm that post-hoc detection does not create.

The tool that exists and was never shipped

OpenAI reportedly built a text watermarking system with very high effectiveness given sufficient text, had it ready for around a year, debated it internally for roughly two, and has not released it.

The stated reasoning is worth sitting with, because it is not cynical. One concern was that watermarking would stigmatise AI use as a writing aid for non-native English speakers, the exact population that post-hoc detection already misclassifies at high rates. The others were ecosystem effects and the commercial risk that users would simply move to an unmarked provider.

That last point is the whole problem in one sentence. Voluntary marking is a prisoner’s dilemma. Any provider who marks unilaterally hands a feature to every provider who does not. Which is why the answer, if there is one, is regulatory rather than voluntary.

What C2PA actually promises, and what an audit found

C2PA is not a detector and it is a mistake to shelve it next to one. It is a signed manifest: this file was created by this device or application at this time, and these edits were applied. The specification has moved through several versions, with 2.2 published in 2025 and later revisions since.

For photojournalism, evidence handling and stock media this is a serious and useful architecture. For text it is close to irrelevant, for a mundane reason: a manifest is metadata attached to a file, and text moves by copy-paste. Nobody submits an essay as a signed container. They paste it into a box.

There is also a security question that deserves more attention than it has received. A 2026 analysis by researchers at UMBC and Hacker Factor concluded that the specifications fail to achieve several of their claimed security goals, identifying issues including timestamps not being cryptographically bound to the content they date, inconsistent certificate revocation checking, validators disagreeing on whether the same file is valid, exclusion ranges permitting regions to be altered without invalidating the signature, and certificate expiry making signed content unverifiable within months, which sits badly with legal record retention.

Their conclusion was that C2PA should not yet be relied upon for high-stakes uses such as financial disclosures, journalism or legal evidence, and that premature adoption could make the misinformation problem worse by attaching unearned authority to a green tick. That is a specific, checkable critique from named researchers, and if you are planning to build a workflow on Content Credentials, it is the paper to read before you commit.

The regulation that forces the issue

Article 50 of the EU AI Act became applicable on 2 August 2026, and it is the first binding requirement anywhere that generative systems mark their output.

• Article 50(1). People must be told when they are interacting with an AI system, unless it is obvious.
• Article 50(2). Providers of generative AI must ensure outputs, including text, audio, image and video, are marked in a machine-readable format and detectable as artificially generated. Assistive editing that does not substantially alter the input’s semantics is excluded.
• Article 50(3). Deployers of emotion recognition or biometric categorisation must inform the people exposed to it.
• Article 50(4). Deepfakes must be disclosed, and AI-generated text published to inform the public on matters of public interest must be labelled. There is an exemption where the content underwent human review with clear editorial responsibility.

Systems already on the market when the obligations landed have until 2 December 2026 for the machine-readable marking requirement specifically. Penalties reach €15 million or 3% of worldwide annual turnover, whichever is higher. The European Commission has published a voluntary Code of Practice on marking and labelling, including a three-icon set distinguishing fully AI-generated content, partially AI-modified content, and general AI involvement.

Two observations that matter more than the compliance detail.

First, note who is regulated: generators, not detectors. There is no binding accuracy or transparency standard governing detection products anywhere. A detector vendor can claim any number it likes. A model provider now cannot decline to mark.

Second, note the editorial exemption in 50(4). Text that went through human review with clear editorial responsibility does not require the label. That is the regulation encoding the same principle every sensible institutional policy has landed on independently: what matters is not whether a machine was involved but whether a named human took responsibility for the output.

What we think a working system looks like

Not one mechanism. Four layers, each doing the job it is actually capable of.

1. Marking at source, mandated. The only approach with a principled long-run answer, now legally required in the EU for systems placed on that market. It handles the large majority of casual, unmodified output, which is the large majority of the volume.
2. Provenance credentials for files. Genuinely valuable for images, video and audio, where the manifest travels with the container and the transformations are meaningful. Weak for text. Read the security literature before you trust the badge.
3. Statistical detection for everything else. Unmarked models, rewritten text, stripped metadata. This is a large residual and it will stay large. Detection’s honest role is to cover it imperfectly, at a known error rate, as one input among several, not to render verdicts.
4. Disclosure and accountability as the actual policy. Every serious institutional framework (Google’s spam policies, publisher submission rules, the AI Act’s editorial exemption) converges on the same test. Not “was a machine involved” but “did a human take responsibility, and is the result any good.”

The fourth layer is the one that survives every technical development, which is a strong hint that it was the right question all along.

If you are planning around this

If you publish into the EU market and use generative systems, the marking and disclosure obligations are live now and the residual grace period for existing systems ends in December 2026. If you are building a verification workflow, do not architect it around watermark detection alone, because unmarked models and a single paraphrase pass both defeat it. If you are buying Content Credentials as an evidentiary control, read the security critique first and decide whether your threat model tolerates the gaps.

And if you are building policy, build it around disclosure and named human responsibility. That is where the regulation landed, it is where publishers landed, and it is the only formulation that does not need rewriting every time a model ships.

We covered the detection mechanics and who detection wrongly accuses separately.