Run a clean block of GPT-4 output through a detector and it lights up with confidence. Take that same text, rewrite a fifth of the words by hand, and the detector starts to hesitate. The cleaner the AI sample, the easier the catch. The more a human touches it, the murkier the verdict. That inversion sits at the heart of how these systems work.
Why Editing Breaks the Signal
Photo by Detectors don’t read for meaning. They read for pattern. Most measure perplexity, a score of how predictable each word is given the words around it. Machine text tends to be smooth and predictable. Human text wanders, doubles back, and surprises itself. The detector is essentially listening for a kind of flatness that pure AI output tends to carry.
Editing roughens that flatness. When a person swaps a word here, splits a sentence there, and reorders a paragraph, they scuff the surface the detector was scanning. The underlying structure may still be machine-shaped, but the telltale smoothness is gone. What’s left is a hybrid: part machine, part human, belonging fully to neither category the tool was trained to recognize.
Edited drafts are the hard case because the detector was taught to sort two clean piles. Edited AI text is a third pile it never learned.How the Models Actually Work
Tools like GPTZero and Originality.ai lean on two signals.
Photo by One is perplexity, already described above. The other is burstiness, the variation in sentence rhythm across a passage. Human writing tends to be bursty, with long sentences crashing into short ones. Machine writing holds a steadier pace.
Both signals assume the text is all one thing. The training data reflects that assumption: raw model outputs on one side, unassisted human writing on the other. The edited middle ground, where most real AI-assisted work actually lives, rarely makes it into benchmark sets. The model has no learned shape for it.
One more habit matters. A detector returns a probability, not a ruling. A score reading “73% likely AI” is an estimate, yet it gets treated as a verdict. That gap between estimate and verdict is where most real-world damage happens.
What the Benchmarks Show
The accuracy numbers tell a consistent story when you line up text types side by side.
Photo by -
Raw AI output: detected at high rates. SurferSEO’s tool correctly flagged 99.4% of AI-generated articles in one study [Graphite].
-
Edited or paraphrased AI text: performance falls to roughly 60% to 85% accuracy [Edcafe].
-
Human-written technical and academic prose: prone to false positives, especially from non-native English writers.
A study of five commercial detectors, testing AI-generated replicas of nearly 6,000 research papers, reported false negative rates from 0.3% all the way to 99.6% [Digital Trends]. The same family of tools, the same kind of text, and results that scatter from nearly perfect to nearly useless.
Researchers have a name for one cause of this. A “lexical complexity attack,” where AI output is rephrased with more sophisticated vocabulary, was shown to cut detector effectiveness sharply [Digital Trends]. That’s editing, described in the language of an exploit.
“AI detection is conceptually broken, procedurally unfair, and methodologically indefensible.” (Edcafe)
The judgment is harsh, and the scatter in the benchmarks helps explain why someone would reach it.
Where Detection Is Heading
The signals visible now point away from reading finished text and toward marking it earlier.
Photo by Google DeepMind and OpenAI have both published work on watermarking, embedding faint signals at the moment text is generated so that moderate editing doesn’t wash them out. No production tool implements this reliably yet, but the research direction is steady.
A second path looks past word choice to structure. Token-level edits, swapping individual words or phrases, are easy to make. The deeper shape of an argument is harder to disguise. Early work suggests paragraph-level transition patterns may differ between AI and human writers even after sentences have been polished. The catch is the arms race: any new signal becomes the next target for editing.
The most durable approach may abandon inference entirely. Provenance tracking, logging where a document came from rather than guessing after the fact, sidesteps the whole problem of reading tea leaves in finished prose. As AI-assisted writing spreads, with AI-generated articles reaching 39% of the web within twelve months [Graphite], knowing a document’s origin starts to matter more than detecting its texture.
The next time a detector hands back a confident percentage on a piece someone clearly revised by hand, hold it loosely. That score is most trustworthy on text least like what writers actually ship. Before trusting any tool’s verdict, it’s worth checking whether it publishes accuracy figures for edited text, not just for raw machine output. A detector that only proves itself on clean AI is answering a question almost no one is really asking.
🔖
- Edcafe, on how manual editing and paraphrasing drop detector accuracy to roughly 60 to 85 percent
- Digital Trends, on a five-detector study reporting false negative rates from 0.3 to 99.6 percent and the lexical complexity attack
- Graphite, on SurferSEO detection accuracy and AI-generated articles reaching 39 percent of the web
