Quick Answer: An AI detector estimates how likely a passage is to be machine-written. It does this in one of three ways: measuring how predictable the words are to a language model (perplexity-style statistics), running a classifier trained on human and AI examples, or checking for a watermark the generating model embedded on purpose (such as Google DeepMind's SynthID Text). The first two produce a probability, never proof. They make real mistakes: a peer-reviewed Stanford study found seven detectors flagged a majority of essays written by non-native English speakers as AI, and OpenAI withdrew its own classifier in 2023 for low accuracy. Treat a detector score as a reason to look closer, then decide using process evidence such as drafts and version history.
AI detectors use one of three methods, and the first two return a probability, not proof.
What an AI Detector Actually Measures
A detector never sees the writer. It sees a block of text and returns a score. Everything it concludes comes from statistical patterns in the words themselves, and human writing and model writing overlap heavily in those patterns. That overlap is the root of every accuracy problem discussed below.
There are three broad families of detection, and they behave very differently.
1. Perplexity and burstiness (statistical signals)
A language model assigns a probability to every next word. Perplexity measures how surprised a model is by a piece of text. Text a model would likely have produced itself scores low perplexity, because each word is the "expected" choice. Human writing tends to include more unexpected word choices.
Burstiness measures variation across a document. People tend to mix long and short sentences and shift their phrasing as they go. Model output is often more uniform from sentence to sentence.
GPTZero, launched in January 2023 by Princeton student Edward Tian, popularized these two terms. GPTZero now says it no longer uses perplexity and burstiness for its detection and moved to a deep-learning architecture, keeping the metrics as explanatory indicators. The idea still underpins much of the field.
A more refined statistical method is DetectGPT (Mitchell et al., Stanford, ICML 2023). It slightly rewrites a passage many times and checks whether the original sits at a local peak of the model's probability. Machine-generated text tends to; human text tends not to. It needs no training data, but it does need access to model probabilities.
Weakness: simple, formulaic, or carefully edited human writing also reads as predictable. That is exactly what produces false positives.
2. Trained classifiers
Most commercial AI content detectors today are classifiers. The vendor collects large sets of human-written and AI-written text, then trains a model to tell them apart. The output is a probability or a percentage of the document estimated to be AI-written.
Strengths: they can pick up patterns no single metric captures, and they improve as the vendor retrains on output from newer models.
Weaknesses: they are only as good as their training data. OpenAI itself noted that neural classifiers perform poorly on text unlike what they were trained on. A classifier tuned on one model's style can miss another's, and new models ship faster than training sets update. Most vendors also do not publish how their classifier works, so a user cannot inspect why a passage was flagged.
3. Watermarking
Watermarking works at the source. The model that generates the text deliberately biases its word choices in a pattern that is invisible to readers but statistically detectable by anyone holding the key.
The best-documented example is SynthID Text from Google DeepMind. The method was published in Nature in October 2024 and released as open source. Google describes it as a "tournament sampling" step applied during generation, with no retraining required, and a production implementation ships in Hugging Face Transformers (version 4.46.0 and later). Its detector returns one of three states: watermarked, not watermarked, or uncertain.
Strengths: when present, a watermark is far stronger evidence than a style guess, because it was put there on purpose.
Weaknesses: it only works on text from a model that applied that specific watermark. Google states that detection accuracy drops considerably when the text is thoroughly rewritten, though it tolerates minor edits. Text from an unwatermarked model carries no signal at all, so "not watermarked" never means "human."
Comparing the Three Detector Types
| Statistical (perplexity-style) | Trained classifier | Watermark detection | |
|---|---|---|---|
What it checks | How predictable the wording is to a model | Learned patterns from labeled human and AI text | A hidden signal embedded at generation |
Works on any model's output | Partly, best on models similar to the scoring model | Partly, depends on training data | Only on the model that applied the watermark |
Output | Probability | Probability or % AI | Watermarked / not / uncertain |
Main false-positive risk | Plain, formulaic, or non-native writing | Writing styles absent from training data | Very low when the key is used correctly |
Defeated by paraphrasing | Often | Often | Weakened by heavy rewriting |
Explains its decision | Partially | Rarely | Yes, statistically |
A fourth approach, from research rather than products, is retrieval: the AI provider stores what its model generated and checks whether a submitted text matches it. Krishna et al. (NeurIPS 2023) found this caught most paraphrased outputs in their tests, but only the provider that holds the database can run it.
Are AI Detectors Accurate?
They are accurate enough to be a useful signal and not accurate enough to be a verdict. Three well-documented facts make the case.
OpenAI withdrew its own classifier
OpenAI released an AI text classifier in January 2023 and shut it down on July 20, 2023, adding a note that it was "no longer available due to its low rate of accuracy." At launch, OpenAI reported that on its own challenge set the tool correctly flagged 26% of AI-written text as "likely AI-written," while mislabeling human text as AI 9% of the time. It also said the tool was unreliable on texts under 1,000 characters. The company that built the model being detected could not build a reliable detector for it.
Non-native English writers get flagged far more often
In "GPT detectors are biased against non-native English writers" (Liang, Yuksekgonul, Mao, Wu and Zou, published in the journal Patterns in 2023), Stanford researchers ran seven detectors on 91 TOEFL essays written by non-native speakers and 88 essays by US eighth graders. The detectors classified the US essays almost perfectly, yet flagged the TOEFL essays as AI-generated at an average false-positive rate of about 61%. All seven detectors agreed on 18 of the 91 essays, and 89 of 91 were flagged by at least one. The authors linked this to lower perplexity: writers with a more limited vocabulary produce more predictable text. You can read the paper on PubMed Central.
Seven detectors on 91 human-written TOEFL essays. Source: Liang et al., Patterns (2023).
The same logic applies beyond language learners. Technical writing, legal boilerplate, short answers, and heavily templated text are all predictable by nature.
Paraphrasing defeats most detectors
The Krishna et al. study built a paraphrasing model and used it to rewrite AI text. At a fixed 1% false-positive rate, DetectGPT's detection rate fell from 70.3% to 4.6%, and the paraphrased text also evaded watermarking, GPTZero, and OpenAI's classifier in their tests. In practice, this means detectors catch careless use more than deliberate evasion, which is the opposite of what an integrity process needs.
OpenAI's launch numbers (2023) and the Krishna et al. paraphrasing result (NeurIPS 2023).
Vendors acknowledge the low-score problem
Turnitin's own help documentation says it shows an asterisk instead of a number for AI scores in the 1% to 19% range, because its testing found a higher incidence of false positives there. That is a useful admission: small percentages are noise. Vanderbilt University went further and disabled Turnitin's AI detector in August 2023, citing false-positive risk at scale, the lack of transparency about how it works, and bias against non-native speakers (Vanderbilt's guidance).
How to Use an AI Detector Responsibly
If you check whether text is AI generated, these practices keep a score from turning into a wrongful accusation.
- Never act on a score alone. A percentage is a prompt for a conversation, not evidence of misconduct.
- Ignore low scores. If the vendor itself says scores under 20% are unreliable, treat them as zero.
- Check length. Short passages give detectors too little signal. OpenAI drew the line at 1,000 characters for its tool; many vendors set their own minimums.
- Consider who wrote it. Non-native speakers, students writing to a strict template, and technical writers are at higher risk of false positives. Weigh the score accordingly.
- Use more than one method, and know which type each is. A watermark check is different evidence from a style score.
- Read the flagged passages. Look for things detectors cannot judge: invented citations, facts that do not match the source material, a sudden shift in voice from the writer's earlier work.
- Record what you did. Note which tool, which version or date, and the score. Results change as vendors retrain, so the same text can score differently a month later.
A detector score starts a review; process evidence decides it.
What Schools and Editors Should Do Instead
Detection answers the wrong question. The useful question is "can this person show how the work was made?" Process evidence answers it far more reliably than any score.
For teachers and schools
- Ask for version history. Google Docs and Microsoft Word both keep revision history. A genuine draft shows hours of typing, deleting, and restructuring. A pasted-in essay shows one large insertion.
- Grade the process, not only the product. Require an outline, a rough draft, and a revision with notes. Each stage is hard to fake in a consistent way.
- Talk to the student. A five-minute conversation about the argument, the sources, and why they chose a particular structure reveals understanding quickly. Approach it as a check-in.
- Design assignments around specifics. Prompts tied to an in-class discussion, a local event, or the student's own experience are harder to outsource. Vanderbilt's guidance recommended in-class writing and topics drawn from class discussion.
- Write a clear AI policy. Say what use is allowed (brainstorming, grammar help, none) and how students should disclose it. Most disputes start with an unclear rule.
For editors and publishers
- Ask writers for working files and notes. Interview recordings, source lists, and draft history are normal parts of the job and easy to provide.
- Fact-check harder than you style-check. The real risk in AI-assisted copy is fabricated quotes, statistics, and citations. Verifying them protects you whether or not a model was involved.
- Set a disclosure rule. Decide what assistance is acceptable and require writers to state it. Then judge the work on accuracy and quality.
If You Were Falsely Flagged
Gather your version history, drafts, notes, and sources. Ask which detector was used and what score it gave. Point to the documented limitations above, then offer to walk through your work in person.
FAQ
How do AI detectors work?
They score text in one of three ways: by how predictable the wording is to a language model (perplexity and related statistics), by a classifier trained on labeled human and AI examples, or by checking for a watermark that the generating model embedded deliberately. The first two produce probabilities. Only a watermark check gives strong evidence, and only for text from a model that applied that watermark.
Are AI detectors accurate?
They are useful signals with real error rates. OpenAI withdrew its own classifier in 2023 for low accuracy, a Stanford study found detectors flagged about 61% of non-native English essays as AI, and research shows paraphrasing can drop detection sharply. Use a score to decide where to look, never as proof.
Why do AI detectors flag human writing?
Human writing that is plain, predictable, or formulaic looks statistically similar to model output. Non-native English writers, students following a strict template, and technical writers are especially affected, because their word choices are more predictable to a language model.
Can an AI detector prove someone used ChatGPT?
No. A style-based detector gives a likelihood, and its errors run in both directions. Process evidence such as version history, drafts, and a conversation about the work is far more reliable. A watermark match is stronger evidence, but most text carries no watermark at all.
What is SynthID Text?
SynthID Text is Google DeepMind's open-source method for watermarking text at generation time. It adjusts word selection through a process called tournament sampling and comes with a detector that reports watermarked, not watermarked, or uncertain. Heavy rewriting weakens it, and it says nothing about text from models that did not apply it.