A detector's own numbers and an independent test disagreed

The classroom note

A teacher deciding whether to trust an AI-detection score is choosing between two very different kinds of document. On its own site, current as retrieved on 16 September 2026, GPTZero advertises '99% Accuracy' in its hero section and, elsewhere on the same page, '99.9% Accuracy', alongside a claim of detecting '95.7% of AI texts while only incorrectly predicting 1% of human texts as AI' on what it describes as the RAID benchmark. The vendor also states it has 'reduced AI detection's false positive rate on TOEFL texts to 1.1%' for English-language learners. These are the company's own marketing figures, not an independent audit of the tool a teacher is about to rely on for a real accusation.

What the evidence says

A 2023 peer-reviewed study, Testing of Detection Tools for AI-Generated Text by Weber-Wulff and colleagues, tested 14 detection systems, GPTZero among them, against a purpose-built set of human-written, ChatGPT-written and obfuscated texts scored by the authors themselves. GPTZero classified 29 of 54 test cases correctly, a reported accuracy of 54 percent, ranking eighth of the 14 tools tested; the study's overall conclusion was that 'available detection tools are neither accurate nor reliable' and that accuracy fell further once obfuscation techniques, such as paraphrasing, were applied. The two figures, above 99 percent and 54 percent, are not measuring the same thing: the vendor's numbers come from benchmarks it selected or helped design, while the independent study built its own document set specifically to probe weaknesses.

The implementation question

What a detector score can actually support depends on what was tested and how the result is used. GPTZero's own methodology notes describe testing on 'a never-before-seen set of human and AI articles,' a reasonable design choice but not the same as testing on the borderline, edited or paraphrased student writing a real classroom produces. The independent study's lower figure came from exactly that kind of adversarial material. Using either number as a disciplinary threshold on its own treats a benchmark score as if it were a courtroom-grade measurement of a specific student's specific paragraph, which neither document claims it is.

What holds and what fails

What holds is that both documents agree detectors are imperfect; even GPTZero's own site states 'no AI detector can ever truly be 100% perfect.' What is likely to fail, editorially, is any school procedure that cites a vendor's headline percentage as the reason a specific piece of work was judged AI-generated, since that percentage was measured on the vendor's chosen material, not the disputed document. The gap between above 99 percent and 54 percent is large enough that the choice of benchmark, not just the tool, decides the story a score tells.

• Which benchmark, if any, produced the accuracy figure being quoted to justify a decision in this school?
• Has this detector, or any detector, been tested on writing similar to what this student actually submitted?
• Does the school's policy treat a detector score as evidence to weigh, or as a verdict on its own?

A detector's marketing page and an independent test of the same tool can both be accurate about what they measured, while supporting entirely different levels of confidence in any single accusation.