A study found detectors mislabelled non-native writers as AI

The classroom note

A non-native English speaker's TOEFL essay and a native-speaking US eighth-grader's classroom essay make an unglamorous but pointed test set. Researchers led by Weixin Liang and James Zou put both sets of writing through seven widely used AI-text detectors, Originality.AI, Quill.org, Sapling, OpenAI's own detector, Crossplag, GPTZero and ZeroGPT, in work first posted as a preprint in April 2023 and published that July in the journal Patterns. Ninety-one TOEFL essays and 88 native-written essays went through unmodified. The detectors flagged an average of 61.22% of the non-native essays as AI-generated, and 97.80% of those essays were flagged by at least one tool. Native essays were misclassified only 5.19% of the time on average.

What the evidence says

This was a small, targeted test of existing commercial and free detector tools against real, human-written essays, not a study of any particular school's disciplinary process. The peer-reviewed Patterns article, whose research was supported in part by the US National Science Foundation, the National Institutes of Health, the Silicon Valley Foundation and the Chan-Zuckerberg Initiative, adds a set of explicit recommendations the preprint framed more cautiously: the authors “strongly caution against the use of GPT detectors in evaluative or educational settings” for non-native writers given the false-positive rate, and suggest detectors are better used as a student-facing check on writing style than as an accusation tool. The authors call the underlying dataset small and describe the work as a pilot needing larger, more diverse validation, and note most tools tested at the time relied on older language-model approaches rather than the newest models.

The implementation question

The mechanism is measurable and specific: these detectors infer AI authorship partly from low sentence-to-sentence perplexity, meaning predictable, low-variation phrasing, which is also a known feature of writing by people composing in a second language. A school treating a detector's score as proof of misconduct is asking a tool calibrated on native-English patterns to judge writing it was never shown enough of during its own testing.

What holds and what fails

The finding holds as evidence that, on this small sample and with the detectors available in 2023, non-native English writers faced a meaningfully higher false-positive rate than native writers on the same task. It does not establish the exact error rate of any specific detector version in use today, since tools have changed since 2023, and the sample of 91 and 88 essays is too small to generalise precisely to every context; the authors say so directly. What holds regardless of the exact numbers is the underlying mechanism: nothing about how these tools work guarantees they treat all writers' English equally.

• Has this detector, in its current version, been tested against non-native and disabled writers' work, and what was the false-positive rate?
• Is a detector score being used as a single piece of evidence, or as an automatic verdict?
• Would a flagged student have any route to be heard before a consequence is applied?

A tool that is wrong more often for some students than others is not a neutral instrument, whatever its accuracy claim on the vendor's homepage.