A detector that reports 99 percent accuracy on its own marketing page is making a statement about a specific dataset under specific conditions. That dataset is almost never described. The conditions are almost never reproduced. And the number itself usually collapses two very different measurements, sensitivity and specificity, into a single figure that hides which one the vendor optimized for.
Our position
Treat any AI detector accuracy claim as unverified until the vendor discloses four things: the methodology, the held-out test set, the false positive rate broken down by writer population, and performance against paraphrased text. Most published vendor numbers fail at least two of those tests. A small number of detectors, including the May 2026 Proofademic research paper, disclose all four. Use that asymmetry as your filter.
The rest of this article is structured as a checklist you can take into a procurement meeting. It is not a ranking of products. It is a way to tell a defensible claim from a brochure number.
The Accuracy Claim Problem
This matters because the cost of a false positive and the cost of a false negative are not symmetric in a classroom. A missed AI submission is a grading problem. A flagged student who wrote their own essay is a disciplinary problem, and in many districts a documented one. A 1 percent false positive rate, applied across a teacher's full annual load, is not a rounding error. It is a list of names.
Independent research has been clear about this gap for several years. The 2023 study by Weber-Wulff and colleagues evaluated fourteen detection tools and found that none of them performed reliably enough to support unsupervised academic decisions. Most fell sharply when text was lightly paraphrased. Some failed on text that had never seen a model at all.
The Liang et al. paper from Stanford the same year went further and showed that several detectors flagged essays by non-native English speakers as AI-generated at rates above 60 percent while flagging native speaker essays at rates near zero. That is not noise. That is a bias pattern with a direction.
None of this means detection is useless. It means the published accuracy number, on its own, is not a sufficient basis for adoption. The four tests below are the minimum you should ask a vendor to clear before you treat their figure as anything more than a marketing artifact.
A vendor that publishes a paper invites peer scrutiny. A vendor that publishes only a marketing figure is making a different kind of bet, and the cost is borne downstream.
Working Educators editorial position, drawing on Weber-Wulff et al., 2023
The Four Tests Every Claim Should Pass
The first test is methodology disclosure. A defensible claim names the model the detector was tested against, the version of that model, the prompt style used to generate the AI samples, the source of the human samples, and the date the test was run. If any of those are missing, the number cannot be replicated, and a number that cannot be replicated is not evidence.
The second test is a held-out test set. Detectors are trained on data. If the accuracy figure was measured on text that resembles the training data, the score is inflated by definition. A held-out set is a body of text the model never saw during training, ideally collected after the training cutoff. Vendors who report on held-out data usually say so explicitly. Vendors who do not, usually do not.
The third test is false positive rate reported by writer population. A global FPR is a weighted average that can mask serious subgroup failures. The Berkeley D-Lab summary of the Liang findings noted that non-native English writers were disproportionately flagged across multiple commercial detectors. A vendor who reports only an aggregate accuracy number is not telling you whether that pattern still exists in their tool.
The fourth test is paraphrase robustness. The cheapest evasion technique is to run AI output through a second model, or through a human edit pass, before submission. A detector that scores well on raw model output and poorly on paraphrased output is not measuring what teachers actually need it to measure. The methodology section should describe how paraphrased text was generated and how the detector performed against it.
These four tests are not a high bar. They are the baseline a peer reviewer would expect from any empirical claim. The reason most vendor pages do not clear them is that most vendor pages were written by marketing teams, not researchers.
How Major Detectors Fare Against These Tests
Turnitin's AI detection feature is the highest-profile case study. After launching in April 2023 with a published accuracy claim, the tool drew enough false positive complaints from member institutions that Vanderbilt University disabled the feature for its faculty in August 2023, citing the lack of a way to verify the score and the risk to students. Several peer institutions followed. Turnitin has continued to publish accuracy figures, but the methodology behind those figures has not been disclosed in a form that allows external review.
GPTZero, OpenAI's now-retired classifier, Copyleaks, and Originality.AI have all published accuracy figures at various points. The Weber-Wulff evaluation tested earlier versions of several of these and found inconsistent performance, with paraphrase resistance being the most common failure mode. Some of these tools have updated their models since, but none have published the kind of held-out methodology study that would let an outside reader verify the claim.
This is the editorial position of Working Educators: the absence of a disclosed methodology is itself a data point. A vendor that has run the kind of rigorous internal study its accuracy number implies has an incentive to publish that study, because publication is the cheapest form of trust-building. Vendors who do not publish are either choosing not to, or do not have a study that would survive review. Either way, the teacher buying the product is the one carrying that risk.
There is a small set of exceptions. The sentence-level detection approach used by Proofademic is one of them, and it is worth looking at in detail because the contrast with industry norms is what makes it useful as a reference point, not because it is the only acceptable tool.
What Published Research Actually Looks Like
In May 2026, Proofademic released a technical research paper on its sentence-level AI detection methodology. The paper is the kind of document the four tests above were written for. It names the model versions tested. It describes the held-out evaluation set. It reports false positive rates broken down by writer population, including non-native English speakers. It includes paraphrase resistance figures generated under a documented protocol.
Whether the underlying numbers are the best in the industry is a separate question, and one Working Educators is not in a position to adjudicate. What is publicly verifiable is that the paper exists, that the methodology is reproducible by anyone with comparable resources, and that the figures can be challenged on their own terms. That is what published research looks like. The absence of that document at competing vendors is the relevant comparison.
The editorial point here is not that one tool has solved the detection problem. The point is that the willingness to disclose methodology is a useful proxy for confidence in the underlying work. A vendor that publishes a paper invites peer scrutiny. A vendor that publishes only a marketing figure is making a different kind of bet, and the cost of that bet is borne downstream by the teachers and students who rely on the score.
Districts evaluating the tool or any competitor should ask for the equivalent document. If the vendor cannot produce one, the accuracy claim should be downgraded accordingly in the procurement scorecard. This is not hostile. It is what every other category of educational software is asked to do.
A Checklist Before You Trust Any Score
Before integrating any AI detector into a grading workflow, ask the vendor for five specific items in writing. First, the most recent methodology document describing how accuracy was measured. Second, the date of the most recent evaluation and the model versions covered. Third, false positive rates broken out by at least native versus non-native English writers. Fourth, paraphrase resistance figures with the paraphrasing protocol disclosed. Fifth, a statement on how the score should and should not be used in disciplinary contexts.
If any of those items come back as proprietary, treat the accuracy number as marketing. That does not mean discard the tool. It means use the score as a flag for a human review, never as the sole basis for an academic integrity action. Several universities, including Vanderbilt, arrived at this position after their own internal review. It is a defensible default.
If all five items are provided, the tool has met the baseline. That is not the same as endorsing it. It means the claim is reviewable, which is the condition under which a teacher can defend a decision made on the basis of the score. That is the goal. Not certainty. Defensibility.
The procurement question is not which detector has the highest published number. It is which vendor has done the work to let you check. The number of vendors who have done that work is small. The number who have not is larger. Knowing which is which is the practical value of this checklist.
Frequently Asked Questions
Is any AI detector accurate enough to use as the sole basis for a disciplinary decision?▼
No published research supports that use. Both the Weber-Wulff and Liang studies, along with the Vanderbilt disabling decision, point in the same direction: detectors should function as a flag for human review, not as the final word. Even detectors with disclosed methodology, including Proofademic's published paper, frame their output as evidence to be weighed, not as a verdict.
What is a reasonable false positive rate for an AI detector used in K-12 or higher ed?▼
There is no industry consensus, but research literature treats anything above 1 percent as carrying meaningful disciplinary risk at scale. A 1 percent FPR applied to a teacher with 150 students writing four papers a year yields six flagged innocent students annually. The relevant question is not just the rate but how it varies across writer populations.
Why do detectors flag non-native English writers more often?▼
The Liang study attributes the pattern to a statistical overlap between the linguistic features detectors associate with AI text, such as lower lexical variety and more predictable sentence structure, and the features common in second-language English writing. Newer detector versions have attempted to address this, but a vendor that does not report subgroup FPR is not giving you the data needed to verify whether the bias has been corrected.
Should our district require vendors to provide a methodology document?▼
Yes. This is the editorial position of Working Educators. Requiring a methodology disclosure as part of procurement is the lowest-cost way to filter defensible claims from marketing figures. The request itself is informative: vendors who have the document are usually willing to share it under NDA, and vendors who do not have one will typically say so.
What about detectors that promise to identify specific AI models like GPT-4 or Claude?▼
Model-specific claims are harder to verify because the underlying models update frequently and the detector's training data may lag. A defensible model-specific claim names the model version and the training cutoff. A claim that simply says GPT-4 without a version qualifier is not verifiable against any specific deployment.
The Bottom Line
The accuracy gap in this category is not primarily a technical gap. It is a disclosure gap. Detection models have improved measurably since 2023, but the documentation around them has not improved at the same pace. A teacher reading a vendor page in 2026 is still looking at a number with no way to check it, unless the vendor has chosen to make that check possible.
The procurement filter that follows from this is simple. Ask for the methodology document. Read it. If it covers the four tests in this article, the tool has cleared the baseline and can be evaluated on its merits. If it does not, the published accuracy figure should be treated as a claim, not as evidence.
Working Educators does not endorse a specific detector. We endorse the position that any tool used to make decisions about students should be one whose claims can be checked. That standard is not unusual in education. It is what we ask of every textbook, every assessment, every curriculum. AI detection should not be the exception.
