Fluent is not the same as supported: checking every citation in an AI-drafted review
A clinical literature review drafted with a general-purpose AI assistant read well. Checked against its sources, citation by citation, it got the source wrong in 27 places, and in 103 more it said more than its sources carry.
The assistant found the sources, starting from the results of an AI literature search service, kept notes on them, and wrote the review from its notes. The review cites 258 sources. Counting each statement once for every source it cites, there are 895 statement–reference pairs.
We checked every pair against the full text of the cited source, or, for 12 sources, against the abstract. AI readers did the reading, and their verdicts quote the passages they rest on. 805 pairs were read a second time, among them every pair with a number in it and every pair the first reading had not found fully supported.
| Pairs | What the check found |
|---|---|
| 733 | Supported, directly or on a defensible reading of the sources cited together |
| 103 | The sources carry only part of the statement |
| 27 | The statement gets the source wrong |
| 32 | Could not be decided from the source text available |
The 27 are the clearest errors, and 25 of them concern a number. In 19, the figure is in the source but attached to the wrong group, setting, measure or denominator: a share of 512 children reported as a share of all 563; a median from one treatment arm reported for the whole trial; a figure for shoulder and elbow together reported for the shoulder. In 6, the figure itself differs from the source. Of the other two, one attributed the review’s own reasoning to the guidelines it cited, and one pointed to the wrong paper.
In the 103, the sources carry part of the statement but not all of it. Typically a sentence draws on several sources, and one element of it, a number, an interval or an “always”, appears in none of them. It is the kind of gap a peer reviewer would likely query.
Citation errors are not new: across published medicine, about one quotation in six does not support its claim (Baethge and Jergas, Research Integrity and Peer Review, 2025). This draft did not escape them, and its fluency did not show where they were. A misplaced number read exactly like a correct one; only setting each statement beside the passage it cites showed the difference.
The figures describe one review, read by AI against its sources. They show what a full check found here, not a rate for AI-drafted reviews in general. Each of the 27 errors was checked again against its source before the review was corrected.
Julie is organised around that comparison. Every statement she extracts arrives with the sentence and the page it came from, and in her drafts no citation or figure appears without a trace back to its source.
Tell us about your project, and what you would expect from Julie: what she should do for you, and how reliable she would need to be.
All news