Checking the reference: 50 numbers, no numerical corrections
A source check of four published COPD trials confirmed every numerical entry in an AI-assisted reference set. Six descriptive entries needed clarification. The distinction matters when we assess what AI can contribute to research.
Before measuring how well a system extracts evidence, we need confidence in the reference against which its answers will be checked. For this part of Julie’s evaluation work, an AI assistant helped prepare 104 reference entries from four published randomised COPD trials. A medical colleague then checked every entry against the printed articles, with the proposed answers visible.
The reference included numerical values, records of information that was not printed, and descriptions of what each result represented. These categories tell different parts of the story:
| What was checked | What the review found |
|---|---|
| 50 numerical entries, including values derived from published numbers | All confirmed; no numerical correction required |
| 17 entries recording information as not printed | All confirmed |
| 37 descriptive entries | 31 clear; six flagged for clarification |
The six questions concerned descriptions of the results, such as the treatment arm, timing, or the meaning of a reported value. Four concerned wording or where the supporting information appeared in the article. In two, the reference description was more specific than the paper explicitly stated.
The lead researcher checked all six against the source text in a second review. The reference values remained unchanged, and clarifications were recorded in the notes. The six clarifications resolved questions about the descriptions; they did not involve changing numerical values.
This is a useful result with a defined scope. One reviewer checked the complete reference, and the second review covered the six flagged entries. The review was not blinded, and the data were not independently extracted by two human readers. Seeing a proposed answer can influence a reviewer’s judgement. The result therefore provides confirmation of this reference set; it cannot establish inter-reader agreement or an overall accuracy rate for Julie.
The practical lesson is that accurate numbers and careful descriptions both deserve attention. Here, the human check found no numerical corrections to make, while it identified two descriptions that went beyond the paper’s explicit wording. Those distinctions can disappear behind a single headline percentage.
For researchers, the useful question is what was checked, against which source, and what required clarification. In these four trials, the answer is specific: 50 numerical entries confirmed without correction, six descriptive questions resolved, and a record of the human review behind that conclusion.
All news