Exclusion carries the greater responsibility: a 300-record screening pilot
In a 300-record development sample from a systematic review in progress, Julie’s screening kept 71 of the 72 records that three AI reviewers all judged eligible or possibly eligible, and 77 of the 84 that at least two did. The figures measure agreement with a machine reference, not accuracy.
The two kinds of screening error do not cost the same. A record kept by mistake costs a full-text check; a record excluded by mistake is a study lost to the review. The lead researcher of the review states it as a rule: at title and abstract, exclusion carries the greater responsibility. Where the pilot showed that the eligibility criteria left room for doubt, the criteria were clarified on that principle before the run reported here.
Julie assesses every record twice against the review’s criteria. Here, a record counts as kept when it was not excluded at this stage: both assessments included it, or it was queued for a decision.
To see which eligible records the screening excludes, three further AI reviewers, separate from the screening, assessed the same 300 records on their own, without seeing its results. The records were drawn at random from the review’s search. A record counted as eligible for a reviewer when that reviewer judged it eligible or possibly eligible. Their judgements predate the latest clarification of the criteria and were not changed afterwards, so the reference and the screening worked from slightly different wording. We report two combined references (the records all three reviewers judged eligible, and those at least two of them did) and each reviewer alone.
| Records | Kept by the screening |
|---|---|
| 72 that all three reviewers judged eligible | 71 (98.6%; lower bound 93.6%) |
| 84 that at least two reviewers judged eligible | 77 (91.7%; lower bound 84.9%) |
| All 300 in the sample | 96 (32%): 7 included by both assessments, 89 queued for a decision; the other 204 excluded |
Against each reviewer alone, the screening kept 74 of 75, 77 of 86 and 80 of 89. The lower bounds in the table are exact one-sided 95% confidence bounds; they show how much uncertainty references of this size leave.
This is a development result with a defined scope. The screening was refined on these 300 records, with knowledge of where it had disagreed with this reference, so the figures are in-sample and likely to be optimistic. The reference is made of AI judgements that can share blind spots with the screening, so the figures describe agreement between machine assessments, not accuracy against human judgement. And “kept” stops at this stage: the figures say nothing about how a researcher would decide the 89 queued records. They cannot establish Julie’s sensitivity in this review.
That is the task of the independent validation. It uses 800 further records drawn at random from the rest of the search, none of them used in development, and two human reviewers will label each of them, blind to each other’s work and to all machine output.
Update, 27 October 2026. The validation sample changed after this article appeared. Later the same day, the lead researcher decided not to set the 300 development records aside when drawing it, so 34 of the 800 records drawn are development records. A pre-specified analysis repeats the measurement on the other 766.
For researchers, the useful questions are which records were measured, against which reference, and what “kept” means. Here the answers are specific: 300 records drawn at random, a reference of three AI reviewers whose judgements were left unchanged, and the share kept stated for every way of reading that reference.
All news