Julie’s screening, measured in your own review

An instrument you did not build earns trust the way any other does: by being measured against a reference. For screening, the reference is human judgement, and the figure that matters most is how many eligible records the screening kept. A record excluded by mistake is a study lost to the review.

You set the terms before the first sheet goes out: how many records to check, or all of them; one reviewer, two, or two with an adjudicator; whether your reviewers work blind; and, if you wish, a pass mark. Julie records these choices before the check begins. If you change them later, the earlier version stays on the record beside the new one.

The records are drawn from those Julie kept and from those she excluded, so the check reaches the exclusions, where a lost study would be. A blind sheet shows a reviewer neither Julie’s decision nor anyone else’s answer. Where two reviewers disagree, the adjudicator receives only those records, without the earlier answers or Julie’s decisions.

The report gives the share of eligible records Julie kept, estimated for all the records she screened rather than only the sample, with a conservative lower confidence bound. Beside it: the agreement between your reviewers, how often a record she excluded proved eligible, how many records needed adjudication, the time the check took, and Methods text for the manuscript. With a pass mark set in advance, the report states whether it was met; without one, it gives the figures without a verdict.

The check covers screening at title and abstract and, when you work with Oxford CEBM, at full text.

Julie’s own screening is being measured the same way in the review described in our screening pilot: two human reviewers are labelling 800 records, blind to each other’s work and to all machine output.

All news→