From four trials to a manuscript draft: Julie’s first complete run with meta-analysis
For the first time, Julie took real data through the whole chain: four published randomised COPD trials went through data extraction, meta-analysis and GRADE certainty assessment to a manuscript draft. Here is what we measured along the way, and what the measurements do and do not show.
Two days ago we described how we checked a reference set for four published trials of tiotropium against placebo in COPD. That reference was built for this run. It is the yardstick against which Julie’s extracted numbers were measured before anything was pooled.
The statistics, tested on their own. Before using new data, we tested the calculations separately. From the study data of a published Cochrane review of tiotropium in COPD, Julie’s meta-analysis reproduced all 17 published quantities for two outcomes (pooled estimates, confidence limits, heterogeneity statistics and test values), and an independent calculation arrived at the same numbers. This checks the arithmetic, not the reading of articles.
The reading. Two AI readers extracted the data from the four trial reports independently. Their answers were scored against the checked reference. Of its 50 numerical entries, 48 correspond to a value the extraction records, and those 48 were scored:
| What was measured | Reader 1 | Reader 2 |
|---|---|---|
| Printed numbers matched (of 48) | 43 | 42 |
| Values returned that differed from the paper | 0 | 0 |
| Information correctly recorded as not reported (of 17) | 17 | 17 |
The numbers the readers did not get right were mostly counts of serious adverse events. One reader left them empty; the other returned conflicting candidates rather than a single value. Before pooling, those entries were filled from the checked reference by a recorded human decision. The two readers agreed on 88 of 102 compared entries (86%). Agreement between two AI readers is not accuracy; it shows where a person should look first.
The rest of the chain. From there the run continued as a review would. The analysis plan was signed before pooling. Lung function (FEV1) and serious adverse events were each pooled from two trials, and the certainty of each was rated with GRADE. The result was a draft with Methods, Results and Discussion. The draft is not finished: it hands three judgements on certainty and interpretation back to the authors, and it records the decisions taken along the way.
What this does and does not show. The scope is defined. The four trials were chosen for the test from a published review; no search or screening was part of this run. The extracted numbers were measured against a reference that one colleague checked against the printed articles, with the proposed answers visible. The readers are AI models, and 48 numbers from four trials is a small sample. This is a first complete run, not a validation. The pooled results are in the draft below. They are a sample of the output, not a clinical finding: four trials selected for a test do not answer a clinical question.
The draft, in four pages
Methods, pooled results and GRADE certainty for lung function and serious adverse events, as the run produced them.
Download the PDF, 141 KBWhat we take from it. The statistics can be checked separately from the reading, and they should be. The numbers that were not right were not invented: they were missing or conflicting, and both kinds can be seen and resolved by a person before they reach a pooled estimate.
How meta-analysis works in Julie All news