A test that can fail: Julie and two recommendations a guideline graded differently
A published guideline graded one recommendation A and another B. Working from seven papers and the wording of the two recommendations, not their grades, Julie drafted an A and a C. The order held and no false A appeared; the second grade came out one step lower, for reasons worth arguing about.
One recommendation makes a weak test: a system that grades everything A would pass it. Two from the same guideline, graded differently, can show whether a system tells stronger evidence from weaker. Ours came from the EULAR/PReS recommendations on Still’s disease (Fautrel et al., Annals of the Rheumatic Diseases, 2024), which use the same levels and grades as Julie’s Oxford CEBM route. One gives IL-1 and IL-6 inhibitors priority, to avoid prolonged glucocorticoid use. The other asks for them to be started as early as possible once the diagnosis is established.
The second holds the trap. Strong randomised trials show that these drugs work, but, as the guideline itself notes, no formal trial has addressed when to start them. A system that lets those trials lend their strength to the timing question would award an A. Before the run we wrote that down as the failure to look for and fixed the level we expected for each paper. We also added two papers that should not get through: a double-blind randomised trial in another disease, and a study of diagnosis only.
| What we compared | Set before the run | Julie |
|---|---|---|
| Seven full-text papers | Five kept, two kept out | Five kept, two kept out |
| Level of evidence, five papers | A reference level for each | Four of five matched |
| Priority recommendation | Guideline: 1b, grade A | 1b, grade A |
| Timing recommendation | Guideline: 2b, grade B | 4, grade C |
The A was not spotless. Of the 14 passages Julie linked to it, 13 supported the recommendation and one contradicted it: a randomised trial in adults that missed its primary endpoint. That passage stayed on the record, and the draft grade carries a note of minor inconsistency.
The C is where it gets interesting. No randomised trial was counted towards it, so the trap did not catch her. But the draft stopped below the guideline because of how she read the timing evidence. Both of her AI readers took the main timing study, which compares patients treated earlier and later within one cohort, as a case series without a control. The reference we had fixed says 2b, so this is the one paper of five where she missed it, and we count it as a miss rather than rewrite the reference afterwards. And the most direct results on timing, which relate delay to non-response, were classed as prognostic rather than therapeutic, and were not counted towards a treatment recommendation. What remained was the authors’ own conclusions and one uncontrolled result.
Who is right is a fair question. Does a comparison of earlier and later treatment within one cohort have a control? Is delay to treatment evidence about treatment, or about prognosis? The task force drew on comparisons with patients whose treatment was delayed, called the evidence on timing low-level, weighed it with efficacy and safety, and arrived at 2b and grade B. A draft built from the levels of the papers cannot do that weighing. Stricter is not the same as right, and this is no criticism of the guideline. In Julie every such grade is marked as a draft and the decision belongs to the panel. Both readings are on our list to review.
What this does and does not show. Five papers and two questions show whether the chain works and whether its output is plausible where the answer is published; they do not measure accuracy. The papers were picked by hand for the test, with no search or title and abstract screening. The guideline’s grades rest on far more: the systematic review of treatments behind it included 128 full texts, 25 of them randomised trials (Bindoli et al., Annals of the Rheumatic Diseases, 2024). The human decisions in the run were simulated and marked as such, and no panel met. An AI assistant proposed the reference level for each paper, and a clinician confirmed them before the run, with the proposals visible. That clinician works on Julie and is a co-author of the guideline and of one of the five trials, so this was neither a blinded nor an independent comparison. The result says nothing about how to treat Still’s disease. The run cost $4.47 in model usage, within a $10 ceiling approved beforehand.
What we take from it. A published answer does not tell you that a system is right. It tells you where the system differs, and why, in terms a methodologist can argue with. That only works if the reasons can be read: each of Julie’s draft grades can be followed back to the passages behind it, including those left out, so a panel can see a disagreement like this before it decides.
Tell us about your project, and what you would expect from Julie: what she should do for you, and how reliable she would need to be.
All news