How it works
Nine stages, and where you decide
Julie follows the stages a systematic review actually has. What follows is that work from your side: what she prepares, what comes back to you, what you decide, and how the review carries on from your decision.
EXAMPLE · A GUIDELINE UPDATEIt crosses this chain once. The F2F consensus meeting you already have in the calendar is one of the decisions on the way, and it arrives with the panel's reading material already prepared.
- 1 research questions
- 2 eligibility criteria
- 3 search strategy preparation
- ↓ approve, run and export searches
- 4 title and abstract screening
- ↓ retrieving the full texts
- 5 full-text eligibility check
- 6 extraction and rating
- 7 material for the panel
- ↓ consensus meeting
- 8 linking evidence to recommendations
- 9 manuscript draft
From your question to a search strategy
Julie helps turn your research questions into draft search strategies for PubMed, Embase and CENTRAL in the Cochrane Library. She asks you to clarify the scope and prepares a strategy for each database. You review and approve the final strategies, then run the searches and export the results.
The search settings and your approval stay with the strategy. If the question or strategy changes, the new version comes back to you for review.
Two standards, one workflow
Julie works with both Oxford CEBM and GRADE. You choose the standard for each project at the outset, because it shapes what is extracted, how it is rated and what you receive. Oxford CEBM rates each study, with a level of evidence and a strength of recommendation. GRADE rates the body of evidence for each outcome of each question, as high, moderate, low or very low certainty, with the reasoning set out domain by domain.
The stages below are the same on both routes, and where the GRADE route differs, the stage says so. On GRADE, extraction shows which outcomes of each question each study informs, and missing outcome data is never a reason to set a study aside; the panel works from an Evidence-to-Decision framework rather than a separate brief, with the direction, strength and rationale of each recommendation recorded under a person's name.
Both standards can also be applied to the same literature. Each then produces its own workbook and its own methods report, and the manuscript cites one of them, by your explicit choice. On either route, a review can start from scratch or update an existing set of recommendations with new evidence.
Stage by stage
1. Research questions
- What happens
- You write them
- What you have afterwards
- the question set every later stage reads
2. Eligibility criteria
- What happens
- Julie drafts them from your questions, you settle the wording
- What you have afterwards
- the criteria every screening decision rests on
3. Search strategy preparation
- What happens
- Julie clarifies the scope with you and prepares a draft strategy for each database
- What you have afterwards
- draft strategies for PubMed, Embase and CENTRAL, with their search settings, ready for your review
↓ Approve, run and export searches
- What happens
- You approve the final strategies, run the searches in the databases and bring back the exports
- What you have afterwards
- the records to screen against your criteria
4. Title and abstract screening
- What happens
- each record assessed twice, you decide where they differ
- What you have afterwards
- the retained set, the upper half of the PRISMA flow, and the worklist of full texts to obtain
↓ Retrieving the full texts
- What happens
- You obtain what you can and say what you could not
- What you have afterwards
- a retrieval record with a stated reason for every report missing from it
5. Full-text eligibility check
- What happens
- On CEBM, Julie checks each full text against your questions before anything is rated, and you can review and correct how a study relates to them. On GRADE, extraction shows which outcomes of each question each study informs, and a study missing data for an outcome is not excluded for that reason
- What you have afterwards
- which studies carry evidence, and a recorded reason for each one set aside
6. Extraction and rating
- What happens
- Julie extracts and rates, each study assessed twice
- What you have afterwards
- on CEBM, a level of evidence for each study; on GRADE, a certainty rating for each outcome of each question; each traceable to the quotations it rests on
7. Material for the panel
- What happens
- Julie prepares it from the rated evidence: on CEBM, a brief; on GRADE, an Evidence-to-Decision framework with the certainty of the evidence already entered
- What you have afterwards
- on CEBM, a brief the panel can read beforehand and a dossier per question; on GRADE, an evidence profile, a Summary of Findings table and the framework the panel completes
↓ Consensus meeting
- What happens
- You and the panel decide, you record the outcome. On GRADE, the direction, strength and rationale of each recommendation go into the Evidence-to-Decision framework, under a person's name
- What you have afterwards
- the agreed recommendations and the record of what was decided
8. Linking evidence to recommendations
- What happens
- on CEBM, Julie links supporting and contradicting evidence to the wording the panel actually agreed, including amendments; on GRADE, each recommendation is tied to the certainty of the evidence behind it
- What you have afterwards
- the evidence behind each recommendation, traceable to verified quotations
9. Manuscript draft
- What happens
- Julie drafts, behind the decisions above
- What you have afterwards
- a drafted discussion, a record of coverage, and the author review still to do
| Stage | What happens | What you have afterwards |
|---|---|---|
| 1. Research questions | You write them | the question set every later stage reads |
| 2. Eligibility criteria | Julie drafts them from your questions, you settle the wording | the criteria every screening decision rests on |
| 3. Search strategy preparation | Julie clarifies the scope with you and prepares a draft strategy for each database | draft strategies for PubMed, Embase and CENTRAL, with their search settings, ready for your review |
| ↓ Approve, run and export searches | You approve the final strategies, run the searches in the databases and bring back the exports | the records to screen against your criteria |
| 4. Title and abstract screening | each record assessed twice, you decide where they differ | the retained set, the upper half of the PRISMA flow, and the worklist of full texts to obtain |
| ↓ Retrieving the full texts | You obtain what you can and say what you could not | a retrieval record with a stated reason for every report missing from it |
| 5. Full-text eligibility check | On CEBM, Julie checks each full text against your questions before anything is rated, and you can review and correct how a study relates to them. On GRADE, extraction shows which outcomes of each question each study informs, and a study missing data for an outcome is not excluded for that reason | which studies carry evidence, and a recorded reason for each one set aside |
| 6. Extraction and rating | Julie extracts and rates, each study assessed twice | on CEBM, a level of evidence for each study; on GRADE, a certainty rating for each outcome of each question; each traceable to the quotations it rests on |
| 7. Material for the panel | Julie prepares it from the rated evidence: on CEBM, a brief; on GRADE, an Evidence-to-Decision framework with the certainty of the evidence already entered | on CEBM, a brief the panel can read beforehand and a dossier per question; on GRADE, an evidence profile, a Summary of Findings table and the framework the panel completes |
| ↓ Consensus meeting | You and the panel decide, you record the outcome. On GRADE, the direction, strength and rationale of each recommendation go into the Evidence-to-Decision framework, under a person's name | the agreed recommendations and the record of what was decided |
| 8. Linking evidence to recommendations | on CEBM, Julie links supporting and contradicting evidence to the wording the panel actually agreed, including amendments; on GRADE, each recommendation is tied to the certainty of the evidence behind it | the evidence behind each recommendation, traceable to verified quotations |
| 9. Manuscript draft | Julie drafts, behind the decisions above | a drafted discussion, a record of coverage, and the author review still to do |
No evidence, no rating
A study is rated only once it has been shown to yield usable evidence, so an unsuitable paper cannot carry a full evidence level through the rest of the review. What fell out, and why, is on the record, which is also what the PRISMA box for excluded reports needs.
Decisions along the way
-
01
Adjudicating the screening
Every record is assessed twice against your criteria. Disagreements and uncertain cases come to you for a documented decision. Your screening decisions are recorded under your name.
-
02
Confirming the full texts you obtained
Julie gives you the worklist of full texts to obtain, named article by article. You obtain what you can and tell her what you could not, and why. The evidence work starts from that confirmation, and your reasons become the review's account of the reports it could not retrieve.
-
03
The full-text eligibility check
You review the full-text eligibility results, with a recorded reason for each study set aside. The record shows which studies contribute evidence to your review and which need your attention. On GRADE, the record shows which outcomes of each question each study informs, and missing outcome data is never a reason to set a study aside.
-
04
The consensus meeting and its record
On CEBM, Julie prepares what the panel reads beforehand: a brief written to be shared, with a dossier behind it for each question. After the meeting you record what was decided for each recommendation, accepted, amended, rejected or deferred, and give the final wording of anything amended. The evidence work behind the recommendations waits for that record.
On GRADE, the panel works from an Evidence-to-Decision framework instead, with the certainty of the evidence already entered. The direction, strength and rationale of each recommendation are recorded there, under a person's name.
When approval is needed
-
05
Approving what a paid stage will cost
No stage that costs money starts on its own. Julie estimates it first, on your own corpus and at no cost, and waits for the ceiling you set and your approval. Declining costs nothing and breaks nothing. A review has several paid stages, so this decision comes back more than once.
-
06
Releasing material that is still incomplete
If a question failed or was never processed, Julie will not hand you a document that reads as finished. She shows you what is missing and in what state, because a question that found no evidence is a finding and one whose processing broke is not. Releasing it anyway is yours to decide, and what goes out is marked partial.
So the claim is not that the chain runs end to end. It is this: the automated stages run end to end, and where a decision is yours, a named person makes it and that decision leaves a trace.
If you are running a systematic review, sitting on a guideline panel, or working in an evidence synthesis unit, tell us what you are working on.