Agreement was moderate between data-based and opinion-based assessments of biases affecting randomised trials within meta-analyses

Rebecca M. Turner, Kirsty M. Rhodes, Hayley E. Jones, Julian Higgins, Jess Haskins, Penny Whiting, Asbjørn Hróbjartsson, Debbi Caldwell, Richard Morris, Barney C. Reeves, Helen Worthington, Isabelle Boutron, Jelena Savović

Research output: Contribution to journalArticlepeer-review

6 Downloads (Pure)


Background: Randomised trials included in meta-analyses are often affected by bias caused by methodological flaws or limitations, but the degree of bias is unknown. Two proposed methods adjust trial results for bias using: (1) empirical evidence from published meta-epidemiological studies; or (2) expert opinion.

Methods: We investigated agreement between data-based and opinion-based approaches to assessing bias in each of four domains: sequence generation, allocation concealment, blinding and incomplete outcome data. From each sampled meta-analysis, a pair of trials with the highest and lowest empirical model-based bias estimates was selected. Independent assessors were asked which trial within each pair was judged more biased on the basis of detailed trial design summaries.

Results: Assessors judged trials to be equally biased in 68% of pairs evaluated. When assessors judged one trial as more biased, the proportion of judgements agreeing with the model-based ranking was highest for allocation concealment (79%) and blinding (79%) and lower for sequence generation (59%) and incomplete outcome data (56%).

Conclusions: Most trial pairs found to be discrepant empirically were judged to be equally biased by assessors. We found moderate agreement between opinion and data-based evidence in pairs where assessors ranked one trial as more biased.
Original languageEnglish
JournalJournal of Clinical Epidemiology
Early online date13 May 2020
Publication statusPublished - 2020


  • Meta-analysis
  • Systematic reviews
  • Randomised trials
  • Bias


Dive into the research topics of 'Agreement was moderate between data-based and opinion-based assessments of biases affecting randomised trials within meta-analyses'. Together they form a unique fingerprint.

Cite this