TY - JOUR
T1 - Are interventions in reproductive medicine assessed for plausible and clinically relevant effects?
T2 - A systematic review of power and precision in trials and meta-analyses
AU - Stocking, Katie
AU - Wilkinson, Jack
AU - Lensen, S
AU - Brison, D R
AU - Roberts, Stephen A
AU - Vail, Andy
N1 - Funding Information:
1Department of Medical Statistics, Manchester University NHS Foundation Trust, Manchester, M23 9LT, UK 2Centre for Biostatistics, Division of Population Health, Health Services Research and Primary Care, University of Manchester, Manchester M13 9PL, UK 3Department of Obstetrics and Gynaecology, University of Auckland, 1142, New Zealand 4Medical Research Council Clinical Trials Unit, University College London, London, WC1V 6LJ, UK 5Department of Reproductive Medicine, Manchester University NHS Foundation Trust, Manchester Academic Health Science Centre, Manchester, M13 9WL, UK 6Maternal and Fetal Health Research Centre, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Sciences Centre, Manchester, M13 9WL, UK © The Author(s) 2019. Published by Oxford University Press on behalf of the European Society of Human Reproduction and Embryology. This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact [email protected]
Publisher Copyright:
© 2019 The Author(s).
PY - 2019/4/1
Y1 - 2019/4/1
N2 - STUDY QUESTION: How much statistical power do randomised controlled trials (RCTs) and meta-analyses have to investigate the effectiveness of interventions in reproductive medicine?SUMMARY ANSWER: The largest trials in reproductive medicine are unlikely to detect plausible improvements in live birth rate (LBR), and meta-analyses do not make up for this shortcoming.WHAT IS KNOWN ALREADY: Effectiveness of interventions is best evaluated using RCTs. In order to be informative, these trials should be designed to have sufficient power to detect the smallest clinically relevant effect. Similar trials can subsequently be pooled in meta-analyses to more precisely estimate treatment effects.STUDY DESIGN, SIZE, DURATION: A review of power and precision in 199 RCTs and meta-analyses from 107 Cochrane Reviews was conducted.PARTICIPANTS/MATERIALS, SETTING, METHODS: Systematic reviews published by Cochrane Gynaecology and Fertility with the primary outcome live birth were identified. For each live birth (or ongoing pregnancy) meta-analysis and for the largest RCT in each, we calculated the power to detect absolute improvements in LBR of varying sizes. Additionally, the 95% CIs of estimated treatment effects from each meta-analysis and RCT were recorded, as these indicate the precision of the result.MAIN RESULTS AND THE ROLE OF CHANCE: Median (interquartile range) power to detect an improvement in LBR of 5 percentage points (pp) (e.g. 25-30%) was 13% (8-21%) for RCTs and 16% (9-33%) for meta-analyses. No RCTs and only 2% of meta-analyses achieved 80% power to detect an improvement of 5 pp. Median power was high (85% for trials and 93% for meta-analyses) only in relation to 20 pp absolute LBR improvement, although substantial numbers of trials and meta-analyses did not achieve 80% power even for this improbably large effect size. Median width of 95% CIs was 25 pp and 21 pp for RCTs and meta-analyses, respectively. We found that 28% of Cochrane Reviews with LBR as the primary outcome contain no live birth (or ongoing pregnancy) data.LARGE-SCALE DATA: The data used in this study may be accessed at https://osf.io/852tn/?view_only=90f1579ce72747ccbe572992573197bd.LIMITATIONS, REASONS FOR CAUTION: The design and analysis decisions used in this study are predicted to overestimate the power of trials and meta-analyses, and the size of the problem is therefore likely understated. For some interventions, it is possible that larger trials not reporting live birth or ongoing pregnancy have been conducted, which were not included in our sample. In relation to meta-analyses, we calculated power as though all participants were included in a single trial. This ignores heterogeneity between trials in a meta-analysis, and will cause us to overestimate power.WIDER IMPLICATIONS OF THE FINDINGS: Trials capable of detecting realistic improvements in LBR are lacking in reproductive medicine, and meta-analyses are not large enough to overcome this deficiency. This situation will lead to unwarranted pessimism as well as unjustified enthusiasm regarding reproductive interventions, neither of which are consistent with the practice of evidence-based medicine or the idea of informed patient choice. However, RCTs and meta-analyses remain vital to establish the effectiveness of fertility interventions. We discuss strategies to improve the evidence base and call for collaborative studies focusing on the most important research questions.STUDY FUNDING/COMPETING INTEREST(S): There was no specific funding for this study. KS and SL declare no conflict of interest. AV consults for the Human Fertilisation and Embryology Authority (HFEA): all fees are paid directly to AV's employer. JW declares that publishing research benefits his career. SR is a Statistical Editor for Human Reproduction. JW and AV are Statistical Editors for Cochrane Gynaecology and Fertility. DRB is funded by the NHS as Scientific Director of a clinical IVF service.PROSPERO REGISTRATION NUMBER: None.
AB - STUDY QUESTION: How much statistical power do randomised controlled trials (RCTs) and meta-analyses have to investigate the effectiveness of interventions in reproductive medicine?SUMMARY ANSWER: The largest trials in reproductive medicine are unlikely to detect plausible improvements in live birth rate (LBR), and meta-analyses do not make up for this shortcoming.WHAT IS KNOWN ALREADY: Effectiveness of interventions is best evaluated using RCTs. In order to be informative, these trials should be designed to have sufficient power to detect the smallest clinically relevant effect. Similar trials can subsequently be pooled in meta-analyses to more precisely estimate treatment effects.STUDY DESIGN, SIZE, DURATION: A review of power and precision in 199 RCTs and meta-analyses from 107 Cochrane Reviews was conducted.PARTICIPANTS/MATERIALS, SETTING, METHODS: Systematic reviews published by Cochrane Gynaecology and Fertility with the primary outcome live birth were identified. For each live birth (or ongoing pregnancy) meta-analysis and for the largest RCT in each, we calculated the power to detect absolute improvements in LBR of varying sizes. Additionally, the 95% CIs of estimated treatment effects from each meta-analysis and RCT were recorded, as these indicate the precision of the result.MAIN RESULTS AND THE ROLE OF CHANCE: Median (interquartile range) power to detect an improvement in LBR of 5 percentage points (pp) (e.g. 25-30%) was 13% (8-21%) for RCTs and 16% (9-33%) for meta-analyses. No RCTs and only 2% of meta-analyses achieved 80% power to detect an improvement of 5 pp. Median power was high (85% for trials and 93% for meta-analyses) only in relation to 20 pp absolute LBR improvement, although substantial numbers of trials and meta-analyses did not achieve 80% power even for this improbably large effect size. Median width of 95% CIs was 25 pp and 21 pp for RCTs and meta-analyses, respectively. We found that 28% of Cochrane Reviews with LBR as the primary outcome contain no live birth (or ongoing pregnancy) data.LARGE-SCALE DATA: The data used in this study may be accessed at https://osf.io/852tn/?view_only=90f1579ce72747ccbe572992573197bd.LIMITATIONS, REASONS FOR CAUTION: The design and analysis decisions used in this study are predicted to overestimate the power of trials and meta-analyses, and the size of the problem is therefore likely understated. For some interventions, it is possible that larger trials not reporting live birth or ongoing pregnancy have been conducted, which were not included in our sample. In relation to meta-analyses, we calculated power as though all participants were included in a single trial. This ignores heterogeneity between trials in a meta-analysis, and will cause us to overestimate power.WIDER IMPLICATIONS OF THE FINDINGS: Trials capable of detecting realistic improvements in LBR are lacking in reproductive medicine, and meta-analyses are not large enough to overcome this deficiency. This situation will lead to unwarranted pessimism as well as unjustified enthusiasm regarding reproductive interventions, neither of which are consistent with the practice of evidence-based medicine or the idea of informed patient choice. However, RCTs and meta-analyses remain vital to establish the effectiveness of fertility interventions. We discuss strategies to improve the evidence base and call for collaborative studies focusing on the most important research questions.STUDY FUNDING/COMPETING INTEREST(S): There was no specific funding for this study. KS and SL declare no conflict of interest. AV consults for the Human Fertilisation and Embryology Authority (HFEA): all fees are paid directly to AV's employer. JW declares that publishing research benefits his career. SR is a Statistical Editor for Human Reproduction. JW and AV are Statistical Editors for Cochrane Gynaecology and Fertility. DRB is funded by the NHS as Scientific Director of a clinical IVF service.PROSPERO REGISTRATION NUMBER: None.
KW - clinical trials
KW - infertility
KW - meta-analysis
KW - power
KW - randomised controlled trials
KW - subfertility
KW - systematic reviews
UR - http://www.scopus.com/inward/record.url?scp=85064115362&partnerID=8YFLogxK
UR - http://www.mendeley.com/research/interventions-reproductive-medicine-assessed-plausible-clinically-relevant-effects-systematic-review
U2 - 10.1093/humrep/dez017
DO - 10.1093/humrep/dez017
M3 - Article
C2 - 30838395
SN - 0268-1161
VL - 34
SP - 659
EP - 665
JO - Human reproduction (Oxford, England)
JF - Human reproduction (Oxford, England)
IS - 4
ER -