Scolaris Content Display Scolaris Content Display

Cochrane Database of Systematic Reviews Editorial

Why do we need evidence‐based methods in Cochrane?

Authors' declarations of interest

Version published: 24 July 2015

https://doi.org/10.1002/14651858.ED000102

If systematic reviews are to provide the information that people need when making decisions about health and social care, we need to be confident that the methods used to plan, conduct, and report these reviews result in valid and reliable evidence that fairly represents the sum of knowledge. This year's Annual Cochrane Methods Symposium at the Cochrane Colloquium (colloquium.cochrane.org) focuses on why the methods used to produce reviews for evidence‐based care should themselves be evidence‐based.

As systematic reviewers, we devote long hours to scrutinising the evidence underpinning clinical practice. Yet how often do we question whether there is good evidence for the methods we use? Our confidence in the findings of more than 35,000 systematic reviews[1] rests on the evidence base underpinning the methods we use. Just as there are consequences arising from the choices we make about health and social care interventions, so too are there consequences when we choose the methods to use in systematic reviews. The DerSimonian and Laird random effects method offers an example of this.[2] Despite this being the most commonly used method, statistical simulation studies have shown that other methods perform better in certain circumstances.[3, 4] However, further research is required to establish how often the use of an alternative method leads to different review conclusions and under what circumstances more than one method should be used. Methods research can allow us to make well‐informed decisions about the most appropriate methods through the provision of information that allows us to understand trade‐offs in terms of bias, efficiency, usability, and resource use.

Over recent decades, the evidence base for systematic review methods has been growing, with some areas having received considerable attention. Examples of methods research includes descriptive studies of reviews[5, 6, 7] and research using varied designs into the effects of alternative methods strategies, including search filters for eligible studies,[8, 9] screening records retrieved from bibliographic databases,[10] meta‐analysis methods,[11, 12, 13] prioritisation of updating of reviews,[14, 15] and ways to summarise the findings of reviews.[16, 17] However, many uncertainties and much debate remains, as illustrated by the findings of systematic reviews of methodology research, such as those conducted under the auspices of the Cochrane Methodology Review Group. These have included reviews relevant to searching for eligible studies,[18, 19, 20] obtaining data,[21] and investigating biases in the review process.[22]

The Methods Symposium will present a series of case studies to illustrate how methods have been developed, evaluated, and implemented for systematic reviews, within and beyond Cochrane. Discussion will cover different approaches to evaluating methods (assessing their strengths and weaknesses), the quality of evidence required to change methods, and how we bring about change in the methods used by people preparing, maintaining, and disseminating reviews. In addition, we will consider how best to determine where methods development is required and how to prioritise that development. There will be opportunity to discuss how research into review methods might be facilitated, using approaches such as the Studies Within A Review (SWAR) programme.[23]

Similar to research evaluating the effects of clinical interventions, there will be uncertainties and varied interpretations arising from research evaluating the effects of methods. What constitutes enough evidence for one person to implement a new method will not be enough for another. As systematic review methods proliferate, there will be an increasing impetus for reviewers, and review groups, to implement new methods, which they may deem have sufficient evidence. However, consistency of review methods has always been a desired and notable feature of Cochrane Reviews[24] and provides some guarantee to review users that the standard methods are appropriate and that the ensuing evidence is valid. This leaves Cochrane walking a tightrope: balancing consistency of methods with review authors' autonomy to use alternative methods. Cochrane is working toward developing processes to manage review developments. The Methods Symposium will allow for debate and discussion on these issues and we encourage participation of reviewers, editors, and methodologists.

The 2015 Cochrane Methods Symposium will take place on 2 October 2015 at the Palais Niederösterriech, Vienna, Austria. Find out more on the Cochrane Methods website.

Information

DOI:
https://doi.org/10.1002/14651858.ED000102Copy DOI
Database:
  1. Cochrane Database of Systematic Reviews
Published:
  1. 24 July 2015
Copyright:
    Copyright © 2015 The Cochrane Collaboration. Published by John Wiley & Sons, Ltd.

Article Metrics

Altmetric:

Topics

Authors

  • Joanne E McKenzie

    School of Public Health & Preventive Medicine, Monash University, Melbourne, Australia

    [email protected]

    Cochrane Statistical Methods Group

  • Mike J Clarke

    Northern Ireland Network for Trials Methodology Research, Queen's University Belfast, UK

    [email protected]

    Cochrane Methodology Review Group

  • Jackie Chandler

    Cochrane, London, UK

    [email protected]

Declarations of interest

The authors have completed the Unified Competing Interest form (www.icmje.org/coi_disclosure.pdf). JM receives salary support from Monash University/Australian National Health and Medical Research Council, via an Australian Public Health Fellowship to undertake methodological research in evidence synthesis. JM has also received grant funding from Cochrane, and is a convenor of the Cochrane Statistical Methods Group and a member of the Methods Board and Methods Executive for Cochrane. MC's post, salary and research grant income depends on the conduct of systematic reviews and other evaluations of health and social care, as well as activities related to the investigation of the methods for these evaluations. MC is also Co‐ordinating Editor of the Cochrane Methodology Review Group, which has no dedicated funding but which provides editorial oversight for all Cochrane Methodology Reviews. JC is employed as Methods Co‐ordinator by Cochrane.

Provenance and peer review

This editorial was commissioned and was not externally peer reviewed.

References

  1. University of York. Centre for Reviews and Dissemination. About the databases. www.crd.york.ac.uk/CRDweb/aboutpage.asp (accessed 22 July 2015).
  2. DerSimonian R, Laird N. Meta‐analysis in clinical trials. Controlled Clinical Trials1986;7(3):177–88. doi.org/10.1016/0197‐2456(86)90046‐2
  3. Sidik K, Jonkman JN. A comparison of heterogeneity variance estimators in combining results of studies. Statistics in Medicine2007;26(9):1964–81. doi.org/10.1002/sim.2688
  4. Veroniki AA, Jackson D, Viechtbauer W, Bender R, Knapp G, Kuss O, et al. Recommendations for quantifying the uncertainty in the summary intervention effect and estimating the between‐study heterogeneity variance in random‐effects meta‐analysis. In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews2015; Suppl 1 (in press).
  5. Bastian H, Glasziou P, Chalmers I. Seventy‐five trials and eleven systematic reviews a day: how will we ever keep up?PLOS Medicine2010;7(9):e1000326. doi.org/10.1371/journal.pmed.1000326
  6. Davey J, Turner RM, Clarke MJ, Higgins JP. Characteristics of meta‐analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross‐sectional, descriptive analysis. BMC Medical Research Methodology2011;11:160. doi.org/10.1186/1471‐2288‐11‐160
  7. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLOS Medicine2007;4:e78. doi.org/10.1371/journal.pmed.0040078
  8. Lefebvre C, Glanville J, Wieland LS, Coles B, Weightman AL. Methodological developments in searching for studies for systematic reviews: past, present and future?Systematic Reviews2013;2:78. doi.org/10.1186/2046‐4053‐2‐78
  9. McKibbon KA, Wilczynski NL, Haynes RBfor the Hedges Team. Retrieving randomized controlled trials from medline: a comparison of 38 published search filters. Health Information and Libraries Journal2009;26(3):187–202. doi.org/10.1111/j.1471‐1842.2008.00827.x
  10. Edwards P, Clarke M, DiGuiseppi C, Pratap S, Roberts I, Wentz R. Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine2002;21(11):1635–40. doi.org/10.1002/sim.1190
  11. Deeks JJ. Issues in the selection of a summary statistic for meta‐analysis of clinical trials with binary outcomes. Statistics in Medicine2002;21(11):1575–600. doi.org/10.1002/sim.1188
  12. Inthout J, Ioannidis JP, Borm GF. The Hartung‐Knapp‐Sidik‐Jonkman method for random effects meta‐analysis is straightforward and considerably outperforms the standard DerSimonian‐Laird method. BMC Medical Research Methodology2014;14(1):25. doi.org/10.1186/1471‐2288‐14‐25
  13. Langan D, Higgins JP, Simmonds M. An empirical comparison of heterogeneity variance estimators in 12 894 meta‐analyses. Research Synthesis Methods2015;6(2):195–205. doi.org/10.1002/jrsm.1140
  14. Pattanittum P, Laopaiboon M, Moher D, Lumbiganon P, Ngamjarus C. A comparison of statistical methods for identifying out‐of‐date systematic Reviews. PLOS One2012;7(11):e48894. doi.org/10.1371/journal.pone.0048894
  15. Takwoingi Y, Hopewell S, Tovey D, Sutton AJ. A multicomponent decision tool for prioritising the updating of systematic reviews. BMJ2013;347:f7191. doi.org/10.1136/bmj.f7191
  16. Maguire LK, Clarke M. How much do you need: a randomised experiment of whether readers can understand the key messages from summaries of Cochrane Reviews without reading the full review. Journal of the Royal Society of Medicine2014;107(11):444–9. doi.org/10.1177/0141076814546710
  17. Rosenbaum SE, Glenton C, Nylund HK, Oxman AD. User testing and stakeholder feedback contributed to the development of understandable and useful Summary of Findings tables for Cochrane reviews. Journal of Clinical Epidemiology2010;63:607–19. doi.org/10.1016/j.jclinepi.2009.12.013
  18. Beynon R, Leeflang MM, McDonald S, Eisinga A, Mitchell RL, Whiting P, et al. Search strategies to identify diagnostic accuracy studies in MEDLINE and EMBASE. Cochrane Database of Systematic Reviews2013;(9):MR000022. doi.org/10.1002/14651858.MR000022.pub3
  19. Hopewell S, Clarke MJ, Lefebvre C, Scherer RW. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database of Systematic Reviews2007;(2):MR000001. doi.org/10.1002/14651858.MR000001.pub2
  20. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews2011;(8):MR000026. doi.org/10.1002/14651858.MR000026.pub2
  21. Young T, Hopewell S. Methods for obtaining unpublished data. Cochrane Database of Systematic Reviews2011;(11):MR000027. doi.org/10.1002/14651858.MR000027.pub2
  22. Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, et al. Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions. Cochrane Database of Systematic Reviews2014;10:MR000035. doi.org/10.1002/14651858.MR000035.pub2
  23. Anonymous. Education section – studies within a review (SWAR). Journal of Evidence‐based Medicine2012;5(3):188–9. doi.org/10.1111/j.1756‐5391.2012.01193.x
  24. McKenzie JE, Salanti G, Lewis SC, Altman DG. Meta‐analysis and The Cochrane Collaboration: 20 years of the Cochrane Statistical Methods Group. Systematic Reviews2013;2(1):80. doi.org/10.1186/2046‐4053‐2‐80