Skip to main content
SpringerLink
Log in

Between Black and White: Examining Grey Literature in Meta-analyses of Psychological Research

  • Original Paper
  • Published:
Journal of Child and Family Studies Aims and scope Submit manuscript

Abstract

The inclusion of grey literature in meta-analyses and reviews is controversial. We examine both the advantages and challenges of including grey literature in meta-analyses. An exemplar meta-analysis of behavioral parenting interventions on parent behavior, child behavior, and parent adjustment outcomes is used to demonstrate these issues. It also explores the influence of the inclusion of grey literature on outcomes, including whether effect sizes are affected by the inclusion of grey literature and describes the challenges of searching for grey literature using traditional search engines such as Google and Yahoo. Homogeneity and publication bias are also examined. Based on these results, recommendations are presented for meta-analysts and researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Abby, M., Massey, M. D., Galandiuk, S., & Polk, H. C, Jr. (1994). Peer review is an effective screening process to evaluate medical manuscripts. Journal of the American Medical Association, 272, 105–107. doi:10.1001/jama199403520020031008.

    Article  PubMed  Google Scholar 

  • American Academy of Child and Adolescent Psychiatry (AACAP) (2006) Policy statement on evidence-based practice. Retrieved Sept 10, 2012 from http://aacap.org/cs/root/policy_statements/ evidence_based_practice.

  • American Psychological Association. (APA) (2005). Policy statement on evidence-based practice in psychology. Retrieved from http://www.apa.org/practice/resources/evidence/evidence-based-statement.pdf.

  • American Psychological Association. (APA). (2010). Publication manual of the American Psychological Association. Washington, DC, APA.

  • American Psychological Association, Task force on the future of psychology. (2009). 2009 Presidential Task Force on the Future of Psychology Practice Final Report. Retrieved from http://www.apa.org/ pubs/info/reports/future-practice.pdf.

  • Auger, C. P. (1998). Information sources in grey literature (guides to information sources). (4th ed.). London: Bowker Saur.

    Google Scholar 

  • Banks, M. A. (2010). Blog posts and tweets: The next frontier for grey literature. In D. J. Farace & J. Schopfel (Eds.), Grey literature in library and information studies. Berlin: De Gruyter Saur.

    Google Scholar 

  • Barwick, M., Boydell, K. M., Stasiulis, E., Ferguson, H. B., Blase, K., & Fixsen, D. (2008). Research utilization among children’s mental health providers. Implementation Science, 3, 19. doi:10.1186/1748-5908-3-19.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bellefontaine, S. P., & Lee, C. M. (2013). A Meta-Analytic Examination of Behavioral Parenting Interventions in the Treatment of Children’s Behavior Problems. Manuscript submitted for publication.

  • Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K. (2006). State-of-the-evidence reviews: Advantages and challenges of including grey literature. World views on Evidence-based Nursing, 3, 55–61. doi:10.1111/j17416787200600051x.

    Article  Google Scholar 

  • Butler, A. C., Chapman, J. E., Forman, E. M., & Beck, A. T. (2006). The empirical status of cognitive-behavioral therapy: A review of meta-analyses. Clinical Psychology Review, 26, 17–31. doi:10.1016/jcpr200507003.

    Article  PubMed  Google Scholar 

  • Coad, J., Hardicare, J., & Devitt, P. (2006). Searching for and using grey literature. Nursing Times, 102, 35–36.

    PubMed  Google Scholar 

  • Conn, V. S., Valentine, J. C., Cooper, H. M., & Rantz, M. J. (2003). Grey literature in meta-analyses. Nursing Research, 52, 256–261.

    Article  PubMed  Google Scholar 

  • Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., et al. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. The Journal of the American Medical Association, 269, 2749–2753. doi:10.1001/jama199303500210049030.

    Article  Google Scholar 

  • Council of Editors. (2011). Summary report of journal operations, 2010. American Psychologist, 66, 405–406. doi:10.1037/a0023978.

    Article  Google Scholar 

  • De Graaf, I., Speetjens, P., Smit, F., de Wolff, M., & Tavecchio, L. (2008). Effectiveness of the triple P positive parenting program on parenting: A meta-analysis. Family Relations, 57, 553–566. doi:10.1177/0145445508317134.

    Article  Google Scholar 

  • Decullier, E., Lhéritier, V., & Chapuis, F. (2005). Fate of biomedical research protocols and publication bias in France: Retrospective cohort study. BMJ, 331, 19. DOI: http://dx.doi.org/10.1136/bmj.38488.385995.8F.

  • Dretzke, J., Davenport, C., Few, E., Barlow, J., Stewart-Brown, S., Bayliss, S., et al. (2009). The clinical effectiveness of different parenting programmes for children with conduct problems: A systematic review of randomized controlled trials. Child and Adolescent Psychiatry and Mental Health, 3, 1–10. doi:10.1186/1753-2000-3-7.

    Article  Google Scholar 

  • Dwan, K., Altman, D. G., Arnaiz, J. A., Bloom, J., Chan, A. W., Cronin, E., et al. (2008). Systematic review of the empirical evidence of study publication bias and outcome reporting bias. Public Library of Science ONE, 3, e3081. doi:10.1371/journal.pone.0003081.

    PubMed  PubMed Central  Google Scholar 

  • Emrick, C. D. (1975). A review of psychologically oriented treatment of alcoholism. Journal of Studies on Alcohol, 36, 88–108.

    PubMed  Google Scholar 

  • Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting and Clinical Psychology, 16, 319–324.

    Article  Google Scholar 

  • Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517. doi:10.1037/0003066X335517a.

    Article  Google Scholar 

  • Fletcher, R., Freeman, E., & Matthey, S. (2011). The impact of behavioural parent training on fathers’ parenting: A meta-analysis of the triple p—positive parenting program. Fathering, 9, 291–312. doi:10.3149/fth0903291.

    Article  Google Scholar 

  • Frater, J., Myohanen, L., Taylor, E., & Keith, L. (2007). What would you tell me if I said grey literature? The New York academy of medicine grey literature report. Journal of Electronic Resources in Medical Libraries, 4, 145–153.

    Article  Google Scholar 

  • Furlong, M., McGilloway, S., Bywater, T., Hutchings, J., Smith, S. M., & Donnelly, M. (2012). Behavioural and cognitive-behavioural group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years. Cochrane Database for Systematic Reviews, 2012(2), 1–344. doi:10.1002/14651858CD008225pub2.

    Google Scholar 

  • Gilgun, J. F. (2005). The four cornerstones of evidence-based practice in social work. Research on Social Work Practice, 15, 52–61. doi:10.1177/1049731504269581.

    Article  Google Scholar 

  • Glass, G. V. (1977). Integrating findings: The meta-analysis of research. Review of Research Education, 5, 351–379. doi:10.3102/0091732X005001351.

    Google Scholar 

  • Gokhale, P. (1997). Grey Literature Varieties–Definitional Problems. In Third International Conference on Grey Literature: Perspectives on the Design and Transfer of Scientific and Technical Information, 13–14 November 1997.

  • Hamilton, J. (2005). Clinician’s guide to evidence-based practice. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 494–498.

    Article  PubMed  Google Scholar 

  • Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6, 203–217.

    Article  PubMed  Google Scholar 

  • Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for moderators in meta-analysis. Psychological Methods, 9, 426–445. doi:10.1037/1082-989X94426.

    Article  PubMed  Google Scholar 

  • Hopewell, S., & Clarke, M. (2001). Methodologists and their methods: Do methodologists write up their conference presentations or is it just 15 minutes of fame? International Journal of Technology Assessment in Health Care, 17, 601–603.

    PubMed  Google Scholar 

  • Hopewell, S., McDonald, S., Clarke, M. J., & Egger, M. (2008). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Library, 1–18.

  • Hunsley, J., & Mash, E. J. (2008). Developing criteria for evidence-based assessment: An introduction to assessments that work. In J. Hunsley & E. J. Mash (Eds.), A guide to assessments that work (pp. 3–14). New York: Oxford University Press.

    Google Scholar 

  • Kaminski, J. W., Valle, L. A., Filene, J. H., & Boyle, C. L. (2008). A meta-analytic review of components associated with parent training program effectiveness. Journal of Abnormal Child Psychology, 36, 567–589. doi:10.1007/s1080200792019.

    Article  PubMed  Google Scholar 

  • Levant, R. F. (2006). Making psychology a household word. American Psychologist, 61, 383–395.

    Article  PubMed  Google Scholar 

  • Light, R. J., & Smith, P. V. (1971). Accumulating evidence: Procedures for resolving contradictions among different research studies. Harvard Educational Review, 41, 429–471.

    Google Scholar 

  • Lundahl, B., Risser, H. J., & Lovejoy, M. C. (2006). A meta-analysis of parent training: Moderators and follow-up effects. Clinical Psychology Review, 26, 86–104. doi:10.1016/jcpr200507004.

    Article  PubMed  Google Scholar 

  • Lundahl, B. W., Tollefson, D., Risser, H., & Lovejoy, M. C. (2008). A meta-analysis of father involvement in parent training. Research on Social Work Practice, 18, 97–106. doi:10.1177/1049731507309828.

    Article  Google Scholar 

  • Martin, J. L. R., Perez, V., Sacristan, M., & Alvarez, E. (2005). Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia. European Psychiatry, 20, 550–553. doi:10.1016/jeurpsy200503011.

    Article  PubMed  Google Scholar 

  • Maughan, D. R., Christiansen, E., Jenson, W. R., Olympia, D., & Clark, E. (2005). Behavioral parent training as a treatment for externalizing behaviors and disruptive behavior disorders: A meta-analysis. School Psychology Review, 34, 267–286.

    Google Scholar 

  • Mays, G. P., & Hogg, R. A. (2012). Expanding delivery system research in public health settings: Lessons from practice-based research networks. Journal of Public Health Management Practice, 18, 485–498. doi:10.1097/PHH0b013e31825f75c9.

    Article  PubMed  PubMed Central  Google Scholar 

  • McAuley, L., Pham, B., Tugwell, P., & Moher, D. (2000). Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? The Lancet, 356, 1228–1231.

    Article  Google Scholar 

  • McCart, M. R., Priester, P. E., Davies, W. H., & Azen, R. (2006). Differential effectiveness of behavioral parent- training and cognitive-behavioral therapy for antisocial youth: A meta-analysis. Journal of Abnormal Child Psychology, 34, 527–543. doi:10.1007/s1080200690311.

    Article  PubMed  Google Scholar 

  • McLeod, B. D., & Weisz, J. R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72, 235–251. doi:10.1037/0022-006X722235.

    Article  PubMed  Google Scholar 

  • Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999). Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. The Lancet, 354, 1896–1900. doi:10.1016/S01406736(99)041495.

    Article  Google Scholar 

  • Moher, D., Hopewell, S., Schulz, K. F., Montori, V., Gøtzsche, P. C., Devereaux, P. J., et al. (2010). CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trial. British Medical Journal, 340, c869–c897. doi:10.1136/bmjc869.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nowak, C., & Heinrichs, N. (2008). A comprehensive meta-analysis of triple p—positive parenting program using hierarchical linear modeling: Effectiveness and moderating variables. Clinical Child and Family Psychology Review, 11, 114–144. doi:10.1007/s1056700800330.

    Article  PubMed  Google Scholar 

  • Pappas, C., & Williams, I. (2011). Grey literature: Its emerging importance. Journal of Hospital Librarianship, 11, 228–234. doi:10.1080/15323269.2011.587100.

    Article  Google Scholar 

  • Piquero, A. R., Farrington, D. P., Welsh, B. C., Tremblay, R., & Jennings, W. G. (2009). Effects of early family/parent training programs on antisocial behavior and delinquency. Journal of Experimental Criminology, 5, 83–120. doi:10.1007/s112920099072x.

    Article  Google Scholar 

  • Posner, B. P., & Bellefontaine, S. P. (2011).ReSearch [Computer Software].

  • Provos, N., McClain, J., Wang, K. (2006, November). Search worms. Paper presented at the 4th Workshop on Recurring Malcode, Fairfax, VA. Retrieved from http://www.citi.umich.edu/u/provos/papers/ search_worms.pdf.

  • Reyno, S. M., & McGrath, P. J. (2006). Predictors of parent training efficacy for child externalizing behavior problems—a meta-analytic review. Journal of Child Psychology and Psychiatry, 47, 99–111. doi:10.1111/j.14697610200501544x.

    Article  PubMed  Google Scholar 

  • Roth, B. J. (2010). An academic perspective on grey literature. Archaeologies: Journal of the World Archaeological Congress, 6, 337–345. DOI: 10.1007/s11759-010-9141-9.

  • Royle, J., & Blythe, J. (1998). Promoting research utilisation in nursing: The role of the individual, organisation, and environment. Evidence-Based Nursing, 1, 71–72. doi:10.1136/ebn1371.

    Article  Google Scholar 

  • Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996). Evidence-based medicine: What it is and what it isn’t. British Medical Journal, 312, 71–72. doi:10.1136/bmj312702371.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sacks, H. S., Reitman, D., Pagano, D., & Kupelnick, B. (1996). Meta-analysis: An update. Mount Sinai Journal of Medicine, 63, 216–224.

    PubMed  Google Scholar 

  • Schulz, K. F., Altman, D. G., Mohe, D., & for the CONSORT Group. (2010). CONSORT 2010 Statement: Updated guidelines for reporting parallel group randomised trials. British Medical Journal, 340, 698–702. doi:10.1136/bmjc332.

    Article  Google Scholar 

  • Scullion, P. A. (2002). Effective dissemination strategies. Nurse Researcher, 10, 65–67.

    Article  PubMed  Google Scholar 

  • Serketich, W. J., & Dumas, J. E. (1996). The effectiveness of behavioral parent training to modify antisocial behavior in children: A meta-analysis. Behavior Therapy, 27, 171–186. doi:10.1016/S0005-7894(96)80013X.

    Article  Google Scholar 

  • Seymour, D. J. (2010). Sanctioned inequity and accessibility issues in the grey literature in the United States. Archaeologies: Journal of the World Archaeological Congress, 6, 233–269. DOI: 10.1007/s11759-010-9144-6.

  • Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 36, 753–760.

    Google Scholar 

  • Smith, M., Glass, G., & Miller, T. (1980). The benefits of psychotherapy. Baltimore: John Hopkins University Press.

    Google Scholar 

  • Thomas, R., & Zimmer-Gembeck, M. J. (2007). Behavioral outcomes of parent-child interaction therapy and triple p-positive parenting program: A review and meta-analysis. Journal of Abnormal Child Psychology, 35, 475–495. doi:10.1007/s1080200791049.

    Article  PubMed  Google Scholar 

  • Weisz, J. R., Weiss, B., Alicke, M. D., & Klotz, M. L. (1987). Effectiveness of psychotherapy with children and adolescents: A meta-analysis for clinicians. Journal of Consulting and Clinical Psychology, 55, 542–549. doi:http://www.wjh.harvard.edu/~jweisz/pdfs/1987c.pdf.

    Article  PubMed  Google Scholar 

  • Wilson, D. B., & Lipsey, M. W. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Bulletin, 6, 413–429.

    Google Scholar 

  • Wood, J. (2008). Methodology for dealing with duplicate study effects in meta-analysis. Organizational Research Methods, 11, 79–95. doi:10.1177/1094428106296638.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Psychology, University of Ottawa, Ottawa, ON, K1N 6N5, Canada

    Sarah P. Bellefontaine & Catherine M. Lee

Authors
  1. Sarah P. Bellefontaine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Catherine M. Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sarah P. Bellefontaine.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bellefontaine, S.P., Lee, C.M. Between Black and White: Examining Grey Literature in Meta-analyses of Psychological Research. J Child Fam Stud 23, 1378–1388 (2014). https://doi.org/10.1007/s10826-013-9795-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10826-013-9795-1

Keywords

Search

Navigation