http://orcid.org/0000-0003-4912-8307
Figures
Abstract
Background
The recommendations of experts who write review articles are a critical determinant of the adaptation of new treatments by clinicians. Several types of reviews exist (narrative, systematic, meta-analytic), and some of these are more vulnerable to researcher bias than others. Recently, the interest in nutritional interventions in psychiatry has increased and many experts, who are often active researchers on this topic, have come to strong conclusions about the benefits of a healthy diet on depression. In a young and active field of study, we aimed to investigate whether the strength of an author’s conclusion is associated with the type of review article they wrote.
Methods
Systematic searches were performed in PubMed, Web of Science, Cochrane Database of Systematic Reviews, and Google Scholar for narrative reviews and systematic reviews with and without meta-analyses on the effects of diet on depression (final search date: May 30th, 2020). Conclusions were extracted from the abstract and discussion section and rated as strong, moderate, or weak by independent raters who were blind to study type. A benchmark on legitimate conclusion strength was based on a GRADE assessment of the highest level of evidence. This systematic review was registered with PROSPERO, number CRD42020141372.
Findings
24 narrative reviews, 12 systematic reviews, and 14 meta-analyses were included. In the abstract, 33% of narrative reviews and 8% of systematic reviews came to strong conclusions, whereas no meta-analysis did. Narrative reviews were 8.94 (95% CI: 2.17, 36.84) times more likely to report stronger conclusions in the abstract than systematic reviews with and without meta-analyses. These findings were similar for conclusions in the discussion section. Narrative reviews used 45.6% fewer input studies and were more likely to be written by authors with potential conflicts of interest. A study limitation is the subjective nature of the conclusion classification system despite high inter-rater agreements and its confirmation outside of the review team.
Conclusions
We have shown that narrative reviews come to stronger conclusions about the benefits of a healthy diet on depression despite inconclusive evidence. This finding empirically underscores the importance of a systematic method for summarizing the evidence of a field of study. Journal editors may want to reconsider publishing narrative reviews before meta-analytic reviews are available.
Citation: Thomas-Odenthal F, Molero P, van der Does W, Molendijk M (2020) Impact of review method on the conclusions of clinical reviews: A systematic review on dietary interventions in depression as a case in point. PLoS ONE 15(9): e0238131. https://doi.org/10.1371/journal.pone.0238131
Editor: Lisa Susan Wieland, University of Maryland School of Medicine, UNITED STATES
Received: April 10, 2020; Accepted: August 9, 2020; Published: September 16, 2020
Copyright: © 2020 Thomas-Odenthal et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: Without relevance to this work, P. Molero reports to have received research grants from the Ministry of Education (Spain), the Government of Navarra (Spain), the Spanish Foundation of Psychiatry and Mental Health and AstraZeneca; he is a clinical consultant for MedAvanteProPhase and has received lecture honoraria from or has been a consultant for AB-Biotics, Janssen, Novumed, Roland Berger, and Scienta. This does not alter our adherence to PLOS ONE policies on sharing data and materials. The other authors declare no competing interests.
Introduction
New treatments do not always deliver on their promise [1]. If a new treatment that looks promising is widely adopted, the long-term effects are often less than expected from the initial evaluations [2]. This process usually takes many years, sometimes decades to complete, and it affects pharmacotherapy as well as psychotherapy [3]. Recent years have seen a marked increase of research on the effects of diet on depression. The broader field of nutrition and psychopathology has been labeled ‘nutritional psychiatry’ [4]. To date, a sizable amount of research exists that investigates the effects of a diet on the treatment and prevention of depression [5,6].
Treatment recommendations and guidelines should optimally rely on the evidence from multiple randomized controlled trials (RCTs), particularly RCTs that are large and well-conducted [7]. Meta-analyses and systematic reviews of RCT data provide the highest level of evidence as they synthesize and evaluate the available evidence with high standards of conduct and reporting before coming to an informed decision [8,9]. An advantage of meta-analyses over systematic reviews is that they statistically pool the available evidence and assess publication bias and between-study heterogeneity [10]. To date, one systematic review and one meta-analysis exist on the effects of dietary interventions on depressive symptoms as assessed through RCTs. The systematic review [11] presented mixed results with almost half of the studies showing a null-effect. The meta-analysis [12] showed a small positive effect of dietary interventions on depressive symptoms, but this effect is difficult to interpret because of the presence of publication bias and heterogeneity among study outcomes.
The RCTs included in the aforementioned systematic review [11] and meta-analysis [12] are typically of short duration (e.g., 8–12 weeks), and have problems with statistical power, blinding, expectation bias, and attrition [13–15]. In the event proper evidence from RCTs is absent, well-conducted prospective cohort studies may serve to provide the best available evidence [9,16,17]. The available meta-analyses of cohort studies report statistical associations between a healthy diet–in particular, the Mediterranean diet–and the incidence of depression over time [5,18]. However, reversed causation, undetected biases, and residual confounding may underlie such relationships [14,15,18,19]. Hence, these study limitations preclude strong conclusions.
Notwithstanding the limited evidence, many authors of narrative reviews come to firm conclusions about the effects of a diet on depression (e.g.: “Studies have shown that diet and nutrition play a significant role in the prevention and clinical treatment of depression” (page 10) [20]). On the contrary, authors of systematic reviews seem to come to less firm conclusions (e.g., "The results of this meta-analysis suggest that healthy pattern may decrease the risk of depression, whereas western-style may increase the risk of depression. However, more randomized controlled trails and cohort studies are urgently required to confirm this finding" (page 373) [6]). So, the clinical implementation of diet to treat or prevent depression may depend on which conclusions and recommendations are adopted by clinicians [2].
We aim to investigate whether research methods that are more sensitive to researcher bias, like narrative reviews, are more likely to overstate the benefits of a treatment–in this case, a healthy diet for depression–than research methods that are less sensitive to researcher bias, like systematic reviews with and without meta-analyses. We hypothesize that narrative reviews report more positive conclusions and recommendations about the benefits of a healthy diet on depression than systematic reviews with and without meta-analyses because narrative reviews lack the systematic method for searching and evaluating the evidence. We also hypothesize that systematic reviews without meta-analyses report more positive conclusions and recommendations than systematic reviews with meta-analyses because systematic reviews without meta-analyses do not evaluate the evidence statistically. In case different review types indeed come to different conclusions, we will explore the number of input papers, various indicators of impact, and potential conflicts of interest as possible explanations for the existence of this relationship.
Method
As a guideline for conducting and reporting this systematic review, we followed the PRISMA [21] statement (see S1 Checklist). A protocol for this review is registered at PROSPERO, number CRD42020141372 (date of registration: April 28th, 2020).
Search strategy
We systemically searched for meta-analyses, systematic reviews, and narrative reviews in the electronic medical databases PubMed (1964–2020), Web of Science (1974–2020), and Cochrane Database of Systematic Reviews (2005–2019) as well as the preregistration platforms PROSPERO and OSF (a free, open source web application) and the preprint servers OSF and BioRxiv from inception to May 30th, 2020. We also performed a non-systematic search in Google Scholar to identify articles that were not captured by our main search. We used the following search terms: (diet OR food) AND ("depressive*" OR depression OR "mental health" OR "mental disorder*") AND ("systematic review" OR meta-analysis OR review). We also screened the reference lists of the included articles for eligible articles. The complete search strategy is presented in S1 Text.
Inclusion criteria
We retained peer-reviewed narrative/literature reviews, perspectives/(expert) opinions, systematic reviews, and meta-analyses on the potential effects of dietary patterns (e.g., healthy or unhealthy diet) or food groups (e.g., fruits, vegetables, or fish) on depression [22,23] and/or depressive symptoms (as measured by self-report depression symptom scales). Articles must have been written in English, German, Dutch, Spanish, or French to be included.
Study selection
Two members of the review team (FT-O and MM) independently screened titles and abstracts of each study for eligibility. In a next round of selection, the full-text of articles was assessed for eligibility. Disagreement about the selection was resolved through consensus.
Data extraction
FT-O and MM independently extracted the data using a prior designed extraction form. The extraction form was pilot-tested and refined accordingly. From each article, we extracted data on: a) publication date; b) number of studies included; b) whether it was a narrative review, systematic review, or meta-analysis; c) conclusions and recommendations; d) the effect sizes from meta-analyses (i.e., odds ratios, hazard ratios, relative risks, and their respective 95% confidence intervals); e) the funding sources; f) the number of input papers; g) indicators of impact (i.e., Altmetric score, impact factor of journal, number of citations); h) whether studies were written by authors with financial conflicts of interest, that is, whether authors reported to have received funding by food industry companies (e.g., Woolworth, Nestle, Taki Maki) or meat or dairy research or marketing companies (e.g., Meat and Livestock Australia); and i) whether studies were written by authors with allegiance bias. We operationalized allegiance bias as being a member of the International Society for Nutritional Psychiatry Research (ISNPR) because this society has recently published very strong conclusions about the potential role of diets on the treatment and prevention of depression [24].
From the systematic reviews with and without meta-analyses, we further extracted data on a) participant characteristics (e.g., country, total number of participants), b) method for assessing dietary patterns or food groups (e.g., diet quality scores or indexes), c) method for assessing dietary intake (e.g., food frequency questionnaires or 24-h dietary recall, food record), d) comparators (e.g., different diets or relative use), e) type of outcome (e.g., diagnosis or depressive symptoms), f) study design, and g) length of follow-up (for systematic reviews and meta-analyses of RCTs and longitudinal studies).
Conclusion and recommendation classification
Conclusions were defined as an overall summary or interpretation of the main findings, and recommendations as an endorsement to treat or prevent depression through a diet. Conclusions and recommendations were extracted from both the abstract and discussion section. For recording the conclusions and recommendations and to reduce bias in this, we adapted the method by Antman et al [2]. One author collected all the conclusions and recommendations, while another author classified the conclusions and recommendations blind to study type into suitable categories (in this case: strong, moderate, or weak). After this, the first author categorized all the conclusions and recommendations accordingly using this classification system.
We identified strong conclusions through keywords, like “compelling support” or “key modifiable targets” and when authors claimed the existence of a causal relationship between diet and depression. An example of a strong conclusion would be “diet and nutrition are central determinants of mental health”. Weak conclusions do not mention the existence of a causal relationship but, at a maximum, an association. These conclusions often come together with contrastive statements, like “however, further research is needed to establish this relationship”. Keywords for weak conclusions are “suggest” or “may/might be”. A moderate conclusion is a mix of strong and weak keywords, or weak keywords in combination with causal statements, like “diet appears to confer some protection” or “diet may prevent”. A strong conclusion together with a contrastive statement also resembles a moderate conclusion. Inter-rater reliability was assessed over random selections of conclusions within the author team as well as two neutral investigators blind to study type to further reduce potential bias. Disagreement was discussed and resolved through consensus.
Quality assessment
We assessed the methodological quality of the included systematic reviews and meta-analyses with the AMSTAR II [25] tool. Based on each item of the quality assessment, we assessed an overall inter-rater agreement. Furthermore, we graded the certainty of the evidence in this field using the GRADE [26] approach. The certainty of the evidence reflects the extent to which we are confident that the estimate of the effect is correct–in this case, the effect of a diet on depression–and can be classified as very low, low, moderate, and high [7]. We did this to obtain a benchmark for the strength of conclusions regarding the association between diet and depressive symptoms as well as the prevention and treatment of depression through a healthy diet. The GRADE assessment was based on a meta-analysis on results derived from RCTs–the highest level of evidence. An evidence profile was generated [27].
Statistical analyses
Agreement regarding the classification of conclusions was assessed through Cohen's Kappa (κ) and rank-correlation coefficients calculated over random samples of ten randomly chosen conclusions ordered from the weakest to the strongest conclusion. Ordinal regression analyses were run to test whether narrative reviews reported stronger conclusions than systematic reviews or meta-analyses. In a similar manner, associations between study type or strength of conclusions and methodological quality were assessed. In post-hoc analyses, the potential associations between study types or strength of conclusions and the number of input studies, number of citations, journal impact factor, Altmetric scores, ISNPR membership, and food industry funding were explored. To assess potential differences regarding the number of input studies and indicators of impact among the study types, independent t-tests were performed, or Mann-Whitney U exact tests in case the normality assumption was violated. Ordinal regression analyses were also conducted to assess putative associations between potential financial or non-financial conflicts of interest and study types or strength of conclusions. The significance level was set at an α level of 0.05, one- or two-tailed, depending on whether we tested a hypothesis or not. Odds ratios (ORs) and their respective 95% confidence intervals (CIs) were used as the measure for effect size. Analyses were performed in SPSS version 25 [28].
Results
Our initial search for meta-analyses, systematic reviews, and narrative reviews yielded 1,868 records after duplicates were removed. After reading the titles and abstracts, we excluded 1,787 records. Another 31 records were excluded after applying the inclusion and exclusion criteria (see S1 Table for the reasons for exclusion per study). Hence, we included a total number of 50 records, among which were 14 meta-analyses, 12 systematic reviews, and 24 narrative reviews/expert opinions (see Fig 1 for a flow-chart).
All systematic reviews and meta-analyses assessed the effects of a diet on depression in the healthy adult population [5,6,11,18,29–48] except for two systematic reviews that assessed children and adolescents [49,50]. The average number of participants included per systematic review with and without meta-analyses was 128,271. In 84% of the cases, the study was performed in a Western country. The reviews mostly included prospective cohort, cross-sectional, or case-control studies, whereas five reviews investigated both observational studies and RCT [31,38–41], and one reviewed RCTs alone [11]. Most reviews included studies investigated the impact of diets, such as Mediterranean, healthy diets, or unhealthy diets. Four studies investigated the effects of fish consumption [29,33,45,46] and another four studies the effects of fruit and vegetable intake [40,42,47,48]. Food intake of participants was measured with food-frequency questionnaires (FFQs), 24-h dietary recalls, diet history questionnaires, or other standardized or non-standardized food intake questionnaires. Depression outcome was measured with standardized self-report depression scales, such as the Beck Depression Inventory or Center for Epidemiological Studies–Depression, formal diagnoses, or antidepressant medication intake. All systematic reviews and meta-analyses found positive effects of healthy diets or food groups on depression in observational studies, or negative effects of unhealthy diets or food groups. The only included systematic review of RCTs concluded that only half of the included RCTs showed significant effects for healthy diets on depression in the treatment, relative to the control group, while the other half reported null effects [11]. A full overview of the basic characteristics and results of the included meta-analyses and systematic reviews can be found in S2 Table.
We categorized the conclusions and recommendations into “strong”, “moderate”, and “weak” (see S3 Table) with high inter-rater agreement (average κ = 0.67, SE = 0.10, P < 0.0001; from four rater pairs on ten different randomly chosen papers [κ‘s per rater pair were 0.55, 0.63, 0.69, and 0.70]). Strong and significant average rank-correlation coefficients calculated over random samples of ten randomly chosen conclusions, ordered from the weakest to the strongest conclusion by four reviewers, further validated the reliability of our conclusion categorization (Kendall’s Tau = 0.72, P < 0.0001 [Tau’s per rater pair were 0.69, 0.78, 0.81, and 0.82]). Note that in all instances, conclusions were assessed blind to the articles from which they were derived. Furthermore, in case there was a discrepancy between assessors, these always involved differences between neighboring classifications (e.g., weak vs. moderate and never weak vs. strong).
Tables 1 and 2 show the conclusion classification by study type of the abstract and discussion section, respectively. In the abstract, narrative reviews were 8.94 (95% CI: 2.17, 36.84) times more likely to report stronger conclusions than meta-analyses and systematic reviews, and systematic reviews were 3.43 (95% CI: 0.66 17.85) times more likely to report stronger conclusions than meta-analyses (P = 0.001, see Table 1). In the discussion, narrative reviews were 3.01 (95% CI: 0.95, 9.58) times more likely to report stronger conclusions than meta-analyses and systematic reviews, and systematic reviews were 2.06 (95% CI: 0.39, 10.83) times more likely to report stronger conclusions than meta-analyses (P = 0.048, see Table 2). After removing expert opinion papers (n = 3) from the analysis, these associations remained the same. Similarly, the associations did not change when Rahe et al [36] was treated as a systematic review instead of a meta-analysis as this study only assessed heterogeneity statistically and showed individual but not pooled effect estimates. Sensitivity analyses also showed that these patterns of results were not due to a particular study (data not shown).
Narrative reviews also appeared to report stronger recommendations in both the abstracts and discussion sections but these associations were not statistically significant (ORabstract = 1.67, 95% CI: 0.29, 9.69, P = 0.569, see Table 3; ORdiscussion = 4.53, 95% CI: 0.98, 20.96, P = 0.053, Table 4). After including the reviews into these analyses that provided no recommendations, these association became significant (ORabstract = 3.28, 95% CI: 1.28, 8.42, P = 0.014; ORdiscussion = 3.78, 95% CI: 1.72, 8.32, P = 0.001).
Inter-rater agreement regarding the AMSTAR assessment of the methodological quality of the included meta-analyses and systematic reviews was high (κ = 0.91, SE = 0.02 [~83% agreement], P < 0.0001; see S4 Table for details). Overall, most systematic reviews and meta-analyses were of critically low quality, except for two meta-analyses that were of low quality and one meta-analysis of moderate quality. Associations between methodological quality and conclusions and recommendations were not calculated due to a lack of variance in the quality among the study types. A more lenient quality assessment, for instance, by excluding items assessing whether studies were preregistered (item 2), whether they had language restrictions (item 4), or whether they included a list of excluded, but potentially relevant studies (item 7) did not increase variation in methodological quality to such an extent that analyses with study types were feasible.
ISNPR members were more likely to have written narrative reviews (OR = 5.10, 95% CI: 1.37, 19.05, P = 0.015) and to have reported stronger conclusions (ORabstract = 4.50, 95% CI: 1.22, 16.55, P = 0.024; ORdiscussion = 4.03, 95% CI: 1.10, 14.71, P = 0.035) and recommendations (ORabstract = 18.00, 95% CI: 1.27, 255.74, P = 0.033) relative to non-members (see S5 Table). ISNPR membership was also associated with a larger likelihood of having potential financial interests (P < 0.01, see Table N in S5 Table). Furthermore, narrative reviews used 45.6% fewer primary input papers (mean = 11.08, standard deviation (SD) = 6.98) than systematic reviews and meta-analyses (mean = 20.35, SD = 12.18; P = 0.002). No statistically significant associations existed between industry funding and study types (OR = 2.43, 95% CI: 0.54, 10.87, P = 0.246) or strength of conclusions (ORabstract = 3.01, 95% CI: 0.65, 13.95, P = 0.158; ORdiscussion = 3.60, 95% CI: 0.74, 17.48, P = 0.112). There were no significant differences in number of citations, Altmetric scores, and journal impact factors as a function of study type.
All associations remained similar after controlling for publication year, number of input papers, or journal impact factor.
GRADE assessment of meta-analysis of RCTs
The GRADE approach was applied to obtain a benchmark for the strength of conclusions regarding the association between diet and depressive symptoms. Initially, we wanted to apply the GRADE assessment on Firth et al [12] as they present the only meta-analysis of studies reporting on the effects of dietary interventions on depressive symptoms derived by means of RCTs. However, we noted two crucial errors in the article that, when corrected, would lead to a substantially different result. An erratum for this study has already been published [51]; however, we think the meta-analysis has not been corrected to a sufficient degree. We, therefore, decided to rerun this meta-analysis following the exact same approach of Firth et al [12] to obtain corrected results as well as the input material for the GRADE assessment: One notable difference relative to Firth et al [12] is that we have applied an extended final search date (August 3rd, 2019 versus December 3rd, 2018). Another difference is that we did not pool all the data in one analysis. Instead, we analyzed the data separately to answer the following questions: 1) can a healthy diet prevent depression? 2) can a healthy diet treat depression? and 3) is a healthy diet associated with a reduction in depressive symptoms over time? This was done to reduce between-study heterogeneity and because of the GRADE requirement that questions are formulated regarding specific populations, interventions, comparators, and outcomes.
The meta-analyses revealed no evidence for the hypotheses that a diet can treat or prevent depression. A small statistically significant benefit of a healthy diet on depressive symptoms was found in association studies that did not specifically aim to prevent or treat depression (see Table 5). Yet, substantial and significant between-study heterogeneity (I2 = 49%) was observed. Cumulative meta-analysis also showed that in 2000, 2005, 2010, and 2015 there was no ground to formulate strong conclusions regarding an effect of a diet on depression. For further information on this meta-analysis, we refer to S2 Text.
A full overview of the GRADE evaluation and the reasons for up- or downgrading the evidence can be found in S1 Appendix. In sum, GRADE indicated very low to low certainty-evidence for the proposed associations between diet and depression that are under study here (see Table 5). These findings, thus, indicate that strong conclusions about the potential effects of diet on depression are not warranted.
Discussion
This systematic review reports substantial discrepancies in the strength of conclusions reported over study types. Narrative reviews were more likely to report stronger conclusions and recommendations regarding the benefits of a healthy diet on depression relative to systematic reviews and meta-analyses, whereas systematic reviews were slightly more likely to report stronger conclusions and recommendations than meta-analyses. In fact, no single meta-analysis came to a strong conclusion regarding the supposed effect of diet on depression. In line with this was the result of a GRADE evaluation of the highest level of evidence, which dictated that the certainty of the evidence is low regarding the prevention, and very low regarding the treatment of depression through a healthy diet as well as the association between diet and depressive symptoms over time. An AMSTAR assessment revealed that the methodological quality of meta-analyses and systematic reviews was mainly critically low. Hence, we can conclude that a substantial part of narrative reviews, and a minor part of systematic reviews, overstate the benefits of a healthy diet on depression.
Although we can only speculate on explanations underlying the biased conclusion formulation, we did find some informative correlates of it. Narrative reviews used 45.6% fewer input studies. This finding could indicate that authors selectively cited easily accessible or mainly positive input material [52]. Selective citation practices could be justified if authors who came to strong conclusions would cite the scientifically strongest articles only. Yet, this seems unlikely since our GRADE assessment showed that no such strong articles exist because of the presence of serious risk of bias in the study designs and executions. Additionally, our meta-analyses of RCTs indicated that there is no evidence for the notion that a diet can treat depression or prevent it from occurring. Financial and non-financial conflicts of interest may also affect narrative reviews more than other types of reviews. ISNPR members wrote 30% of all papers, 45.8% of narrative reviews, and only two meta-analyses [5,12] on this topic. One of these meta-analyses [5] was included in this systematic review, while the other [12] was excluded because of crucial errors favoring the authors' study hypotheses. As in all areas of health research, many factors may underlie the formulation of biased conclusions, ranging from a drive to make a positive contribution to personal experiences to financial interests [53–56].
Food industry-funded studies may also more likely to report stronger conclusions than non-funded studies. The effect size of this association was large (OR = 3.60), but it did not reach statistical significance. Authors tend to underreport their financial conflicts of interest [57], but we do not know if this is also the case in nutritional mental health research. Yet, a recent debate about a new dietary recommendation on red meat consumption disclosed that financial conflicts of interest may indeed play a role also in the nutritional field [58].
Authors of narrative reviews tend to report stronger conclusions in the abstracts than in the discussion sections among the study types. Word limits of abstracts may force authors to generate more generic conclusions [59]. As most readers may only read the title and abstract, we think the abstract should already convey the right message to begin with. When scientific experts conclude that “diet and nutrition are central determinants of mental health” and that “nutrition is a crucial factor in the high prevalence and incidence of mental disorders” (page 271) [24], the data that underlie these conclusions should be convincing. This is not yet the case. As a consequence, the general media have disseminated grossly overstated conclusions [60,61]. A healthy diet has few downsides, but patients may inaccurately believe that they are themselves responsible for their depression (e.g., “My bad dietary habits made me depressed”).
A strength of this systematic review is that we used a transparent and quantitative method for identifying study limitations and their potential sources of bias. We applied a systematic method for searching and extracting relevant data with high inter-rater agreements. We used rigorous, scientific, and standardized instruments to assess the certainty of the evidence (GRADE) and the methodological quality (AMSTAR) to gauge which conclusions are most likely correct. Furthermore, we obtained the highest level of evidence regarding the effects of diet on depression and reduced between-study heterogeneity by pooling, through a meta-analysis, results from RCTs based on their populations, interventions, comparators, and outcomes (i.e., prevention, treatment, association), which increases the validity of our findings [62].
A limitation of our work is the subjective nature of our conclusion classification system, although this was done with high inter-rater agreements, which was confirmed outside of the review team. Secondly, the GRADE approach may be less applicable for lifestyle interventions as the evidence from large and well-conducted RCTs is often absent; more lenient rules for appraising the evidence have been suggested (e.g., HEALM [63]). Thirdly, the present research is limited to the field of diet and depression but similar inferences may apply to different exposures, like nutraceuticals, and different outcomes, like cardiovascular health (our GRADE evaluation of other patient-relevant health outcomes already indicated low to very low certainty-evidence also in these fields of study; see S1 Appendix). Lastly, there may be sub-samples in the population for whom diet may directly affect mood (e.g., people with celiac disease) [64]. However, this was hardly, if ever, acknowledged in the generic conclusions that we encountered and was, therefore, also not investigated here. Future research should investigate all this and invest in large-scale and long-term randomized controlled dietary intervention and prevention trials.
Our work indicates that conclusions derived from systematic reviews with meta-analyses should be favored over those derived from less rigorous scientific methods, like narrative reviews. Awareness of this should be high on the agenda of journal editors and reviewers to reconsider publishing narrative reviews before meta-analytic reviews are available. Our work may also encourage researchers to use systematic reviews instead of narrative reviews to protect themselves against their own biases. The preregistration and open access of such work may further reduce these researcher biases.
Supporting information
S2 Text. Details on our meta-analysis of RCTs.
https://doi.org/10.1371/journal.pone.0238131.s003
(DOCX)
S2 Table. Basic characteristics and results of assessed systematic reviews.
https://doi.org/10.1371/journal.pone.0238131.s005
(DOCX)
S3 Table. Reported conclusions and recommendations of included systematic reviews with classified strength.
https://doi.org/10.1371/journal.pone.0238131.s006
(DOCX)
S5 Table. Association between study types and strength of conclusion, number of input papers, indicators of impact, and potential conflicts of interest.
https://doi.org/10.1371/journal.pone.0238131.s008
(DOCX)
S1 Appendix. GRADE evaluation of three patient-relevant healthcare questions.
https://doi.org/10.1371/journal.pone.0238131.s009
(DOCX)
Acknowledgments
We thank our colleagues from the Universities of Leiden and Navarra for the discussions about the topic of this manuscript on formal and informal occasions. We also want to thank Anouk Mentink, Cristina Vidal Adroher, Liv Caro Henrich, Marta Santos Burguete, and Mirjam Christina Reidick for their help in classifying conclusions and/or proof-reading of earlier versions of his manuscript.
References
- 1. Fanelli D, Ioannidis JPA. US studies may overestimate effect sizes in softer research. Proc Natl Acad Sci U S A. 2013;110: 15031–15036. pmid:23980165
- 2. Antman EM. A Comparison of Results of Meta-analyses of Randomized Control Trials and Recommendations of Clinical Experts. JAMA. 1992;268: 240. pmid:1535110
- 3. Fanelli D. “Positive” results increase down the hierarchy of the sciences. PLoS One. 2010;5.
- 4. Sarris J. Nutritional Psychiatry: From Concept to the Clinic. Drugs. 2019;79: 929–934. pmid:31114975
- 5. Lassale C, Batty GD, Baghdadli A, Jacka F, Sánchez-Villegas A, Kivimäki M, et al. Healthy dietary indices and risk of depressive outcomes: a systematic review and meta-analysis of observational studies. Mol Psychiatry. 2019;24: 965–986. pmid:30254236
- 6. Li Y, Lv M-R, Wei Y-J, Sun L, Zhang J-X, Zhang H-G, et al. Dietary patterns and depression risk: A meta-analysis. Psychiatry Res. 2017;253: 373–382. pmid:28431261
- 7. Zhang Y, Akl EA, Schünemann HJ. Using systematic reviews in guideline development: The GRADE approach. Res Synth Methods. 2019;10: 312–329.
- 8.
UC Library Guides. Evidence-Based Practice in Health: Hierarchy of Evidence. 2019 [cited 6 Sep 2019]. Available: https://canberra.libguides.com/c.php?g=599346&p=4149721
- 9.
National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for developers of guidelines. NHMRC. 2009 [cited 17 Sep 2019]. Available: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=AEFFDA62A5245D6D07F060B56789ED5A?doi=10.1.1.177.4984&rep=rep1&type=pdf
- 10. Crowther M, Lim W, Crowther MA. Systematic review and meta-analysis methodology. Blood. 2010;116: 3140–3146. pmid:20656933
- 11. Opie RS, O’Neil A, Itsiopoulos C, Jacka FN. The impact of whole-of-diet interventions on depression and anxiety: A systematic review of randomised controlled trials. Public Health Nutr. 2015;18: 2074–2093. pmid:25465596
- 12. Firth J, Marx W, Dash S, Carney R, Teasdale SB, Solmi M, et al. The Effects of Dietary Improvement on Symptoms of Depression and Anxiety: A Meta-Analysis of Randomized Controlled Trials. Psychosom Med. 2019;81: 265–280. pmid:30720698
- 13. Molendijk ML, Fried EI, Van der Does W. The SMILES trial: do undisclosed recruitment practices explain the remarkably large effect? BMC Med. 2018;16: 243. pmid:30591065
- 14. Jacka FN. Nutritional Psychiatry: Where to Next? EBioMedicine. 2017;17: 24–29. pmid:28242200
- 15. Trepanowski JF, Ioannidis JPA. Perspective: Limiting Dependence on Nonrandomized Studies and Improving Randomized Trials in Human Nutrition Research: Why and How. Adv Nutr. 2018;9: 367–377. pmid:30032218
- 16. Temple NJ. How reliable are randomised controlled trials for studying the relationship between diet and disease? A narrative review. Br J Nutr. 2016;116: 381–389. pmid:27267302
- 17. Blumberg J, Heaney RP, Huncharek M, Scholl T, Stampfer M, Vieth R, et al. Evidence-based criteria in the nutritional context. Nutr Rev. 2010;68: 478–484. pmid:20646225
- 18. Molendijk M, Molero P, Ortuño Sánchez-Pedreño F, Van der Does W, Angel Martínez-González M. Diet quality and depression risk: A systematic review and dose-response meta-analysis of prospective studies. J Affect Disord. 2018;226: 346–354. pmid:29031185
- 19. Vandenbroucke JP. Observational Research, Randomised Trials, and Two Views of Medical Science. PLoS Med. 2008;5: e67. pmid:18336067
- 20. Huang Q, Liu H, Suzuki K, Ma S, Liu C. Linking What We Eat to Our Mood: A Review of Diet, Dietary Antioxidants, and Depression. Antioxidants. 2019;8: 376.
- 21. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6: e1000097. pmid:19621072
- 22.
American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 4th ed. Washington, DC: Author; 2013.
- 23.
World Health Organization. International classification of diseases for mortality and morbidity statistics. 11th ed. 2018. Available: https://icd.who.int/browse11/l-m/en
- 24. Sarris J, Logan AC, Akbaraly TN, Amminger GP, Balanzá-Martínez V, Freeman MP, et al. Nutritional medicine as mainstream in psychiatry. The Lancet Psychiatry. 2015;2: 271–274. pmid:26359904
- 25. Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358: j4008. pmid:28935701
- 26. Guyatt GH, Oxman AD, Schünemann HJ, Tugwell P, Knottnerus A. GRADE guidelines: A new series of articles in the Journal of Clinical Epidemiology. J Clin Epidemiol. 2011;64: 380–382. pmid:21185693
- 27.
GRADEpro GDT. GRADEpro Guideline Development Tool [Software]. McMaster University (developed by Evidence Prime, Inc.). 2015. Available: Available from gradepro.org
- 28.
IBM Corp. IBM SPSS Statistics for Macintosh, Version 25.0. Armonk, NY: IBM Corp.; 2017.
- 29. Yang Y, Kim Y, Je Y. Fish consumption and risk of depression: Epidemiological evidence from prospective studies. Asia-Pacific Psychiatry. 2018;10: e12335. pmid:30238628
- 30. Nicolaou M, Colpo M, Vermeulen E. Association of a priori dietary patterns with depressive symptoms: a harmonized meta-analysis of observational studies. Psychol Med. 2019. Available: in press
- 31. Salari-Moghaddam A, Saneei P, Larijani B, Esmaillzadeh A. Glycemic index, glycemic load, and depression: a systematic review and meta-analysis. Eur J Clin Nutr. 2019;73: 356–365. pmid:30054563
- 32. Shafiei F, Salari-Moghaddam A, Larijani B, Esmaillzadeh A. Adherence to the mediterranean diet and risk of depression: A systematic review and updated meta-analysis of observational studies. Nutr Rev. 2019;77: 230–239. pmid:30726966
- 33. Murakami K, Sasaki S. Dietary intake and depressive symptoms: A systematic review of observational studies. Mol Nutr Food Res. 2010;54: 471–488. pmid:19998381
- 34. Quirk SE, Williams LJ, O’Neil A, Pasco JA, Jacka FN, Housden S, et al. The association between diet quality, dietary patterns and depression in adults: a systematic review. BMC Psychiatry. 2013;13: 175. pmid:23802679
- 35. Sanhueza C, Ryan L, Foxcroft DR. Diet and the risk of unipolar depression in adults: Systematic review of cohort studies. J Hum Nutr Diet. 2013;26: 56–70. pmid:23078460
- 36. Rahe C, Unrath M, Berger K. Dietary patterns and the risk of depression in adults: A systematic review of observational studies. Eur J Nutr. 2014;53: 997–1013. pmid:24468939
- 37. Rahimlou M, Morshedzadeh N, Karimi S, Jafarirad S. Association between dietary glycemic index and glycemic load with depression: a systematic review. Eur J Nutr. 2018;57: 2333–2340. pmid:29744611
- 38. Altun A, Brown H, Szoeke C, Goodwill AM. The Mediterranean dietary pattern and depression risk: A systematic review. Neurol Psychiatry Brain Res. 2019;33: 1–10.
- 39. Arab A, Mehrabani S, Moradi S, Amani R. The association between diet and mood: A systematic review of current literature. Psychiatry Res. 2019;271: 428–437. pmid:30537665
- 40. Tuck N-J, Farrow C, Thomas JM. Assessing the effects of vegetable consumption on the psychological health of healthy adults: a systematic review of prospective research. Am J Clin Nutr. 2019;110: 196–211. pmid:31152539
- 41. Ljungberg T, Bondza E, Lethin C. Evidence of the Importance of Dietary Habits Regarding Depressive Symptoms and Depression. Int J Environ Res Public Health. 2020;17: 1616.
- 42. Głąbska D, Guzek D, Groele B, Gutkowska K. Fruit and Vegetable Intake and Mental Health in Adults: A Systematic Review. Nutrients. 2020;12: 115.
- 43. Psaltopoulou T, Sergentanis TN, Panagiotakos DB, Sergentanis IN, Kosti R, Scarmeas N. Mediterranean diet, stroke, cognitive impairment, and depression: A meta-analysis. Ann Neurol. 2013;74: 580–591. pmid:23720230
- 44. Lai JS, Hiles S, Bisquera A, Hure AJ, McEvoy M, Attia J. A systematic review and meta-analysis of dietary patterns and depression in community-dwelling adults. Am J Clin Nutr. 2014;99: 181–97. pmid:24196402
- 45. Li F, Liu X, Zhang D. Fish consumption and risk of depression: A meta-analysis. J Epidemiol Community Health. 2015;70: 299–304. pmid:26359502
- 46. Grosso G, Micek A, Marventano S, Castellano S, Mistretta A, Pajak A, et al. Dietary n-3 PUFA, fish consumption and depression: A systematic review and meta-analysis of observational studies. J Affect Disord. 2016;205: 269–281. pmid:27544316
- 47. Liu X, Yan Y, Li F, Zhang D. Fruit and vegetable consumption and the risk of depression: A meta-analysis. Nutrition. 2016;32: 296–302. pmid:26691768
- 48. Saghafian F, Malmir H, Saneei P, Milajerdi A, Larijani B, Esmaillzadeh A. Fruit and vegetable consumption and risk of depression: Accumulative evidence from an updated systematic review and meta-Analysis of epidemiological studies. Br J Nutr. 2018;119: 1087–1101. pmid:29759102
- 49. Khalid S, Williams CM, Reynolds SA. Is there an association between diet and depression in children and adolescents? A systematic review. Br J Nutr. 2016;116: 2097–2108. pmid:28093091
- 50. O’Neil A, Quirk SE, Housden S, Brennan SL, Williams LJ, Pasco JA, et al. Relationship between diet and mental health in children and adolescents: A systematic review. Am J Public Health. 2014;104: e31–e42.
- 51. The Effects of Dietary Improvement on Symptoms of Depression and Anxiety: A Meta-Analysis of Randomized Controlled Trials: Erratum. Psychosom Med. 2020;82. Available: https://journals.lww.com/psychosomaticmedicine/Fulltext/2020/06000/The_Effects_of_Dietary_Improvement_on_Symptoms_of.13.aspx
- 52. Schmidt LM, Gøtzsche PC. Of mites and men: Reference bias in narrative review articles: A systematic review. J Fam Pract. 2005;54: 334–338. pmid:15833223
- 53. Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Med. 2005;2: e124. pmid:16060722
- 54. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. Relationship between Funding Source and Conclusion among Nutrition-Related Scientific Articles. Katan M, editor. PLoS Med. 2007;4: e5. pmid:17214504
- 55. Ioannidis JPA, Trepanowski JF. Disclosures in Nutrition Research. JAMA. 2018;319: 547. pmid:29222543
- 56. Cope MB, Allison DB. White hat bias: examples of its presence in obesity research and a call for renewed commitment to faithfulness in research reporting. Int J Obes. 2010;34: 84–88.
- 57. Forsyth SR, Odierna DH, Krauth D, Bero LA. Conflicts of interest and critiques of the use of systematic reviews in policymaking: an analysis of opinion articles. Syst Rev. 2014;3: 122. pmid:25417178
- 58. Rubin R. Backlash Over Meat Dietary Recommendations Raises Questions About Corporate Ties to Nutrition Scientists. JAMA. 2020;323: 401–404.
- 59. DeJesus JM, Callanan MA, Solis G, Gelman SA. Generic language in scientific communication. Proc Natl Acad Sci. 2019;116: 18370–18377. pmid:31451665
- 60. Fleming A. Nutritional psychiatry: can you eat yourself happier? In: The Guardian [Internet]. 2019 [cited 18 Dec 2019]. Available: https://www.theguardian.com/food/2019/mar/18/can-you-eat-yourself-happier-nutritional-psychiatry-mental-health
- 61. Schiffman R. Can What We Eat Affect How We Feel? In: The New York Times [Internet]. 2019 [cited 18 Dec 2019]. Available: https://www.nytimes.com/2019/03/28/well/eat/food-mood-depression-anxiety-nutrition-psychiatry.html
- 62. Barnard ND, Willett WC, Ding EL. The Misuse of Meta-analysis in Nutrition Research. JAMA. 2017;318: 1435. pmid:28975260
- 63. Katz DL, Karlsen MC, Chung M, Shams-White MM, Green LW, Fielding J, et al. Hierarchies of evidence applied to lifestyle Medicine (HEALM): introduction of a strength-of-evidence approach based on a methodological systematic review. BMC Med Res Methodol. 2019;19: 178. pmid:31429718
- 64. van Hees NJM, Van der Does W, Giltay EJ. Coeliac disease, diet adherence and depressive symptoms. J Psychosom Res. 2013;74: 155–160. pmid:23332531