Defining the question
(1) Is it clear why the central problem is important?
(2) Are there clear conceptual definitions for the key variables, and are the variables available from the primary studies appropriate given the conceptual definitions above?
(3) Are there a sufficient number of studies available for addressing the primary research questions, but not so many that it will be impossible to synthesize their results with a reasonable effort in a reasonable amount of time?
(4) Is it clear which research designs are appropriate for answering the questions?
(5) Have decision rules been developed for inclusion/exclusion of studies?
Collecting the data
(6) Are multiple search strategies and online databases used to find relevant studies?
(7) Are alternative terms for each key variable used in searches?
(8) Is the set of retrieved studies assessed for relevance by consistent application of the a priori decision rules for inclusion/exclusion?
(9) Was extraction of information from included studies done rigorously and reliably?
(10) If different types of studies are included, are they identified by appropriate coding information?
(11) Are the methods used to extract data from studies justifiable, clearly documented, and repeatable?
(12) Have appropriate methods been used for dealing with missing data (e.g., variances, sample sizes)?
Synthesizing the data if a meta-analysis is performed
(13) Is an appropriate effect size metric used? Is this decision clearly explained and justified by the problem and/or data structure? If a nonstandard effect size metric is used, are its statistical properties and distributional assumptions understood, clearly explained, and justified?
(14) Are average effect sizes and confidence intervals reported? Is an appropriate model used for the analyses?
(15) Were the studies weighted by the inverse of the variance? If no weighting or another weighting scheme was used, was it explained and justified acceptably?
(16) Is the degree of heterogeneity of effect sizes examined?
(17) Are critical features of studies tested as potential moderators of study outcomes? If more than one moderator is tested, has that been taken into account appropriately in the analyses? Has the data structure, including possible confounding of moderators, and nonindependence of moderators or outcomes been taken into account?
Testing the robustness of results
(18) Are sensitivity analyses conducted and used in interpreting the results?
(19) Is the possible impact of missing data or overrepresented data in the evidence base considered?
Presenting the findings
(20) Are clear, high quality graphs and tables used to clarify the results?
(21) Were the procedures, results and conclusions of the research synthesis clearly and completely documented, and justified?
Interpreting the results
(22) Are the generalizability and limitations of the findings discussed?
(23) Is a distinction made between study-generated and review-generated evidence?
(24) Are the findings and their interpretation considered in light of their biological, and/or practical significance?
(25) Are major areas identified where more primary studies are needed?