Summer 2008 Program on Meta-analysis: Synthesis and Appraisal of Multiple Sources of Empirical Evidence
Scientific Context
Seldom is there only a single empirical research study relevant to a question of scientific interest. However, both experimental and observational studies have traditionally been analyzed in isolation, without regard for previous similar or other closely related studies. A new research area has arisen to address the location, appraisal, reconstruction, quantification, contrast and possible combination of similar sources of evidence. Variously called meta-analysis, systematic reviewing, research synthesis or evidence synthesis, this new field is gaining popularity in diverse fields including medicine, psychology, epidemiology, education, genetics, ecology and criminology. Statistical methods for combining results across independent studies have long existed, but require renewed consideration, development and wider dissemination by inclusion in the mainstream statistics curriculum. The possibility that the due consideration of all relevant evidence should be accepted as standard practice in statistical analyses deserves investigation.
The combination of results from similar studies is often known simply as 'meta-analysis'. Common examples are combining results of randomized controlled trials of the same intervention in evidence-based medicine; of correlation coefficients for a pair of constructs measured similarly across studies in social science; or of odds ratios measuring association between an exposure and an outcome in epidemiology. More complex syntheses of multiple sources of evidence have developed recently, including combined analyses of clinical trials of different interventions, and combined analysis of data from multiple microarray experiments (sometimes called cross study analysis). For straightforward meta-analyses, general least-squares methods may be used, but for complex meta-analyses, the technical statistical approach is not so obvious. Often likelihood and Bayesian approaches provide very different perspectives; and in practice the possible benefits of more complex approaches may be hard to discern as many meta-analyses are compromised by limited or biased availability of data from studies as well as by varying methodological limitations of the studies themselves.
The presence of multiple sources of evidence has long been a recognized challenge in the development and appraisal of statistical methods — from Laplace and Gauss to Fisher and Lindley. In the 1980s Richard Peto argued that a combined analysis would be more important than the individual analyses, a view taken still further by Greenland who has suggested that that individual study publications should not attempt to draw conclusions at all, but should instead only describe and report results, so that a later meta-analysis can more appropriately assess the study's evidence fully informed by other study designs and results. Will combined analyses actually replace individual analyses (or at least decrease their impact)? If so, it is time to re-examine the perennial problems of statistical inference in this context.
Three Challenges
- To substantiate and clarify how existing statistical methodology can effectively combine multiple sources of evidence, given perfect conduct and reporting of all studies.
- To identify statistical areas in need of development or improvement, both in theory and application, for the practical situations of studies having methodological limitations and studies providing biased or incomplete data.
- To identify and develop material and pedagogy for undergraduate and graduate programs in statistics, to allow future statisticians to deal effectively with multiple sources of evidence, and to motivate further development of new methodology.
This program comprises two weeks of research, mixing tutorials, research presentations and working group activities on the subject. The goal of this program is three-fold: 1) to bring the area to the attention of statistical researchers, whose expertise is critical to substantiate and clarify the necessary statistical theory and methodology; 2) to nurture the necessary interdisciplinary collaboration and communication between statistical researchers and statisticians who currently work or plan to work with basic and applied science researchers and 3) to provide an entry point into the field to interested students and faculty, and to allow researchers already specialized in the domain to exchange recent results and information.
Program Leaders: Keith O'Rourke (Duke University), Joseph Beyene (University of Toronto), Vanja Dukic (University of Chicago), Julian Higgins (UK Medical Research Council, Cambridge), Peter Hoff (University of Washington), Ken Rice (University of Washington) and Dalene Stangl (Duke University).
Description of Activities
Workshop
The meeting consists of five days of conference culminating in the formation of Working Groups, followed by a week of Working Group meetings and research.