Abstract
Although use of the standardized mean difference in meta-analysis is appealing for several reasons, there are some drawbacks. In this article, we focus on the following problem: that a precision-weighted mean of the observed effect sizes results in a biased estimate of the mean standardized mean difference. This bias is due to the fact that the weight given to an observed effect size depends on this observed effect size. In order to eliminate the bias, Hedges and Olkin (1985) proposed using the mean effect size estimate to calculate the weights. In the article, we propose a third alternative for calculating the weights: using empirical Bayes estimates of the effect sizes. In a simulation study, these three approaches are compared. The mean squared error (MSE) is used as the criterion by which to evaluate the resulting estimates of the mean effect size. For a meta-analytic dataset with a small number of studies, theMSE is usually smallest when the ordinary procedure is used, whereas for a moderate or large number of studies, the procedures yielding the best results are the empirical Bayes procedure and the procedure of Hedges and Olkin, respectively.
Article PDF
Similar content being viewed by others
References
Cohen, J. (1969).Statistical power analysis for the behavioral sciences. New York: Academic Press.
DerSimonian, R., &Laird, N. (1986). Meta-analysis in clinical trials.Controlled Clinical Trials,7, 177–188.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research.Educational Researcher,5, 3–8.
Glass, G. V. (1977). Integrating findings: The meta-analysis of research.Review of Research in Education,5, 351–379.
Glass, G. V., MacGaw, B., &Smith, M. L. (1981).Meta-analysis in social research. Beverly Hills, CA: Sage.
Goldstein, H., Rasbash, J., Lewis, I., Draper, D., Browne, W., Yang, M., Woodhouse, G., &Healy, M. (1998).A user’s guide to MLwiN. London: University of London, Multilevel Models Project.
Hedges, L. V. (1981). Distribution theory for Glass’s estimator of effect size and related estimators.Journal of Educational Statistics,6, 107–128.
Hedges, L. V. (1982). Estimation of effect size from a series of independent experiments.Psychological Bulletin,92, 490–499.
Hedges, L. V. (1983). A random effects model for effect sizes.Psychological Bulletin,93, 388–395.
Hedges, L. V., Cooper, H., &Bushman, B. J. (1992). Testing the null hypothesis in meta-analysis: A comparison of combined probability and confidence interval procedures.Psychological Bulletin,111, 188–194.
Hedges, L. V., &Olkin, I. (1985).Statistical methods for meta-analysis. Orlando, FL: Academic Press.
Hedges, L. V., &Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis.Psychological Methods,3, 486–504.
Hunter, J. E., &Schmidt, F. L. (1990).Methods of meta-analysis: Correcting error and bias in research findings. London: Sage.
Hunter, J. E., Schmidt, F. L., &Jackson, G. B. (1982).Meta-analysis: Cumulating findings across research. Beverly Hills, CA: Sage.
Raudenbush, S. W., &Bryk, A. S. (1985). Empirical Bayes metaanalysis.Journal of Educational Statistics,10, 75–98.
Rosenthal, R. (1991).Meta-analytic procedures for social research. Newbury Park, CA: Sage.
Rubin, D. B. (1981). Estimation in parallel randomized experiments.Journal of Educational Statistics,6, 377–400.
Van den Noortgate, W., &Onghena, P. (2003). Multilevel metaanalysis: A comparison with traditional meta-analytical procedures.Educational & Psychological Measurement,63, 765–790.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Van Den Noortgate, W., Onghena, P. Estimating the mean effect size in meta-analysis: Bias, precision, and mean squared error of different weighting methods. Behavior Research Methods, Instruments, & Computers 35, 504–511 (2003). https://doi.org/10.3758/BF03195529
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03195529