Question about publication bias
Hello,
For background, I feel that I fully understand the concept of publication bias; the sources that go into it and how to detect and measure it. The issue I have is how to think about publication bias in the following situation:
Let’s say I have 20 studies collected from a research group. These 20 studies are all published and the raw data is available. After extracting the effect sizes (Cohen’s D in this case) and the standard error of them and test for publication bias I find evidence that there it is indeed likely that there is publication bias. But what if I have all the raw data for the studies and pool them (i.e., something akin to a within-lab individual participant data meta-analysis). So, instead of having 20 independent t-tests I now compile the data and do one t-test of the full dataset. Would it then matter that there is publication bias when looking at the effect sizes for each of the studies separately? In other words, would this compilation of data be somewhat of a “cure”, or would the data still be contaminated by publication bias?
Note here that I am only interested in evaluating the effect within this research group, so I am not interested in whether other researcher’s “similar” studies could provide help provide a more balanced view.
Thank you in advance,
Sincerely,
Philip Millroth
Comments
I think that if there's publication bias then the effect sizes in the published studies are inflated relative to to the effect sizes for all studies, including published and unpublished. Therefore, if you pool the data from the biased, published studies, the pooled effect size will still be inflated.
R