Correcting for multiple tests
Hello,
I want to use multiple Bayesian Mann-Whitney-U tests to compare two experimental groups before the experiment (age and stuff like this, 11 comparisons in total).
How do I correct for multiple comparisons? The test allows me to adjust the priors, however, I have no idea how I should to this.
Has anyone any tips or literature recommendations?
Cheers,
Max
Comments
Hi Max,
For a correction I'd recommend the procedure we use for our post-hoc ANOVA comparisons. Some background literature is cited in here: https://psyarxiv.com/s56mk
Cheers,
E.J.
Hey,
Thanks that was really helpful! Now I know how to adjust my p(H0i), however, I'm a bit lost when it comes to implement this in JASP. It allows me to adjust the distribution of the prior, how do I adjust p(H0i)? Alternatively, how can I calculate the width of the Cauchy to reflect p(H0i)?
Cheers,
Max
JASP gives you the Bayes factor (determined in part by the width of the Cauchy); this is independent of the multiplicity correction. Then, with the BF in hand, you adjust the prior odds; multiplying the BF with the prior odds yields the posterior odds, which is then corrected for multiplicity.
E.J.
So, if I have a BF10 of 1.703 and a p(H0i) = 0.5^{2/11} = 0.88 (Westfall); then I multiply BF with p(H0i), which gives me a BF of 1.5;
This BF10*p(H0i) is my corrected BF that I report and interpret?
Just trying to make sure that I understand correctly :)
Cheers,
Max
BF is a ratio, so the prior probability component needs to be in ratio form too. So you multiply
BF10 with p(H1)/p(H0), which in your case equals (1-.88)/.88 = 0.136, so you get
1.703 * 0.136 = 0.232. This is the posterior odds; converting this to a posterior probability means you have
0.232 / 1.232 = 0.188 for H1, which leaves 0.812 for H0.
E.J.
Ah okay the calculation of the posterior odds makes sense, thanks!
But where does the 1.232 come from? Also, 0.188 and 0.812 are P(M|data) in JASP, aren't they? How do I get from that to BF10?
I think, I understand the calculation now:
P(H1|data) = O(H1)/(O(H1)+O(H0)) = 0.232 / 1.232 = 0.188
P(H0|data) = 1 - P(H1|data) = 0.812
BF10 = P(H1|data) / P(H0|data) = 0.23
Is that correct?
If the odds are O, the associated probability is O / (O+1). JASP gives you the BF, so you would have to multiple with the prior odds yourself.
Cheers,
E.J.
Ah okay that makes sense, thanks E.J.!