#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

# Correcting for multiple tests

Hello,

I want to use multiple Bayesian Mann-Whitney-U tests to compare two experimental groups before the experiment (age and stuff like this, 11 comparisons in total).

How do I correct for multiple comparisons? The test allows me to adjust the priors, however, I have no idea how I should to this.

Has anyone any tips or literature recommendations?

Cheers,

Max

• Hi Max,

For a correction I'd recommend the procedure we use for our post-hoc ANOVA comparisons. Some background literature is cited in here: https://psyarxiv.com/s56mk

Cheers,

E.J.

• edited April 9

Hey,

Thanks that was really helpful! Now I know how to adjust my p(H0i), however, I'm a bit lost when it comes to implement this in JASP. It allows me to adjust the distribution of the prior, how do I adjust p(H0i)? Alternatively, how can I calculate the width of the Cauchy to reflect p(H0i)?

Cheers,

Max

• JASP gives you the Bayes factor (determined in part by the width of the Cauchy); this is independent of the multiplicity correction. Then, with the BF in hand, you adjust the prior odds; multiplying the BF with the prior odds yields the posterior odds, which is then corrected for multiplicity.

E.J.

• So, if I have a BF10 of 1.703 and a p(H0i) = 0.5^{2/11} = 0.88 (Westfall); then I multiply BF with p(H0i), which gives me a BF of 1.5;

This BF10*p(H0i) is my corrected BF that I report and interpret?

Just trying to make sure that I understand correctly :)

Cheers,

Max

• BF is a ratio, so the prior probability component needs to be in ratio form too. So you multiply

BF10 with p(H1)/p(H0), which in your case equals (1-.88)/.88 = 0.136, so you get

1.703 * 0.136 = 0.232. This is the posterior odds; converting this to a posterior probability means you have

0.232 / 1.232 = 0.188 for H1, which leaves 0.812 for H0.

E.J.

• edited April 9

Ah okay the calculation of the posterior odds makes sense, thanks!

But where does the 1.232 come from? Also, 0.188 and 0.812 are P(M|data) in JASP, aren't they? How do I get from that to BF10?

• I think, I understand the calculation now:

P(H1|data) = O(H1)/(O(H1)+O(H0)) = 0.232 / 1.232 = 0.188

P(H0|data) = 1 - P(H1|data) = 0.812

BF10 = P(H1|data) / P(H0|data) = 0.23

Is that correct?

• If the odds are O, the associated probability is O / (O+1). JASP gives you the BF, so you would have to multiple with the prior odds yourself.

Cheers,

E.J.

• Ah okay that makes sense, thanks E.J.!