Correction for multiple testing of post hoc Bayesian t-tests

NilsPixa

Dear JASPER’s,

I have a question regarding correction for multiple testing of post hoc Bayesian t-tests, e.g., in a Bayesian RM-ANOVA:

The output in JASP provides tables of Post Hoc Tests, including values for the Prior Odds, Posterior Odds, BF10,U, and error%. While the Posterior Odds are corrected for multiple testing (following Westfall et al., 1997), the BF is uncorrected as indicated by U. However, am I right to calculate a corrected BF by dividing the (corrected) Posterior Odds by the Prior odds? For instance, I have a Prior Odd of 0.320, and a Posterior Odd of 368.130 and a BF10,U of 1152.177 provided in JASP; calculating a corrected BF10 (by dividing 368.130/0.320) will result in a BF10 of  1.150,406. The new BF10 is slightly lower, but is that a proper way to correct the BF for multiple testing?

Is that right, and can we say than that the BF10 is corrected for multiple testing?

 

I appreciate any help you can provide.

Best

Nils    

EJ

Hi Nils,

It is conceptually cleaner to say that the BF is just the BF -- for an assessment of the evidence does not matter how many hypotheses you are testing. But if you were testing many hypotheses, in a haphazard fashion, this usually indicates that each test by itself involves a low probability H1. So the Bayesian adjustment for multiple testing is to lower the prior odds. This in turn affects the posterior odds, but the change from prior to posterior odds (i.e., the BF) is independent of the number of other hypotheses that are tested.

Cheers,

E.J.

NilsPixa

Hi E.J.,

Thanks for your quick reply. Okay, so it is fine to report the provided BF, U in the table of post-hoc t-tests, e.g., to disentangle an interaction effect of a RM-ANOVA with multiple (let's say three) groups?

Best

Nils

EJ

Well, if you would just report BF_U you would ignore the post-hoc character of the test. So I would report the BF_U but also the posterior probabilities (which are corrected for the post-hoc nature of the test)

NilsPixa

Well, yes, that is also a fine recommendation. Maybe that can be inserted in the preprint of van Doorn et al.The JASP Guidelines for Conducting and Reporting a Bayesian Analysis.

How would you abbreviate or label the posterior odds in a manuscript (to be readerfriendly 😉)?

Thanks and best

Nils

EJ

Lindley (1985) uses the zero, so "0", but a lower-case "o" seems to be more common (https://en.wikipedia.org/wiki/Odds)

NilsPixa

Okay, many thanks E.J.!

Best

Nils