How to calculate corrected BF manually?

KevinvSchie

Hi all,

So I have run a Bayesian repeated measures ANOVA followed up with post-hoc tests (actually, the comparisons are planned, but nevertheless). JASP gives me uncorrected Bayes Factors, but I'd much prefer to report corrected Bayes Factors (given the number of tests that are performed). I assume there is a way I can calculate this using the prior odds and the corrected posterior odds; I am just not sure how (by now it is evident that I'm merely a pragmatic Bayesian). I've added the output, so if someone here could just show me how to do this for one of the postdoc test, I'd be happy!

EJ

Yes, this is based on the Westfall paper. If you send me an Email at EJ.Wagenmakers@gmail.com I can give you the thesis of Tim de Jong who explains how it's done.

KevinvSchie

It took me a while to get back to this. I've read through the relevant parts of Tim's thesis, but it is not completely clear to me how I can calculate a corrected BF. Could you maybe explain to me how to do it for eyes moving vs. eyes fixed. Clearly, there are four different conditions in this design and it seems that the Westfall correction takes that into account. Would greatly appreciate some help!

EJ

So the prior odds have been adjusted according to the Westfall formula. If you multiply this by the uncorrected "regular" BF you will get the posterior odds, which you can interpret as a "corrected" BF.

KevinvSchie

Hi EJ,

just wanted to check something real quick. In the note under the post hoc table it states "note. the posterior odds have been corrected for multiple testing by fixing to 0.5 the prior probability that the null hypothesis holds across all comparisons (Westfall, Johnson, & Utts, 1997)". However, I'm multiplying the prior odds by the uncorrected (regular) BF. I am not sure exactly how any correction is taken into account.

EJ

Yes, the uncorrected BF needs to be multiplied by the prior odds to give the posterior odds. The correction for multiplicity is in the prior odds.

As an aside, Tim's thesis is now on PsycArXiv: https://psyarxiv.com/s56mk/
Tim will do a blog post soon on his thesis, and I'll encourage him to explain in detail, with a concrete example, how the Westfall correction plays out.

Cheers,
E.J.

KevinvSchie

That would be great! Also, thanks so much for explaining!

martinangelusp

Thanks EJ and KevinvSchie.

The questions, answers/comments above really helped and saved me quite a lot of stress and time. Just bespoke to what I was looking for.

Thanks once again.

Bayes4Food

I stumbled upon this thread in search for some thoughts on multiple comparison. I got Tim's thesis but was wondering whether the blog post was written after all? I couldn't find that. Would be very much appreciated!

EJ

hmm I don't think the blog post happened...but it would just have summarized the paper

E.J.