Bayesian Repeated Measures ANOVA Post Hoc t-test help

Comments

• JohnnyB

  October 2020

  Hi Charlotte,

  Unfortunately the Bayesian ANOVA does not have the feature to compare groups of an interaction effect in the post hoc tests (this is available in the frequentist version).

  What you could do is to make some dummy variables with the compute column function and/or by filtering. Then you can indeed run individual t-tests for each comparison, using the desired prior specification. Please let me know if you run into any trouble =)

  Cheers

  Johnny
• chilly

  October 2020

  Hi Johnny,

  Thank you for your response!

  I've done what you suggest above, but what prior options do I need to select for it to replicate a posthoc test? I'm guessing 'informed' and then 'Cauchy'? But what do I put in location and scale?

  

  Thanks again for your help.

  Charlotte
• JohnnyB

  October 2020

  Hi Charlotte,

  If I understand correctly, the Bayesian post hoc test corrects for multiplicity by adjusting the prior odds. In a normal comparison, the prior odds are set 50-50, so the Bayes factor equals the posterior odds. However, if the prior odds are adjusted, the Bayes factor stays the same but the posterior odds will be different. So to take into account the multiplicity, it's better to look at the posterior odds, since these will contain the correction.

  Your interaction post hoc will contain 4 comparisons, so to correct the Bayes factors you get from the T-test module, you multiply them by 0.4142.

  This is from the post hoc test footnote:

  The posterior odds have been corrected for multiple testing by fixing to 0.5 the prior probability that the null hypothesis holds across all comparisons (Westfall, Johnson, \& Utts, 1997). Individual comparisons are based on the default t-test with a Cauchy (0, r = 1/sqrt(2)) prior. The "U" in the Bayes factor denotes that it is uncorrected.

  Cheers,

  Johnny
• rohanp16

  October 2020 edited October 2020

  I have a similar, but slightly more complicated, question @JohnnyB and would appreciate your insights.

  I am running a 3 way Bayesian ANOVA with 2 within-subject factors (say, A and B) and 1 between-subject factor (say, C). The analysis of effects indicates strong evidence (BFincl is greater than 3) for the B * C predictor. What would be the best way to follow this up?

  1) Should I run a A*B ANOVA separately for each level C and interpret only the B main effect (and similarly, a A * C for each level of B and interpet only the C main effect)? That is, should I still include A in the follow-up ANOVA and only interpret the effect for which I have a statistical basis?

  Or,

  2) Create new columns where all levels of A are averaged and then just conduct t-tests? That is, B1 vs B2 for each level of C and C1 vs C2 for each level of B?

  Overall, it's still not clear to me, from any tutorial paper or JASP blog-post, what the best way is to follow-up an interaction effect in a Bayesian ANOVA.
• JohnnyB

  October 2020

  Hi Rohan,

  That's a good question, and not something routinely tackled by the literature.

  Suggestion 1) is basically doing a simple main effects analysis. Where in the frequentist framework you would correct for multiple comparisons by using a different SS term and DF, you could maybe apply the same reasoning as for the Bayesian post hoc test (adjust the prior odds, and look at the posterior odds for a model instead of the BF).

  Suggestion 2) is more straightforward doing the post hoc analysis manually, so you can follow the procedure above.

  The difference between 1) and 2) is that in 2) you specifically compare each group(subset) to each other, whereas in 1) you look at the more general main effect. When your factors only have 2 levels each, these are equivalent.

  How to properly follow up in Bayesian ANOVA's is an interesting issue though, and is something we could tackle with our lab in the future!

  Cheers,

  Johnny
• rohanp16

  October 2020

  Thanks for that prompt response Johnny, much appreciated!

  I now have more clarity on how to proceed and will proceed with 2). My only question is, how did you end up with 0.4142 as the correction factor for the BF_U with 4 comparisons?

  Indeed, your lab has done a lot to further Bayesian analyses (and better statistical inference in general), would be keen to hear what comes up in this regard.

  Cheers,

  Rohan
• JohnnyB

  October 2020

  Hi Rohan,

  That's great to hear!

  For the calculation, you can look at this paper, pages 13-15, where they coincidentally also use 4 conditions =)

  Cheers,

  Johnny
• rohanp16

  October 2020

  Thanks for pointing me in the right direction Johnny. For anyone else reading this and interested to know,

  The formula is (1-0.5^(2/m))/0.5^(2/m), where m is the number of levels of a factor.

  With 4 levels of a factor, it is (1-0.5^(2/4))/0.5^(2/4) = 0.414 and is the prior odds seen in the paper.

  With 6 levels of a factor, it is (1-0.5^(2/6))/0.5^(2/6) = 0.260, and so on and so forth...

  Okay, so this has opened up a few more questions.

  My skim reading of Tim's thesis on this issue, is that JASP uses the Westfall (1997) approach (as it says in the footnote when 'null control' is checked), where the focus is on the number of groups, m, rather than the number of comparisons, k. This makes sense to me when one is following up a main effect in an ANOVA (as there are dependencies), but what about following up an interaction effect? I can't clearly see what a 'group' would be and how I would implement the Westfall approach manually... Sticking with the above example,

  B1 vs. B2 for C1

  B1 vs. B2 for C2

  C1 vs. C2 for B1

  C1 vs. C2 for B2

  I understand that Jeffreys (1938) is too conservative but in light of the above conundrum, I think that it might be the only way to go currently? If I do go via the Jeffreys route, is my understanding correct that the formula is,

  (1-0.5^(1/k))/0.5^(1/k), where k is the number of comparisons?

  So, for the above 4 t-tests, my correction factor would be 0.189?

  Thank you :)
• JohnnyB

  October 2020

  Hi Rohan,

  I should have introduced my post by saying that I'm definitely not an expert on these post hoc corrections. What you're saying makes sense - it would be better to take into account the number of comparisons rather than the number of levels, to make it more general.

  For 4 levels of a factor, a post hoc test will yield m(m-1)*0.5 = 6 comparisons, which according to Jeffreys ought to be corrected by 0.1225, so there is quite a large discrepancy between both approaches (one being too conservative, one being too generous, or both).

  My initial post stating you have 4 groups was based on the 2x2 setup:

  1. B1, C1
  2. B1, C2
  3. B2, C1
  4. B2, C2

  But you are not making all 6 comparisons between these 4 groups, so I would expect a decreased penalty for multiple comparisons compared to the main effect with 4 levels posthoc test. Again, this is just my hunch, and of course interactions warrant a different treatment from main effects. I guess it depends on how careful you want to be.

  Cheers

  Johnny
• rohanp16

  October 2020

  For 4 levels of a factor, a post hoc test will yield m(m-1)*0.5 = 6 comparisons, which according to Jeffreys ought to be corrected by 0.1225, so there is quite a large discrepancy between both approaches (one being too conservative, one being too generous, or both).

  Indeed, quite a large discrepancy between the two approaches - 0.414 (Westfall) vs. 0.1225 (Jeffreys).

  But you are not making all 6 comparisons between these 4 groups, so I would expect a decreased penalty for multiple comparisons compared to the main effect with 4 levels posthoc test

  I completely agree here but this is where I find the Westfall approach confusing. With Jeffreys, because it is based on the number of comparisons, the correction is relatively straightforward. Main effect with 4 levels has 6 comparisons, so the correction factor is 0.1225. If we are doing only 4 comparisons, such in the case of following up a 2 x 2 interaction effect, then it is 0.189. On the other hand, with the Westfall approach, following up a main effect is straightforward as the number of 'groups' of a factor is well-defined. For a factor with 4 levels, when one is making 6 comparisons, the correction is 0.414. However, for the interaction effect with "4 groups", as you mention, the Westfall approach would say the correction is 0.414, which doesn't seem right considering only 4 comparisons are being made (vs. the 6 following up a main effect). As you say, one would expect a decreased penalty here.

  I should have introduced my post by saying that I'm definitely not an expert on these post hoc corrections.

  Is there someone else in the lab who would be one? @EJ or @vandenman perhaps?