Discrepancy in results between frequentist and Bayesian repeated measures Anova

chuber

Hi all!

I've this dataset that shows a quite puzzling discrepancy between a frequentist rmAnova and its Bayesian version. In particular one 2 x 2 interaction effect has F(1,17) = 21.5, p < .001, partial eta^2 = .56, generalized eta^2 = 0.0015, while the BF in favour of the alternative is BF = 0.31, calculated as the "Baws" factor, aka BF for effects "across matched models" in JASP (see here). The complete design 2 x 2 x 2 x 2.
My suspicion for the reason for this discrepancy is that the interaction effect size is very very small (the generalized eta^2 is indeed very small) but consistent, so the freq. Anova suggests there is an effect, and the BF says there is none, because it is that small and the default priors don't allow for detection of such small effects with this sample size (18).

What would you conclude in such a case? I guess there is no clear answer, but any expert opinion is greatly appreciated.

Thanks,
Chris

You can get the data as tab-delimited text file in wide format here.

chuber

Perhaps useful: I ran the anovaBF with increasing number of iterations (up to 10^6), because the errors were quite large for some models (~2-10%, one even ~30%). However, that didn't change much.

EJ

Yes, that explanation sounds right. Sometimes there can also be discrepancies because the model is misspecified (e.g., data are not normally distributed). The more fundamental problem is that "interaction" is vague -- all non-additive patterns are included, and the interaction model is punished for this vagueness. Julia Haaf has been working on specifying more informed interaction, where you list the ordering of cell means. Julia will join us in January, and I hope she will help implement such informative ANOVAs in JASP.

Cheers,
E.J.

chuber

Good to know - thanks!