The inconsistent results of Bayesian analysis and the classical frequentist analysis

Zhenzhen

Hi all,

I am new for Bayesian analysis. When I analyze the data in JASP, the results of one classical ANOVA analysis revealed a significant interaction effect, while the Bayes factor for the interaction effect is quite small (F(1, 34) = 7.869, p = .008, ηp2 = .188, BF10 = 1.977). Also, another classical ANOVA analysis revealed no significant main effect, but the Bayes factor is even bigger than the previous one (F (1.63, 55.26) = 3.251, p = .056, ηp2 = .087, BF10 = 2.848). I have learned that Bayesian analysis is more conservative, so the "significant " results in classical ANOVA might have no strong evidence in Bayesian analysis. But I feel confusing that why a relatively small p value (.008) can coincide with a low BF value (BF10 = 1.977) while a relatively bigger p value (.056) can coincide with a bigger BF value (BF10 = 2.848)?

Best,

Zhenzhen

EJ

Dear Zhenzhen,

First off, the p=.056 vs BF10 = 2.848 result is pretty much as expected. The anomaly is the p=.008 vs BF10 = 1.977 outcome. What models did you compare exactly, to get at the interaction BF? Also, is this a repeated-measures ANOVA? We have a paper coming out that discusses some issues that may be relevant:

van Doorn, J., Aust, F., Haaf, J., Stefan, A., & Wagenmakers, E.-J. (2021). Bayes factors for mixed models. Manuscript submitted for publication. https://psyarxiv.com/y65h8

I'll attend these authors to your post as well.

Cheers,

E.J.

JohnnyB

Hi Zhenzhen,

To give a more precise answer, I would need some more information on your setup: how many factors did you include, are these factors measured between or within subjects, and which specific models are being compared in your Bayes factor? You could also post a screenshot from the JASP output, or save the JASP file and upload it here as a zip file.

Kind regards,

Johnny