BIC Bayes Factors multilevel model comparisons
Dear EJ and others,
We have run a longitudinal study trying to detect changes over time in several cognitive variables. We have done so using multilevel analyses including random effects for each individual in a frequentist manner. However, our main idea would actually be that the two groups that we compare in our study do not change differently over time. Therefore, a reviewer suggested that the addition of BIC BF comparing a model with/without a group interaction would be useful to "test the null hypotheses" being that the added value of an interaction term in our model is indeed not very much (referring to the Wagenmakers et al., 2007 paper on pvalue problems). This seems like a super easy, yet useful Bayesian addition in our paper, and a clear benefit is that anyone could understand what we've done.
However, I have gone back and forward in my thoughts on the BIC BF method, as it seems like a rather strict method to test if the interaction has any added value in the model. Would you think this would indeed be of added value or is it (too) biased in quantifying evidence for H0?
Thank you so much in advance.
Carolien
Comments
Hi Carolien,
I've gone back and forth on the BIC as well. My current thinking is that *is* reasonable as a point of reference, and especially appropriate for default analyses that do not commit to a strong prior position. For instance, I assume that fractional Bayes factors will give similar results (you might want to check out the work by Joris Mulder and the BFPack R package). See also
@ARTICLE{RouderEtAl2009Ttest,
AUTHOR = {Rouder, J. N. and Speckman, P. L. and Sun, D. and Morey, R. D. and Iverson, G.},
TITLE = {{B}ayesian $t$ Tests for Accepting and Rejecting the Null Hypothesis},
JOURNAL = {Psychonomic Bulletin \& Review},
YEAR = {2009},
volume = {16},
pages = {225--237},
}
where BIC does not look to bad.
Cheers,
E.J.
Hi EJ,
Thank you so much for your quick and helpful advice. I will check out the references you've suggested.
I think more advanced Bayes Factors will indeed point in the same direction (though maybe not as strongly). The BIC BFs values do make sense in our case, but I just wanted to make sure that we are not making a huge mistake by adding them to our paper.
Best wishes,
Carolien
Follow up: For anyone interested in this topic, I've found a really nice tutorial: https://rpubs.com/lindeloev/bayes_factors.
In my case it turned out that the BIC BFS were actually really inflated when I compared them to BFs computed using the brms package. My conclusion would be that for multilevel models adopting BIC BFs seem quite tricky (as is also suggested in the paper recommended by EJ).