overwhelmed by how to report results
The more I read in the forum, the more confused i get. With a simple design, results are still manageable. For t tests, one can report BF10 or BF01, done. For a simple 2x2, one can describe the model that provides the best evidence (maybe its for a main effect, maybe its for an interaction, maybe both are good), one can report the best or various. Alternatively, if the BF10 is arbitrary, one can look at BF01. Report BF01 instead, done. (Maybe I remember wrong, but in the workshop I believe it was said that the model comparison is whats better than just looking at effects and reporting BFincl?? though for simple designs it wouldnt make much difference, right?).
For a more complicated design, which results in e.g. 166 models, it becomes more tricky.
From the workshop, I remembered to look at the models first, compare to null. That might very straightforward identify that the model including a main effect is best, the others arent even comparable, wonderful, I'll report that BF10 for that model, maybe it even is consistent with my p value, winning.
But with 166 models, I might have a lot of them providing extreme evidence. So I can then compare to the best model, and see whether they really are as good as the best model against the null. But maybe they are, so then I need to understand what drives these models (many containing interactions).
So from the forum, it appears I should go into the output for effects. So here, I can look at effects across all models, hopefully that identifies some main effects. If so, wonderful, I'll report the BFincl for whatever effects win. I might or might not have a strong BFincl for any many effect, but importantly I have evidence for an interaction. Though, this is not yet real evidence for the interaction as in this analyses the models that contain an interaction also contain the main effects. So to evaluate the interaction, I need to look at the output of effects across matched models (or Bawes) - so this strips away the effect from the models.
Thus, this analysis wont ever provide strong BFincl for main effects (since they were stripped, correct), but instead allows me to understand whether the interaction exists and wasnt just driven by a main effect, correct?
So having identified maybe some main effects, some interactions, do I then stick to the BFincl, or should I return to the model comparison?
I guess my confusion comes from my various data sets:
1) While for one main effect I get a BF10=80, when I look at BF01 instead, various models (incl interactions) provide extreme BF01. When I look at analysis of effects across all models, that main effect has a BFincl=13. When I look at analysis of effects across matched models, that main effect has a BFincl=90, while the interactions have weak BFincl. So the interactions in BF01 were driven be the main effect. But how is my BFincl for the main effect now 90 if in this Bawes analyis the models are stripped of the effect? So I'd say the strongest evidence is for that main effect, reporting that model BF10, but do I also report the BFincl, and which one of the two?
2) On a different data set, again 166 models, my BF10 are weak, but the BF01 for main effects =10 and various other models are extreme. Here analysis of effects across all models show only effect of the fourway interaction with BFincl = 743. So again, to evaluate the interaction I look at effects across matched models, and here beautifully, that interaction turns into a BFincl = 1.743E+8. So though I actually have a strong BF01 (though only =10) for the null regarding main effects, there is extreme BFincl for that interaction. So I should just report the evidence for the interaction, I feel like I'm then not telling the whole story here?
3) In another data set, again 166, I have a wonderful BF10 for the model incl factor1 + factor2 + factor1*factor2. In effects across all models, each of the BFincl tell the same story, both factors = 2.8E+13 and their interaction = 2E+14. When I strip away the effect across matched models, the interaction comes out nicely with BFincl = 9.3E+15. So this all makes sense to me, each factor has strong evidence, and their interaction is there too.
But again, do I report the model, the BFincl across models or across matched models?
Sorry about the essay, I just really want to get this right.
MANY MANY THANKS!