Very strong evidence in Post-Hoc but no main effect
Dear all,
I conducted a three-way repeated measures ANOVA (2x2x4) and did not observe a main effect for one of the factors, which is named "region of interest" (ROI). However, upon running a post-hoc test, I discovered strong evidence indicating a significant difference between the two levels of this factor (BF10 > 100). Any insights or ideas as to why this might be the case would be greatly appreciated.
Thank you for your input.
Best regards,
María Paula
Comments
Could there be assumption-violations? It's difficult to tell what might be going on without seeing the actual data.
R
Yes the data would be good, but screenshots of the relevant tables would be good too. Did you use a recent version of JASP? We updated the model specification a year ago, see for instance https://jasp-stats.org/2022/07/29/bayesian-repeated-measures-anova-an-updated-methodology-implemented-in-jasp/
I did use the recent version of JASP and the specifications in the 2022 paper. I also included the comparison of only matched models that Sebastian suggested as specified in the paper of 2020. Based on the Q-Q plots I believe there was not an assumption that has been violated (correct me if I am wrong).
This is the analysis of the effects:
Post hoc
Q-Q plots:
Am I missing something important? Please let me know, and thank you so much for your quick replies.
Best,
María
Could there be any rows with missing data that are excluded from the ANOVA but (incorrectly) included in the post hoc test?
R
There are not missing values in the data, from what I checked just now. As far as I know, it is not possible to check or correct some assumptions (like non-sphericity) in the Bayesian ANOVA features in JASP as it is now. Maybe there will lie the issue? What other option may have caused this?
I wonder, what is the result of a frequentist repeated measures ANOVA on the same data? Does it still produce an incongruence between main effect and post hoc?
R
Do you have a raincloud plot of the data, showing the effect?
This is the rain cloud plots of ROI.
And regarding the frequentists ANOVA: with corrections for not met assumptions:
The frequentist says it is not significant
This is a little puzzling, because the uncorrected BF ought to be just a t-test BF, and that particular t-value would not give such a huge BF (but something anecdotal in favor of H0). I've asked the team.
Thank you so much for your time and caution in this matter! I will be looking forward to your response.
Best wishes,
María
So suppose you take all of the "visual" data points, and compare it to all of the "motor" data points, collapsing across all other factors -- for that t-test, what is t and n?
I did it like you mentioned:
For the frequentist analysis I got a significant p-value and for the BF I got a similar one to the post-hoc test value.
The n is 152.
Ah, so this is t=-4.747, and that matches with the BF10. So the problem, it seems, is that in our Bayesian post-hoc test we average across all the other factors and levels, and this is not what the frequentist test does. I'll see whether the team has something to add.
Alright. Once again, thank you so much for your prompt responses! :)