Interaction is significant, but simple effects is not; Bayes Factor hints towards null hypothesis
Hello, everyone,
I am currently running a 2 x 2 x 2 repeated measures ANCOVA in JASP. I have 2 within factors (animacy, word pairing) and one between factor (group).
I have found a significant interaction of animacy * group. However, when i use the “simple effects” option in JASP, it yields that both simple effects are not significant. I have plotted the interaction using the estimated marginal means using R; it is a slight crossover interaction. However the crossover is not as intense to make the simple effects non significant I believe. I have run additional pairwise post-hoc comparisons for the animacy by group interaction (using Bonferroni correction). The pairwise comparisions yielded two significant differences, one of which was that the two levels of the animacy factor for one of the groups (community-dwellers) showed a significant difference.
I additionally ran a Bayesian analysis for the ANCOVA and found that the BFincl for the animacy * group interaction was only 0.311, hinting towards the null hypothesis.
Could these issues be due to power? (N = 56; n group 1 = 26; group two = 30). What should I do now? Which additional analysis could I run and which analysis do I report? How do I interpet the interaction?
Thanks in advance!
ANCOVA
https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/optimized/3X/5/f/5f453364764044743b43b10ef6ec781abe7d9b5d_2_690x358.png
Post Hoc & Simple Effects
https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/original/3X/1/0/10ce62a7a3faa5e0bb66bd410df65284a2891abc.png
Bayesian analysis
https://canada1.discourse-cdn.com/flex030/uploads/mc_stan/optimized/3X/4/0/4006818a0320ab17c64649606e27112621ebcb03_2_690x366.png
Comments
First, is there a reason why you did not use the JASP ANCOVA analysis to make the graph? There's a question in my mind as to whether the means in the graph are estimated from an ANCOVA model.
Second, I think the use of covariates adds a lot of complexity. Did you check to see whether the ANCOVA assumptions are roughly met? (e.g., see https://www.lehigh.edu/~wh02/ancova.html)
Third: Conceptually, the simple main effects should agree with *Uncorrected Post-hoc tests, not corrected ones (I don't recall whether JASP permits uncorrected post-hocs).
Fourth, I would start with a simpler analysis--no covariates--and try to interpret that first, before going to a more complex model.
R
Finally, I think you should post s JASP file. It could be some modified or made-up data rather than your real data. This is because it's not clear, for example, whether you pooled the error terms, and whether that choice was consistent for the post-hocs and the simple main effects.
R