RM ANOVA - extrem diverging main effects - interpretation of the interaction effect

Comments

• EJ

  January 2017

  Hi Markus,

  In JASP we use the marginality principle, which means that when an interaction is present, so are the constituent main effects. The most straightforward test is to compare the full model against the model with only the main effects. When one or both main effects do not receive a lot of support, you can wonder whether this is the right comparison, and the inclusion probability may make more sense. In your case, both will probably point to H0, so yeah, I think you have some evidence in favor of H0.

  Cheers,
  E.J.
• Markus

  January 2017

  Great, thank you very much E.J.!
  Best,
  Markus
• Markus

  January 2017

  Hi,
  I have a problem that I can not handle. I found a significant interaction effect (p < .02) based on classical repeated measures ANOVA analysis but I got a insensitive BFinclusion = 1.6 result based on Bayes analysis. Now my biggest problem is
  how I should handle this in a paper. When I would publish the results without the BFs everything would be fine, but no I have to find the right words when I include the BFs. Because now a reviewer could possible say that my study was not designed well enough to find the effect and rejects the paper.

  Another problem here is that I can not explain what the prior in the repeated measures ANOVA really means (r scale fixed effect of: 0.5; r scale random effects: 1; and r scale covariates: 0.354).

  Thanks for any advice or help!
  Best,
  Markus
• EJ

  January 2017

  I was a bit caught up and did not respond too quickly. I have now responded to a similar question you just asked. Sorry about my tardiness. I try to answer quickly but sometimes diapers and deadlines get in the way.
  Cheers,
  E.J.
• FSella

  January 2017

  Hi,

  I saw the discussion and I thought to join in.

  I am new to Bayesian analysis and I would like to have a feedback on the Bayesian ANOVA I have just run (see attached image). I am interested in assessing the evidence for the model that includes the interaction term. The model with the interaction Time * Group has a BF10 = 2.120e+7 whereas the model with the two main effects Time + Group has a BF10 = 639811.912. It is my understanding that the comparison of the model with the interaction and the model with the two main effects can be achieved with a simple division (due to transitivity): BF10 Interaction / BF10 main effects (i.e., 2.120e+7/639811.912 = 33). The BF = 33 suggests a strong evidence in favour of the model with interaction. Am I right?

  Thanks in advance for any help.

  Francesco

  
• EJ

  January 2017

  Hi Francesco,
  Completely correct!
  E.J.
• FSella

  January 2017

  Hi EJ,

  thanks for the immediate reply.

  We have analysed the performance of two small clinical groups (6 training vs 6 controls) across nine time points. Again, it is my understanding that Bayesian approach is more appropriate with small sample sizes as long as the priors (we used the default in JASP) are reasonable (with large large samples likelihood tends to dominate whereas priors may have a stronger influence with small sample sizes). Now, I am planning to run a series of Bayesian t-tests (still in JASP) in order to obtain a BF for each comparison (training vs. control for each time point). I would also double check the results using the BEST package to obtain a posterior distribution of the difference between groups' means.

  However, the Editor that handles this paper asked us to run a classic mixed ANOVA (Ws:Time; Bs: Group) with bootstrapping along with non-parametric analysis (Mann-Whitney) for the post-hoc comparisons. Is there any motivation to use Bayesian analysis over NHST in case of small sample sizes, or would it be better in this case to not use Bayesian analysis?

  Thanks in advance

  Best wishes

  Francesco
• EJ

  February 2017

  Hi Francesco,

  It is always a good idea to use Bayesian inference. But at the very least you can argue that the two paradigms are complementary, and it is counterproductive to ignore the Bayesian outcomes. So my advice is to heed the editor's advice, but report both.

  Cheers,
  E.J.
• peterose1

  March 2017

  Is the result of Francesco's division (BF10 = 33) strong evidence of a pure interaction effect? It seems to me to state only that a model including the interaction term is more strongly supported than one without it.
• EJ

  March 2017

  Hi peterose1,

  JASP does not allow "pure" interaction effects (meaning without the constituent main effects). This is the principle of marginality, discussed in part II available at https://osf.io/ahhdr/

  Cheers,
  E.J.
• peterose1

  March 2017

  Thank you. I am downloading he paper now. But meanwhile, I conclude my other interpretation is correct: "a model including the interaction term is more strongly supported than one without it." Yes?
  (I am BoomChicago dad.)
• EJ

  April 2017

  Yes, that's correct.