Welcome!

Sign in with your CogSci, Facebook, Google, or Twitter account.

Or register to create a new account.

We'll use your information only for signing in to this forum.

Supported by

Interpretation help (RM ANOVA)

Hi,
unfortunately, I have a problem that I can not handle. I found a significant interaction effect (p < .02) based on classical repeated measures ANOVA analysis but I got an anecdotal BFinclusion interaction score of ~1.6 based on Bayes analysis. Now my problem is how I should handle this in a paper. When I would publish the results without the BFs everything would be fine, but now I have to find the right words when I include the BFs. Because now a reviewer could possibly say that my study was not designed well enough to find the effect (or results are "worthless" because I can not say whether H0 or H1 is supported) and rejects the paper?

Another problem in this context is that I can not explain what the prior in the repeated measures ANOVA really means (r scale fixed effect of: 0.5; r scale random effects: 1; and r scale covariates: 0.354).

Thanks for any advice or help!
Best,
Markus

Comments

  • EJEJ Posts: 201

    Hi Markus,

    Well, I would just be transparent. Sometimes you do get these conflicts and, in my opinion, they urge caution. If you had a specific contrast in mind then you ought to test that (I believe Richard has a blog post showing how this can be done; we are working to implement something like that into JASP but haven't done so yet). With respect to the prior scales, the settings are explained in the relevant papers. I think you may not have covariates and random effects, so then the only thing to explain is the r scale for the fixed effects. This is based on the width of a multivariate Cauchy. It has been chosen so that the results are consistent with a t-test in case of two conditions in a between-subjects design. I would not attempt to explain this but just mention they are the default settings.

    Cheers,
    E.J.

  • MarkusMarkus Posts: 15

    Hi E.J.

    Thank you very much for your response. I posted my questions a second time because I thought that nobody would recognize it as a separate post.
    Regarding my problem above, is it allowed to base my interpretation on the BFInclusion score? In other words, reporting the BF which compares the interaction model against the two main effects model and the BF that compares the interaction model against all cadidates models, but focussing on the BFinclusion score?

    Thank you very much!
    Best,
    Markus

  • EJEJ Posts: 201

    Hi Markus,

    When you have few models, I am in favor of including the entire tables, perhaps as a supplement.

    Cheers,
    E.J.

    Thanked by 1Markus
  • MarkusMarkus Posts: 15

    Hi E.J.,
    regarding my post above. When my interaction model has an anecdotal BF10 but my BFInclusion is moderate, which one of these two should I give more weight in my interpretation? Because for me it makes a difference to say the interaction effect gets weak or (at least) moderate support.

    Thank you very much!
    Best,
    Markus

  • EJEJ Posts: 201

    The reason for in the increased support in the inclusion method may be due to the fact that some models (like the null model, or the model with only one factor) perform very poorly. I am not so sure that this effect is of interest to you.
    Cheers
    E.J.

    Thanked by 1Markus
  • eduardeduard Posts: 600
    edited January 18

    Hi EJ,

    I have a similar issue as Markus. Also my Bayesian ANOVA is not as convinced of the existence of an effect than the classical ANOVA. However, in my case, the difference is rather large. The classical ANOVA (df=19) yields an F=9.00 and p = .007, whereas the BF for this effect is 0.6, so providing even anecdotal evidence for the Null (the jasp output and the figure of the means incl. within-subject 95% CI are attached!). From looking at single-subject data, I can say that the effect is indeed small (~10ms), but rather consistent over subjects. Only one subject is showing the opposite effect but three times as strong as everyone else. However, this alone is no reason for exclusion because the overall performance of that subject is still within 2SD of the sample mean.

    I was wondering whether it is possible to have these to analyses to diverge so strongly, or whether it is more likely that an error must have happened somewhere along to road. And if it really is possible, do you know what the reasons for that could be, also given my data in particular? I understand that Bayesian stats tend to be in general a little more conservative than the classical ones, but why exactly is that?

    This experiment is the third in a series of very similar ones, and the effects so far were always rather strong and consistent between Bayesian and classical approach. So, I was also wondering whether it is possible in JASP or R to provide the outcome of earlier ones as priors in later analyses? In another discussion, I read that simply multiplying the BF doesn't work. Is there a way?

    Finally, what is your recommendation for how to tackle the issue? Just being transparent, along the lines of "classical ANOVA finds an effect, however this is not supported by Bayes", or would you take more measures? I was also running a t-test between the two conditions where I expected the effect to originate from and found moderate support for my hypothesis.

    Your opinion is very much appreciated!

    Thanks,

    eduard

  • EJEJ Posts: 201

    Hi Eduard,

    I assume the interest is in the interaction? In general BFs are less enthusiastic because they look at both sides of the coin --H0 and H1-- instead of just focusing on H0. Indeed, multiplying BFs is not allowed, as it uses the prior again. So the correct approach, as you suggest, is to compute BFs using the updated distributions. This is not yet possible in JASP.

    Being transparent is always good. However, perhaps you can achieve more informative results by not just testing an interaction "in general", but opt for a more informative contrast. I believe Richard has a blogpost on that. In addition, sometimes we see big differences between the two paradigms when particular assumptions are not met (outliers, heterogeneity of variances, etc). So you could check that too. Maybe Richard likes to weight in as well.

    Cheers,
    E.J.

  • eduardeduard Posts: 600

    Hi EJ,

    Thanks for your reply.

    I assume the interest is in the interaction?

    Indeed the interaction is what matters most.

    opt for a more informative contrast. I believe Richard has a blogpost on that.

    Do you happen to mean this blog post? In a 2x2 design, wouldn't this boil down to a simple t-test?

    compute BFs using the updated distributions. This is not yet possible in JASP.

    Is it possible directly in R with the BayesFactor package?

    sometimes we see big differences between the two paradigms when particular assumptions are not met

    Just for sakes of clarity, if some assumptions would not be met, this would mostly concern the outcome of the classical ANOVA?

    Thanks again,

    Eduard

  • EJEJ Posts: 201

    Hi Eduard,
    I'm not sure what tests would be most effected by a violation of assumptions. It feels a little like comparing apples and oranges, but perhaps it can be done. Yes I meant that blog post -- or the next one, http://bayesfactor.blogspot.nl/2015/01/multiple-comparisons-with-bayesfactor-2.html. What I'm saying is that your interaction can be specified more exactly as a specific ordering of means (equality and inequality constraints).
    E.J.

  • eduardeduard Posts: 600

    This looks interesting. I'll give it a try.

    And a last thing. Provided that neither this more specific analysis turns out to support our hypothesis, how much of a problem would it be to just try to publish the data nevertheless? (Of course, this is a highly subjective question. I just wondered how it might appear to reviewers.)

    In any case, thanks for your support. Very helpful.

    Eduard

  • EJEJ Posts: 201

    I don't think it's a problem at all. Did you see this paper by Etz and Lakens about not every study needing to provide picture-perfect results? Besides, I think you should only be applauded for being transparent. And my guess is that this will happen.
    Cheers,
    E.J.

Sign In or Register to comment.