Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Scale for the interaction in an ANOVA model

Hello everyone,

I'm a bit confused about the rscaleFixed value that is applied to an interaction. The interaction is coded as the product of two main effects, with the latter coded as sqrt(2)/2 implying the former is coded as 1/2. Does that mean that, for a fixed rscaleFixed value entered into the BF package, there are different scales applied to the main effects and the interactions?

I know that this isn't the case when a whichModels is set to "bottom" because the BF package is coded to default to a ttest in this case (so the difference in coding doesn't matter). But what if whichModels is set to "top"?

Comments

  • edited May 18

    I figured this out. The answer is "yes", different scales are applied to the main effects and the interactions.

    For clarity, I want to mention that my original post did not provide a complete description of the setting that I was asking about. I was referring to the case of a 2x2 fixed effects ANOVA. As it turns out, the difference in coding is more obvious when you do a 2x2x2x2 fixed effects ANOVA.

    I simulated data with the constraint that the sums of squares for ALL main effects and interactions is zero (each cell has exactly the same distribution). Here's the aov() summary:

    Df Sum Sq Mean Sq F value Pr(>F)
    F1 1 0.00 0.000 0 1
    F2 1 0.00 0.000 0 1
    F3 1 0.00 0.000 0 1
    F4 1 0.00 0.000 0 1
    F1:F2 1 0.00 0.000 0 1
    F1:F3 1 0.00 0.000 0 1
    F2:F3 1 0.00 0.000 0 1
    F1:F4 1 0.00 0.000 0 1
    F2:F4 1 0.00 0.000 0 1
    F3:F4 1 0.00 0.000 0 1
    F1:F2:F3 1 0.00 0.000 0 1
    F1:F2:F4 1 0.00 0.000 0 1
    F1:F3:F4 1 0.00 0.000 0 1
    F2:F3:F4 1 0.00 0.000 0 1
    F1:F2:F3:F4 1 0.00 0.000 0 1
    Residuals 144 75.46 0.524

    So there's no variance explained by any of the terms. Here's the BF outputs for with whichModels set to "top":

    Bayes factor top-down analysis

    When effect is omitted from F1 + F2 + F3 + F4 + F1:F2 + F1:F3 + F2:F3 + F1:F4 + F2:F4 + F3:F4 + F1:F2:F3 + F1:F2:F4 + F1:F3:F4 + F2:F3:F4 + F1:F2:F3:F4 , BF is...

    [1] Omit F1:F2:F3:F4 : 2.488178 ±4.05%
    [2] Omit F2:F3:F4 : 4.447647 ±24.45%
    [3] Omit F1:F3:F4 : 3.250185 ±5.16%
    [4] Omit F1:F2:F4 : 3.353204 ±5.62%
    [5] Omit F1:F2:F3 : 3.305366 ±4.36%
    [6] Omit F3:F4 : 4.74362 ±3.53%
    [7] Omit F2:F4 : 4.696784 ±3.53%
    [8] Omit F1:F4 : 4.419148 ±5.93%
    [9] Omit F2:F3 : 4.721122 ±3.53%
    [10] Omit F1:F3 : 4.344021 ±4.76%
    [11] Omit F1:F2 : 4.305297 ±4.33%
    [12] Omit F4 : 5.551244 ±3.75%
    [13] Omit F3 : 6.582877 ±11.44%
    [14] Omit F2 : 6.392093 ±3.53%
    [15] Omit F1 : 6.382501 ±3.52%

    Against denominator:

    DV ~ F1 + F2 + F3 + F4 + F1:F2 + F1:F3 + F2:F3 + F1:F4 + F2:F4 + F3:F4 + F1:F2:F3 + F1:F2:F4 + F1:F3:F4 + F2:F3:F4 + F1:F2:F3:F4

    Bayes factor type: BFlinearModel, JZS

    The error %s are quite high so there's some noise, but it's pretty clear that there are different Bayes Factors for the main effects, the two-way interactions, the three-way interactions, and the four-way interaction. This is because there are different parameterizations for the different interactions (they are coded as the product of the main effects involved; this can be verified by using model.matrix()). Also, a careful reading of Rouder, Morey, Speckman, and Province (2012) section 6 suggests that no adjustment is made for the difference in scales across the main effects, two-way interaction, three-way interactions, ...., and n-way interactions.

    This strikes me as an odd choice. On the one hand, it's kind of consistent with the sparsity-of-effects principle: interactions are in some sense automatically assumed to be of a lower magnitude.

    However, there's some strange features. For example, my intuition is that as the number of factors grows, the evidence for the nth-way interaction also grows such that a trivial n-way interaction can be supported by the data despite a traditional ANOVA showing that it accounts for very little of the variance. The reason that this happens is that the nth-way interaction (when there are two levels per factor) is coded as (1/sqrt(2))^n, so increasing n decreases the numbers used to code the interaction. This effectively makes the scale for the Cauchy distribution for the interaction smaller. Hence, more evidence for the alternative.

    I'm pretty sure that these quirks would go away if the lengths of the interaction vectors were normalized to length one, but I haven't finished doing the math.

  • Dear blindreviewer,

    The questions you raise (also in an earlier post) are of course highly relevant. The Rouder et al. paper is not an easy read, and it would be worthwhile to dissect the reasoning and the resulting performance much more than has been done so far. In my lab we are currently debating similar issues regarding the Rouder et al. ANOVA (not the interactions). A lot of this boils down to pretty subtle modeling choices. As far as specific questions on the Rouder et al. paper are concerned, the chances of getting a meaningful answer are highest if you contact Jeff or Richard directly. Richard does not spend all his time on the forum (as I do :-)) but if you send him an Email and receive a meaningful reply, please post it here!

    Cheers,
    E.J.

  • Many thanks, E.J. I found a few additional subtle modeling choices (as you call them) that appear to matter greatly under certain conditions. I will contact Jeff or Richard directly about them, and the issue posted above. Should the discussion prove helpful (I'm not simply making silly mistakes), I will ask for their permission to post info here so everyone benefits.

Sign In or Register to comment.