Effects of Multicollinearity and JASP?
I have encountered an issue that involves Bayesian ANOVAs in JASP and multicollinearity. I do not believe there to be an issue with JASP, but rather with the frequentist approach. Below I outline the finding, and I hope that you can help me make sense of this. It would be greatly appreciated.
The Issue:
We are looking to examine how people integrate probability and outcome values when they have multiple outcomes. We thus have a factorial design as follows:
Probability of outcome 1 (P1) with three levels: .1,.5,.9
Value of outcome 1 (V1 three levels): 10, 50, 90
Probability of outcome 2 (P2 three levels): .1,.5,.9
Value of outcome 2 (V2) three levels: 10, 50, 90
P1,V1,P2, and V2 are factorially crossed to create 81 prospects where the two outcomes are independent of each other.
Next, we apply a six-term statistical linear ANOVA/regression model:
P1,V1, Interaction of P1;V1, P2, V2, Interaction of P2;V2.
A frequentist analysis of this model on simulated data from a strict additive agent (i.e., an agent that simply adds P1,V1,P2,V2) will produce output only for the four main effects, even though there is multicollinearity and the parameter variances are inflated (as revealed by analysis of tolerance and variance inflation factors). This is not surprising. However, when conducting the same model in a Bayesian framework in JASP, the output will produce evidence for the interactions (P1;V1 or P2;V2) close to 10% of the time (we ran the analyses 100 times).
My intuition from this is that the frequentist analysis fails to model the parameter variance and produce results that are not trustworthy. Instead, one needs to model this uncertainty as in the Bayesian analyses and conduct Monte-Carlo simulations of the probability of producing Type I errors (or Type II. Am I right in this intuition? I have a hard time making mathematical sense of this and have not found any guiding references to point me in the right direction.
Thank you in advance and best regards,
Philip Millroth