Two methods to test within-subject interaction using Bayesian ANOVA in JASP
Hi everyone,
I have two within-subject variables in my experiment: Condition (two levels: con1 and con2) and Position (five levels: pos1, pos2, pos3, pos4 and pos5). I plan to examine whether there is Condition x Position interaction.
In the framework of classical (Frequentist) ANOVA, there are two methods to investigate this interaction. One method (which is the most commonly used method) is to add both variables and their interactions into the repeated measures ANOVA analysis, and check the F value and p value for the interaction. The second method is to first calculate the difference between the two levels of Condition (e.g., con1 - con2) at each level of Position, through which I can create a new variable named Diff_Condition with five within-subject levels (pos1, pos2, pos3, pos4 and pos5). Then I can perform a one-way repeated measures ANOVA on Diff_Condition and check the F value and p value.
The F value and p value are the same for the two methods when I use classical ANOVA (as shown in JASP):
However, results for the two methods are quite different when I use Bayesian repeated measures ANOVA in JASP. This is the results for the first method:
which seems to support null hypothesis for the interaction term.
This is the results for the second method:
which strongly supports alternative hypothesis.
I was wondering whether results for the two methods are quite different in Bayesian ANOVA. Thank you very much!
Comments
Dear Xiaohupsy,
Thanks for posting this! Together with @frederikaust and @EJ I'm working on a project where we try to resolve some discrepancies between the frequentist and Bayesian repeated-measures ANOVA. Would it be possible to share your data with us so we can check if the same is happening here as is happening in other data sets?
Kind regards,
Don
Hi Don,
Thank you for your reply!
I have sent my data to your email address obtained from this webpage (https://jasp-stats.org/team/don-van-den-bergh/).
Sincerely,
Xiao Hu
Dear Xiao,
My apologies for taking so long to reply.
I've taken a look at the data and it appears that the same is happening as in another post (see https://forum.cogsci.nl/discussion/4855/strange-behavior-with-bayesian-rm-anova). With JASP <= 0.16.2 I get these results:
The inclusion Bayes factor indicates evidence against including condition (con) and the interaction between condition and position (con * pos) but indicates evidence in favor of including position (pos). However, the frequentist RM-ANOVA shows different results:
This is exactly as you already pointed out in your first post.
The difference between the two analyses is caused by the random slopes (or lack thereof). The frequentist RM-ANOVA considers random slopes for all but the highest order repeated-measures interaction. The Bayesian RM-ANOVA does not do this. JASP has unfortunately inherited this behavior from the underlying R package BayesFactor.
In the next release of JASP (0.16.3, which should be out in a few weeks) we have modified the Bayesian RM-ANOVA to always include the random slopes as the frequentist analysis does. The new results are as follows:
Now, there is inconclusive evidence for condition (BFincl = 1.863) which matches the non-significant p-value. The evidence in favor of including position is still overwhelming, while the evidence in favor of including the interaction between condition and position is somewhere between moderate and strong. The Bayesian results are now in agreement with frequentist results.
If anything is unclear or if you have any further questions, please let me know!
Kind regards,
Don