uninformative ANCOVAS in JASP
Hi everyone! I was wondering if bayesian ANCOVAS work properly in JASP (v. 0.8) and if they are; how to interpret them?
As they are now nothing really changes when adding a covariate. BF10 stays the same and the analysis instead favours the inclusion of both the fixed factor and the covariate; telling us nothing if the fixed factor is relevant when controlling for the covariate. I've added the table below where the intended study is if the group differences are meaningful even when controlling for variable 1.
Model Comparison - Stuff
Models P(M) P(M|data) BF M BF 10 % error
Null model 0.250 1.844e -235 5.531e -235 1.000
Group 0.250 1.630e -157 4.891e -157 8.843e +77 7.161e -83
Variable 1 0.250 1.004e -54 3.011e -54 5.444e +180 0.005
Group + Variable 1 0.250 1.000 2.989e +54 5.424e +234 0.935
Expected results (and the one you get with a frequentist analysis) is that BF10 remains > 1 but reduced when adding a covariate. As of now the analysis adds nothing of value (the results are identical to an ANOVA without the interaction effect) and I can't help but feel this is not the intended function.
So do anyone know if this is indeed the intended function and how to interpret the results or if it isn't then if theres a workaround possible using JASP (I've been considering doing a regular ANOVA and count everything but the interaction effect as nuisance but I'm not confident in my understanding of blocked designs).
Thanks for your time and hopefully awesome help 
Comments
Hi 2Elephants,
Can you send a .jasp file or a screenshot of the output?
[As an aside, you can't use ANCOVAs to control for a covariate that differs between groups. This is a common misunderstanding. You can use ANCOVA to explain away error variance that is attributable to a covariate that does not differ between groups.]
Cheers,
E.J.
Certainly; here's the picture.
I added the ANOVA as the "baseline" without the covariate and then the frequentist ANCOVA that has the expected results when the marginal means are adjusted as expected. And your note about the misunderstanding is appreciated; I've recently started to understand the processes properly and still fall into some traps.
And here "Group" is the factor of interest, and "Variable1" is the covariate?
Yes exactly. To clarify on the design: Group contains two groups of subjects that have each given 640 measures on Dependent variable. Variable 1 predicts the dependent variable. So the question that I'm trying to answer is do group one and two differ in Dependent variable even when you take into account that they have some differences in Variable 1.
OK. So this is the way I see it. In the Bayesian analysis, it is more about model comparison than about parameter estimation. So we have the model without any predictors: the null model. Then we can compare this to the model with only the covariate (model 1, featuring only Variable 1). This gives us BF10.
Next we add the Group variable (model 2: featuring both Group and Variable 1). This gives BF20. We can then obtain the Bayes factor that compares model 1 to model 2 by transitivity: BF21 = BF20/BF10. This Bayes factor indicates the extent to which the Group variable is important, over and above the influence of the covariate.
A similar model-based perspective (albeit not in the context of ANCOVA) was recently promoted by Rouder and Morey in a PBR paper: http://pcl.missouri.edu/sites/default/files/Rouder.etal_.pbr_.2016.pdf
Cheers,
E.J.
So I think I get your point browsing through the linked article. Since the full model is preferable over the model that only includes Variable 1 it suggest that taking both into account is preferred and by then removing the influence of the covariate we end up with BF21=2.871e+20 which would be the intuitive number from a frequentist viewpoint. I'm still not following in regards to the accuracy of this number if it's on the decimal correct or just a good approximation? But it seems sensible given the data.
Anyways thank you very much for your help! Much appreciated
If I'm correct then BF21 = 5.521e+234/5.444e+180 = 1.014144e+54.
This is a massive number (a 1 followed by 54 zeros). The percentage error on the component BFs is low, so no need to worry about accuracy.
E.J.
Haha yes that would be correct. Thats what you get for trying to use your phone as a calculator. Well my thanks again.