Slightly different outcomes everytime I execute the same analysis

Feldkamp27

Currently, I am writing my Masterthesis in the faculty of Human Factors & Engineering Psychology. For my data analysis I am using JASP in order to execute a bayesian repeated measures anova. However, I got the problem, that everytime I execute the analysis anew with exactly the same data, I receive slightly different outcome values e.g. for the bayes factor or credible interval. I do not have an explanation for this. Is this usual?

Moreover, my bayes factor and credible interval values of main effect also change when I add a covariate. This has huge consequences for the main effects found, as somehow there occurs an effect but merely when no covariate is added. But again, I do not know why that is the case. 

Another question I have also concerns the model averaged posterior summary. How is the mean and standard deviation built here, as these are not the actual means or SD`s of the scores. Is there a certain distribution when bayesian statistics is applied? I could not find that information in the student guide, unfortunately.

I would really appreciate, if you could help me with this problem.

Kind regards,

Niklas

EJ

Hi Niklas,

1. The results are based on a numerical procedure, so some variability is inevitable;
2. To understand the impact of your covariate you could do some exploratory analyses;
3. The model-averaged posterior is a weighted combination of the posterior distributions from each of the models, with the weights determined by the posterior model probability.
4. We have some tutorial papers on the way.

Cheers,

E.J.

Feldkamp27

Hello E.J.

Thanks for your quick response!

Regarding your second point... of course, I could do some exploratory analyses but that is not the point here. Sorry, if my description of the problem was not distinct enough. It is not about the interaction term built with the covariate (e.g. system 1 and 2 ad IV and sense of agency as covariate and Trust as DV) . It is only about the main effect of system 1 and 2 on trust that changes when a covariate is added to the calculation in general and not to the model directly, you know? So, when I calculate an ANOVA without adding an covariate to the field for covariate on the left side (console or whatever that is called) and then look at the output for the main effect of system 1 and 2 on Trust , then this differs from the main effect of system 1 and 2 on Trust when I add a covariate in the field for covariate in the console. And again, I do not mean the interaction effect of system 1 and 2 and sense of agency on trust. I only mean the main effect of system 1 and 2 that is different when a covariate is filled in in the field for covariate on the left side.

Kind regards,

Niklas

Feldkamp27

P.s. I mean, the main effect of x->y still should be the same regardless whether I add a covariate z or not, right?

EJ

Hi Niklas,

Well, I have not Googled this, but it seems to me that when you add a covariate, you are changing the main effects. Suppose you test IQ scores of dogs to those of cats, but you have reason to believe that "number of pets per day" is a confounding variable. You add that as a covariate, in the hope that it will reduce the unexplained variability and hence make the IQ effect come out more clearly.

Cheers,

E.J.