"Bayes factor changes with every run of the data (no seed set)"

rrilke

I could really use some help with replicating my own results/ understanding a comment I got from a journal. 

I used the Bayesian ANOVA section in JASP for two models: 

(1) A main effect where X1 predicts Y.

(2) A full model with X1, X2, X3 and all their interactions predicting Y

When I run the analysis for (1) I always get the same result. When I run the analysis for (2) I get slightly different BF10 every time I run it. 

A journal we submitted a paper to is re-running and reviewing our analysis and results. The comment they gave us is “Bayes factor changes with every run of the data (no seed set). Please indicate to show the full list in JASP in the tab ‘Limit No. Models Shown’ as the result was not visible when settings were set as default.” 

Could you please help me understand what they mean by this comment? And how can I set the analysis such that it always produces the same results? 

I could also not find “limit no. models shown”, and in the results of 2 all possible models are shown in the results. 

The JASP version I am using is 0.16.3.0

Thank you very much!

EJ

First of all, it is recommended to use the latest version. If you wait a day or so, you can get the brand new 0.17.3. Then:

1. Under "Additional options" -> "Repeatability", you can set a seed, which will yield the same result.
2. Under Lim. no models shown (in the GUI input panel, just underneath "credible interval"), do not select "show best 10" but select "no".

EJ

rrilke

Thank you!

Indeed, setting the seed helped to produce the same results again.

I still could not find the "Lim. no models shown". Under "credible interval" I see plots -- model averaged posteriors, Q-Q plot of residuals, and posterior R2, and order.

could it be that an older version of JASP does not contain this option?"

EJ

Yes, that is possible! I recommend to use the latest version of JASP.

E.J.