Variability of Bayes Factors
Hi everyone,
I recently read a paper (Pfister, 2021) about the variability of Bayes factors and notably the fact that repeating a Bayesian ANOVA on the same dataset resuts in dramatical changes in Bayes factors. The author of the paper suggested to repeat the analysis a large amount of time and to report the trimmed mean of the Bayes factors. I would like to know whether it is possible to do that in JASP. I saw that there is Repeatability option in the Additionnal Options of the Bayesian ANOVA module where one can specify a seed. However, I am not sure how to use it and I feel that it is not really what I am looking for. I have no doubt that it is possible to use R to deal with this question but I am not familiar enough with R to do so by myself. Can someone help me with that please? Thank you very much in advance.
Best regards
Nico
Comments
Hi Nico,
First, it is generally not a good idea to average BFs. For instance, averaging 1/3 and 3 does not yield 1. So averaging the log BFs would be better. In addition, the variability in ANOVA is quantified by a % error, which you can bring down by increasing the number of samples under Advanced options.
Indeed, repeatability is not what you are looking for: the results will always be the same (good for reproducibility) but you hide the very variability you wish to reveal and quantify. I am sure you can do this in R (but again, I would average the logs!) Let me email the R master.
Cheers,
E.J.
Hi Nico,
I took a look at that paper and noticed that it ignores the error estimate on the BF, which is relevant here. I can't be sure without reproducing their figures, but I suspect the rare outliers in their simulations would be accompanied by large error estimates, so I'd echo what EJ said about paying attention to that, and potentially increasing the number of samples.
Because all samples are independent, doubling the number of samples is the same as doing it twice and taking the average, with the added benefit that you get an estimate of error as well.
Hi E.J. and Richard,
thanks for your answers. When you say I have to increase the number of sample, is that for the posterior samples or for the numerical accuracy option? Where can I get the error of estimate that you mentionned?* Thanks again for your help.
Best regards,
Nico
*Update: Sorry my question was not clear enough on that point. I am working with BFexcl. I found the % error for the regular BF. So my question should have been "Is there a way to get it for BFexcl"? Thanks
This is a question for EJ, since BFexcl is his thing. Conceptually, yes, since BFexcl is a function of the individual BFs, but it might be a bit difficult/time consuming to calculate.
Thanks Richard. I think it will be convincing enough for my reviewers if I increase the number of samples to decrease the %error. To be sure, the number of samples I have to increase is the one I higlighted in red below, isn't it? thanks again
Yes (you can also try it out and see). The BFexcl and BFincl do not come with this error%, but if the individual ones have small error then so will the derivatives.
E.J.
Thanks for your answer EJ and thanks again Richard. Your both helped me a lot.
Hi everyone, I was wondering whether averaging the error percentages for all the models included in a given model averaging is useful to have a rough idea of the error percentage that we could get in the BMA? I guess actually computing the error percentages for all BFincl in the BMA requires more complex calculation but I thought that maybe at a conceptual level it could be useful. I would be interested to have your opinion about that. Thank you very much
My first thought is that you'd have to weigh the error percentages with the posterior model probabilities, such that error percentages have a bigger impact when they are associated with models that exert a large effect on the end result.
Cheers,
E.J.
Hi E.J., thanks for your answer. Ok, that makes sense. I will do that.
Cheers
Nico