Fluctuating Bayes Factors when recalculating based on same data
I've noticed that the Bayes Factors for Bayesian Repeated Measures can fluctuate quite a bit when running the same analysis again on the same data. If my understanding is correct, some fluctuations are to be expected, because the analysis is not deterministic. However, the fluctuations are sometimes substantial.
For example, for the attached data file, the BF inclusion for target-match × distractor-match varies from 1.3 to 2.5. And I'm fairly sure that once the BF was actually above 3.0, which is what prompted me to look into it in more detail. (@Cherie reanalyzed this data based on a reviewer comment, and now suddenly there was evidence for this interaction where previously there was none!)
So how should we best deal with this? I realize of course that we're talking about relatively small fluctuations, and that zero evidence never becomes very strong evidence. But still, the fluctuations are sufficiently large to affect the interpretation.
For future releases, would it be possible to reduce the fluctuations by increasing the number of repetitions that JASP uses to estimate the Bayes Factors? (I assume that something like that happens behind the scenes.)
PS. Two example results for the same data:
There's much bigger issues in the world, I know. But I first have to take care of the world I know.