a jump in BF after increasing the number of participants

LiorAbramson

Dear forum,

We have conducted an experiment in which we continuously increased the number of participants, and every once in a while examined the Bayes factor. We decided to stop the recruitment once the BF reached a substantial conclusion (BF higher than 3 or lower than 1/3), thus supporting one of two hypotheses (H1 or H0).

In practice, what happened is that for a while (until recruiting approximately 400 participants) the BF ranged between 1 and 2, providing inconclusive results again and again. In the ninth iteration of subjects' recruitment (after adding 50 more participants), the BF increased substantially (BF= approximately 11 in favor of H1).

Is such a jump common in Bayesian hypotheses testing? If so, is there an analytical explanation for why such jumps in the value of the BF occur?

Thank you very much,

Lior

EJ

Hi Lior,

1. It would be informative if you could show the sequential plot so we can assess the size of the jump;
2. Very large jumps suggest that the observations that gave rise to it may be an outlier
3. An evidential threshold of 3 is not a lot. Under equal prior probability, the posterior model probability for the disfavored hypothesis would still be 25%.

Cheers, E.J.

LiorAbramson

Dear E.J,

Thank you for your help!

I attach the plot, and would love to hear if you think this jump is reasonable. I would also like to add that in terms of the descriptive patterns, there was no substantial change after the last iteration. That is, the means and SEs were stable across the entire recruitment procedure. I think it indicates that the last batch of participants did not include outliers

Thanks again,

Lior

EJ

Hi Lior,

A change from 2.5 to 12, based on about 80 additional participants, is not such a large jump. What is remarkable about this sequential plot is not the "jump", but rather the fact that up to 420 or so participants you have such stable (and evidentially weak) BFs. I've seen this before but only when the first batch of data is dominated by one condition (so evidentially not a lot can be learned).

Cheers,

E.J.

LiorAbramson

Hi E.J,

Thanks again. I would like to ask a few more questions to make sure I am following.

First, when you say "batch is dominated by one condition", do you mean that there were more participants in one condition compared with the other conditions? if so, this is not the case in our data...

Second, what do you think the stable pattern of data until 420 participants means? Do you think it means that there is something different in the last batch, or, can I conclude that the results are nevertheless reasonable? I saw similar patterns in other papers as well, although the "jump in BF" was after much less participants, so I am not sure if this is a right comparison.

Third, do you perhaps know on a formal answer or a reference for the claim that the jump from 2.5 to 12 is not considered large?

Thank you,

Lior