Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Sample size planning with multiple comparisons in open-ended sequential BF design

Hi,

My question is relating to multiple comparison and sample size planning. As I understand, when we have multiple comparisons, we can control prior probability of H0, depending on the number of comparisons. For example, if one has five comparisons, the prior probability of H0 = (1/2)^(2/5) = 0.758 and the prior probability of H1 = (1 - 0.758) = 0.242. Then, the corrected posterior probability of P(H1|y)/P(H0|y) can be referred as evidence for H1 relative to H0, which can be obtained by multiplying uncorrected BF10 with the corrected prior probability (= 0.242/0.758).

Now I am making an experiment of “open-ended sequential BF design” with multiple comparisons. For such design, one needs to set the threshold for evidence of H0 and H1 to stop data collection. For example we stop data collection when BF10 exceeds 3 or when BF10 gets less than 0.3. But with multiple comparisons, should the threshold be corrected?

If so, increasing the prior probability of H0 makes testing more conservative. It is problematic because evidence for H0 is easier to be gotten as the number of comparisons increases. So, do we need to select a hypothesis depending on the two values. One is based on correction of prior probability of H0 and another is based on correction of prior probability of H1?

For example, if we have five comparisons and want to set the threshold as BF10 = 3 and 0.3, two values are evaluated. One value is calculated by multiplying uncorrected BF10 with 0.242/0.758, and judged whether it exceeds 3 or not. Another value is calculated by multiplying uncorrected BF10 with 0.758/0.242, and judged whether it is less than 0.3 or not. Then, when the value goes beyond the thresholds in one of those judgements, data collection is stopped.

Is such method wrong for open-ended sequential BF design?

Thank you very much,