Sample size planning with multiple comparisons in open-ended sequential BF design
Hi,
My question is relating to multiple comparison and sample size planning. As I understand, when we have multiple comparisons, we can control prior probability of H0, depending on the number of comparisons. For example, if one has five comparisons, the prior probability of H0 = (1/2)^(2/5) = 0.758 and the prior probability of H1 = (1 - 0.758) = 0.242. Then, the corrected posterior probability of P(H1|y)/P(H0|y) can be referred as evidence for H1 relative to H0, which can be obtained by multiplying uncorrected BF10 with the corrected prior probability (= 0.242/0.758).
Now I am making an experiment of “open-ended sequential BF design” with multiple comparisons. For such design, one needs to set the threshold for evidence of H0 and H1 to stop data collection. For example we stop data collection when BF10 exceeds 3 or when BF10 gets less than 0.3. But with multiple comparisons, should the threshold be corrected?
If so, increasing the prior probability of H0 makes testing more conservative. It is problematic because evidence for H0 is easier to be gotten as the number of comparisons increases. So, do we need to select a hypothesis depending on the two values. One is based on correction of prior probability of H0 and another is based on correction of prior probability of H1?
For example, if we have five comparisons and want to set the threshold as BF10 = 3 and 0.3, two values are evaluated. One value is calculated by multiplying uncorrected BF10 with 0.242/0.758, and judged whether it exceeds 3 or not. Another value is calculated by multiplying uncorrected BF10 with 0.758/0.242, and judged whether it is less than 0.3 or not. Then, when the value goes beyond the thresholds in one of those judgements, data collection is stopped.
Is such method wrong for open-ended sequential BF design?
Thank you very much,
Comments
With multiple comparisons I personally would adjust the BF threshold that signals when sufficient evidence for H1 has accrued. This will mean that the thresholds become asymmetric, because the BF one for H0 can remain the same.
EJ
Thank you very much for your replying. ah…your approach is simpler than mine!
I have a further question. When stopping data collection by adjusting the threshold of BF for H1 (say, in the case with 5 levels and adjusting by Westfall approach, changing BF = 3 to 9.39 = 3 * 0.5^(2/5) / (1-0.5^(2/5))), can stopping data collection with the threshold of BF for H0 (keeping 1/3) indicate null-effect? For providing evidence of null-effect, Is it not necessary to adjust the threshold of BF for H0 to stop data collection (0.11 = 1/3 * (1-0.5^(2/5) / 0.5^(2/5) instead of 1/3). This question may be silly thpough….