Bayesian Statistics: Correcting for multiple testing?
Dear all,
recently, I found out that multiple testing can become a problem also with Bayesian statistics, if the tests are not independent. I have already collected data for my study and would like to ask you, if a correction for alpha error rates is necessary and if so, how to do that with JASP?
The design is the following:
Independent Variable (IV): Participants answer three different questionnaires. For analysis, we split them into two groups per questionnaire (high vs. low scores) based on a pre-defined cut-off. In sum, there are 6 groups (3 questionnaires x 2 subgroups).
Dependent Variable (DV): All participants complete 3 different tasks. Each task refers to one of the three questionnaires.
Then, for each of the 6 groups a Bayesian one-sample t-test of the respective DV against the expectation value under chance is performed. We considered each questionnaire (IV) plus the respective task (DV) as independent experiment with a Bayesian hypothesis for or against an effect.
Problem: The three questionnaires highly intercorrelate. This means, there is a high probability that the same participants are in more than one of the high-score subsamples. This is kind of a dependency, but does this influence the error rates in a Bayesian design or how can I correct for that?
Thank you very much for your support!
Comments
Hi jaspuser23,
The primary Bayesian correction for multiplicity is in the prior model probability (see for instance https://psyarxiv.com/s56mk). If the tests are not independent that does complicate matters, and in general it is always best to account for all of the data in one model (instead of first selecting subsets and then testing those). But the primary tool is adjusting the prior model probability, and the de Jong work summarizes methods on how to deal with this.
Cheers,
E.J.
Thanks a lot for your answer, EJ! The work you linked is very helpful!