Sequential analysis and frequentist statistics
Hi everybody! I'm planning an experiment for which I'm considering, with my co-authors, to define the sample using sequential analysis. We would like to pre-register our study indicating that data collection will continue until a BF of a given value (e.g., 5) will be reached in support of either H1 or H0. As far as I understood, in a Bayesian framework this is perfectly fine.
Given that in psychology and neuroscience fields reviewers are in general more familiar with the frequentist statistics than the Bayesian one, it is recommended to report boh BF and p-values (e.g., see section "Reporting both frequentist and Bayesian results" in Keysers et al., 2020, https://doi.org/10.1038/s41593-020-0660-4).
I would like to ask if this is possible when defining the final sample size using sequential analysis. In other words, in this case can we run frequentist statistics (additionally to the Bayesian one, to get a p-value) despite the fact that we have stopped data collection according to results (BF in the sequential analysis) generated by the same datataset?
Thank you for your help!
Matteo
Comments
Hi Matteo,
I would say that's a clear "no": the usual p-value is computed under a fixed-N sample plan, which is definitely not the one you intend to follow. What I would do instead is report the probability of obtaining misleading evidence, as outlined in the papers with Felix Schoenbrodt and Angelika Stefan. This gives the long-term operating characteristic of the design, so it has a strong frequentist flavor.
Cheers,
E.J.