Backsliding from Bayes
For the past few years I've been moving toward incorporating Bayesian statistical analysis into my research and teaching. However, the more I learn about the standard implementations of Bayesian analysis, the more of a let-down I feel--somewhat like a religious convert on the verge of backsliding into a life of sin.
To me, the main attraction of Bayesian analysis is that, in principle, it allows us to answer the kind of statistical question that we would really like to answer. Specifically, it allows us to answer the question, “Given the data, what’s the likelihood that the alternative hypothesis is true (i.e., that the effect size is non-zero) relative to the likelihood that the null hypothesis is true?” That would be the posterior odds. But in JASP and R, the emphasis is on outputting the Bayes Factor, which is not about the likelihoods of hypotheses, but about the likelihoods data: the likelihood of the data under the presumption of true alternative hypothesis, relative to the likelihood of the data given a true null hypothesis. We could, in principle, find the probabilities (which I see as the ultimate promise of the Bayesian approach) by multiplying the Bayes factors by the prior probabilities. But the stickler is that the prior probability for the alternative hypothesis and for the null hypothesis are ill-defined. The priors are well-defined for the range of possible values of the population parameter in question (e.g., the difference between two population means). But since that range can be infinite, it is not so easy to precisely specify the likelihood that that the population parameter has some non-zero value.
So the end result is that, at least in JASP and in every other implementation I’m aware of, we aren’t given the two posterior probabilities—of the alternative and of the null hypothesis. Instead, we’re given the continuous distribution of posterior probabilities, under the presumption of a somewhat arbitrary (though not unreasonable) distribution of prior probabilities. I‘m therefore unable to draw the kind of statistical conclusion I initially set out to draw, which is: “Given the data, he alternative hypothesis is ___ times as likely to be true as is the null hypothesis.” I feel that if I can’t do that, I might as well slide back to the easy and familiar sin of null-hypothesis testing.
Bowling Green State University