How to briefly interpret/report the Bayes factor in papers

senne

We recently used JASP to calculate the Bayes factor for a couple of experiments in a study that is now under revision. Thank you for this very helpful and intuitive software!

One of our Reviewers now suggests that we misinterpreted the Bayes factor. To be honest, we are a bit confused/uncertain on how to reply. Although the comment does make sense to us, we did inspire our phrasing on an earlier paper using the JASP Bayes factor and were wondering if we could consult your expertise on this?

Specifically, we mentioned:
"The BF01 was 11.499, suggesting that these data are 11.499 more likely to be observed under the null hypothesis."

The reviewer commented:
"According to my understanding, the Bayes factors tell the relative odds that the (in this case) null hypothesis is correct relative to the alternative hypothesis, given these data. Namely, it is not the probability of the data (given the hypothesis) which is what null hypothesis testing tell us, but the probability of the hypothesis given the data."

MSB

To my understanding, you were correct in your interpretation.
In order to calculate the odds that the null is correct relative to the alternative, you would have to multiply the BF01 by the prior odds of both hypotheses being correct... (I haven't seen anyone do this in a paper though, is this common practice?)

M

EJ

Hi Senne,

You were correct, and so is M above. It is the reviewer who has misunderstood.

Cheers,
EJ

senne

Thank you for your quick and helpful replies!

Best,

Senne

sebastiaan

Hi guys,

While I generally take @EJ's word as gospel on these matters, I'm confused here. My understanding has always been that of the reviewer: The BF is a likelihood ratio that expresses how likely one hypothesis (say H0) is relative to another hypothesis (say H1) given a set of data. This is also how the Bayes factor is described in this article:

• http://www.psychologicalscience.org/index.php/publications/observer/2015/september-15/bayes-for-beginners-probability-and-likelihood.html

Which says:

To decide which of two hypotheses is more likely given an experimental result, we consider the ratios of their likelihoods. This ratio, the relative likelihood ratio, is called the “Bayes Factor.” œ

Reading the discussion above, it's almost like the following statements are equivalent (or at least confused) if we assume that H0 and H1 are equally likely to begin with:

1. BF01 expresses the likelihood of H0 relative to H1 given the data.
2. BF01 expresses the probability of the data given H0, relative to H1.

Are these two statements equivalent? If so, then at least option 1 is the more common interpretation, right? That's what the reviewer means; that's what's in the Psych Sci article; and that's also my understanding. But option 2 seems to be what @MSB and @EJ mean.

My apologies if my comment only adds to the confusion. But I think this warrants a few extra words.

Cheers,
Sebastiaan

EJ

Hi Sebastiaan,

Statements 1 and 2 are not identical. They can be numerically the same but only if the models are equally plausible a priori. Consider for instance ESP. In a given experiment, the data may be 10 times more likely under H1 (there is ESP) than under H0 (there is no ESP). In other words, H1 predicted the data from the experiment better than H0. This is the Bayes factor: the relative plausibility of the data under H1 versus H0. But this does not mean that we can conclude that it is 10 times more likely that people have ESP! The a priori probability of ESP is very very low, so a posteriori (combining the prior odds with the BF) the plausibility of ESP is still low, even though the experiment provided some evidence in its favor.

Cheers,
E.J.

sebastiaan

Right, so then my statement was correct?

Reading the discussion above, it's almost like the following statements are equivalent (or at least confused) if we assume that H0 and H1 are equally likely to begin with:

Or, phrased differently, you could say: BF01 is a likelihood ratio that reflects the likelihood of H0 compared to H1 given a set of data, but given only this set of data and not taking into account any other data that might affect the likelihood of the hypotheses.

So the reviewer is not really incorrect, then. He's just interpreting the BF how most people would interpret it, implicitly assuming that H0 and H1 are equally likely to begin with.

EJ

Well yes, the BF equals the posterior odds under equal prior odds. But the reviewer was not saying this or implying it; from what I understood, the reviewer was trying to convince the authors that the theoretically correct interpretation of the BF is in terms of posterior odds, not as an average likelihood ratio. So the reviewer was saying that the BF is about p(H|data) instead of p(data|H), whereas it is exactly the other way around.
Cheers,
E.J.

sebastiaan

So the reviewer was saying that the BF is about p(H|data) instead of p(data|H), whereas it is exactly the other way around.

Fair enough, that's indeed what he says. But so does Gallistel in his Psych Sci column. Would you say that he makes the same mistake of confusing p(H|data) with p(data|H)?

EJ

Gallistel writes "To decide which of two hypotheses is more likely given an experimental result, we consider the ratios of their likelihoods." The issue is how much emphasis to place on "given an experimental result". If you interpret this as "the evidence that is in the data" or "the degree to which the data change our beliefs" then he is correct. If he means it defines p(H0|data)/p(H1|data), then he is incorrect.

FSella

Hi,

I jump in the discussion maybe a little bit late, but I am actually facing the same issue. I ran some analyses using JASP (BTW, thanks to the team for the amazing software) and reported BF10 in the paper. Now, I would like the reviewers (and myself ) to correctly get the meaning of BF.

That's what I wrote at the beginning of the Results section:

"Statistical analyses were conducted using the free software JASP using default priors (JASP Team, 2017). We reported Bayes factors expressing the probability of the data given H1 relative to H0 (i.e., values larger than 1 are in favour of H1) assuming that H0 and H1 are equally likely".

Does it sound right?

Thanks in advance for any help.

Best wishes

Francesco

EJ

Hi Francesco,

You are reporting the BF, which, as you indicate, express the relative likelihood of the data under the models at hand. So that means that you can eliminate the text "assuming that H0 and H1 are equally likely" -- the analysis does not assume that.

Cheers,
E.J.

FSella

Many Thanks E.J.!