How to compute the Bayes Factor inclusion in JASP?

StefanW

Hi!

I read this blog:

https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp

I tried to follow the calculations and everything worked fine.

However, I got stuck when I tried to understand the article by van den Bergh et al. (2020): A tutorial on conducting and interpreting a bayesian ANOVA in JASP.

https://www.cairn-int.info/revue-l-annee-psychologique-2020-1-page-73.htm

I can reconstruct the BF_M in table 2 (I think, given rounding errors?): For each effect, the BF_M is OR with P(incl) and P(incl|D). Eg., for Model S, .6 and 0.445 gives 0.547.

For Table 3, the authors claim that the BF inclusion is inspired by the blog. I can figure out the P(incl) and P(incl|D) from Table 2, but it seems that the BF_inclusion is not simply the OR of these probabilities (i.e., it is not computed as the OR in Table 2).

I thought that maybe it is computed as in the blog (only comparing P(incl|D) between models), but this does not seem to be true, either.

Does anybody know how to get the BF_inclusion in Table 3? Can it be computed from values in Table 1?

EJ

[Solution at the end]

Strange. I can get the second number. So, in R:

prior.odds <- .4/.6

posterior.odds <- .1524/(1-.1524)

posterior.odds/prior.odds

[1] 0.2697027

[The table gives 0.28, but that is probably just rounding]

I'm not really getting the third number:

prior.odds <- .2/.8

> posterior.odds <- .3033/(1-.3033)

> posterior.odds/prior.odds

[1] 1.741352

[The table gives 2.018]

And I am nowhere near the first number:

prior.odds <- .4/.6

> posterior.odds <- .6872/(1-.6872)

> posterior.odds/prior.odds

[1] 3.295396

[The table gives 72.76, which seems way to high given the probabilities]

So I've downloaded the jasp file (https://osf.io/f8krs/). JASP does give the result that is in the table. So the prior and posterior inclusion probabilities are perhaps not the ones we think they are. Note that for matched models, the table note explains that this is " Compares models that contain the effect to equivalent models stripped of the effect. Higher-order interactions are excluded."

So for the "O" factor this means we compare the models "O" and "O+I" against "Null" and "I". This means the prior inclusion odds is 0.4/0.4 = 1 (each model has prior inclusion probability 0.2). The posterior inclusion probability is (.534+.147)/(.007+.002) = 75.67. That's not exactly right, but this number is very sensitive to the rounding of .007 and .002, so it looks correct to me.

It does seem somewhat strange to me to have the prior and posterior inclusion probabilities in the table disconnected from the inclusion Bayes factor. So this may be something we ought to remedy. For the matched models, the prior inclusion probability is 0.5 by construction (I think). Anyway, something to discuss.

Don is on vacation but I'll attend him to this.

Cheers,

E.J.