Bayesian model averaging

nicolas_m · July 2017

Hi everyone, I read the two-part paper on bayesian inference in psychology (Wagenmakers et al., in press) and I found it very interesting. In the Part 2, the authors described BMA (p. 22-23) and they concluded that "averaged across all candidate models, the data strongly support inclusion of both main factors Disgust and Fright" based on the two BFinclusion (46.659 and 1.413e+8, respectively). I would like to be sure that my understanding of the BFInclusion is correct.

Is it right to say that based on the BFInclusion = 46.659, the data are 46.659 more likely if we consider that disgust has an effect on the DV than if we consider that it has no effect at all?
-Is the formula to compute BFInclusion similar to those used to compute BF10?
-If I have to present results in a paper, do you recommend to present the BMA rather than the full model comparison? Or is it better to present both of the analyses?

Thank you very much in advance

EJ · July 2017

Hi Nicolas,

Generally I'd advocate reporting both analyses. The BF inclusion formula is similar to BF10 in that it quantifies the change from prior odds to posterior odds. But in the BF inclusion case, those odds are computed for all the models that contain the variable of interest vs those that do not.

As far as the interpretation goes, yes, something like that -- I'd explicitly indicate that you contrasted the models with and without the variable of interest.

Cheers,
E.J.

nicolas_m · July 2017

Hi E.J.,

Thank you very much for your answer. I read your interesting discussion about BF Inclusion with Sebastiaan (here http://forum.cogsci.nl/index.php?p=/discussion/2996/odd-bayesian-repeated-measures-seems-biased-against-interactions-and-in-favor-of-main-effects) and his presentation of what he called "Baws Factors" (here : https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp). Is it right to say that Baws Factors are just BF Inclusion computed using a different bayesian model averaging procedure than those used in JASP?

Thanks you very much

Best regards,

Nicolas

EJ · July 2017

Hi Nicolas,

The "Baws Factors" should be in the next version of JASP; they are computed by neglecting certain models. So the procedure is the same, but what differs is the set of models that is averaged over.

E.J.

nicolas_m · July 2017

Hi E.J.,

Thank you very much.
Have a nice day

Nicolas

nicolas_m · September 2017

Hi E.J., I was happy to see the last updates to JASP including BMA across matched models. However, when I run my analysis with JASP I don't find the same BFincl as when I compute the BAWS factors by hand. I think that I have probably made a mistake when computing them by hand. I would like to understand where my mistake is. Could you provide the exact formula used to compute the BAWS in JASP please? Thanks a lot

Nicolas

EJ · September 2017

Hi Nicolas,

Sure, happy to oblige. I'll ask Tim to send the relevant code, or post it here.

Cheers,
E.J.

nicolas_m · September 2017

Thanks E.J.
I talked about that with Sebastiaan. I attach a file with my JASP results and my by-hand results. I obtained the by-hand results by applying the procedure proposed by Sebastiaan to compute BAWS Factors (https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp).
Thanks again for your help

TimKDJ · September 2017

It's a little tricky to give an excerpt of the code as it uses calculated results from other functions.
Let's give it a try though!

I attached the code to this post in a .zip file.
To demonstrate what the calculated results from the other functions look like I also include a .RData file.
This .RData file was generated with a Bayesian RM ANOVA performed on the JASP Bugs dataset.

The column names look a little funky, because they contain (escaped) unicode characters.
Normally the columns are base-64 encoded, but for clarity purposes I decoded them.
If you mimic the structure of the objects effects.matrix / model$interaction.matrix / effectNames you should be able to run the code.

Hopefully you'll find it helpful!
If you have any other questions do not hesitate to ask.

TimKDJ · September 2017

Could you maybe send me the .jasp file itself, Nicolas?
(t.dejong@jasp-stats.org if you want to keep it private)

nicolas_m · September 2017

Hi Tim, thanks for your help and for the script. I will send you the .jasp file.

TimKDJ · September 2017

Omitting the prior inclusion probabilities,

IV 1:

in = 0.020 + 0.003 = 0.023
out = 0.094 + 0.014 = 0.108
Incl. BF = 0.023 / 0.108 = 0.213

IV 2:

in = 0.014 + 0.003 = 0.017
out = 0.094 + 0.020 = 0.114
Incl. BF = 0.023 / 0.108 = 0.149

IV 1 * IV 2:

in = 0.868
out = 0.003
Incl. BF = 0.023 / 0.108 = 289

Obviously there is some discrepancy because of rounding, but it seems to match to me.

nicolas_m · September 2017

Thanks Tim. However, for IV 1 * IV 2, I don't understand how your sum of P(M|data) for in-models and for out-models are not equal to 0.868 and 0.003 respectively. JASP indicate Incl. BF = 250.453 for this effect whereas my by-hand calculation indicates 285.33. Intuitively, I thought that the discrepency should be smaller if it is merely due to rounding.

EJ · September 2017

Hi Nicolas,

Not 100% sure, but these high BFs translate into very small probabilities, where rounding can make a big difference. To check, you could take an example where the BFs are modest.

E.J.

nicolas_m · September 2017

Thanks for your answer E.J., I think I finally got it. Your explanation makes sense with the very small discrepancies observed for the BF associated with the main effects.

Thanks to both of you. Best regards

Nicolas

TimKDJ · September 2017

Glad it makes sense to you now

I indeed made a little mistake with the copy-pasting earlier.
It should be

IV 2:

Incl. BF = 0.017 / 0.114 = 0.149

IV 1 * IV 2:

Incl. BF = 0.868 / 0.003 = 289

To demonstrate how strong the effect of rounding is with low probabilities (as EJ mentioned); IV 1 * IV 2 is (roughly) bounded between:
Lower: 0.868 / 0.0034 = 255
Upper: 0.868 / 0.0025 = 347

nicolas_m · September 2017

Hi Tim, thanks very much. It is crystal-clear thanks to you guys.
Best Regards
Nicolas

m_linde_18 · June 2018

Hi,
I am doing a Bayesian RM-ANOVA in R using the "BayesFactor" package. I am not an absolute expert, so I used JASP with the same data to check whether I did everything correctly. Luckily, all Bayes factors were roughly the same. Then I saw the inclusion BF across matched models in JASP (suggested by Sebastiaan Mathôt). I know that the "BayesFactor" package is the basis for JASP functionality. However, I could not find a function or an easy way to calculate the inclusion BF across matched models in R. When I googled that, I found the present forum thread. I looked at the "matchedModels.R" code that was posted here. However, I have problems extracting the effects.matrix, model$interaction.matrix, and effectNames from my BFBayesFactor object in R. Stated differently, I have an object of the Bayesian RM-ANOVA in R. Assuming that I have this object and the "matchedModels.R" code that was provided above, how can I get the effects.matrix, model$interaction.matrix, and effectNames and, ultimately, the inclusion BF across matched models in R?
Thank you in advance for your help!
M.L.

EJ · June 2018

Ah, this is something that Tim did for us.
Tim is on a trip around the world and may rejoin us, but that will take a few more months. Of course our JASP/R code online should demonstrate how this is done...
E.J.

MSB · June 2018

M.L.,

I've previously posted some of my custom code here in the forum (here>>) that includes a function for computing inclusion BFs.

m_linde_18 · June 2018

Thanks a lot MSB! That is exactly what I was looking for. I have one question, though: Is there a fundamental reason why you decided to calculate the log of the inclusion BF?
M.L.

MSB · June 2018

No particular reason - just found that is makes it easier to do other calculation with it later (which I do sometimes).

You can set the print to show the regular BFs:

incBF <- inclusionBF(BF)
print(incBF, logBF = F)

Or you can extract them yourself

exp(incBF$log.BF.Inc)

EJ · June 2018

Thanks MSB, I forgot this for a moment
E.J.

m_linde_18 · June 2018

Great. That solves everything! Thank you.
M.L.

Howdy, Stranger!

Categories

Bayesian model averaging

Comments

Howdy, Stranger!

Quick Links

Categories

Bayesian model averaging

Comments