Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Bayesian model averaging

Hi everyone, I read the two-part paper on bayesian inference in psychology (Wagenmakers et al., in press) and I found it very interesting. In the Part 2, the authors described BMA (p. 22-23) and they concluded that "averaged across all candidate models, the data strongly support inclusion of both main factors Disgust and Fright" based on the two BFinclusion (46.659 and 1.413e+8, respectively). I would like to be sure that my understanding of the BFInclusion is correct.

  • Is it right to say that based on the BFInclusion = 46.659, the data are 46.659 more likely if we consider that disgust has an effect on the DV than if we consider that it has no effect at all?
    -Is the formula to compute BFInclusion similar to those used to compute BF10?
    -If I have to present results in a paper, do you recommend to present the BMA rather than the full model comparison? Or is it better to present both of the analyses?

Thank you very much in advance

Comments

  • Hi Nicolas,

    Generally I'd advocate reporting both analyses. The BF inclusion formula is similar to BF10 in that it quantifies the change from prior odds to posterior odds. But in the BF inclusion case, those odds are computed for all the models that contain the variable of interest vs those that do not.

    As far as the interpretation goes, yes, something like that -- I'd explicitly indicate that you contrasted the models with and without the variable of interest.

    Cheers,
    E.J.

  • Hi E.J.,

    Thank you very much for your answer. I read your interesting discussion about BF Inclusion with Sebastiaan (here http://forum.cogsci.nl/index.php?p=/discussion/2996/odd-bayesian-repeated-measures-seems-biased-against-interactions-and-in-favor-of-main-effects) and his presentation of what he called "Baws Factors" (here : https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp). Is it right to say that Baws Factors are just BF Inclusion computed using a different bayesian model averaging procedure than those used in JASP?

    Thanks you very much

    Best regards,

    Nicolas

  • Hi Nicolas,

    The "Baws Factors" should be in the next version of JASP; they are computed by neglecting certain models. So the procedure is the same, but what differs is the set of models that is averaged over.

    E.J.

  • Hi E.J.,

    Thank you very much.
    Have a nice day

    Nicolas

  • Hi E.J., I was happy to see the last updates to JASP including BMA across matched models. However, when I run my analysis with JASP I don't find the same BFincl as when I compute the BAWS factors by hand. I think that I have probably made a mistake when computing them by hand. I would like to understand where my mistake is. Could you provide the exact formula used to compute the BAWS in JASP please? Thanks a lot

    Nicolas

  • Hi Nicolas,

    Sure, happy to oblige. I'll ask Tim to send the relevant code, or post it here.

    Cheers,
    E.J.

  • Thanks E.J.
    I talked about that with Sebastiaan. I attach a file with my JASP results and my by-hand results. I obtained the by-hand results by applying the procedure proposed by Sebastiaan to compute BAWS Factors (https://www.cogsci.nl/blog/interpreting-bayesian-repeated-measures-in-jasp).
    Thanks again for your help

  • It's a little tricky to give an excerpt of the code as it uses calculated results from other functions.
    Let's give it a try though! :)

    I attached the code to this post in a .zip file.
    To demonstrate what the calculated results from the other functions look like I also include a .RData file.
    This .RData file was generated with a Bayesian RM ANOVA performed on the JASP Bugs dataset.

    The column names look a little funky, because they contain (escaped) unicode characters.
    Normally the columns are base-64 encoded, but for clarity purposes I decoded them.
    If you mimic the structure of the objects effects.matrix / model$interaction.matrix / effectNames you should be able to run the code.

    Hopefully you'll find it helpful!
    If you have any other questions do not hesitate to ask.

  • Could you maybe send me the .jasp file itself, Nicolas?
    (t.dejong@jasp-stats.org if you want to keep it private)

  • Hi Tim, thanks for your help and for the script. I will send you the .jasp file.

  • Omitting the prior inclusion probabilities,

    IV 1:

    in = 0.020 + 0.003 = 0.023
    out = 0.094 + 0.014 = 0.108
    Incl. BF = 0.023 / 0.108 = 0.213
    

    IV 2:

    in = 0.014 + 0.003 = 0.017
    out = 0.094 + 0.020 = 0.114
    Incl. BF = 0.023 / 0.108 = 0.149
    

    IV 1 * IV 2:

    in = 0.868
    out = 0.003
    Incl. BF = 0.023 / 0.108 = 289
    

    Obviously there is some discrepancy because of rounding, but it seems to match to me.

  • Thanks Tim. However, for IV 1 * IV 2, I don't understand how your sum of P(M|data) for in-models and for out-models are not equal to 0.868 and 0.003 respectively. JASP indicate Incl. BF = 250.453 for this effect whereas my by-hand calculation indicates 285.33. Intuitively, I thought that the discrepency should be smaller if it is merely due to rounding.

  • Hi Nicolas,

    Not 100% sure, but these high BFs translate into very small probabilities, where rounding can make a big difference. To check, you could take an example where the BFs are modest.

    E.J.

  • Thanks for your answer E.J., I think I finally got it. Your explanation makes sense with the very small discrepancies observed for the BF associated with the main effects.

    Thanks to both of you. Best regards

    Nicolas

  • edited September 2017

    Glad it makes sense to you now :)

    I indeed made a little mistake with the copy-pasting earlier.
    It should be

    IV 2:

    Incl. BF = 0.017 / 0.114 = 0.149
    

    IV 1 * IV 2:

    Incl. BF = 0.868 / 0.003 = 289
    

    To demonstrate how strong the effect of rounding is with low probabilities (as EJ mentioned); IV 1 * IV 2 is (roughly) bounded between:
    Lower: 0.868 / 0.0034 = 255
    Upper: 0.868 / 0.0025 = 347

  • Hi Tim, thanks very much. It is crystal-clear thanks to you guys.
    Best Regards
    Nicolas

  • Hi,
    I am doing a Bayesian RM-ANOVA in R using the "BayesFactor" package. I am not an absolute expert, so I used JASP with the same data to check whether I did everything correctly. Luckily, all Bayes factors were roughly the same. Then I saw the inclusion BF across matched models in JASP (suggested by Sebastiaan Mathôt). I know that the "BayesFactor" package is the basis for JASP functionality. However, I could not find a function or an easy way to calculate the inclusion BF across matched models in R. When I googled that, I found the present forum thread. I looked at the "matchedModels.R" code that was posted here. However, I have problems extracting the effects.matrix, model$interaction.matrix, and effectNames from my BFBayesFactor object in R. Stated differently, I have an object of the Bayesian RM-ANOVA in R. Assuming that I have this object and the "matchedModels.R" code that was provided above, how can I get the effects.matrix, model$interaction.matrix, and effectNames and, ultimately, the inclusion BF across matched models in R?
    Thank you in advance for your help!
    M.L.

  • Ah, this is something that Tim did for us.
    Tim is on a trip around the world and may rejoin us, but that will take a few more months. Of course our JASP/R code online should demonstrate how this is done...
    E.J.

  • M.L.,

    I've previously posted some of my custom code here in the forum (here>>) that includes a function for computing inclusion BFs.

  • Thanks a lot MSB! That is exactly what I was looking for. I have one question, though: Is there a fundamental reason why you decided to calculate the log of the inclusion BF?
    M.L.

  • No particular reason - just found that is makes it easier to do other calculation with it later (which I do sometimes).

    You can set the print to show the regular BFs:

    incBF <- inclusionBF(BF)
    print(incBF, logBF = F)
    

    Or you can extract them yourself

    exp(incBF$log.BF.Inc)
    
  • Thanks MSB, I forgot this for a moment
    E.J.

  • Great. That solves everything! Thank you.
    M.L.

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games