Replication Bayes factors in R [BayesFactor package]

Comments

• MSB

  May 2020

  library(BayesFactor)

  
  

  # say you have 2 data sets

  iris_1 <- iris[1:75, ]

  iris_2 <- iris[-(1:75), ]

  
  

  # To get a replication BF you need:

  
  

  ## 1. BF of the first data set

  BF1 <- lmBF(Sepal.Length ~ Sepal.Width + Petal.Length,

        data = iris_1)

  
  

  ## 2. BF of the combined data sets

  BF_total <- lmBF(Sepal.Length ~ Sepal.Width + Petal.Length,

           data = rbind(iris_1,iris_2))

  
  

  ## 3. Divide the latter by the former:

  extractBF(BF_total, logbf = F, onlybf = T) / extractBF(BF1, logbf = F, onlybf = T)

  #> [1] 2.564315e+32

  
  

  
  

  # (compare to)

  lmBF(Sepal.Length ~ Sepal.Width + Petal.Length, data = iris_2)

  #> Bayes factor analysis

  #> --------------

  #> [1] Sepal.Width + Petal.Length : 2.170426e+18 ±0%

  #> 

  #> Against denominator:

  #>  Intercept only 

  #> ---

  #> Bayes factor type: BFlinearModel, JZS
• lyambailey

  May 2020 edited May 2020

  Thanks so much, this works perfectly!

  One follow up question: I'm wondering if it would make sense to use this approach iteratively to compute a replication Bayes factor based on multiple previous experiments? So, assuming there are three experiments, the first two showing the same effect, and I wanted to use the (combined) posteriors from the first two experiments as a prior for the third, could I use...

  # 3 data sets

  iris_1 <- iris[1:50, ]

  iris_2 <- iris[50:100, ]

  iris_3 <- iris[-(1:50),]

  
  

  ## 1. BF of the first TWO data sets

  BF_first_two <- lmBF(Sepal.Length ~ Sepal.Width + Petal.Length,

        data = rbind(iris_1, iris_2))

  
  

  ## 2. BF of the combined data set from all three experiments

  BF_total <- lmBF(Sepal.Length ~ Sepal.Width + Petal.Length,

           data = rbind(iris_1, iris_2, iris_3))

  
  

  ## 3. Divide the latter by the former:

  extractBF(BF_total, logbf = F, onlybf = T) / extractBF(BF_first_two, logbf = F, onlybf = T)
• MSB

  May 2020

  Yes, this should generally work for any K datasets from replications - but note that all data must be from exact replications for this kind of analysis to make sense.

  Good luck!
• lyambailey

  May 2020

  @MSB I'm curious what you mean by "exact replication" in this context. I'm in a situation where Experiment 1 compared performance between two conditions (let's call them condition A and condition B), and Experiment 2 included the same conditions (and used the same paradigm / method of assessing performance / etc etc) PLUS one extra control condition (condition C). So you could say Experiment 2 was an exact replication, with the exception of one additional condition. Moreover the effect I was trying to replicate was still between conditions A and B.

  Would this still satisfy the criteria for an exact replication (in the context of using replication BF)? If not, would it make sense to exclude data from the new condition, and simply compute the replication BF for the contrast between conditions A and B in Experiment 2?
• MSB

  May 2020

  I don't think so - because the second model's parameters are not the same as the parameters of the first (it has one more). But @EJ would probably know best (:
• EJ

  May 2020

  Well if you throw away condition C then everything is fine

  E.J.
• lyambailey

  May 2020

  Amazing, thank you both so much!