BF Hypothesis Testing Question

GeorgesLemaitre

Hi,

I am writing for the first time a paper within the Bayes framework, and I feel rather comfortable now up to the level of Linear Regression. However, i do have a question regarding how to test the following hypothesis, and if i need to rephrase it.

H:  The negative relationship between X and A is stronger than the negative relationship between X and B.

(A, B being two dimensions of the same construct). As of now i have conducted Bayes correlations using JASP and the BF for XA is about 34 and the for BF for XB about 4. Would comparing the BFs suffice? or is there a "difference test" that i need to know about.

Moreover, I am conducting about 20 correlation tests at once (we are dealing with multiple characteristics), Do i have to worry about multiple testing, and should I specify this in the paper?

Greetings, from the Big Bang,

Georges Lemaitre

EJ

Hi George,

1. About multiplicity: yes, I do think a correction is in order, unless you want to go fully subjective Bayes -- with carefully specified prior plausibilities for all hypotheses involved, there is no need for a correction. Some relevant references and remarks here: https://psyarxiv.com/s56mk
2. If you want to test whether the correlation between X and A is larger then between X and B, you have two complications: (a) X features in both correlations, so they are not independent; (b) in order to address your question you need a test for the difference in correlation -- simply comparing the BFs is not correct, for the same reason that comparing p-values is not correct (http://www.ejwagenmakers.com/2011/NieuwenhuisEtAl2011.pdf)

Cheers,

E.J.

GeorgesLemaitre

Hi E.J.,

Thanks for the input, will do.

Just another question, and i will go. What are good places to get feedback on our methods section? The issue at hand is that while NHST experts are everywhere (myself included), i am feeling quite uneasy jumping directly into peer-review with interpretations of Bayesian methods that I am only learning.

Greetz,

Georges

EJ

Hi Georges,

Well you could post it here, or email some people for feedback.

Cheers,

E.J.

GeorgesLemaitre

Good morning,

thanks. What I will probably do, is post the preprint on PsyArXiv and link it here for the good people here to look over the methods... It'll take some time though :)

Have a good day!

Georges

PS: Jasp has been godsent for all trying to learn about Bayesian inference. Even old dogs like me. I am very curious however about the reception it will have with the reviewers :) Especially in organizational psych, where bayesian methods are not really present :)

PSS: having hypothesis stating the H0 is absolutely new and awesome to me ...

Papi

Hello George,

I think the new R package BFpack can do what you are looking for, see section 2.2 in the preprint https://arxiv.org/abs/1911.07728v1

If you are into R: I used this code to compare pairs of correlations:

library(polycor) # install.packages("polycor")
library(BFpack) # install.packages("BFpack")

# compute some correlations using data from the package:
cors <- hetcor(memory[memory$Group == "HC", -ncol(memory)])

# check out correlations:
cors$correlations

# compute Bayes factor to compare two correlations that seem to be rather different
BF(
 cors,
 hypothesis = "Del_with_Im = Fas_with_Del"
)

# compute Bayes factor to compare two correlations that are similar
BF(
  cors,
  hypothesis = "Cat_with_Wmn = Fas_with_Wmn"
)

Note that the BFpack package does not return the Bayes factor, but posterior probabilities (based on prior odds specified using the prior argument; the default is 1:1), which I found quite odd, but well, you can compute the Bayes factor from the posterior probabilities.

Note that my code does not come with any warranty, however. I checked out the package last week and only now tried to use it to compare correlations. You should definitely verify it; the documentation was not 100% clear to me.

Cheers

Martin

GeorgesLemaitre

Hi Papi (lol)

thank you for the code and insight, will run it the first thing tomorrow! Have a good evening...

Cheers back!

Georges