Settings for one-sided Bayesian chi-square test
I have a question about the settings for a one-sided Bayesian 2 x 2 chi-square test.
In an Experiment, participants took part in two performance tests. The outcome is accurate (1) or inaccurate (0). In these performance tests, participants’ perspective at encoding and test was either matching (1) or mismatching (0). We hypothesized that accuracy would be higher under the matching than the mismatching condition. Thus, my alternative hypothesis is that group 1 < group 2.
I am unsure which "Bayes Factor" settings to choose, because my findings suggest a different interpretation in the classical chi-square test than the Bayesian chi-square test.
The classical chi-square test shows statistically significant result for performance test 1, but not for performance test 2.
Next. I conduced the Bayesian chi-square test with group 1 < group 2 and BF01. For both performance tests, the Bayes Factors suggests that there is strong evidence for the null hypothesis – but that this evidence is even stronger evidence for performance test 1 than performance test 2. This confuses me because the classical chi-square test showed a smaller p-value and larger effect size for performance test 1 than performance test 2. This is what makes me wonder if any of my settings are incorrect.
I have attached the results of the analyses.
Thanks a lot in advance!
Comments
Hi Mels,
Sorry for the tardy reply. For the test of two proportions, I would use the Bayesian A/B test -- I think it is more in line with the hypothesis you want to examine. See https://psyarxiv.com/z64th and https://onlinelibrary.wiley.com/doi/10.1002/sim.9278 for instance. It is possible for p-values and Bayes factors to show this non-monotonicity, because they are related through a sqrt(n) term; so if sample size differs between experiments than a lower p-value can be associated with more evidence for the null hypothesis.
Cheers,
E.J.