Bayesian chi-square test
Hello,
I have a few questions regarding the Bayesian chi-square test. Namely, I want to conduct a chi-square test of homogeneity (using the contingency tables option), comparing 10 groups (countries) on one yes/no variable = a 10x2 contingency table. What I need is an omnibus test of differences between these 10 countries, to see if any of them is different from the others when it comes to the yes/no variable.
I read the latest JASP manual for this and found it very helpful, however, I wasn't entirely sure which sampling scheme to select when conducting the analysis. The data collection was conducted separately in each of the 10 countries, and we determined in advance how many participants we would recruit at minimum in each country (for the purposes of another analysis), however, this was not a strict stopping criterion and in some countries the data collection continued for as long as it could time-wise. Therefore, I was not sure whether independent multinomial sampling (as it is defined in the manual) fully applies here, as we did not precisely fix the country sample sizes in advance, but rather just set a minimum for each (and most countries ended up having sample sizes larger than this minimum). However, I don't think Poisson sampling applies either, as we did have this within-country approach and didn't just collect data as an overall dataset for as long as we could. Could you please help me in determining this?
My second question is whether this 10x2 contingency table is suitable for analysis with the current version of JASP. Namely, when I try to run the test (with the hypothesis that group 1 is not equal to group 2), I get a Bayes factor but with a note under the table saying that the "proportion test is restricted to 2x2 tables". Is the analysis I want to conduct (see first paragraph) possible in JASP given that there are ten groups, and the hypothesis only allows specifying relationships between "group 1 and group 2"?
Thank you in advance!
Mirna
Comments
Hi Mirna,
E.J.
Hi E.J.,
Thank you for the response!
As the .jasp file is too large to share here, I included it and an example of my data in this folder: https://drive.google.com/drive/folders/1_hyJFx96fZmkoxdsK3VGxtYLyeWsdtro?usp=sharing
(The data in the csv are randomly generated, but the categories of variables mirror those in the real dataset.)
Mirna
Hi,
I have an additional question regarding the "prior concentration" option in the Bayesian chi-square window. I read the Jamil et al. (2016) paper as suggested in other comments on the forum, but still do not fully understand this concept. Could you please help me in understanding how this affects the analysis, and how I can determine the appropriate setting for my purposes?
Moreover, since I want to conduct 20 of these Bayesian chi-square tests (comparing the countries on 20 different yes/no variables), is it possible/necessary to correct for this large number of tests?
Hi Mirna,
Sorry for the tardy response. I will take a look at your JASP file soon.
Well the prior concentration determines how close the prior distribution is with respect to the point of equal proportions. For instance, For instance, if we have a 2x2 table (a test of two proportions) and the prior concentration is 1, this means we have a prior distribution that assigns equal mass to all combinations of the two proportions. If the prior concentration is set higher, more mass will be located near values where the two proportions are equal. Think of the prior as a landscape; with a prior concentration of 1, the landscape is completely flat; the higher the prior concentration, the more the prior mass will concentrate on a ridge along the diagonal.
Now about the multiplicity when you test 20 different yes/no variables: if you are a purely subjective Bayesian you don't need to correct for multiplicity -- but you do need to think very deeply about the prior plausibility of each of your tests, and take that into account. For objective Bayesians, there are different ways to correct. I recommend the thesis from Tim de Jong for an overview: https://psyarxiv.com/s56mk
Cheers,
E.J.
Hi Mirna,
I took a look at your JASP file and it seems fine. I am not sure what that message about the proportion test means, but I will look into it. For more examples and background information see https://link.springer.com/article/10.3758/s13428-016-0739-8
Cheers,
E.J.
Hi E.J.,
Thank you so much for the thorough explanation and for checking my JASP file!
I'm glad to hear that the message about the proportion test is not an issue. However, I would just like to check how the "group 1 is different from group 2" hypothesis in JASP extends to my 10-group example: Does the test examine whether all groups are different from each other, or whether at least one stands out from the rest in some way (as would be the case in the frequentist version)?
Regarding the prior concentration, I think I will go with the default option of 1 for all tests as I do not have any specific prior information on what to expect regarding how the proportions will compare to each other. Is this a valid approach?
When it comes to correcting for multiple tests, I saw your answer on this post: https://forum.cogsci.nl/discussion/2020/post-hocs-planned-contrasts-in-bayesian from some years ago. I also looked at Tim de Jong's thesis. From what I gathered, it seemed to me that my analysis plan falls into the category that you describe in the first part of your answer on the link, where all tests are pre-planned to test separate hypotheses, so there might not be a need for correction. Could you please point me to whether I am misunderstanding something?
Best regards,
Mirna
E.J.
I see. So just like with ANOVA, if the test implies a difference, we wouldn't know where exactly the difference is (or whether it is necessarily between all groups) without doing some sort of post-hoc testing?
exactly
E.J.
Thank you so much for your help and clarifications, E.J.!