Correlation Matrix, multiple comparisons and non-parametric data
Hello.
I have been given a large data-set and plan to perform some exploratory analyses for publication. I want to present these analyses as such, and can see that the frequentist approach is problematic. My undergrad studies were all SPSS based, so I'm trying to teach myself some alternatives, but this is no easy task! I'm hoping for a little help in these early days please!
I want to illustrate the relationship between one questionnaire score and several other questionnaire scores. The data are all ordinal. I also want to control for a couple of nuisance variables. As there is no partial correlation function yet in JASP, I have calculated the standardized (Z) residuals using my nuisance variables, for each of my questionnaire scores.
My questions are:
1) Are there any issues if I run the Kandall's tau-b Bayesian correlation matrix with the Z-residuals? (or can anyone point me to any references on partial correlation, as I've not found anything explicit)
2) What are peoples thoughts on presenting a correlation matrix where both the frequentist info and Bayes factors? I'm guessing that many readers in the Psychology world will not currently be able to interpret the Bayes factors yet; however, I will use Wetzels and Wagenmakers (2012) evidence categories in the descriptive text also.
3) If it doesn't seem too crazy to present the frequentist values, I plan to identify which p-values survive Bonferroni correction. Is there any explicit instruction on how to consider multiple tests in the Bayesian method?
Thank you for helping. I've looked through the forum, but not found an answer yet.
Graham.
Comments
Hi Graham,
(1) I am not sure whether those Z-residuals are ordinal, or how they have been obtained. At any rate, regular Pearson correlations on ordinal data become less problematic the more data you have. But JASP now has Bayesian Kendall's tau as well, for ordinal data.
(2) I think reporting both metrics is sensible and transparent.
(3) The multiplicity correction is discussed in Scott & Berger (2006, 2010). It is not implemented in JASP yet, but it has high priority for us.
Cheers,
E.J.
Thanks for your reply E.J. That's really helpful. I'll look up the paper you suggest.
For the Z-residuals, I did this as there is no partial correlation option - but I need to control for 3 nuisance variables. I regressed my 3 nuisance variables (IVs) against the score from my questionnaire of interest (DV), and calculated the Z-residuals from this. I did this for each questionnaire that will be in the correlation matrix. I then used these values, instead of the original questionnaire scores, as input for the kendall's tau-b (some of the variables are not normally distributed, as well as originally being ordinal). Does this sound a sensible workaround?
Thank you to you and the team for this excellent tool.
Yes, it is probably not perfect, but looks like an acceptable workaround. Perfect would be ordinal regression I guess, where you first enter the nuisance variables and then see whether the addition of the variable of interest matters.
E.J.