Computing internal consistency for ordinal variables in JASP

sharonschroen

Dear community,

Together with two other researchers, I am analyzing the internal consistency of a newly developed self-esteem scale, that exists of 3 questions in the short version and 5 questions in the long version. Question answers are rank ordered, so the questions are of the ordinal measurement level. I used the McDonald's omega to compute the internal consistency of the questions, but I am starting to wonder if this is the best procedure? Is there any other internal consistency I could compute in JASP (or maybe another software), that might be better suitable for our set of questions?

We also have low levels of the McDonald's omega, about .310, which is lower than we hoped for. Of course, this could have something to do with the low number of questions, with the questions not measuring a unidimensional concept or with limited understanding of the questions by respondents (although we consider the last two options not that likely). Has any of you ever struggled with unexpected low levels of internal consistency? What was the reason for that?

Thank you in advance for your response! It is greatly appreciated!

Best,

Sharon

julius

Dear Sharon,

which program or software did you use to compute omega? There is a version of omega (as found in JASP as well) that assumes the scale to be unidimensional. How large is your sample size? And how many levels does your response have? Several reasons for the low value come to mind.

1) With very few items (and possibly small sample size, and few response categories) it is very hard to distinguish systematic variance from error variance (noise)

2) as you already noted, the unidimensional model does not fit (did you check by means of a CFA?) and the scale is not internally consistent. However, it might be that the scale is multidimensional (but that is unlikely given the few items)

Other reliability coefficients are alpha, or lambda2 (can be found in JASP). Those are usually not very different from the omega value, so it might be unlikely that they yield very different results. It might be worthwhile to obtain an uncertainty interval (both Bayesian and frequentist can be found in JASP) for your omega (or alpha) estimate and thereby get a sense for the noise in your data.

Let me know how it goes.

Best,

Julius