Reporting measurement invariance

Sinan

Using JASP, I conducted a confirmatory factor analysis on a multi-country dataset, by grouping the sample by country. I would like to report configural, metric, and scalar invariance but all I see in the output is detailed statistics for each country. What value should I report to discuss measurement invariance? I would really appreciate any advice on where to look at and what to report.

EJ

This is a good question. I'll pass it on to our experts. My initial gut-level response was to say "why not use SEM"? But maybe we ought to expand our CFA to make this possible (or maybe the upcoing version can do this -- we'll know soon enough I hope).

Team, this is a relevant reference: https://www.psychologicalscience.org/observer/testing-for-measurement-invariance

Cheers,

E.J.

JMBKoch

This is indeed a good question, and this will be a relatively long answer accordingly.

In order to test for measurement invariance, you need to fit a sequence of models. First, in the configural model all parameters (i.e. factor loadings, intercepts, and residual variances) are allowed to vary between groups, although the factor structure of the model is identical. This means that the same latent variables are measured in the countries, but the way in which they are measured may still differ, such that meaningful comparisons between groups are impossible. To illustrate, think of comparing 2 groups on their average body temperature, where one group has been measured in Celcius and one in Kelvin. Both measures the same latent variable, but a comparison would be meaningless, without transforming one of the two scores first. Next, in the metric (or "weak") invariant model the factor loadings are constraint to be equal across groups. This means that the latent variables share the same 'metric' or scale across groups. This, in turn, allows to formally (by means of a statistical test) compare structural parameters, such as regression coefficients or correlations between countries. Scalar (or "strong") invariance means that also the intercepts (the score on an item when the score on the latent variable is 0, i.e. the 'base-level' scores on the items) are equal across groups. This in turn allows to also meaningfully compare the means of the latent variables between groups. In a strict invariance model, the residual variances of the items are equal across groups too. This means that the groups share the same random measurement error. You will most likely not need this to be true and usually it is not tested for strict invariance in practice.

There are different view points on when we can conclude that there is (a certain form of) measurement invariance. Some say that as soon as the model in question (e.g. one where the factor loadings are constraint to be equal across groups) shows good fit, we can conclude that there is this form of measurement invariance (e.g. metric invariance). This can easily be done using jasp's CFA function. You can just fit the sequence of models by choosing the respecting model under 'Invariance testing' in the 'Multigroup CFA' tab. Then based on the output of the model fit table the fit of the respecting models can be evaluated (here, the additional fit statistics, that you can chose under 'Additional Output' are important too). If a certain model fits, it can be concluded that there is that particular type of invariance.

In the second approach, we don't check whether the model fits well, but whether it fits the data 'as well' as the previous model, by means of a chi-square difference test. The underlying logic is that if a model with the constraints imposed does not fit the data significantly worse than the previous model, the constraints do not make the model a significantly worse simplification of reality than the previous one. Using this test, we thus test the Null-Hypothesis that the respecting parameters are equal across the groups. With a non-significant result, we cannot reject that Null-Hypothesis. Thus, we conclude that there is a particular form of measurement invariance, when the chi-square difference test against the previous model in the sequence is not significant. A big advantage of this latter approach is that in practice you may have a configural model to start off, that already shows poor fit. If you have to continue with such imperfect model for some practical reason, you would then also have poor fit for the subsequent models, and thus per default conclude that there is no measurement invariance. Comparing relative fit might however show that, for instance, the metric invariant model does not fit the data significantly worse than the configural one. This would mean that - while there is no overall good fit - constraining the loadings to be equal might still be an adequate constraint to make. Of course, there may also be cases where the constraint model does fit the data significantly worse than the previous one, while actually having good overall model fit. In such cases you would thus conclude that there is no measurement invariance based one the second approach, whereas you would conclude that there is measurement invariance with the first approach. Such inconsistencies are unfortunately unavoidable when conducing statistical tests to make 'yes-no' decisions, so it's important to keep this in mind.

Unfortunately, the second approach will only be possible using the new SEM module, which as far as I know is about to launch towards the end of August. Here, a CFA model can be fitted using the syntax from the R-package 'lavaan', whereby a grouping variable can be specified under the 'Multiple Group Modeling' Tab. In order to fix parameters to be equal across groups, they need to be given a label in the lavaan syntax. Lavaan interprets a single label as one label for all groups, i.e. as constraining the parameter to be the same across groups. The new SEM module will allow to fit a sequence of models easily, which are then automatically compared with the chi-square difference tests in the output tab. Hence the output you need to formally test for measurement invariance using the second approach will appear automatically, as soon as the right sequence of models is fitted. In the current SEM module, this is unfortunately not as straightforward yet. If you want to try out the second approach, you can try out the beta of the new SEM module using a nightly build of jasp (see https://static.jasp-stats.org/Nightlies/). Alternatively, as a quick fix, you could conduct the chi-square difference test manually. For this you need to substract the Chi-square of the constraint model from that of the unconstrained model (i.e. the chi-square of each model from the one of its previous model), and the same with the degrees of freedom. The resulting values can then be pasted into a webtool such as https://www.danielsoper.com/statcalc/calculator.aspx?id=11 to conduct the final test. If you need further assistance in this, you can best email me under: j.m.b.koch@protonmail.com.

Regards,

Michael

anabelasantos

You also use the CFI, SRMR and RMSEA differences (manually tested) to test measurement invariance between models.

See this reference for guidance

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504.

https://doi.org/10.1080/10705510701301834

All the best,

Anabela