RE: SEM in JASP

Comments

• patc3

  August 4

  Can you show a screenshot for a bit more context? I'm in JASP 0.95 now and I don't see what you're describing in the output

  You can sometimes find info in the help file for any particular analysis:

  

  
  

  That being said, the baseline model (unless I recall this incorrectly) is a model where all observed variables are uncorrelated, and the chi2 test is a test of whether a correlation matrix with all 0s (i.e. the baseline model) is different from the actual correlation matrix in the sample. This is basically always significant, as usually there will be some correlations between your observed variables.

  Sometimes you'll see user model (instead of baseline model), this compares the correlation matrix implied by your model to the actual correlation matrix in the sample. Again if it's significant it means the correlation matrix in the sample is different from the one implied by your model (i.e. you're not reproducing the correlation matrix *exactly*--it's a test of exact/perfect model fit). This will often be significant as well, and it depends partly on sample size, such that you're more likely to get a significant test as sample size increases.

  So usually a low p-value for the chi-square test of exact fit is 1) irrelevant for the baseline model (though in reality it's "good", it indicates that some of the variables in your sample are correlated); and 2) "bad" for your own (user) model (it indicates your model doesn't reproduce exactly the observed correlation matrix), but it's the norm more than the exception
• Michael_Jasper

  August 4

  I'm using using JASP version 0.19.1.0.

  No direct info in the help file but has a link (to https://osf.io/xkg3j), which seems to show different output for chi-squared than the below.

  Please note in the screenshot below the heading "baseline test" but on the row it says "Model 1".

  I don't know why I would ever be shown just something for the baseline model as a result, and so I presume this is a comparison of the model to the baseline model (?), but that is a presumption and I just don't know. Is a low p-value here good or bad?

  

  
  
• patc3

  August 4

  OK yes I see the table header now, sorry.

  So I think writing baseline model there is an error, this is the user model, so what I wrote in my previous post still stands. This is a test that compares the correlation matrix implied by your model to the observed correlation matrix (so a low-value is "bad" in the sense that your model doesn't reproduce the relationships between your observed variables exactly, but as I wrote above, it's often significant, and even more so if you have a large sample).

  From the documentation:

  

  The saturated model is a model with 0 df that reproduces the sample correlation matrix exactly, i.e. it's a test of whether your model reproduces the sample correlation matrix exactly (significant = it doesn't, not significant = no evidence that it doesn't).
• Michael_Jasper

  August 4

  Do you think if I was to download a later version of JASP this error (it being labelled "Baseline model") would disappear? What do you think about me just dropping reporting this statistic given this ambiguity and that it seems overly sensitive and so isn't a very good measure surely? Just reporting Goodness of Fit index and maybe a couple of relative measures such as NFI and CFI atop? I'm actually revisiting this work after some time and I don't know where I first got the sense that this chi-squared was a relative measure comparing to the baseline model, I thought that it come from contact with JASP, but I've searched my email and can't pull anything up, so maybe I just made it up, I don't know. All very confusing. Especially given the labelling in JASP itself (as per the screenshot).
• patc3

  August 4

  No the error (if it is one) isn't corrected in 0.95 (the newest version).

  The chi square test of exact fit is almost always reported, so you should probably report it (look at other articles in your field to see what it commonly done). If the other fit indices are OK then the chisquare being significant shouldn't be a big deal.
• Michael_Jasper

  August 4

  When you say official documentation are you talking about: https://osf.io/xkg3j ?

  One thing that confused me about that is, by my reading, its interpretation of Example 1 contradicted that?

  I'm not sure if the comparison is to a completely perfect model or to a completely imperfect model.

  Might there be some margin to triangulate what is being reported by using one of the relative measures such as CFI or NFI?
• patc3

  August 4

  No the documentation directly in the software when you click the i button next to the analysis name in the analysis panel (middle panel of the 3 panels)

  What's the problem with example 1 in the pdf you linked?

  The chi square compares your model to a perfect model (a model that reproduces the sample correlation matrix exactly). It's called a test of exact fit. The null hypothesis is that your model reproduces the sample correlation matrix. If you reject the null hypothesis (low p-value, p<.05) it means your model DOESN'T reproduce the sample correlation matrix exactly, i.e. it's not perfect.

  You should report other indices as well (like you said, CFI TLI GFI, whatever you fancy/is common in your field).

  I'm not sure how to help you more than that. Feel free to read and re-read my posts up to now.
• patc3

  August 4

  Example 1 seems to be interpreted correctly in the pdf you linked:

  (the "baseline model" in newer JASP versions, an apparent mistake):

  

  the corresponding report:

  

  the test is significant, the model doesn't reproduce the sample correlation matrix (capital sigma I assume); they even mention the sensivity of the chi square to large samples. Then they report other indices which indicate adequate fit.
• Michael_Jasper

  September 1

  For the chi-squared test-of-exact-fit in JASP what happens when FIML is used to "gloss over" some missing data? Is this test comparing the model against only the original data OR the original data plus the added synthetic data?

  (thank you for all your help prior, earlier in this thread, it really helped)
• Michael_Jasper

  September 3

  @patc3 @EJ

  Just bumping my last post (directly above) on this thread. Thank you.
• patc3

  September 3

  FIML doesn't impute data, so the chi square has the same interpretation (it is again a test of whether your model is able to reproduce the sample correlation matrix exactly)