Using Generalized Linear Mixed Model for Signal Detection Theory
Wright et al. (2009, Behavior Research Methods; https://link.springer.com/article/10.3758/BRM.41.2.257) and others have suggested calculating signal detection theory indexes using multilevel generalized linear mixed modeling with probit as link function. The approach sounds very interesting and could be a great complement to GLMM approaches. I tried replicating example findings using GLMM in JASP (see https://rstudio-pubs-static.s3.amazonaws.com/480255_9baa652276b540d0a239188b9513a026.html#(6) or https://mvuorre.github.io/posts/2017-10-09-bayesian-estimation-of-signal-detection-theory-models/#example-data), but results are very different. Possibly the R code of Wright (mlmsdt) works different than what JASP uses. Is there another way to conduct SDT using GLMM in JASP and/or could it perhaps become a new JASP module?
Kind regards,
Vincent.
Comments
Dear Vincent,
The GLMM approach to signal detection theory is indeed interesting, and it definitely would be a nice addition to JASP. I'm unsure whether we currently have the capacity to implement this analysis, but I added it to a list for future development. I can't promise anything, but you can track the issue here: https://github.com/jasp-stats/jasp-issues/issues/1358
Regarding the issues reproducing the GLMM analyses outlined above, could you provide more details regarding the mismatched models? I quickly checked the first GLMM model outlined in the second link and the results matched exactly (you can download the JASP analysis file from my google drive: https://drive.google.com/file/d/1ajwZW-X3EV-zoJJh_vqiTLzYa25IUyf4/view?usp=sharing).
Note please that by default JASP uses a different contrast coding than R, as mentioned in the note under the output and the help files (since the module was designed for analyses of factorial designs):
The analysis uses sum contrast encoding for categorical (nominal and ordinal) predictors (R uses dummy encoding by default). This scheme is used for better interpretability of models with interactions. However, the fixed and random effects estimates will differ from those obtained from R with default settings. We advise using the 'Estimated marginal means' section for obtaining mean estimates at individual factor levels. For comparing the mean estimates, use the contrasts option.
To bypass the automatic re-coding of contrasts, you have to change the 'isold' variable to a scale. The 0, 1 values will be then treated as the actual levels.
Cheers,
Frantisek
Dear Frantisek,
thank you very much for your efforts and the nice explanation. I had not considered the Bayesian GLMM approach yet, since I could not replicate the results for subject 53 with the "conventional" GLMM as described in the links in the previous message. As I am relatively new to using GLMM, I am finding out about usage and interpreting parameters on the fly. Your explanations and analysis file help a lot.
Thanks also for adding the analysis to the list of possible future features. Perhaps if other uses express their interest as well, the chances of it becoming a part of JASP will increase.
Kind regards,
Vincent.
Dear Frantisek,
I would like to expand the issue a bit. The previous issue was a step up to my main aim, which is to use linear or Bayesian GLMM to analyse recognition responses in a signal detection theory way (see previous posts) across different within-subject factors. Apart from overall model fit and population-level estimates of c and d', I would also like to be able visualize estimates of individual subjects and crossed factor levels. When running the analysis (using MCMC GLMM, as you also used), JASP raises an error message after adding the crossed factors. Running the model for each level separately gives me interpretable estimates for c and d' but I lose the ability to statistically compare between levels / factors. I also cannot find how to estimate / visualize subject-specific (random effects) estimates. It's possible that as newbee to this type of model I lack the insight in how to properly conduct the analysis I want, but perhaps some features that I am looking for are not yet supported in JASP?
Any advice on how to tackle this problem would be very helpful.
Kind regards,
Vincent.
Hi Vincent,
Crossed factors should not be an issue in the current JASP implementation - could you please send me a JASP file with the analysis that produced the issue? We will be releasing a new version in a few months so I can get it fixed.
You can visualize the individual subjects estimate across different levels of the factor in the Plots tab. The option aggregates the values for the individual levels in the selected Random effect grouping factors for each level of the selected fixed effects.
Best,
Frantisek
Dear Frantisek,
the .jasp datafile containing the data and the MCMC GLMM models can be downloaded at https://drive.google.com/file/d/1LK221kUYHWS9BFUyYDHmot7P9DRi20i_/view?usp=sharing. I added some notes clarifying the experimental design and the variables. The failing model with one added factor is also included in the .jasp file. Thank you again for your help.
Kind regards,
Vincent.