Custom Contrasts in JASP RM Anova: different df calculation (and error pooling) in 0.14.1 and 0.16.3

Farina

As I am currently running a replication study, I wanted to replicate some MIXED-ANOVA results from last year (JASP version 0.14.1). However, when I did so using my actual version of JASP (0.16.3), the same input led to strongly changed degrees of freedom and error terms in the tests for custom contrasts. Here are the results:

results0.14.1.pdf

results0.16.3.pdf

I thought both analyses should reproduce the same results (they were performed on the same data-set). But they differ regarding dfs and errors for the custom contrasts (but not in any other parameters). Does anyone know if JASP changed something in the calculation of contrasts for MIXED Anova? And if so: It is documented somewhere, so that I can refer to it? What is the difference between the two approaches?

I also posted a potential bug on GitHub, but did not get an answer yet: https://github.com/jasp-stats/jasp-issues/issues/1946

But because time is a bit pressing I wanted to ask here again if anyone has an idea or an explanation for this behavior of JASP?

Thank you very much in Advance!

EJ

I'll ask our expert!

Cheers,

E.J.

Farina

Dear E.J.,

Thank you very, very much!!! I am already looking forward to your reply!

Farina

JohnnyB

Hi @Farina ,

Since version 16, we use the contrast function of the emmeans package (see https://rdrr.io/cran/emmeans/man/contrast.html) to compute the df's/SE for custom contrast analyses, which is an improvement on how it was done before. The difference in df and SE causes a slight difference in the test statistic/p-value, as you noted, although I would not say qualitatively different results.

Kind regards,

Johnny

Farina

Hi @JohnnyB ,

thank you very much for your answer! Could you tell me, what package was used before version 16? So that I am able to compare? I think, I do not understand the different procedures fully by now (one is more t-test related, the other more a multivariate approach, I assume), but if I have both references, I hopefully am able to work this out for me.

Either way, though, it's good to know that these discrepancies are actually due to a deliberate and intentional change in calculation on JASP's part and are even an improvement (and were not some problem on my computer).

Best regards,

Farina

JohnnyB

Hi @Farina ,

Your assumption is correct - before we were doing manual t-tests (so no particular package), with corresponding df's, so I was also glad we were able to upgrade the procedure.

Kind regards

Johnny

Farina

Hi @JohnnyB ,

again, thank you very much for your reply! That helped me a lot!

And also thanks to you and all the persons who are constantly developing JASP and providing us with such a great tool!

Best regards,

Farina