Meta-analysis output differs in Jasp compared to metafor (R)

Michelle1 · July 2020

Hello

Does anyone know why JASP and R have given me different output for the same meta-analysis?

For context, I attempted a meta-regression in Jasp after completing it in R using the metafor package. I uploaded my file to JASP after calculating effect size using the escalc function in metafor and then I plugged in the ES (yi) and SE (vi) values into the model.

The meta-regression coefficients seem similar, so I am not too worried about this. However, the heterogeneity estimates differ more substantially. Here are some examples:

I^2 in R: 36.82%; I^2 in JASP: 97.052%

Tau squared in R: 0.036; Tau squared in JASP: 0.097

H^2 in R: 1.58; H^2 in JASP: 33.927

I've been trying to find some more information about this online, but I can't seem to get anywhere. Can someone explain these differences? Since JASP is built using the metafor package, I assumed these would be the same.

Thank you in advance for your insights!

EJ · July 2020

Before I bring this to the attention of our meta-analysis expert: have you executed the metafor analysis on the exact same ES and SE that were available to JASP? So if you take the csv that was analyzed by JASP, you read it into R, and you execute the metaphor package, do you still get different results?

E.J.

Michelle1 · July 2020

Hi E.J.

This is correct. The results differ even though I am using the same CSV file.

I use the rma function in metafor

Thank you for looking into this!

Michelle

EJ · July 2020

OK, thanks for looking into this. Strange. I'll contact our meta-analysis expert.

Michelle1 · August 2020

Any update on this?

Raoul · August 2020

Hi Michelle,

Could it be that you don't convert the sampling variances (vi) into standard errors (SE) by taking a square root? Otherwise, do you have an example JASP file that illustrates the problem?

Raoul

rick_m · August 2020

I have seen a similar apparent error in the JASP meta-analysis output. It is most clearly seen in the calculation of I^2.

I^2 can be calculated simply as 100* ((Q - df)/Q). You can see in the attached screen shot that I^2 appears to be calculated incorrectly. The values shown below for Q (4.309) and df (3) are correct in the upper table (confirmed by manual calculation). Thus, the value of I^2 should be 30.37%, but it is quite different in the lower table (2.421e -4).

Is this an error in JASP, or have I missed something (more likely...)?

Thanks.

Raoul · August 2020

@rick_m Your formula is correct if you use the DerSimonian-Laird estimator (which is *not* the default). See the plot below. I have not checked wether the calculation of I² is correct for the other options, but I have checked whether it is the same output as from the metafor package, and at least for my example that is indeed the case.

Hope that helps.

rick_m · August 2020

Raoul,

Thanks for this helpful explanation!

Rick

rick_m · August 2020

Raoul,

Just to clarify, I had used the default Restricted ML method when I got the results table I posted. Does this mean that when using the Restricted ML method, I² ≠ 100* ((Q - df)/Q)?

Thanks

Raoul · August 2020

Rick,

Indeed it does. I don't know exactly how I² is calculated when REML is used. A good place to find more information is: http://www.metafor-project.org/doku.php

Raoul

Howdy, Stranger!

Categories

Meta-analysis output differs in Jasp compared to metafor (R)

Comments

Howdy, Stranger!

Quick Links

Categories

Meta-analysis output differs in Jasp compared to metafor (R)

Comments