Meta-analysis output differs in Jasp compared to metafor (R)

Michelle1

Hello

Does anyone know why JASP and R have given me different output for the same meta-analysis?

For context, I attempted a meta-regression in Jasp after completing it in R using the metafor package. I uploaded my file to JASP after calculating effect size using the escalc function in metafor and then I plugged in the ES (yi) and SE (vi) values into the model.

The meta-regression coefficients seem similar, so I am not too worried about this. However, the heterogeneity estimates differ more substantially. Here are some examples:

I^2 in R: 36.82%; I^2 in JASP: 97.052%

Tau squared in R: 0.036; Tau squared in JASP: 0.097

H^2 in R: 1.58; H^2 in JASP: 33.927

I've been trying to find some more information about this online, but I can't seem to get anywhere. Can someone explain these differences? Since JASP is built using the metafor package, I assumed these would be the same.

Thank you in advance for your insights!

EJ

Before I bring this to the attention of our meta-analysis expert: have you executed the metafor analysis on the exact same ES and SE that were available to JASP? So if you take the csv that was analyzed by JASP, you read it into R, and you execute the metaphor package, do you still get different results?

E.J.

Michelle1

Hi E.J.

This is correct. The results differ even though I am using the same CSV file.

I use the rma function in metafor

Thank you for looking into this!

Michelle

EJ

OK, thanks for looking into this. Strange. I'll contact our meta-analysis expert.

Michelle1

Any update on this?

Raoul

Hi Michelle,

Could it be that you don't convert the sampling variances (vi) into standard errors (SE) by taking a square root? Otherwise, do you have an example JASP file that illustrates the problem?

Raoul

rick_m

I have seen a similar apparent error in the JASP meta-analysis output. It is most clearly seen in the calculation of I^2.

I^2 can be calculated simply as 100* ((Q - df)/Q). You can see in the attached screen shot that I^2 appears to be calculated incorrectly. The values shown below for Q (4.309) and df (3) are correct in the upper table (confirmed by manual calculation). Thus, the value of I^2 should be 30.37%, but it is quite different in the lower table (2.421e -4).

Is this an error in JASP, or have I missed something (more likely...)?

Thanks.

Raoul

@rick_m Your formula is correct if you use the DerSimonian-Laird estimator (which is *not* the default). See the plot below. I have not checked wether the calculation of I² is correct for the other options, but I have checked whether it is the same output as from the metafor package, and at least for my example that is indeed the case.

Hope that helps.

rick_m

Raoul,

Thanks for this helpful explanation!

Rick

rick_m

Raoul,

Just to clarify, I had used the default Restricted ML method when I got the results table I posted. Does this mean that when using the Restricted ML method, I² ≠ 100* ((Q - df)/Q)?

Thanks

Raoul

Rick,

Indeed it does. I don't know exactly how I² is calculated when REML is used. A good place to find more information is: http://www.metafor-project.org/doku.php

Raoul