Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

What denominator does the Cohen's d use on JASP??

As I'm getting different results when calculating Cohen's d with SD Pooled as the denominator to the results JASP is giving me.
I just want to make sure I'm reporting it correctly in my write up.

Cheers,

Comments

  • I'll ask Johnny what's up. I think the issue came up before. Do you have a concrete example?
    Cheers,
    E.J.

  • Here's a couple of examples with the JASP d and the SD Pooled d (calculated using Lee Becker's website):
    M1 = 66.5 ± 72.8 M2 = 36.5 ± 30.7 JASP d = 0.84 SD Pooled d = 0.54

    M1 = 10.3 ± 14.8 M2 = 2.3 ± 4.1 JASP d = 0.54 SD Pooled d = 0.73

    As you can see, they're coming up with vastly different results, both increasing and decreasing the ES, some even changing thresholds (from moderate to large for example).

    Cheers

  • Hi Biva,
    I just ran some examples through both JASP and Lee Becker's calculator, and it seems that they provide the same output whenever the sample sizes in both groups are equal. If the sample sizes between groups differ, they indeed provide different results.
    JASP calculates Cohen's d with the following R-code:

    num <- (ns[1] - 1) * sds[1]^2 + (ns[2] - 1) * sds[2]^2
    sdPooled <- sqrt(num / (ns[1] + ns[2] - 2))
    d <- as.numeric((ms[1] - ms[2]) / sdPooled) 
    

    Whereas Lee Becker does the following:

    sdPooled <- sqrt((sds[1]^2 + sds[2]^2)/2)
    d <- as.numeric((ms[1] - ms[2]) / sdPooled) 
    

    So the latter does not weigh the standard deviations by sample size to determine the pooled standard deviation. My guess would be that Lee Becker's calculator rests on the assumption that the sample sizes are equal.
    See also here:
    https://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d
    http://stackoverflow.com/questions/15436702/estimate-cohens-d-for-effect-size
    Andy Field - Discovering Statistics using SPSS

    Cheers,
    Johnny

  • Hi, thanks for the reply Johnny.
    It makes sense everything you've said, but my sample sizes are equal. I double checked to see if I'd missed any participants from the data set, but they're all there.

    Would it be possible to send you a Jasp file with the data set via Dropbox or something?

  • I've also just ran the following through a couple of other online calculators and they all agree with the 0.54 result:
    M1 = 66.5 ± 72.8 M2 = 36.5 ± 30.7 JASP d = 0.84 SD Pooled d = 0.54

    I obviously don't want to report incorrect results, and I'm getting concerned about the rest of my results as well at the minute.

  • Hi Biva,

    Yes, please send us your data set. You can just Email it to EJ.Wagenmakers@gmail.com and I'll forward it to Johnny.

    E.J.

  • Hi,
    that's great, I've sent them from C Kirk

    Cheers

  • Hi Biva,
    Thanks again for your help! I took a look at the JASP files you sent us. I was under the impression that you were conducting two-sample t-tests, as you reported the means and sd's for both groups. However, in the JASP files I saw that you are conducting paired samples t-tests.
    This has repercussions for the way Cohen's d is calculated. For two-sample t-tests, the formula's I described previously are used, but for paired sample tests, the following formula/code is used:

    d <- mean(c1 - c2) / sd(c1 - c2)
    

    where c1 and c2 refer to the scores of the first sample and second sample respectively, so now the denominator is the standard deviation of the difference scores (instead of the pooled standard deviation). One can also compute Cohen's d from the paired samples t-statistic:

    d <- t / sqrt(n)
    

    For instance, for your test on the variables WinSigStrLanded and LossSigStrLanded, the paired samples t-statistic equals 4.443, with n = 28. Now, 4.443/sqrt(28) = 0.84.
    See also:
    https://stats.stackexchange.com/questions/201629/cohens-d-for-dependent-sample-t-test
    http://mandeblog.blogspot.nl/2011/05/cohens-d-and-effect-size.html
    https://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d (the first part of that section is about Cohen's d for two-sample t-tests, and the second part about Cohen's d for paired samples t-tests)
    Please let me know if you have any additional questions, I'd be happy to help!
    Cheers,
    Johnny

  • Ah that explains a lot! That's really helpful, thankyou!

    Further proof - as if it was needed - that I have a lot more reading and learning to do as well!

    Thanks again!

  • Hi Biva,
    Happy to help, I'm glad that it got sorted out!
    Cheers,
    Johnny

  • Hi,

    Sorry for opening again this discussion, but I wonder why JASP team decided to compute Cohen's d for paired samples the way JohnnyB described, with the standard deviation of the difference scores (SD_diff) as denominator. Cummings (2012) argued against that and Dunlap, Cortina, Vaslow and Burke (1996; Psychological Methods) showed that computing d from the t statistic largely overestimates the effect size because it does not account for the correlation between samples.

    I just tried with the data attached (separator is ";").

    • In JASP, paired samples t-test shows Cohens d = 0.780, t = 2.466, and r = 0.962.
    • Following the formula in Table 1 of Dunlap et al. (1996), d = t[2(1 - r)/n] (note the correlation in the formula), we get d = 2.466* sqrt(2* (1-0.962)/10) = 0.215
    • If Cohen's d is computed following Cumming (2012, formula 11.10) and using SDPooled as denominator as JohnnyB posted above (either formula, because n is always the same for paired samples):
      sdPooled <- sqrt((sds[1]^2 + sds[2]^2)/2)
      you get d = (mean1 - mean2) / sdPooled = 5.2 / 21.90 = 0.237.

    If you correct d = 0.237 for bias and get Hedge's g, then you get d_unbiased = 0.217, very similar to the outcome with the formula of Dunlap et al. (I may have lost some decimals here and there). In any case, d is more similar using sdPooled and the Dunlap et al. (1996) formula than with JASP and SD_diff.

    In general, it seems to me that sdPooled may be a better denominator than SD_diff for paired samples, but I wonder what's the opinion of the JASP team in this matter.

    Cheers,

    Karlos

  • Comparing two, small different sized groups the Independent Samples T-Test returns an effect size, Cohen's d =-0.916. Putting the same data into the effect size calculator gives a different answer, Cohen's d = 0.944853. This would suggest that the T-Test is returning the value for Hedges' g and not the advertised Cohen's d.
    With the same data it is possible to run an Independent ANOVA. The post-hoc test gives a Cohen's d of -0.453. There is a note that Cohen's d does not correct for multiple comparisons but here there is just the one comparison so why the different score?
    (I apologise in advance if I am just showing my ignorance of basic statistics and effect sizes.)

  • Hi Cscmac,
    Interestingly enough there are multiple ways of calculating cohen's d. The difference lies in how the standard deviations are pooled. In JASP we have chosen to use the same method as applied by the effsize package in R:

    library(effsize)
    dat <- read.csv("/home/johnny/Downloads/vbpggdw47ucs.csv")
    cohen.d(dat$time..s. ~ dat$sport, hedges.correction = FALSE) # gives -0.9157262
    cohen.d(dat$time..s. ~ dat$sport, hedges.correction = TRUE)  # gives -0.8790971
            #   This is the relevant JASP code, where n1 is sample size, sd1 is standard deviation, and m1 the mean of group 1:
    num <-  (n1 - 1) * sd1^2 + (n2 - 1) * sd2^2
    sdPooled <- sqrt(num / (n1 + n2 - 2))
    if(test==2){ # Use different SE when using Welch T test
      sdPooled <- sqrt(((sd1^2)+(sd2^2))/2)
    }
    d <- (m1 - m2) / sdPooled # Cohen's d
    

    As you can see, the standard deviations are treated differently, based on whether the Welch t-test is used or not. If you go to JASP and tick the Welch test, you can see that the resulting effect size matches the one you found with the online calculator.
    As for the discrepancy with the ANOVA post-hoc test - this was a bug and it has been fixed in a later release of JASP.
    Kind regards,
    Johnny

Sign In or Register to comment.