Effect size classical and Bayesian confidence interval -puzzling?
Hi,
I am unable to convert the classical confidence interval to the classical Cohen-d effect size interval.
Paired t-test file( "moon and aggression) from JASP (version0.16) library used to illustrate question
The mean difference is given as 2.433 and this converts correctly to the Cohen-d effect size 1.666
that is: mean diff/SD = 2.433/1.460 = 1.666 (SD=sqrtN*SE N=15, SE=0.377)
However the 95%confidence limits [1.624, 3.241] are given in standarised units for Cohens-d as [0.861,2.447]
how? I expected the Cohens-d limits to be { [1.112, 2.220] that is [1.624/1.460, and 3.241/1.460] }
or equivalently using the t value =2.145 for 14 df
[1.666 - 2.145/sqrt(15) , 1.666 + 2.145/sqrt(15)] or [1.112 , 2.220] which is the same as i calculated above.
My question is what is the formula in JASP for the conversion and why?
More importantly to me is how do I convert the Bayesian analysis(paired t test) Cohen-d effect size credibility interval limits to absolute units. Help ?
This is a major stumbling block to using Jasp but should be simple once formula are known.
The same problem with Bayesian Independent t test conversion of Cohen-d effect size limits to absolute units. Can you possible give guidance how to get over this impasse?
Otherwise Jasp is a great asset and thanks to the team for having this forum.
Thanks for any help you can give.
declan /qdata
Comments
Hi Declan,
In your calculations you're assuming that Cohen's d has the same standard error as the mean difference, which is not the case, since they operate on different scales. Even though it would maybe make intuitive sense to just divide the lower/upper bounds for the mean difference by sqrt(n), this does not work for computing the confidence interval for Cohen's d. JASP uses the computations from the R-package MBESS (https://cran.r-project.org/web/packages/MBESS/index.html) for computing the CI for Cohen's d (the ci.smd function).
I am not sure what you mean with absolute units. Cohen's d and the t-statistic are meant to be not in absolute units, since they are standardized measures. This is why we also include the raw mean difference and CI for that (i.e., those are on the absolute scale). To go from absolute scale to the standardized scale, you (for instance) divide by the standard error and sqrt(n). For instance, 2.433 / (0.377 * sqrt(15)) to go from mean difference to cohen's d. It's just that for confidence intervals this is a bit trickier because of the involvement of the standard deviation/error in the sampling distribution.
I hope this helps, please let me know if any further clarification is needed!
Kind regards
Johnny
Hi Johnny,
Thank you for taking the time to explain, that while the "raw mean difference 2.433" will convert to the correct Cohens-d 1.666, the limits on the confidence won't convert using the same formula. I will of course accept the Jasp wisdom that it is not as simple as I initially thought.
For the second part of my query above I am still stuck.
"More importantly to me is how do I convert the Bayesian analysis(paired t test) Cohen-d effect size: [ 0.737, 2.344 ] credibility interval limits to Raw units."
In applications of Jasp to physics and medically studies the believable range of Raw credible differences can be informative. Hopefully the Jasp team would consider their inclusion in a post data analysis description tables. I understand from your helpful comment above that the scaling of the limits is not simple, so we will have to depend on the JASP team for the "raw data credible range" inclusion in future editions of Jasp. Also in teaching scenarios the lack of the actual credible range of the raw data must be a considerable limitation to the benefits of using JASP.
Thank you again for your expertise and help.
Declan
.
Hi Declan,
Ah yes, thanks for clarifying!
In the case of the t-test, you can look at the 95% credible intervals for the group means (in case of two groups) or the difference with the test value (one sample t-test) by ticking the box "Descriptives". These are on the scale of the variable itself. You could also include the Descriptives plots in your report, which show the group means and their credible intervals.
Kind regards,
Johnny
Hi Johnny,
In the Bayesian paired t-test, yes it gives the credibility intervals in the pre and post measurement sets (or paired samples) but this is not what most researchers want. I would expect what almost every researcher wants is the credibility interval on the "difference pre and post" which is not in the descriptives table. The Prior and Posterior Bayesian density plot gives at the top, the median and the credibility interval on the difference" in standardized units in the. hence the problem remains. Need the credibility interval on the difference in raw units since this is what the researchers are trying to find out. Would it be it possible that the Jasp team would consider including the required credibility interval in future editions. It would be a great addition to Jasp's Bayesian analysis capability. Manually having to calculate a raw credibility for some of the tests greatly takes away the user friendly aspect of Jasp's graphical interface.
Thanking you,
Declan
Hi Johnny,
A brief look into the book " Modern Bayesian Statistics in Clinical Research" by T.J. Cleophas, and A.H. Zwinderman on the Bayesian paired t-test to see how SPSS handles effect size shows that this software gives the credibility interval on the difference in raw units(not standardized). To clarify above request, it would be great if a future edition of Jasp could give the formula in help file or better still if a Bayesian credibility interval could be added to descriptives table for the actual(non-standardized) difference between the groups-compared. Keep up the good work.
Declan
Hi Johnny,
Could I make a further point to clarify; its really the same point as above but hopefully more meaningful and hopefully a bit clearer.
To describe a typical situation: If one is teaching or trying to recommend Jasp to groups that are moving from frequentist to Bayesian paradigm and typically process their results via GUI software, then Jasp seems ideal and there are a lot inspiring and helpful papers to guide the transition to the Bayesian framework from the Jasp team. The researchers who wish to include a Bayesian perspective in their result analysis often come from other disciplines and are not strong on programing languages such as r on python and are depending on Jasp to be able to report their results in a meaningful way. Normally Jasp is good at this in its approach.
However, using the difference of two means as an example: while as you point out, the descriptives table give the 'before' and 'after' values , mean_before, mean_after, standard deviations in the original units, the Bayesian- Credibility of the 'difference of the means in the original units' (not standarized units) or indeed any other way that a researcher could obtain these credibility limits from Jasp, seems to be unavailable. Accessing Jasp r code does seem appropriate as the program is based on being user friendly. The help file does not give give any way to scale from standardized units to the original units of the variable for the credibility interval. The mean difference scales ok and is not a mystery as shown above(via Cohens d). However the scaling for the credibility interval limits are a mystery and include scaling units which are unknown to the user of Jasp. This is like for instance, quoting the length limits of an object but not being able to specify whether the limits are in cm or feet or meters. (sorry to sound dramatic). Am I missing the point or is this an unreasonable expectation to be able to specify the credibility of the difference measurement in the units of the measurement. A solution to this dilemma would help me to recommend Jasp as a friendly pathway to Bayesian analysis. Can you give any further guidance on this problem , if so I would be grateful.
qdata.
I think this does make sense; you could make this a feature request on our GitHub page (for details see https://jasp-stats.org/2018/03/29/request-feature-report-bug-jasp/)
Cheers,
E.J.