95% CI for Cohen's d in post-hoc tests

vicente_inefo

Is it possible to report a pBonf = 0.03 with a 95% CI that contains a zero value?

I attach a screenshot with the values reported by JASP, please see second row of the attached table.

Thanks in advance.

JohnnyB

Hi @vicente_inefo ,

My guess here is that the p-values are for the t/mean difference value (you can see that the CI for the mean difference does not include 0), which use a slightly different standard error for their standardization and CI computation, which can result in these types of fringe cases. I would think that if the result is this close, there is not very substantial evidence for a difference between Match and TriveBk.

Kind regards,

Johnny

vicente_inefo

Thanks for the reply @JohnnyB ,

Then I think is better not to report the Cohen's d in the paper we are writing because it could be confusing, isn't it?

JohnnyB

Hi @vicente_inefo ,

I think this is a problem that stems from binary reasoning about statistical results - if you hold the 0.05 threshold as very holy, and a p-value below that as plain evidence for a difference, then yes, this Cohen d CI is a bit confusing because it illustrates that there is just not very much evidence in your data for the difference. I would report the full results, and curb the conclusion a bit (i.e., "weak evidence" or "tentative") about the difference, but of course this is very easy for me to say, being at the sideline and all ;-)

Kind regards,

Johnny

vicente_inefo

I understand your point but the reviewers, as a general rule, tend to "suggest" the use of this binary reasoning about statistical results. It's a pitty.