Effectsize comparison with Frequentist and Bayesian Mann Whitney test

Mirjam

Hi all,

I am trying to interpret the results from a one sided Mann Whitney test. I have conducted a Bayesian and a frequentist analysis. The median and CI in the posterior distribution of the Bayesian Mann Whitney differs from the effect size given by the Rank biserial correlation in the frequentist analysis. How can I explain these differences? In my case

comparing group A with B

rrb=0,358 CI (0.199, ∞) corresponds with Bayesian posterior median:0.557 CI (0.21, 0.926)

Comparing group B with C

rrb=0,040 CI (-0.227, ∞) corresponds with Bayesian posterior median: 0,189 CI (0.009, 0.595)

Any ideas how I can interpret these differences. Thanks for your suggestions!!

Mirjam

EJ

Hi Mirjam,

Our Bayesian Mann-Whitney expert is enjoying a holiday, but I have attended him to your post!

Cheers,

E.J.

Mirjam

Thanks a lot!!

Mirjam

andersony3k

Best to wait for the expert to return. However, in playing around with some numbers, it's clear to me that the median 'effect size' in the posterior plot is not the median rank-biserial correlation. This is because notice that the median 'effect size' sometimes goes well above 1.0, whereas the rank-biserial correlation is constrained to vary between -1 and 1.

JohnnyB

Hi @Mirjam ,

As @andersony3k rightfully pointed out, there are different effect sizes at play here. The rank-biserial correlation is a correlation and is bound to be between -1 and 1, while the effect size for the Bayesian Wilcoxon is Cohen's d based on data imputation, so can also be greater than 1.

The method is explained in our paper here, in case you want to read more.

Kind regards,

Johnny

Mirjam

Hi Johnny,

I hope you had a great holiday!

Thanks so much for your answer and the paper! Thanks also to @andersony3k

Kind regards,

Mirjam

andersony3k

@JohnnyB Thanks. From the paper, I'm seeing that the latent-normal assumption applies when you have ranked data that might well have been normal had they not been converted into ranks. However, I believe a major motivation for performing a Mann-Whitney test is when the analyst has good reasons not to accept the normality assumption that underlies a t t-test. In that common situation, latent normality assumption has largely been rejected already, prior to subjecting the decidedly non-normal data to ranking. So in this latter situation, JASP's Bayesian approach wouldn't be quite applicable. Is that correct?

EJ

Let me speak for Johnny, as I know he's busy teaching. The assumption (and yes, it is a parametric assumption) is that the latent distribution is normal. But my current thinking is that this is a relatively mild assumption, since it applied also to any monotonic transformation of the raw data. Suppose for instance your RTs are not normally distributed, but heavily skewed. The latent normal distribution could be on "drift rate" in a diffusion model; this translates to a non-normal distribution on RTs. Another way to view this is that you can take the non-normal raw data, apply any monotonic transformation to achieve (approximate) normality, and then assume that all you know are the ranks.

EJ

andersony3k

@EJ

OK. But maybe the situations in which you need non-parametrics the most are when such a transformation is not possible because the two non-normal distributions have different shapes (as illustrated below).

EJ

Yes, but in these situation a rank-based analysis would not be a good test either...

andersony3k

I think a rank-based analysis is fine in these situations as long as one doesn't treat the test as going beyond ranks in any way. The statistical inference concerns whether, in the population, the Group A values tend to rank differently (i.e., higher or lower ) than do the Group B values. (Nothing to do with means, medians, or anything besides ranks.)