Reporting posterior R² and posterior effect size in Bayesian rmANOVA and paired t-tests
Dear JASP team,
I am using Bayesian repeated-measures ANOVAs and Bayesian paired-samples t-tests in JASP and would like to clarify how best to report effect sizes and uncertainty measures within a Bayesian framework.
For the Bayesian repeated-measures ANOVA, JASP provides a “Posterior R²”, described as the posterior distribution of explained variance. I am considering reporting the model-averaged posterior R² together with its 95% credible interval as a model-level measure of effect size/uncertainty.
I would like to clarify a few points:
- What exactly does the model-averaged posterior R² correspond to in Bayesian repeated-measures ANOVA? Is it the proportion of variance explained by the fixed effects, by the full model including random factors, or by the model-averaged set of models weighted by posterior model probabilities?
- Should posterior R² be interpreted as a model-level measure of explained variance rather than as an effect size for a specific factor?
- JASP seems to summarize posterior R² using a posterior mean and 95% credible interval, whereas Bayesian paired-samples t-tests report the posterior median of the standardized effect size δ with a 95% credible interval. Would it be possible to obtain and report either the posterior mean or the posterior median for both quantities? Reporting a posterior mean for R² and a posterior median for δ seems somewhat inconsistent, and I wonder whether a common summary statistic would be preferable for coherent reporting.
Many thanks for your help and clarifications !
Johan
Johan A. ACHARD
ATER - PhD student in Cognitive Sciences
Université Marie et Louis Pasteur, INSERM, UMR 1322 LINC, F-25000 Besançon, France.
Comments
Forwarded to our expert...
EJ
Hi Johan,
What exactly does the model-averaged posterior R² correspond to in Bayesian repeated-measures ANOVA? Is it the proportion of variance explained by the fixed effects, by the full model including random factors, or by the model-averaged set of models weighted by posterior model probabilities?
Specifically, we compute the predictions like so:
First we sample from the models,
M_s ~ p(M | y)
then we sample from the parameters given a particular model
θ_s ~ p(θ | y, M_s).
For each posterior draw, we make predictions for that model
ŷ_s = X_{M_s} θ_s.
and then we compute the explained variance.
R²_s = Var(ŷ_s) / [Var(ŷ_s) + Var(y - ŷ_s)].
So the reported posterior distribution of R² is based on repeated draws from the model-averaged posterior predictive fit. This includes all fixed and random effects included in the (sampled) model. Because these models are sampled from the posterior they are indeed weighted by posterior model probabilities.
Should posterior R² be interpreted as a model-level measure of explained variance rather than as an effect size for a specific factor?
Yes. It should be interpreted as model-level predictive fit or explained variance. Not as an effect size for a specific factor.
JASP seems to summarize posterior R² using a posterior mean and 95% credible interval, whereas Bayesian paired-samples t-tests report the posterior median of the standardized effect size δ with a 95% credible interval. Would it be possible to obtain and report either the posterior mean or the posterior median for both quantities? Reporting a posterior mean for R² and a posterior median for δ seems somewhat inconsistent, and I wonder whether a common summary statistic would be preferable for coherent reporting.
Yeah that's on me. This is not possible right now, but if you would like to see this in a future version of JASP please create a feature request here.
I hope this clarifies things, please let me know if anything else is unclear!
Cheers,
Don
Thank you very much for your quick reply !
Your explanation is very clear and detailed, thank you.
Regards,
Johan
Johan A. ACHARD
ATER - PhD student in Cognitive Sciences
Université Marie et Louis Pasteur, INSERM, UMR 1322 LINC, F-25000 Besançon, France.