Post Hoc tests for interaction in Bayesian repeated-measures ANOVA

EmilieC

Hello!

I am using JAPS for my analyses, but I have two main elements that appear to be missing in the last version (JASP 0.17.2.1). But perhaps there is another way to do them.

1. My first problem is that in non-bayesian repeated-measures ANOVA, the marginal means display some possible interactions, but not all (see print screen). For instance, I have a significant triple interaction category*perspective*PTSD, but it does not appear in the options for the marginal means.
2. I wanted to run the same versions of the analyses but with the Bayesian version. However, although in the non-bayesian there is the possibility to compute contrast, this option does not appear in the bayesian version (see print screen attached). I know I can run the Bayesian version of independent sample t-tests and paired sample t-tests to overcome the fact that there is not the contrast option available. However, the issue is that I need to run this analyses with a covariate (Age), which is possible to introduce in the repeated-measure ANOVA (bayesian and frequentist), but this covariate can't be introduce (as far as I know) in the paired and independent sample t-tests.

Would you have solutions for these two points?

Capture d’écran 2023-07-03 à 16.36.10.png

Capture d’écran 2023-07-03 à 16.35.52.png

Thank you very much!

Emilie

EJ

I've forwarded this to our expert!

E.J.

JohnnyB

Hi @EmilieC ,

1. You are right - the interactions have disappeared from the marginal means menu and we are currently working on bringing those back!
2. There is a difference between the frequentist and Bayesian anova in how they handle the posthoc tests (contrasts not being available in the bayesian anova). In the frequentist anova, the posthocs and contrasts are based on the estimated marginal means (based on the full model, so also covariates - you can read more about that here). The Bayesian anova simply runs individual t-tests (as mentioned in the table footnote), which means that there it does not matter whether there are any covariates included or not. I will discuss with another team member how feasible it is to switch to a marginal means implementation in the Bayesian anova.

Kind regards

Johnny

EmilieC

Thank you for your answer.

Regarding point n°2, I don't know if you have seen this other post, but it looks like there is an issue with models including covariates, and I don't know at the moment if I can reliably use JASP for my covariate analyses

https://forum.cogsci.nl/discussion/8677/incongruency-main-effects-vs-post-hoc-comparisons#latest

JohnnyB

HI @EmilieC ,

Yes, that one is definitely related, but it's a feature, rather than a bug (and consistent with other analysis software). The blogpost I linked explains this property more clearly, including an example.

Cheers

Johnny

EmilieC

Thank you for your answer,

I have post hoc comparisons for repeated-measures ANOVA, and I am asked to add the associated Bayes Factors. I had consider two options, which apparently are not reliable:

1) If I understand correctly, the Bayesian version will not provide reliable results, as not calculated on the marginal means (at the post hoc option only offers some main effects, not the contrasts for the interactions).

2) If I conduct Bayesian paired comparisons, same issue, as not on marginal means.

So at the moment, there is no proper correspondance between Post Hoc comparisons with the frequentist approach and with the Bayesian approach in JASP? Or is there a third option I forgot to consider that would provide me with these BF?

Thank you

andersony3k

@EmilieC I definitely would call myself a non-expert in Bayesian analysis, especially concerning complex linear models. Of course, even non-experts like myself need sometimes need to make scientific/statistical judgments that go a bit beyond our expertise. It is my judgment that, relative to frequentist models, simple Bayesian models (e.g.,Bayesian t tests) tend to be more extensively worked-out than complex (e.g., ANCOVA) Bayesian models. Therefore, I am more to rely on a Bayesian t test than a Bayesian ANOVA.

Here's my suggestion for some work that someone should do (maybe even get a dissertation out of it?): Figure out a way to generate not just a set of model estimates (estimated marginal means, estimated regression slopes and intercepts, etc.), and not just a set of observed deviations from the model (e.g., residuals), but instead, a *full estimate of what each data point in the data set would be* if the model were correct.

I imagine that such a "full estimate could be obtained simply by adding random noise to the estimated model parameters, except that the whole process would need to be iterated multiple times (to avoid a situation in which the result depends what particular random deviates happen to be added to the model parameter).

Because the full estimate would be a complete data set, it would then be straightforward to conduct Bayesian post-hoc or planned comparisons: Just conduct bunch of individual Bayesian t tests on the desired subsets of the estimated (not actual) data set.

JohnnyB

Hi @EmilieC

If your design is balanced (also in terms of continuous covariates), then the marginal means will be at least highly similar to the descriptive means, so then the paired t-tests will give an accurate idea of the marginal means differences. It's a bit hard to say without having your specific results/data, so I can only give general guidelines here.

@andersony3k that's a great technique indeed, and is known as "posterior predictive check" - where you generate data under a certain model and see how well they match the observed data. So nice intuition ;-)

Kind regards

Johnny