Theoretical questions on JASP and Bayesian approach (RM-ANOVA)

johan_achard

I'd like to share with you some theoretical and practical questions about the two approaches, frequentist and Bayesian, and their use on JASP (especially calculations and interpretations).

First of all, I'm well aware that the methods are different and therefore not necessarily comparable, but the results shouldn't be opposed in the majority of cases, otherwise there would be problems of replication and generalization?

To sum up, frequentists focus on the reliability of the procedures generating their conclusions (the p-value), while Bayesians are interested in the credibility of the hypotheses (Bayes factor). Bayesians look at the H0/H1 ratio to quantify and discuss it.

Do you think that simple one-factor models (which make more accurate predictions) are to be favored for Bayesian RM-ANOVAs, and that currently for RM-ANOVAs with more than two factors the frequentist model would be more reliable/fair?

In this case, we could use Bayesian mixed models rather than B-RM-ANOVA when we have more than 2 VI in repeated measures. But what are the advantages and disadvantages of using one rather than the other?

What's more, Bayesian analysis takes priorities into account. How do you define them for a good anlayse, and how were the default parameters you propose in 'additional options' chosen?

What do you think of the examples "where the use of prior distributions and Bayes factors suggests different conclusions than the classic p-values"?

If the results differ between frequentist and Bayesian, what can and should be reported in the articles and/or how can we justify our choices? The two totally different interpretations?

Furthermore, in the article by van den Bergh et al. (2020) you specify that "the residuals must be normal" (p.80-81), but is this the only assumption for Bayesian RM-ANOVA? Or do Bayesian sphericity tests exist (and would be useful in this approach?) but are not yet available on JASP?

We also noticed earlier (see forum discussion) that Bayesian RM-ANOVA was more sensitive to outliers, how can we see this in JASP quickly?

Thank you for your feedback and your hard work on JASP, it's always very interesting to have your opinion!

Johan

EJ

Dear Johan,

Let me go over your questions one by one:

>First of all, I'm well aware that the methods are different and therefore not necessarily comparable, but the results shouldn't be opposed in the majority of cases, otherwise there would be problems of replication and generalization?

Yes, the methods would generally agree, roughly, due to the interocular traumatic test (when the data are do clear that the conclusion hits you right between the eyes).

>To sum up, frequentists focus on the reliability of the procedures generating their conclusions (the p-value), while Bayesians are interested in the credibility of the hypotheses (Bayes factor). Bayesians look at the H0/H1 ratio to quantify and discuss it.

Yes. Well, the BF is the *change* in credibility (on the odds scale) so the evidence.

>Do you think that simple one-factor models (which make more accurate predictions) are to be favored for Bayesian RM-ANOVAs, and that currently for RM-ANOVAs with more than two factors the frequentist model would be more reliable/fair?

No. In my opinion, the rational/coherent framework is always better. Lindley once compared Bayesian inference to an orchestra, and frequentist inference to a child's rattle. This upsets those who have been using the rattle their entire lives, but I support the general sentiment.

>In this case, we could use Bayesian mixed models rather than B-RM-ANOVA when we have more than 2 VI in repeated measures. But what are the advantages and disadvantages of using one rather than the other?

The B-RM-ANOVA is in fact a linear mixed model.

>What's more, Bayesian analysis takes priorities into account. How do you define them for a good anlayse, and how were the default parameters you propose in 'additional options' chosen?

Generally these options were chosen based on general desiderata. The resulting priors are relatively wide. If you have strong background knowledge it would be a shame not to include it.

>What do you think of the examples "where the use of prior distributions and Bayes factors suggests different conclusions than the classic p-values"?

Those are generally compelling to me. See Lindman, Edwards, & Savage (1963) for instance, and Wagenmakers & Ly (2023 -- https://link.springer.com/article/10.1007/s00407-023-00310-4).

>If the results differ between frequentist and Bayesian, what can and should be reported in the articles and/or how can we justify our choices? The two totally different interpretations?

In those cases I generally advice to report both and acknowledge the uncertainty instead of sweaping it under the rug.

>Furthermore, in the article by van den Bergh et al. (2020) you specify that "the residuals must be normal" (p.80-81), but is this the only assumption for Bayesian RM-ANOVA? Or do Bayesian sphericity tests exist (and would be useful in this approach?) but are not yet available on JASP?

Well the Bayesian model is really a linear mixed effects model. I am not sure whether sphericity is an assumption there.

>We also noticed earlier (see forum discussion) that Bayesian RM-ANOVA was more sensitive to outliers, how can we see this in JASP quickly?

You can't, I think. Would be a good feature request for our GitHub page (for details see https://jasp-stats.org/2018/03/29/request-feature-report-bug-jasp/)

Cheers,

E.J.

johan_achard

Dear EJ,

Thank you for your thorough, clear and comprehensive answers to my questions.

I appreciate your suggestions and help in the Bayesian field,

Johan