Setting Priors in JASP

MSB

Hi all,

From what I understand, one can set the Cauchy prior width acordding to a speculated effect size when preforming a Bayesian t-test, such that if I think my effect should be around 0.6, I would set the Cauchy prior width to 0.6 (correct me if I'm wrong).

What I still don't understand is the Beta* prior width for correlations - can it be used in the same manner?
If I expect an r=~0.6, would I set the Beta* prior width to 0.6?

Or in other words, how do I translate an expected effect size/correlation to Beta* prior width?

Thanks,
Mattan

EJ

Hi MSB,

The best way to understand the shape of the prior is to inspect the plots. A Cauchy prior with of 0.6 still has the center of the mass at zero. When specified as a 2-sided test, this means that 50% of prior mass falls between -0.6 and +0.6. This may sound rather wide but keep in mind that the prior is centered on zero, assigning most mass to values very close to zero. We are working on modules that allow one to specify priors with a location parameter away from zero. Until that time, I recommend to set the width at the default value (at least). Making it smaller causes H1 to behave like H0, leading to an uninformative test.

For the correlation test, parameter rho ranges from -1 to 1. So we take a Beta(a,a) prior (which ranges from 0 to 1), we stretch it so that it ranges from -1 to 1 (this is the Beta* distribution) and then the prior width is given by 1/a. So when you set the prior width to 1/2, you should recognize the shape of a Beta(2,2) distribution.

Cheers,
E.J.

MSB

If I understand you correctly,

If I have a one-tailed hypothesis that rho is (-0.4), I would set the beta* prior to 0.4*2=0.8 and test for a negative correlation - this would give me a distribution with 50% of prior mass between 0.0-(-0.4) and 50% of prior mass between (-0.4)-(-1.0).
Right?

As for t-tests, unfortunately sometimes the default prior is just too wide for some smaller effect sizes (robustly small :-( ).

EJ

Hi MSB,

(1) The interpretation I gave (50% mass between -x and x) only holds for the Cauchy prior, which is used for the t-test. It does not hold for the beta-star prior, which is used for the correlation test. It is best to just inspect the prior distribution visually, which is relatively straightforward in case of rho.
(2) If you really want to test H0: rho=0 against H1: rho=-0.4, then you want a likelihood ratio. The question then becomes, how can you be so sure about the value of -0.4? Anyway, I expect new versions of JASP to include this kind of likelihood ratio testing as well.
(3) For your correlation problem, I would simply use the uniform prior and do a one-sided test, potentially exploring robustness by changing the prior width somewhat.
(4) I cannot stress this enough: if you want to have the alternative hypothesis predict predominantly small effect sizes (so have Cauchy prior width equal to 0.3 or something) you also have to shift the location parameter away from zero. Otherwise the models just become indistinguishable. So there are two opposing forces: (I) a large width can work against H1 when the true effect is relatively small; (II) centering the prior on zero works in favor of H1 when H0 is true. Ideally the user can change both -- this is in the works but not implemented yet. Until it is implemented, my advice is not to stray too much from the default settings. It is fine to go subjective but it can be dangerous to stop halfway.

Cheers,
E.J.

MSB

Hi EJ,

I sure do like the subjective part of Bayesian inference - but I think I understand your point.
Looking forward to all the awesome things coming out in future releases.
Until then - thanks again!

M

ScottPeters

I have a similar question. I'm looking at a Bayes ANCOVA in JASP. I want to try and make sure I'm including as much of an informed prior is possible for my model comparison. I'm testing whether or not a certain level of training results in higher levels of certain practices. There's some prior research to suggest the effect of this training is in the area of .20 - .40 d. So it's greater than a null or uninformed prior. How do I incorporate this prior information into a Bayesian ANCOVA prior in a JASP ANCOVA?

EJ

Hi Scott,
For the t-tests, JASP now has "informed priors" that allow great flexibility in their specification. In particular, they can be centered away from zero (https://arxiv.org/abs/1704.02479). We are working to make the ANOVA and ANCOVA similarly flexible, but this will take some time. I am not 100% sure whether the BayesFactor package in R allows you to implement an order-restriction; in my experience, adding the order-restriction (here: training helps, and does not hurt performance) and setting the prior width together provide a decent approximation to what you would get with a more informed distribution that is not centered on zero.
Cheers,
E.J.

larry

Hi EJ,
I just wanted to check with new informed priors function in JASP. When you use the "normal" section is the "mean" a standardised effect size (Cohen's d), and is the "std" the variability around this effect size (d)?
Also for the "t" section of the informed priors, I have read that the "Oosterwijk Prior" of location 0.350, scale 0.102, and 3 degrees of freedom, is a good model for a small-medium effect. Did you have any other suggestions for a medium-large effect and a large effect?
thanks for any help,
love the program
Larry

EJ

Hi Larry,
1. Yes, the priors are all on delta, the standardized effect size (and sd indicates the uncertainty)
2. We are in the process of eliciting a number of other informed prior distributions.
3. For medium-large and large: interesting question. Of course, with large effects the test becomes redundant (it will be apparent from the data). My first thought is to take the Oosterwijk prior and just shift it. Maybe to 0.650 for medium-large, and 0.950 for large?
Cheers,
E.J.

larry

Thanks EJ,
Really helpful
cant wait for more updates
Larry

cnhstreet

Hi All,

I have a related question. I am reading in quite a few places that the objective/default Cauchy is computationally convenient and gives certain desirable properties for the Bayes factor when the Cauchy scaling is set to 1. JASP uses 1/sqrt(2) for the JZS t-test. I was wondering what has motivated that choice.

Thanks in advance for any help!
Chris

EJ

Hi Chris,

The desirable properties of the Cauchy hold for any scaling. The value of 1 was suggested by Jeffreys but this is not a principled point. The value of 1/sqrt(2) was suggested in the BayesFactor R package to be more reasonable (i.e., more in line with the kinds of effect sizes reported in the literature). The reason why it is exactly 1/sqrt(2) has to do with the generalization to the ANOVA setting.

Cheers,
E.J.

cnhstreet

Hi EJ,
Ah I see, thanks so much for such a quick response! That makes me feel a little more comfortable using an informed prior (noting your points made in other threads about small effect sizes).
Thanks!
Chris

joanne_packer_eaves

Hi,

I was wondering if there was a way to perform a robustness check for ANOVA's and Chi-square tests in JASP? For t-tests it's really straightforward, but there's no option for other types of bayesian test. I was wondering how other people had done this in JASP (e.g. manually changing the priors and then plotting the results), or if there was a reference any where for how to do it?

Thank you,

Jo

EJ

Hi Jo,

No, this has not been implemented yet. It would be straightforward to implement for the chi-square test, but for the ANOVA the problem is that the check may take a very long time. So we probably need to pick just a few points and plot the results for those. If you'd like to make it an issue on our GitHub page we will get to it in due time.
Cheers,
E.J.