#### Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

# Use Bayes posterior as prior for next null hypothesis test

Hi,

I have a number of experiments essentially testing how two experimental conditions do not differently influence the dependent variable. Thus, I am interested in the BF01 I guess. As I conducted the experiments one after the other, it would be fantastic if I could use the posterior of the first Bayesian analysis as the prior for the following analysis, and so on (it's four experiments so far). This way I hope to get a good estimate of how the two conditions do or do not differ. And I'd love to use JASP, because I'm new to using Bayesian statistics.

Now my question is: is this possible to use a previous posterior as a prior for the next (null hypothesis) test? And how would I proceed here?

Cheers, Elena

• edited August 2023

My understanding is that normally you should look at the Bayes Factor from Study 1 and multiply it by 1/1 if the prior odds of the alternative versus the null hypothesis were 1/1 {50/50}. This yields the posterior odds (which is of course identical to the Bayes factor when the prior odds are 1/1). Subsequently, in Study 2, if you're re-examining the exact same hypotheses (I think many people would reject my 'exactness' requirement), you should take the posterior odds derived from Study 1 and multiply that by the Bayes Factor found in Study 2. The net result is the new, Bayesian-updated posterior odds. (If you're working with BF[10] rather than BF[01], it's the posterior odds of the alternative hypothesis relative to the null hypothesis.)

R

• Hi,

thanks for your reply! I'm afraid I need to delve a bit more into understanding the details of Bayesian statistics to know exactly what you mean.

I thought that I could change the default values given in JASP under "Prior" into something that reflects my results from the previous experiment. Does that make sense?

Best, Elena

• I believe that the "prior" options in JASP are not options that allow the user to specify prior odds that are different from 1/1. Instead, you should do the following . . .

If (for example) Study 1 yields BF[10] = 2.3, and Study 2 (a replication of Study 1) yields BF[10] = 1.9, then subsequent to the Study 2 analysis, you should compute (by hand or otherwise) 2.3 * 1.9 = 4.37 which equals the "posterior odds." Thus, having gathered and analyzed the data data from the two studies, the likelihood of the alternative hypothesis being true is now 4.37 times the likelihood of the null hypothesis being true.

R

• This sounds pretty cool and straightforward. Am I correct to assume that it works just the same with BF[01]s and with more of them? That is, Study 1 yields BF[01] = 3.4, Study 2 yields BF[01] = 1.2, Study 3 yields BF[01] = 3.6, and Study 4 yields BF[01] = 5.1. Then the likelihood of the null hypothesis being true is 74.9 times higher than the likelihood of the alternative hypothesis being true?

However... is 74.9 then the Bayes Factor for the four studies combined? This would confuse me, because if I divide the data set of one of the studies and multiply the two resulting BF[01]s, it does not yield the same value like when I compute the BF[01] for the whole data set. Am I not overestimating the combined BF[01] with the multiplication strategy?

• The multiplication works only for point hypotheses, or when the data of the different experiments are independent (in the sense that knowledge about the one experiment does not affect your expectations about the outcome of the other experiment). Let's say you have experiments A and B. Then we have

BF(A,B) = BF(A) * BF(B|A)

The second BF is conditional, that is, it needs to take into account the knowledge obtained from experiment A: this means that the prior distribution needs to be updated. See also https://link.springer.com/article/10.3758/s13428-018-1092-x

Easy example to show that the prior needs to be updated:

We toss a coin. H0: coin is fair (theta = .5); H1: coin is either double-heads or double-tails (theta = 1 or theta = 0). We perform multiple experiments, each consisting of a single toss. Experiment A: the coin lands heads. Now this is equally likely under H0 and H1, so the BF=1. But now consider 1000 successive experiments, each of which will yield "heads". If we multiply without updating the prior distribution, we would conclude that for the complete sequence, BF = 1 (because 1 x 1 x 1.....etc.) Instead, the data are compelling evidence for H1. This comes about because, after Experiment A, the prior distribution for theta under H1 is changed to a point prior on theta = 1 (because the first observation has falsified theta=0); from the second observation onwards, each new observation contributes a factor of 2 in favor of H1. If we have assigned theta a beta distribution, the prior would need to be updated after every observation.

Cheers,

E.J.

• edited August 2023

Hi,

thanks a lot for this detail. So I guess I shouldn't just multiply BF[01]s, because previous experiments might have already led me to expect data in favour of the H0.

In the first part of this cool paper you cite, you write that the updated distribution [Beta(1 + 20,1 + 10)] can be specified in the Summary Stats module of JASP. And I am sorry that just don't understand how. Does it directly translate into a Cauchy, Normal, or t distribution which I would be able to specify under Priors? [Edit: sorry, I guess I can find that information also elsewhere, eg. https://arxiv.org/abs/1704.02479] Anyhow, as I don't have binomial data (but continuous values), this wouldn't work for me anyway.

From the second part of your cited paper, I understand that I could theoretically just combine all data from my four experiments into one large data set and analyse everything in one step. Is that, so far, correct?

However, there is one more pitfall, as the two conditions were tested between-subject in two of the experiments and within-subject in the other two, I guess I also have to run two separate analyses. I guess it would be highly uncommon to just analyse all four data sets as if all of them were obtained from independent samples for the one combined analysis, right?

And could I even use knowledge from the between-subject experiments for informed priors (however I would enter that information...) for the within-subject experiments? The essential research questions are the same: "are slopes more negative under condition A than condition B?" (where I want to quantify the evidence for the null hypothesis, that slopes are on average just the same under both conditions.)

• For continuous priors you have to specify different distributions. I think that in your case I would just select a normal distribution. You can get the best fitting normal distribution from the previous experiment from the mean and 95% central credible interval. Given the complications of design differences this may be more feasible than combining the data.

The effect size delta from the between-subject experiment could be used for the within-subject experiment, I think (but I'll ask our expert to be sure)

• From the combined data of the first and second experiment I have the 95% CI [-0.222, 0.488]. Can I directly derive the mean 0.133 (i.e., mean value of the CI)? Then, my first hunch was to simply use the 95% CI as the new normal distribution (something like: a standard deviation for my following prior normal distribution is 1/4 of the range of the 95% CI, thus: mean = 0.133, std = 0,178). Is it really that simply translatable? (If that really works in principle, I could also specify a 95.4% credible interval before calculating the std...)

Furthermore, then the BF is not really meaningful anymore, right? As my prior distribution starts to strongly mass around 0, the BF[01] gets smaller. Would then the only evidence FOR the H0 be the resulting CI?

• Thanks EJ. I did not know that for typical statistical scenarios--where the alternative hypothesis is a diffuse set of non-zero effect-size possibilities--the statement from the JSP manual, that "Posterior odds Posterior odds = Bayes factor × prior odds" [http://static.jasp-stats.org/Manuals/Bayesian_Guide_v0_12_2_1.pdf], isn't strictly correct since it's an over-simplification.

I do see a conceptual problem, though. "Posterior odds Posterior odds = Bayes factor × prior odds" is a verbal statement of Bayes' theorem! So we are talking about "Bayesian" statistical analyses that do not conform strictly to Bayes' theorem. It might be clearer to describe what we're doing as "quasi-Bayesian" analyses that incorporates only some aspects of Bayesianism.

As an aside, I mentioned my surely-unpopular opinion that prior odds derived from a previous study are only relevant if the current study is an exact replication of the previous study. This is because Bayesian updating (or any kind of updating) of pair of hypothesis requires that the same pair of hypotheses be considered prior to and subsequent to the update. However, if (for example) Study 1 examines the physiological effects of caffeine (from Coca-Cola) on 2nd-year psychology students and Study 2 examines the physiological effects of caffeine (from Pepsi) on 1st-through-3rd year psychology students, Study 1's statistical null hypothesis is different from Study 2's statistical null hypothesis. It is therefore possible for the null hypothesis to be true in one study but false in the other (and the same goes for the alternative hypotheses). For this reason, I think that in real-world *statistical* analysis it's best to always have the prior odds be 1/1, and to "update" by combining all the data into one data set while including Study ID (e.g., 1 vs 2) as an explicit variable. In situations where it seems too difficult or awkward to combine the data across studies, that's probably a good cue that the studies differ in the formulation of their null-hypotheses (and that they differ in the formulation of their alternative hypotheses).

R

• I do think the statement "posterior odds = BF times prior odds" is correct; it is just that the calculation of the BF requires some care, that is, you need to update not just your belief in the rival hypotheses but also in the competing parameter values within the hypotheses itself. This is the only way in which the entire system coheres. For examples see https://www.bayesianspectacles.org/bayes-factors-for-those-who-hate-bayes-factors-part-ii-lord-ludicrus-vampire-count-of-incoherence-insists-on-a-dance/ and https://www.bayesianspectacles.org/bayes-factors-for-those-who-hate-bayes-factors-part-iii-the-coherence-plot/

Cheers,

EJ

• It's my impression that in Bayes' Theorem, all of the information pertaining to prior beliefs is contained in the 'prior odds' term. I contrast the Bayes factor likelihood ratio is only bout the current data. So I think that any conceptualization of the Bayes factor that reaches back prior to the data at hand is a change to Bayes' theorem rather than a mere application of the theorem.

@esixtus Maybe something to take away from this thread is that the degree of consensus within the practice of Bayesian analyses statistics isn't as strong as the consensus within classical, null-hypothesis testing.

R

• There is only a single mathematically correct way to update, if the data are judged to be exchangeable. The second blog post referenced above discusses an example where one analyses a data set of 10 observations in two ways:

(1) all data at once;

(2) in two equally large batches, A and B

The Bayesian inference should be the same in both cases. The only way to achieve this is to do the batch-by-batch analysis by condition on all information, that is, by adjusting the prior distribution.

Of course there is a legitimate debate to be had on whether and to what extent data from prior experiments ought to impact the analysis of the current experiment (maybe there are differences in design or in participant characteristics). If experiments might differ in a meaningful way, a Bayesian random-effects meta-analysis might be the way to go.

Cheers,

EJ

• edited August 2023

I can see that all is not easy, and maybe not unambiguous, from a philosophical point of view. However, I still wonder, whether in principle my application of EJ's instruction could work (my previous comment, from yesterday, 1.57pm) or whether I fundamentally misinterpreted something...?

Cheers,

Elena

• If I may add my two cents:

As far as I understand, you would like to quantify the relative evidence (i.e. Bayes factor) for the effect across all studies, correct?

I, too, would suggest a joint analysis of the raw data. Given that you are combining within- and between-subject studies, a linear mixed model may be appropriate. Currently, JASP supports Bayesian LMM, but does not compute Bayes factors. You could, however, use the R package `BayesFactor` , which powers some of JASP's calculations.

Here's a simplified example for one within- and one between-subject study. The within-subject experiment has only one observation per participant per condition (otherwise we should add random slopes for the manipulation). Also, I'm making the strong assumption that the true effect size is the same in both studies (otherwise we would need consider additional fixed effects for study or study design).

I'll first simulate some fake data for 100 participants in each study.

```n <- 100

subject <- c(
1:n
, rep((n + 1):(2 * n), each = 2)
)

design <- c(
rep("between", n)
, rep("within", 2 * n)
)

condition <- c(
c(
rep(0:1, n / 2)
, rep(0:1, n)
)
)

truth <- data.frame(
par = c("mu", "mu_delta", "sigma_mu", "sigma_e")
, value = c(0.8, 0.4, 0.75, 0.5)
)

m <- rnorm(n * 2, mean = truth[truth\$par == "mu", "value"], sd = truth[truth\$par == "sigma_mu", "value"])
delta <- rep(truth[truth\$par == "mu_delta", "value"], n * 2)

dat <- tibble::tibble(
subject = subject
, condition = condition
) |>
dplyr::mutate(
dv = rnorm(dplyr::n(), mean = m[subject] + (condition-0.5)*delta[subject], sd = truth[truth\$par == "sigma_e", "value"])
, subject = as.factor(subject)
, condition = as.factor(condition)
, design = as.factor(design)
)
```

Now, we can use the `BayesFactor` package to fit two LMM and compare the relative evidence for the effect. I use very diffuse prior distributions here, which will be inappropriate for your application. Make sure, you understand the nature of the fixed and random effect priors and choose appropriate scales for your effect and design as your results will depend on these settings.

```BayesFactor::lmBF(
dv ~ condition + subject
, data = dat
, whichRandom = "subject"
, rscaleFixed = 1
, rscaleRandom = 2
) /
BayesFactor::lmBF(
dv ~ subject
, data = dat
, whichRandom = "subject"
, rscaleFixed = 1
, rscaleRandom = 2
)
```

If you are new to Bayesian LMMs, our recent special issue on the topic may be of interest. Rouder et al. (2023) provide some guidance on the specification of models and prior scales in their example application (see the section "Fine-Grained Specification", p. 60). They also provide some code for their example analysis. Note that their approach of specifying the null model sparked controversy among the contributors to the special issue. Make sure you understand the implications, consider the alternatives discussed in some of the other contributions, and decide which is most suitable for your application.

• Thanks a lot for your response!

I will try that out. It will, however, take quite a while, cause I'm in fact very new to Bayesian LMMs. I was hoping for a "quick and dirty" (but sound) JASP solution with some default priors and just some changed settings. I understand now that this will not work out.

Maybe I'll ask more questions here in a week or in six months...

Cheers

• It seems from this thread that all of JASP's Bayesian analyses may be missing crucial bit of output. In my opinion, the most useful aspect of Bayesian (as opposed to a frequentist) analysis is that the former provides a ready answer to the following question: "Given the data, the alternative hypothesis is __ times as likely as the null hypothesis." In other words, the Bayesian analysis provides the posterior odds for the hypotheses.

In a simple, not-uncommon situation where the prior odds are 1:1, a "traditional Bayesian" analysis provides a Bayes factor (rather than posterior odds) as output would present absolutely no difficulty. The posterior odds (OP) would be identical to the Bayes factor (BF) since OP would = BF * 1.0 = BF. However, in light of the preset thread, it seems that such may not be the case when dealing with the kind of statistical Bayesian statistical analysis involving diffuse alternative hypotheses. Therefore, it seems, JASP's (and R's) offering of a Bayes factor doesn't directly provide the most important output of an equal-priors Bayesian analysis: the odds of the alternative versus the null hypothesis, given the data.

R

• Agreed in general; some of our analyses (for regression and ANOVA) do present posterior model probabilities though. But perhaps we ought to do this for t-tests as well. I guess we are still prioritizing adding more functionality at the moment...

EJ

• Thanks for pointing that out to me. However, below is the JASP output for a Bayesian two-group ANOVA (preceded by a Bayesian t test on the data). I want to know the value of the posterior odds (OP) where: "Given the data, the odds of the alternative hypothesis relative to the null are ______." (*Moreover, in light of the present thread, I expect OP to be something other than just a prior odds of 1.0 multiplied by the present Bayes factor.*) Where in the output do I find that value of OP?

Thanks.

R

• Well the posterior odds would be the ratio of the posterior model probabilities (values in the "p(M|data)" column). I personally think that probabilities are a little easier to grasp than odds (although odds are useful for convenient updating, and for working with extreme outcomes).

EJ

• edited December 2023

Yes, but that ratio appears to always be identical (within the estimation error of the model) to the prior odds of 1:1 multiplied by the Bayes factor, which is simply the Bayes factor (which JASP provides both for the t test output and the ANOVA output). Could this be evidence that with a prior odds of 1:1 it is valid, after all, to compute the posterior odds as an update to the prior odds--i.e., to simply multiply the Bayes factor by 1:1?

R