Creating an informed prior from Bayesian reanalysis of two frequentist publications
Hi,
I'm tying to figure out how to make an informed prior from pre-existing knowledge in the literature. My main analysis is a Bayesian independent samples t-test. I've currently run this using a default uninformed prior (Cauchy with scale = 0.707). I've also searched the literature for research on the same question and found two papers. However, they are both frequentist analyses with sparsely reported results, but I was able to calculate t-statistics for the results I need. I then used JASP to do a Bayesian reanalysis for these two papers using their t-stats and sample sizes. The results are uploaded to OSF here: https://osf.io/qgcw7.
The first paper's reanalysis is called: S.S. Bayesian Paired Samples T-Test: J et al.
The second paper's reanalysis is called: S.S. Bayesian Independent Samples T-Test: S et al.
I wanted to combine the information in both these papers to get an informed prior for my analysis. I've been reading the "Replication Bayes factors from evidence updating" here: https://link.springer.com/article/10.3758/s13428-018-1092-x. But I can't seem to figure it out. I don't think I have enough information from the original publications to be able to "compute the overall t value for the combined data", as was done in Appendix A of this paper. So I'm not sure if I can use this Bayes Factor approach.
I've turned to the “today’s posterior is tomorrow’s prior” approach. The paper says "the posterior for δ in a t test has no known distributional form", but that you can "approximate the posterior on effect size obtained from the t test with a normal distribution; this normal distribution is then used as a prior for the analysis of the replication experiment". Assuming the 95% credible interval could take the place of a 95% confidence interval, I used the posterior 95% interval and the original sample size to calculate the standard deviation of the (normal approximation of) posterior distribution, and I used the median of the posterior as the mean. So taking the mean and standard deviation to approximate the posterior, I used this mean and standard deviation as an informed prior with a normal distribution.
The results are uploaded to OSF here: https://osf.io/qgcw7. The analysis called "S informed by J" is the same analysis as "S.S. Bayesian Independent Samples T-Test: S et al.", but instead of using the default Cauchy prior, I used the informed prior based on the posterior from "S.S. Bayesian Paired Samples T-Test: J et al". And vice versa for the analysis called "J informed by S".
I expected the posterior to be the same for "S informed by J" and "J informed by S", as the order shouldn't matter (I thought), but they have different posteriors. And ultimately, I just want to take the posterior of the combined result forward to my main analysis to use as an informed prior. But given my "S informed by J" and "J informed by S" don't match, I feel I've gone very wrong somewhere.
Could you please advise on how to combine the results from two Bayesian reanalyses to get an overall posterior? And how to take forward the combined posterior result to be an informed prior?
Many thanks,
Luke.