Determining the Best Model in JASP

uhandoko

Hello JASP users, this is my first time using JASP and I have very little knowledge about statistics. I will determine the best ensemble of 12 climate models in simulating rainfall. I have been running with the Bayesian Linear Regression in JASP, but I am still confuse in determining the best model criteria. Is it based on BFm, B10 or R2? Furthermore, if we have decide one best model, which bayesian  linier regression coefficient do we use to predict rainfall? Does it use the mean coefficient, SD coefficient, P (incl / data), or BF inclusion (all in the Posterior Summary table)?

Thank you so much for your help.

uhandoko

Output.docx

This is the output

EJ

Hi Uhandoko,

We are finishing up a tutorial paper on this. I have asked the first author to send you a draft. Quickly though:

1. If you tick "compare to best model", the models will be arrange in decreasing order of predictive performance.
2. If you want to predict anything, the Bayesian course of action is to average over the parameters (so use the full posterior distribution, not just the point estimate) *and* average over the models (so do not use a single model).

Cheers,

E.J.

uhandoko

Output.docx

Thank you very much EJ for your quick response and your kindness will send me a draft about this tutorial.

Sorry, I still don't understand the second answer. As I understand from your answer to predict something with Bayesian regression equation,  the coefficients regression for each variable  using  "mean" column in the posterior summary table. For example, from the output I get, the first ranking model consisted of 12 climate models. So to predict rain, I will use the equation as follows:

Rainfall prediction = 8.687 + (- 0.450 * NorESM) + (- 0.299 * ACCESS) + (- 0.082 * bcc) + (- 0.085 * BNU) + (0.905 * CanESm2) + ... + (0.02 * MRI).

Is it like that?

Best regard..

Uhandoko

EJ

Dear Uhandoko,

There is a whole literature on how to predict exactly. In general, I would say you want the uncertainty surrounding your point prediction, so you'd want to take the distributions for the beta's into account (rather than just focusing on the means). Basically, I would do the following:

1. For each of the models k=1...K, sample from the joint posterior distribution and make a prediction.
2. The number of samples from each model should be proportional to their posterior probability.

This way you take into account uncertainty in the beta's and in the models. But whatever you do, both need to be taken into account or else the prediction will be overconfident.

Cheers,

E.J.

uhandoko

Thank you for the advice for me, EJ

Best regard...

Uhandoko