[Bayesian ANCOVA] number of covariates and multiplicity correction

alhess

Hi,

I am interested in running a Bayesian ANCOVA on a small data set (N=30). I plan to include the nuisance variables gender (random factor) and age (covariate) in the null model. I have 9 other covariates of interest and I wish to identify those that are related to the dependent variable (i.e. explain variance of the dependent variable). Since I do not have any further a priori knowledge about the relationship between these covariates and the dependent variable, this leaves me with a total of 512 possible models that I could be evaluating (and that is without including interaction terms). The BF_10 (comparing to the null model) would be my metric of interest for selecting the best model.

• Given the small size of my data set and the comparably large number of parameters, do I need to be worried about the problem of specifying overly complex models (overfitting) when including all 9 covariates and the 2 nuisance variables in a Bayesian ANCOVA model?
• Should I be correcting for multiple comparisons in this scenario since each individual BF_10 essentially represents a hypothesis test? I did try to find resources on this topic but was only able to find resources discussing the issue of correcting for multiplicity in the case of post hoc tests in an ANOVA setting.

Thank you very much for your help!

Best wishes,

Alex

EJ

Dear Alex,

1. My first response is that you are being too optimistic. With only N=30 and already 2 covariates in the null model, I don't think you can test the impact of another 9 covariates.
2. If you evaluate all possible models, I would recommend model-averaging to ascertain the importance of each covariate (analysis of effects, yielding an inclusion BF for each covariate)

EJ