Which test to use?

ChrisB

Hi,

I work in metabolomics research, and one particular statistical method that I use involves the following process:

1. Conduct a univariate test (e.g., independent samples t-test) on each chemical compound, normally 500 or more
2. Group the chemical compounds based on various factors such as structure
3. For each group of compounds, use a Kolmogorov-Smirnov test with a uniform distribution to see if there is a sufficient positive skew to examine effects at the compound group level (i.e., are there more compounds with p values less than 0.05 than might be expected if there was no effect on compound group?)

I'm wondering if some Bayesian version of this might exist, where BFs are calculated for each individual compound (which is simple enough through t-tests), then some way of agglomerating these values based on the chemical group to get a chemical group-level BF?

Chris

EJ

Hi Chris,

I would think that, within each group of compounds (which I assume is defined up front), you'd have a hierarchical model, with a mean effect and heterogeneity. In other words, you could do a Bayesian meta-analysis.

E.J.

ChrisB

Hi E.J.,

Thank you for such a quick response. The groups of compounds are predefined. I was originally thinking a meta-analysis might work, but I thought that it assumed independence of observations though? I probably should've made it clearer in my original post, a typical study would conduct the 500+ t-tests on the same sample (just different dependent variables for each t-test).

Thanks

Chris

EJ

It's difficult for me without the substantive background knowledge, but I am not sure this is an issue. You basically have 500+ effect sizes delta_i, and we assume that delta_i ~ N(mu,sigma). I guess in the usual setup each test is for an experiment, and across experiments you want to generalize to the population; in your case, each test is for a different DV, so you are not generalizing to other samples (but I don't think that is the intention, or otherwise you may have strong a priori reasons to believe that it will).

E.J.

ChrisB

Hi E.J.

Thank you for your response, and sorry for my delay in getting back to you. I've been thinking about this some more, and how to describe my questions within the context of psychology. We do want to be able to generalize to other samples, which is an issue.

A psychology version of the kind of experiments I'm running/analysing would be something like all participants (split into 2 groups) completing 10 working memory tasks, 10 inhibition tasks, 10 mental rotation tasks, 10 processing speed tasks, etc. - there would be a very high number of DVs, but we can confidently say that tasks 1-10 measure working memory, tasks 11-20 measure inhibition, and so on. At a task level, we then look to see which tasks have p < 0.05. However, what is more interesting is whether working memory as a construct (or inhibition, mental rotation etc.) is different between the two groups.

Currently, this is looked at with a one-sample Kolmogorov-Smirnov test, where the p values that are produced for each construct are compared to a uniform distribution. If there is no construct-level effect, then the distribution of p values should be uniform and the KS test will be non-significant. If there is a construct-level effect, then there should be more small p values, which would make the KS test significant.

In terms of a Bayesian approach to this, I was thinking that BFs could be calculated on each test, then the average BF or the distributions of BFs of each construct could be looked at. To me, it seems like if a construct is not affected by the experimental manipulation, the average BF10 is most likely to be less than 1 (dependent on sample size), and if it is affected then the average BF10 should be greater than 1 (depending on sample size and effect size). The figures attached show the distributions of 1000 logBF10s with 50 simulated samples per group for no effect (Cohen's d = 0), and the same for a large effect (Cohen's d = 0.8). In my mind, this suggests that we have two competing hypotheses that can be quantified in probability distributions (average BF/distribution of BFs when there is no effect < 1 vs. average BF/distribution of BFs when there is an effect > 1), which would mean that a BF can be calculated on this.

I guess my questions at this point are: a) does what I've written make sense, and b) does this seem mathematically reasonable?

Thanks

Chris

EJ

Well you could do things this way. Beware to average the log BFs, not the raw BFs -- the mean of 3 and 1/3 is not 1. However, when you describe the problem as follows:

"A psychology version of the kind of experiments I'm running/analysing would be something like all participants (split into 2 groups) completing 10 working memory tasks, 10 inhibition tasks, 10 mental rotation tasks, 10 processing speed tasks, etc. - there would be a very high number of DVs, but we can confidently say that tasks 1-10 measure working memory, tasks 11-20 measure inhibition, and so on. At a task level, we then look to see which tasks have p < 0.05. However, what is more interesting is whether working memory as a construct (or inhibition, mental rotation etc.) is different between the two groups."

, this screams "SEM model" to me. In general, I would be in favor of setting up a complete model first and then obtaining a single BF that addresses the key question, instead of bringing multiple BFs together in some way. But this is a general preference and it may not be feasible for your problem.

E.J.

ChrisB

Hi EJ,

Thank you for your response. I agree that it does sound a lot like an SEM model would be appropriate - the major problem is that sample sizes are frequently around 6 per group, and we often have hundreds of dependent variables. Effect sizes can in some cases be Cohen's d > 1.5, but these are still very small sample sizes and I'd need to use some pretty informative priors for each variable to get a BF value that isn't anecdotal.

Thanks

Chris