Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Which test to use?

Hi,

I work in metabolomics research, and one particular statistical method that I use involves the following process:

  1. Conduct a univariate test (e.g., independent samples t-test) on each chemical compound, normally 500 or more
  2. Group the chemical compounds based on various factors such as structure
  3. For each group of compounds, use a Kolmogorov-Smirnov test with a uniform distribution to see if there is a sufficient positive skew to examine effects at the compound group level (i.e., are there more compounds with p values less than 0.05 than might be expected if there was no effect on compound group?)

I'm wondering if some Bayesian version of this might exist, where BFs are calculated for each individual compound (which is simple enough through t-tests), then some way of agglomerating these values based on the chemical group to get a chemical group-level BF?

Chris

Comments

  • Hi Chris,

    I would think that, within each group of compounds (which I assume is defined up front), you'd have a hierarchical model, with a mean effect and heterogeneity. In other words, you could do a Bayesian meta-analysis.

    E.J.

  • Hi E.J.,

    Thank you for such a quick response. The groups of compounds are predefined. I was originally thinking a meta-analysis might work, but I thought that it assumed independence of observations though? I probably should've made it clearer in my original post, a typical study would conduct the 500+ t-tests on the same sample (just different dependent variables for each t-test).

    Thanks

    Chris

  • It's difficult for me without the substantive background knowledge, but I am not sure this is an issue. You basically have 500+ effect sizes delta_i, and we assume that delta_i ~ N(mu,sigma). I guess in the usual setup each test is for an experiment, and across experiments you want to generalize to the population; in your case, each test is for a different DV, so you are not generalizing to other samples (but I don't think that is the intention, or otherwise you may have strong a priori reasons to believe that it will).

    E.J.

  • Hi E.J.

    Thank you for your response, and sorry for my delay in getting back to you. I've been thinking about this some more, and how to describe my questions within the context of psychology. We do want to be able to generalize to other samples, which is an issue.

    A psychology version of the kind of experiments I'm running/analysing would be something like all participants (split into 2 groups) completing 10 working memory tasks, 10 inhibition tasks, 10 mental rotation tasks, 10 processing speed tasks, etc. - there would be a very high number of DVs, but we can confidently say that tasks 1-10 measure working memory, tasks 11-20 measure inhibition, and so on. At a task level, we then look to see which tasks have p < 0.05. However, what is more interesting is whether working memory as a construct (or inhibition, mental rotation etc.) is different between the two groups.

    Currently, this is looked at with a one-sample Kolmogorov-Smirnov test, where the p values that are produced for each construct are compared to a uniform distribution. If there is no construct-level effect, then the distribution of p values should be uniform and the KS test will be non-significant. If there is a construct-level effect, then there should be more small p values, which would make the KS test significant.

    In terms of a Bayesian approach to this, I was thinking that BFs could be calculated on each test, then the average BF or the distributions of BFs of each construct could be looked at. To me, it seems like if a construct is not affected by the experimental manipulation, the average BF10 is most likely to be less than 1 (dependent on sample size), and if it is affected then the average BF10 should be greater than 1 (depending on sample size and effect size). The figures attached show the distributions of 1000 logBF10s with 50 simulated samples per group for no effect (Cohen's d = 0), and the same for a large effect (Cohen's d = 0.8). In my mind, this suggests that we have two competing hypotheses that can be quantified in probability distributions (average BF/distribution of BFs when there is no effect < 1 vs. average BF/distribution of BFs when there is an effect > 1), which would mean that a BF can be calculated on this.

    I guess my questions at this point are: a) does what I've written make sense, and b) does this seem mathematically reasonable?

    Thanks

    Chris


  • Well you could do things this way. Beware to average the log BFs, not the raw BFs -- the mean of 3 and 1/3 is not 1. However, when you describe the problem as follows:

    "A psychology version of the kind of experiments I'm running/analysing would be something like all participants (split into 2 groups) completing 10 working memory tasks, 10 inhibition tasks, 10 mental rotation tasks, 10 processing speed tasks, etc. - there would be a very high number of DVs, but we can confidently say that tasks 1-10 measure working memory, tasks 11-20 measure inhibition, and so on. At a task level, we then look to see which tasks have p < 0.05. However, what is more interesting is whether working memory as a construct (or inhibition, mental rotation etc.) is different between the two groups."

    , this screams "SEM model" to me. In general, I would be in favor of setting up a complete model first and then obtaining a single BF that addresses the key question, instead of bringing multiple BFs together in some way. But this is a general preference and it may not be feasible for your problem.

    E.J.

  • Hi EJ,

    Thank you for your response. I agree that it does sound a lot like an SEM model would be appropriate - the major problem is that sample sizes are frequently around 6 per group, and we often have hundreds of dependent variables. Effect sizes can in some cases be Cohen's d > 1.5, but these are still very small sample sizes and I'd need to use some pretty informative priors for each variable to get a BF value that isn't anecdotal.

    Thanks

    Chris

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games