Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Interpreting (volatile) sequential analyses

Hi,

We have a dataset comparing 2 groups of participants (n ~ 350 in each group) on various structural MRI measures. Some analyses suggest moderate evidence for group differences, but I remain a bit puzzled by the sequential plots. We often have such plots:

Here we have moderate evidence for a group difference, but I also understand from this graph that, had we restricted our analyses/data collection to the first 500 participants, we would have concluded that there is no evidence at all. Is this interpretation correct? If yes, I find that a bit unsettling and gives me little faith regarding claiming an actual group difference. Can someone enlighten me on this issue?


Alternatively, we sometimes have such plots:

I find this even more disturbing: I may want to conclude based on the whole sample that there is moderate evidence for a group difference, but this plot suggests that stopping after 400 or 500 participants would have led to very strong vs anecdotal evidence, respectively. I find this sample-size-dependent "volatility" of the evidence quite disturbing. Should I be disturbed indeed? And thus cautious about any interpretation?

Thanks a lot in advance for your help!

All the best,

Guillaume

Comments

  • Dear Guillaume,

    These look a little unusual indeed. For the first graph, it is strange that the BF hardly changes from 1 until after about 400 data points are in. What sometimes causes these graphs to look strange is that the first n rows feature only (or predominantly) participants from a single condition, so that not much is learned about the difference between conditions. The downward slope in the second graph could also be due to batch effects in the data -- it isn't the case that a different analysis method or something like that was used, right?

    Cheers,

    E.J.

  • Dear E.J.,

    thank you so much for your quick reply!

    The underlying data has indeed an unusual structure, sorry I forgot to mention it. The whole (structural MRI) dataset is actually composed of 12 smaller datasets acquired in the context of 12 different studies that we pooled together. So there is heterogeneity (in terms of sample size, acquisition parameters/MRI scanner, stringency of inclusion criteria...), which could create such batch effects. Note that we have used Combat to counter-act this heterogenity the best we could.

    Now knowing this, do you think it could explain the above graphs ? I.e. the top graph would suggest that only a fraction of the datasets provide evidence for a group difference? In that case is it fair to make a claim of a group differences by just reporting the Bayes factor of the whole dataset, i.e. BF10=5 (somehow seems wrong to me)?

    And in the bottom graph, could this mean that various datasets provide evidence in opposite directions, in which case I would also feel uncomfortable reporting an "overall" group difference based on overall BF?

    Final question: this makes me realize that the order of participants is obviously a key driver of the shape of these sequantial plots, right? If yes, would it make sense to randomize their order?

    Thanks again for your thoughts, much appreciated!

    Best,

    Guillaume

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games