Interpreting (volatile) sequential analyses
We have a dataset comparing 2 groups of participants (n ~ 350 in each group) on various structural MRI measures. Some analyses suggest moderate evidence for group differences, but I remain a bit puzzled by the sequential plots. We often have such plots:
Here we have moderate evidence for a group difference, but I also understand from this graph that, had we restricted our analyses/data collection to the first 500 participants, we would have concluded that there is no evidence at all. Is this interpretation correct? If yes, I find that a bit unsettling and gives me little faith regarding claiming an actual group difference. Can someone enlighten me on this issue?
Alternatively, we sometimes have such plots:
I find this even more disturbing: I may want to conclude based on the whole sample that there is moderate evidence for a group difference, but this plot suggests that stopping after 400 or 500 participants would have led to very strong vs anecdotal evidence, respectively. I find this sample-size-dependent "volatility" of the evidence quite disturbing. Should I be disturbed indeed? And thus cautious about any interpretation?
Thanks a lot in advance for your help!
All the best,