Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

BF between 1 and 3, cannot collect more data, meaningful to report ?

I have a question about the interpretation of anecdotal/inconclusive evidence.
My sample size is quite big given the standards in my field (N=75), and I cannot collect extra data. Bayesian t-tests for my contrasts of interest show BFs between 1 and 3, suggesting inconclusive evidence. Some of these go together with p-values around 0.05 in classic t-test (in the unexpected direction).
Can I report/interpret such results in a meaningful way? E.g., can I state "if there is an effect in the population, it is not strong or consistent enough to be detected with this sample"?
I am asking for practical reasons. If given these outcomes, I would absolutely have to collect more data to draw any meaningful conclusions, I cannot report the experiment. Because I cannot collect more data. It would be a pitty, because the conclusion that there is not enough evidence to show an effect, would be in line with two other experiments that showed support for the nullhypothesis and are to go into the same publication.

Comments

  • Hi Hanna,

    Here's my take on this: Inconclusive evidence is a perfectly valid outcome of an experiment, even though perhaps not very satisfying; sometimes life is such that you try to find out something but don't succeed. There is certainly nothing wrong with reporting that.

    Whether it is meaningful, in the sense of theoretically interesting, depends on the context. For example, if you're trying to replicate a paper that showed a huge effect with N=10, then inconclusive evidence with N=75 would suggest that the initial paper at least overestimated the effect—that might be worthwhile. But in other cases it may not be. The interest, or lack thereof, of your result is not primarily a statistical question. How do you feel about it?

    Cheers,
    Sebastiaan

    Thanked by 1HannaG

    There's much bigger issues in the world, I know. But I first have to take care of the world I know.
    cogsci.nl/smathot

  • Hi Hanna,

    This is a great question. Some brief thoughts:
    1. With BFs in between 1 and 3 you have not learned much with respect to the specific model-comparison question you were studying. with N=75, the posterior distribution will be relatively peaked -- so you did learn a lot about the size of the effect (as Sebastiaan also points out). You may also have learned a lot about whether the effect is positive or negative.
    2. You mention that the effect goes in the opposite direction. Was your Bayesian test one-sided? If the effect goes in the opposite direction and is almost significant, I would expect pretty good evidence in favor of the null. This depends of course on how reasonable it is to employ the order-restriction. If the theory you are testing clearly stipulates the order-restriction, then your prior ought to reflect that, imo.

    Cheers,
    E.J.

    Thanked by 1HannaG
  • Dear both,
    Thank you both for your responses. A lot happened last year, and I must have lost track of this thread. But I wanted to give a shout-out to you to thank you for taking the time to answer questions here! Much appreciated and very helpful! I'll try to find out what happened with this data to let you know. I think we reported it as inconclusive evidence.
    Hanna

Sign In or Register to comment.