bayesian t-test - hypothesis
Hi everyone,
I`m a bit unsure whether to conduct/report one-sided or two-sided bayesian t-tests.
In short - We have conducted an experiment, in which participants performed keypresses and were presented with effects that were either congruent or incongruent with prior acquired action-effect associations. Participants performed four blocks of 48 trials each. We expected shorter reaction times in congruent than in incongruent trials (But of course if the effect is the other way around (shorter reaction times incongruent trials than congruent trials) we would like to interpret this as well.
So if i conduct a one-sided bayesian t-test with the hypothesis A (congruent) < B (incongruent)
I get moderate evidence for a congruency effect (congruent < incongruent) in block 1 (BF = 4), but no evidence for such an effect in Block 2-4 (BFs are somewhere between 0.05 and 0.09). And using this approach, there is no way to obtain any evidence for an reversed effect (incongruent < congruent) if there is one.
However if I conduct a two sided bayesian t-test with the hypothesis A(congruent) ≠ B(incongruent) than I get only anecdotal evidence (BF = 2.03) for a congruency effect in block 1, but moderate evidence for a congruence reversal (incongruent < congruent) in block 4 (BF = 7.7)
So, If Im strictly adhering to my hypothesis choosing a one-sided approach I get some evidence for my original hypothesis (in the first block), but I wont "detect" the congruency reversal in the last block. If a choose a two-sided approach, the evidence towards my original hypothesis is quite weak (BF of 2 instead of 4 in the first block) but I "detect" that over time (block 4) the opposite of which I had expected can be observed.
Are there any ideas on how to solve this?
Comments
This is a question of priors - your priors didn't expect a reverse congruency effect, so you didn't think of testing for one.
This is the same as for NHST - if you did a 2-tail test, it would be significant, but in the opposite direction of what you would have expected.
I guess you could do both - a one sided test for hypothesis testing, and a two sided (or even reverse one sided test) as a post hoc test (which, from what you say, is exactly what you've done).
Also note that BF10 of 0.05-0.09 do not indicate "no evidence" for an effect, but evidence against an effect, as BF01 = 1/BF10 = 11-20.
Hi v.b,
Yes, like MSB says. In general, your description seemed fine to me; there is nothing to be solved. If you were not completely committed to the one-sided test then you ought to do the two-sided one.
Cheers,
E.J.
Thanks a lot!
so would it be appropriate to report both (first one sided in the direction of the hypothesis than two sided bayesian t-Test) in an article. Or would your rather recommend to report first a one sided t-test in the direction of the hypothesis and afterwards a reverse one-sided t-test for effects in the opposite direction)?
Both make sense. [Captain Hindsight adds: this is of course also why preregistration is a good idea -- it forces you to come up with these choices beforehand, so that it is clearer which analyses were confirmatory and which ones were inspired by the data.]
I think it is more appropriate to explicitly note that the second "revered" test is post-hoc. If I were reviewing the paper, I would ask about this.
Thanks again :-)
The revered test was definitely inspired by the data and was not planned beforehand (a congruency reversal was not expected). So, I will explicitly note that it was post hoc. Thanks for all your advise.