Post hocs / planned contrasts in Bayesian

Comments

• EJ

  April 2016 edited 1:22AM

  Hi Kristel, Sebastiaan,

  As I was about to answer your post, I realized a similar issue can up earlier in the forum. Here's my previous answer, slightly edited:

  "As far as I am concerned, t-test as usually what researchers want to know if they think carefully about their hypotheses beforehand. In the Bayesian framework there is no need for a correction for the number of tests you entertain; however, there is a correction for conducting tests that you only half-anticipated to carry out, and this is through the often-ignored prior model odds. The extent to which you are able to assess prior model odds when the data have led you to consider a particular t-test is interesting and not really resolved. I personally would be happy to report the t-test but specifically note that it is post-hoc and the data led you to consider this test. The seriousness of the problem also depends on the design. Suppose you have a one-way ANOVA with 100 levels; you see that level 5 differs from level 11, and you t-test this difference. Clearly this is misleading. It would be interesting to develop a Bayesian test to provide a default prior model odds "correction" for post-hoc tests, but I recall an interesting conversation with Richard who argued it is impossible in principle. Maybe I misremember. Richard?"

  I would like to add that there is something conceptually weird about the post-hoc test. A real test can never be post-hoc. If you approach your data with the attitude "let us see what we can find", then you are really in the hypothesis-generating stage, not in the hypothesis-testing stage. Only in the latter stage doe your inferential statistics make sense. I think it is a mistake to view statistics as an "effect-discovery device"; it is really more a "testing pre-planned hypotheses device". So my advice is to think carefully about what you want to test, report the associated t-tests and ANOVAs under the level "pre-planned analyses (it does not hurt to preregister these quickly on aspredicted.org, for instance). The more exploratory analyses can be discussed under the level "post-hoc analyses" with the disclaimer that the data led you to those hypotheses. But maybe Richard has additional suggestions!

  Cheers,
  E.J.
• richarddmorey

  April 2016 edited 1:22AM

  What is it that JASP is not letting you do? BayesFactor should allow you to do it, I think, if you don't mind using R.
• Kristel

  April 2016 edited 1:22AM

  Thanks you all for shining some light onto this!

  Richard, are you suggesting it is possible to do post-hocs (or planned contrasts, as this is in fact less conceptually 'weird', following EJs reasoning) with Bayes Factor using R? Because no, don't mind using it
• frederikaust

  April 2016 edited April 2016

  Hi Kristel,

  I found these blog post by Richard really instructive:

  • http://bayesfactor.blogspot.de/2015/01/multiple-comparisons-with-bayesfactor-1.html
  • https://bayesfactor.blogspot.de/2015/01/multiple-comparisons-with-bayesfactor-2.html

  If I understand them correctly the approach is the one detailed in this paper: http://www.ejwagenmakers.com/2014/MoreyWagenmakers2014.pdf
• MSB

  April 2016 edited 1:22AM

  Hi All,

  Are there any plans to implement into JASP the methods Richard has described in the BayesFactor blog?
  Or should I add this to the list or reasons to finally learn R?

  M
• EJ

  April 2016 edited 1:22AM

  Yes there are plans :-)
  But it can't hurt to learn R!
  E.J.
• MSB

  April 2016 edited 1:22AM

  If i had known that being a psych-graduate would involve so much coding (matlab, e-prime, opensesame, R...) I might have re-considered 8-)

  Thanks!
• Kristel

  April 2016 edited 1:22AM

  The blog posts by Richard are indeed really instructive, thanks a lot! I think I will make that my weekend activity for this week