How about pool error term for RM factors?

Comments

• EJ

  October 2021

  The help file refers to Morey (2008) for details. I also Googled this and also found people who believe it is prudent not to pool the error term (e.g., http://people.umass.edu/~bogartz/on_pooling_error_terms..html). I'll ask Johnny for his input.

  Cheers,

  E.J.
• JohnnyB

  October 2021

  Hi @PerPalmgren,

  The initial RM ANOVA model is a model where the error terms are pooled - this is simply how linear models are set up in these settings. As a consequence, the emmeans R-package (which we use for marginal means, contrasts, and posthoc tests) also works with the pooled error term (see the documentation here).

  You can read further about this in a previous thread:

  That's how the emmeans works: it operates based on the model you specify. In the ANOVA model, there is the assumption of equal variances, so that influences how emmeans handles the pairwise comparisons. We offer a workaround, where (for balanced designs only) in the RM ANOVA you can choose to use the non-pooled error term for RM factors. This basically comes down to running a series of paired samples t-tests.

  I am currently trying to have the anova be able to specify different types of models (i.e., gls), so that we can offer this possibility while still using the lovely emmeans package.

  see for instance https://stats.stackexchange.com/questions/369532/interpreting-the-standard-error-from-emmeans-r

  I feel like the default behavior of emmeans ought to be respected (it makes sense to have pooled errors for followup tests if the original model is also using the pooled error term), hence the pooled error term is the default option. Does this clarify the issue for you?

  Kind regards,

  Johnny
• TPalma

  February 2022

  Hi @JohnnyB ,

  
  

  Would you also advise pool error terms for simple main effects? E.g., in a RM ANOVA, you have a sig. interaction between the RM factor (A) and the between-subjects factor (B) and you follow-up this interaction with simple main effects in which A is the simple effect factor and B the moderator.

  Thanks!

  Tomás
• JohnnyB

  February 2022

  Hi Tomas,

  Yes, I would use the pooled error term by default, since that is in line with the anova model. Only if you have substantial grounds for doing otherwise, I would switch to unpooled.

  Kind regards

  Johnny
• PerPalmgren

  June 2022 edited June 2022

  @JohnnyB When reading the documentation you provided and the article by Bogartz https://people.umass.edu/~bogartz/on_pooling_error_terms..html provided by @EJ I am not 100% convinced of the option of employing pooled error terms. I think it would be easier for JASP users to go with one or the other. One could than in a scientific discourse or statistical debate argue for that this is how the post hoc calculation is done in the statistical program. I think the present choice is very confusing. Further, the article Morey (2008) which is referenced in JASP does not adress the issue of pooled errors.
• andersony3k

  June 2022

  1. Why is non-pooling only an option for the repeated-factors and not for the between-subject factors?

  2. The reason I think non-pooling is necessary is that there should be some way to conduct planned comparisons without having those comparisons be part of an ANOVA. Perhaps jamovi could be modified to include a planned-comparisins analysis with no error-term pooling, rather than include non-pooling as an option in the ANOVA analysis.

  R
• JohnnyB

  June 2022

  @PerPalmgren, I am not sure if both discuss the same issue. It seems the post you reference speaks about the main model in ANOVA, whereas above we were discussing the best course for a follow-up test on a default ANOVA, but I could be mistaken. I feel that the follow-up tests should follow the main model as closely as possible (see this lengthy discussion here, with the author of the emmeans package outlining their arguments). If the default ANOVA is wrong to begin with (as seems to be implied by Bogartz?), a different starting model should be used, such as a GLM.

  However, if it turns out that the Bogartz finding also holds for the pooled SE of post hoc tests on a default ANOVA, then maybe we should indeed reconsider the default value for that option.

  @andersony3k

  1. For the reason outlined above, it has not been high priority to also implement it for between subjects (In most cases where the non-pooled SE seems to the more sensible option, this concerns within subjects factors).
  2. We recently implemented generalized linear models in the Regression module, where you can specify contrasts as well. This might be what you are looking for (note that this module requires the data to be in long data format, whereas the RM ANOVA requires wide data format).

  
  

  Kind regards,

  Johnny
• andersony3k

  June 2022

  For #2. Sometimes the data severely violate the assumptions of a linear model, and so the analyst would like to do a set of planned t tests (instead of fitting the data to a linear model such as an ANOVA or a Generalized Linear Model). JASP supports this if one is willing to do the t tests one at a time. It would be convenient of one could present JASP with a set of factors and have it automatically perform a set of individual (not linear-model-based) t tests.

  R
• JohnnyB

  June 2022

  Hi @andersony3k ,

  Yes that would be a great functionality! We are now exploring the possibilty of specifying a more flexible structure for the SE's (with the nlme package) in ANOVA, which would make this functionality very straightforward to implement. In the meantime, I'm afraid you are stuck with specifying the individual t-tests in the t-test module (a bit of clicking, but wont take that long I reckon).

  Kind regards,

  Johnny
• PerPalmgren

  April 2023 edited April 2023

  @JohnnyB

  Hi,

  Despite being JASP lover and of the opinion that JASP is the most educative and stringent statisitcal program on the market, I still do not agree with using the "Pool error term for RM factors" as the default for posthoc analysis with a RM-ANOVA. Recently in a workshop for doctoral students whom where using both JASP. SPSS and Jamovi there was a lot of confusion as the results differed between the programs (all using the default settings). In my opinion one imperative aspect of science is to to strive for better collective knowledge and understanding of different phenomena of the world. One way is to create a common language and practice. I think JASP would better contribute to that discourse if the setting was harmonized. I understand your opinion (based on above) of using the pool term is to prefer. However, to create coherency in the scientific community, I think JASP would do better to have the Pool error unchecked as default or make information should be much more clear to the user. As pragmatic user I think ii is more important to have comparable and analogous results between programs, especially with regards to P-values and measures of effect sizes.

  Please see attached outputs from JASP, SPSS and Jamovi.

  All the best

  Per

  Pooled error terms.pdf

  
  
• PerPalmgren

  April 2023

  Did also the post hoc in DATAtab this morning (using default) and got:

  

  
  
• EJ

  April 2023

  I think we could probably add a note underneath the table. On the other hand, I am not married to this particular setting, and if most programs have a different default then I don't see why we ought to stick religiously to the current setting.

  E.J.
• andersony3k

  April 2023 edited April 2023

  @PerPalmgren

  Note however that there's currently more consistency *within* JASP than there is within SPSS and within jamovi regarding error-term pooling in repeated-measures versus between-subject ANOVA post hoc tests. In SPSS an jamovi, the post-hocs for a between-subject use pooled error terms whereas the post-hocs in repeated-measures use un-pooled error terms. In contrast, JASP's defaults result in consistent use of pooled error terms across both types of ANOVA models.

  (As an aside, I recall that for many decades SPSS used pooled error terms for repeated-measures. So they must have changed it recently.)

  R
• PerPalmgren

  April 2023 edited April 2023

  @andersony3k

  That is a valid point. However, the attached article advice to refrain from pooled error terms in RM designs.

  https://people.umass.edu/~bogartz/on_pooling_error_terms..html

  But maybe this article is referring to the omnibus test (have only skimmed it through).

  Per😎