[open] JASP: Standard Error in Post-hoc comparisons Repeated Measures ANOVA

Comments

• RaviSelker

  March 2016 edited 7:55PM

  We use the lsmeans package in R (in combination with the afex package) to calculate post-hoc tests for repeated measure. Here's an example of how we use it:

  # load packages
  require(afex)
  require(lsmeans)
  
  # simulate data set
  set.seed(1)
  dataset <- data.frame(subject = rep(1:20, each = 3), 
                        group = rep (c("A", "B", "C"), 20), 
                        measure = c(rnorm(20, .5), rnorm(20, .6), rnorm(20, .7)))
  
  # fit rm anova using the afex package
  model <- aov_car(measure ~ Error(subject/group), data=dataset, type= 3)
  
  # get the least square means
  referenceGrid <- lsmeans(model, ~group)
  
  # pairwise comparisons
  summary(pairs(referenceGrid, adjust="bonferroni"))
  
  # reshape to wide format for JASP and SPSS
  datasetWide <- reshape(dataset, timevar = "group", idvar = "subject", direction = "wide")
  
  # save data set for analysis in JASP and SPSS
  write.csv(datasetWide, "rmDataset.csv")
  

  This code indeed doesn't give the same p-values and SE as SPSS gives. I will contact the author of the afex package next week to see what's happening here. Thanks for your question.
• Nathan

  March 2016 edited 7:55PM

  Ok, thanks for looking into it.
  I'm curious to hear what's going on.
• RaviSelker

  April 2016 edited 7:55PM

  Sorry for not getting back to you sooner. I discussed it with the maintainer of the R package and it appears that the R function assumes that the variances (and thus the SE's) are equal to each other. This makes sense because one of the assumptions in ANOVA is equality of variances. Letting that assumption suddenly go for the post-hoc tests seems like weird behavior from SPSS. I will look into it a bit further to see which method makes more sense (or maybe make an options where you can specify whether you want to assume equal variances or not, but that is gonna take some time).
• Nathan

  May 2016 edited 7:55PM

  Ah, right. I may be wrong, but testing whether the SEs are more or less equal seems different from just setting them as equal. I don't know what's best here, but I would love to hear your thoughts about it when you have time. Thanks for the help so far.
• Dr.billgrisham

  May 2017 edited May 2017

  I had a similar/related problem with the SEM that comes with the marginal means. Th SEM that JASP reports does not match the SEM that SPSS provides for a between subjects design ANOVA. Why is this
• EJ

  May 2017

  Hi Bill,
  I can have someone look into this, but it would certainly help us if you could provide a specific example so we have some concrete numbers to work with.
  Cheers,
  E.J.
• Dr.billgrisham

  May 2017

  Attached is a JASP output of a simple between subjects ANOVA with 2 levels. The SE associated with the marginal means are identical in this output.
  The descriptive statistics show that the SDs are markedly different between the two groups, Given that the SE of the mean is the SD/sqrt of n, the SEs with the marginal means cannot possibly be correct. (The SD values given in the descriptive stats check with Excel.)

  So why does JASP not give the correct SE value with the marginal means?
• EJ

  May 2017

  Hi Bill,
  Thanks. I will look into it, but I can already tell you that this will most likely be an issue with an R package.
  E.J.
• JohnnyB

  May 2017

  Hi Bill,
  You are correct in that the marginal means table produces different standard errors than what you would expect from the descriptives table. JASP uses the "lsmeans" R-package to do the computations for the marginal means table. When there is a balanced ANOVA, the standard errors for estimating the confidence intervals get pooled. Whenever n differs per level of the factor, these standard errors do not get pooled and the marginal means table displays differing standard errors.
  In short, the standard errors in the marginal means table are used for testing whether the marginal means differ from 0, whereas the standard deviations reported in the descriptives table are purely descriptive. However, it would be nice to have an additional tickbox that could be unticked to use the unpooled standard errors in these tests - I will discuss this with the programmers and put it on the JASP bucketlist
  Kind regards,
  Johnny
• FranckM

  August 2023

  Hi,

  I'm bringing up this old discussion, as I've just encountered this problem. JASP gives me the same SE for all posthoc comparisons, whether I check the "pool error terms for RM ANOVAs" box or not. I don't get it. 

  Franck
• JohnnyB

  August 2023

  Hi @FranckM ,

  The box should lead to different SE's for within subjects factors - are you perhaps looking at a between subjects factor that is in your RM design?

  Cheers

  Johnny
• FranckM

  August 2023

  Hi Johnny and thanks for your answer.

  There is no between-subjects factor in my analysis. Here are two screenshots :

  

  

  What do you think?

  Franck