How to assess if people got faster at doing X task over time?

sizzlinkola

Hello all! I'm a newbie to statistics in general and Bayesian especially. I was wondering what would be the best test to use on JASP for the following scenario.

I launched an app to help people get faster at doing X task. The app has been running for 3 months and I have data for each day on the time it takes to do X task. I have this data for even before app launch, as far back as 3 months prior. So I have 3 months before my app and 3 months after my app.

I want to see if there's evidence that people actually got faster at doing X task after using my app. Any suggestions on what tests I should do?

EJ

It is always tricky to pick the right design, because in the current setup there will be practice effects and effects of time (and possibly selection effects as drop-out may be related to performance?), so the comparison is potentially confounded. But OK, let's say these are the data you have. Then it would be best, imo, to have a time series model "pre-app", and a similar model "post-app", and do inference on the relevant model parameter (e.g., the speed of improvement, or the intercept -- depending on your model). In the time series literature, you could search for regime-switching models where you already know the time point of the potential regime switch. Of course the models can become easier if you don't assume learning or autoregression. Best to do some diagnostics first (are the data autocorrelated, pre-app and post-app? Are they stationary?) This gets complicated fast and it is not something we currently have automated in JASP.

Cheers,

E.J.