Timing of expyriment: it is pretty good!
in Expyriment
I removed the dust from my Blackbox toolkit and ran a quick timing evaluation of expyriment, similar to the one performed in the "Mega-Study".
I find that expyriment performs quite well when programmed correctly (see the attached file)
More raw data (.e.g. expeyrment event and data .xpe files) are available at https://github.com/chrplr/bbtkv2_python/tree/main/tests/expyriment
I intend to perform a more thorough investigation of timing when I find the time.
But this is enough to show that the Mega Study is quite misleading.
Comments
Hi Christophe,
yeah, unfortunately, the "Mega-Study" is quite problematic for multiple reasons (which we all outlined in our commentary: https://www.expyriment.org/timing-accuracy/).
Thanks a lot for doing those tests and making them available! I really appreciate that. They are much more in line with the results we have observed in all our previous tests. Your data shows that Expyriment sometimes skips or misses one exact refresh rate, which is not too unusual (at least the missing), but then there also seem to be a few outliers that fall between refresh rates. Those are quite interesting, and I wonder whether they could be measuring errors? It would also be interesting to see the Expyriment test suite results for the visual timing test on that machine. You might be able to improve on your results still by taming the video card's power saving mode and turning off compositing in Ubuntu.
We actually also started an effort to replicate the exact test scenario from the "Mega-Study" at some point in the past, but then Covid hit and that effort was put on hold. If you intend to run more tests, maybe you can have a look at our Expyriment scripts as well here: https://github.com/fladd/collaborative_timing_tests
All the best,
Florian
Thx for your answer.
Indeed, I saw several flaws in the experiment script used for the mega study.
Otherwise, I was also surprised by the durations that were not a multiple of refresh rate. It could be my settings of the opto threshold on the bbtkv2. I need to check with an oscilloscope recording the timecourse of the photodiode.
I did not know the existence https://github.com/fladd/collaborative_timing_tests I actually sent a pull request on experiment-stash for a similar script two days ago... It is probably useless with this repo.
I will run your test with the bbtkv2. One suggestion: maybe we could add to your repo `collaborative_timing_tests` a Markdown file containing a table listing hardware/OS and the results. THis may guide people who want to buy hardware.
Finally I reran my timing test on my current laptop, a Dell Latitute Precission 5530 and, when the Nvidia GPU is selected with `prime-select`, not a single frame is missed on 1000 trials although I can see ~6% misses when the integrated intel driver is selected). All this under Ubuntu 20.04.
Dear @chrplr,
would it be okay if we included a link to those tests (https://github.com/chrplr/bbtkv2_python/tree/6383bea065e45bc082af74db6ef6c81a14ea8470/tests/expyriment) at our documentation website, once the new release is out?
Best,
Florian