[open] test_suite

Comments

• sebastiaan

  March 2013 edited 1:22AM

  I have a question regarding the test_suite script. (...) what exactly does the second item, maximum refresh rate, signal?

  Monitors have a fixed refresh rate, and you want the software to take this into account. For example, if your refresh rate is 100Hz you can never present more than 100 frames per a second. With some configurations (using the legacy backend mostly) OpenSesame may nevertheless claim to do so and report impossibly high framerates. In this case, higher is not better: It simply means that frames are being skipped. Bottom line: The maximum refresh rate should match the monitors refresh rate.

  For a good explanation, see also

  • https://code.google.com/p/expyriment/wiki/Timing#Visual

  How does this influence our experiment?

  It does not necessarily mean that the timing of your stimulus presentation is off. It does, however, mean that it will be difficult to verify the timing afterwards, because the timestamps will not be perfectly reliable.

  In general, the test_suite.opensesame script is a bit outdated, and I should probably replace it by something better. This page contains a few newer test scripts that are specific to the new back-ends (the test_suite is from back when only the legacy back-end was available). If you want to test your system, I'd recommend using these scripts over the test_suite.

  Cheers!

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!
  
• Jákup

  March 2013 edited 1:22AM

  Hello.
  Thank you very much for your reply

  I found the new timing scripts after the fact We'll retest the PCs using those scripts. I was also concerned by the up to 3 ms delay in sketchpad and synth timing, but we are using the legacy backend, so perhaps we can get it down to 1 ms across the board, by using PsychoPy or Expyriment.

  And thanks for the expyriment wiki article. But I'm still a bit confused about the frame rate issue. How important are these max rates, and what could be the influence on our experiment if we use setups with different frame rates?

  Best regards,
  
  Jákup
• sebastiaan

  March 2013 edited 1:22AM

  I was also concerned by the up to 3 ms delay in (...) synth timing

  The timing of sound is in general much less precise than that of display presentation. Unfortunately, it's also very difficult (impossible maybe) to know the actual timing of the sound playback without doing benchmarks with external hardware. Some guys from Swansea have done great testing on this, which you can read about here:

  • http://forum.cogsci.nl/index.php?p=/discussion/261/open-hardware-for-timing-/p1

  How important are these max rates, and what could be the influence on our experiment if we use setups with different frame rates?

  The crucial thing is whether or not the software 'knows' when a display is actually presented on the screen. When you present a display with the legacy back-end, the timestamp that you get is the moment that the canvas.show() function is called. This is inaccurate, because there may be a small delay between the moment that you say show() and the moment that a new display appears, because a new display can only be shown when the monitor starts refreshing from the top (assuming that v-sync is enabled). So the maximum temporal error equals one refresh interval, which is 16.7ms on a typical 60Hz monitor.

  In contrast, psycho and xpyriment use a 'blocking flip'. This means that when you call canvas.show(), the program will block until the monitor starts refreshing from the top, and you get the timestamp of the exact moment at which the display is actually shown. For most purposes (such as games), this blocking is horribly inefficient, but for the purpose of experimenting it's usually what you want.

  Long story short: If OpenSesame reports a frame rate of 1000 fps, then it is clearly not using a blocking flip, because that many frames per second is physically impossible. And therefore, whether the maximum framerate matches the refresh rate is diagnostic for whether a blocking flip is used.

  Does that make sense?

  Practically speaking: The temporal error that we are talking about here does not affect your statistical power in any meaningful way, unless (maybe) you are doing things such as stimulus locked ERPs. So if you're doing a manual RT task, or something along those lines, don't spend too much time worrying about a 10ms jitter if you're using the legacy back-end!

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!
  
• Jákup

  March 2013 edited 1:22AM

  Yeah ours is a regular RT task, using 75 participants in an hour long experiment. So, I guess it will not matter if some of these PCs are off in respects to each other, as the noise will be covered through randomization and statistical power.

  I find that I actually prefer legacy over xpyriment or psycho. It's a lot faster, and doesn't "show at the seems", as it were, in the way the other two do. And as the workstations we're going to use are rather slow and dumb terminals, I think the better running speed of legacy will be preferable to increased temporal measurements.

  Btw. where do I enable v-sync? Is a graphics card or OpenGL settting?

  And, again, thank you very much for your in-depth explanation. This really helped clear up my remaining confusion

  Best regards,
  
  Jákup
• sebastiaan

  March 2013 edited 1:22AM

  Btw. where do I enable v-sync? Is a graphics card or OpenGL settting?

  Some graphics cards have a dedicated configuration panel that allows you to enable/ disable this, but generally speaking v-sync is enabled. You can find out really easily, though. Just present screens of different color in rapid alternation (using a simple experiment) and look for horizontal 'tearing' of the display, i.e. lines that run horizontally through the display more or less at random. If you don't see those, v-sync is enabled.

  Cheers!

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!
  
• Jákup

  March 2013 edited 1:22AM

  Which incidentally is just what the first sequence of the test_suite experiment does

  Thanks again man!

  Best regards,
  
  Jákup