[open] Timing Accuracy and Stimuli Onset Times

rip

Hello,

I'm evaluating OpenSesame for a study that requires accurate timing and I'm trying to get an idea of how accurate the timing is. I've built an experiment that just displays a single image for 20 cycles at durations of 34 and 84ms (which I would think should yield durations of 50 and 100ms respectively taking the added refresh into account). My sequence includes a single sketchpad item and a logger. I am logging the time_sketchpad variable.

The durations that are being logged are frankly bizarre. Setting the sketchpad duration to 84ms yields the following
results (this is a smaller sample):

time_sketchpad -	- onset to onset difference
1837	250
2087	267
2354	283
2637	284

I calculated the onset to onset difference in the second column.

Questions:

1) Why are the durations all greater the 200ms?

2) Why isn't there a constant duration error?

I would expect the onset to onset difference to be ~100.67ms (84ms duration + 16.67ms for missed refresh).

I'm using the Expyriment back-end

Thank you

absurd

How did you design your experiment?

With a loop item?

Then these delays would be explainable, as OS uses a prepare run strategy (see: here). The computer will prepare the whole trial sequence before running it and the additional delay might be the time your computer needs for this preparation. Therefore always put everything that needs to be timed accurately within one sequence.

In other words:
If that's what you did, try it out again by putting everything within a trial (a.k.a. sequence item).

Best,
Johannes

rip

In other words: If that's what you did, try it out again by putting everything within a trial (a.k.a. sequence item).

Everything is in a sequence item. The prepare and run strategy is a poor choice with
sequence items that have only one event in them. It looks like it's a combination of the data logger and stimulus preparation that causing the unexpected onset delays.

The problem is that this method of bulk logging and stimuli preparation is adding
what is essentially a random jitter between each trial in the experiment. This is not going to be acceptable for most of my tasks.

Edwin

The loop and sequence items were designed with one trial in mind. That is: one trial is represented by one cycle in the loop (allowing you to set the specific parameters in that trial with variables), and runs through a single sequence. In this light, the strategy of preparing an entire sequence at once and then running it, makes a lot of sense.

It is true that it adds random jitter to the end/start of a trial, but in most designs this is simply not a problem (it might even be beneficial, e.g. in event-related fMRI).

Even though using one item per sequence in some cases might be a good idea, I expect your experiment can be adapted to fit within one sequence. If you give us some specifics (that is: what do you want your experiment to do?), we might even be able to help you do it.

BTW: I think it's a very good idea to read what the documentation pages have to say about OpenSesame's timing

rip

The loop and sequence items were designed with one trial in mind. That is: one trial is >represented by one cycle in the loop (allowing you to set the specific parameters in that >trial with variables), and runs through a single sequence.

Yes that's how I'm using the sequence.

In this light, the strategy of preparing an entire sequence at once and then running it, >makes a lot of sense.

Wouldn't a more sensible approach be to load the next stimulus while the current one is being presented (a la E-Prime, Presentation, etc...)?

It is true that it adds random jitter to the end/start of a trial, but in most designs this >is simply not a problem (it might even be beneficial, e.g. in event-related fMRI).

In fMRI sure, but even there I would want the jitter durations to be to known ahead of time. With OpenSesame, the loading time varies depending on how much is being logged and how much preloading needs to be done. And what if I actually do want to add a more controlled jitter? It would be difficult to do that being that OS is adding additional (unpredictable) jitter already.

And I disagree that it's "simply not a problem". I think if you told most researchers their stimulus presentation system was adding a random jitter between the trials in
their experiment they wouldn't be very happy.

BTW: I think it's a very good idea to read what the documentation pages have to say >about OpenSesame's timing

I have, I'm just surprised that this is the way it works. Yikes.

Edwin

To clarify: You are not using the sequence in the advised way. You mention in the initial post that you use a single sketchpad per sequence, and calculate the difference between the onsets of two consecutive sketchpads (in two consecutive sequences, or trials). This means you are comparing across cycles, which means you are measuring the preparation time of the next cycle. Hence my referral to the Documentation page, where this is explained.

To solve your problem, add a blank sketchpad (Duration of 0 ms) directly after the stimulus sketchpad (Duration set to the required presentation time). Then look at the onset differences between the two to test whether the timing was accurate.

The only issue you are then left with, is the random jitter between trials. This can be large (tens of milliseconds) if the sequence requires a lot of preparation (e.g. when using large images). However, ITI jitter is a problem only in very few studies (I should add: in my experience). If yours is one of those, I would definitely advise you to use a programming language rather than a graphical interface, as the former gives you full control over your timing.

Would it be more sensible to load the next image during the presentation of the current? For some designs, maybe. But I think your design here is a good example of the potential issues with such an approach: your stimulus requires a longer loading time than a single refresh cycle takes. Therefore, preparation of image n+1 during the presentation of image n would render it impossible to show image n for less than two frames.

PS: to account for non-constants trial preparation durations, add the following code to an inline_script placed at the start of your sequence:

Prepare phase

# timestamp of preparation start
exp.set("trialprepstart", self.time())

Run phase

# calculate preparation duration
exp.set("trialprepduration", self.time()-exp.get("trialprepstart"))

# pause for the required amount of time, e.g. 2 seconds
self.sleep(1990 - exp.get("trialprepduration"))

absurd

Wouldn't a more sensible approach be to load the next stimulus while the current one is being presented (a la E-Prime, Presentation, etc...)?

That would be nice, indeed, though -- according to my experience with Presentation -- this only works well if you have really a well chosen hardware and, most of all, a huge amount of RAM (at least if you present more than just pictures). I did a dummy experiment 2 years ago with visual and auditory stimuli. I set it up both with Presentation and OpenSesame on the same, not very new, PC (with just 4GB RAM -- at that time this was already something). I had jitters between trials both in Presentation and Opensesame. Overall, delays and jitters in Presentation very even larger than in Opensesame.

With a preload strategy, you also have to employ a really accurate and quick "freeing the RAM again" strategy, which is very dependent on how quick your overall hardware setup is (mainboard, RAM, etc...). In many cases this preloading strategy also leads to crashes (which are probably mostly due to running out of memory), which is most of the time far worse than anything else. A preload strategy is also more difficult to maintain if you develop a software that should run on different operating systems, I guess...

Anyway, if you put everything that has to be timed accurately into the same sequence then ITI jitter is no problem. Sometimes putting everything into the same sequence is a little more work and less elegant, that's true. But so far I could always find some solution/trick with the help of the nice community here ;-)

For this it would be necessary, though, that you post some more details of your Opensesame script and explain a little more what you want to achieve in detail.

Best, Johannes.

rip

This means you are comparing across cycles, which means you are measuring the >preparation time of the next cycle. Hence my referral to the Documentation page, >where this is explained.

To solve your problem, add a blank sketchpad (Duration of 0 ms) directly after the >stimulus sketchpad (Duration set to the required presentation time). Then look at the >onset differences between the two to test whether the timing was accurate.

But that wouldn't solve my problem. It's the between cycle jitter that I'm concerned about.
Adding another sketchpad will just push the jitter down one event. It's still going to be there.

With a preload strategy, you also have to employ a really accurate and quick "freeing >the RAM again" strategy, which is very dependent on how quick your overall hardware >setup is (mainboard, RAM, etc...).

Right which is why the "load while presenting" strategy is better. Preloading all the stimuli for the next trial "upfront" certainly uses more memory than preloading the next
event's worth of stimuli during presentation no? It also removes the inter-trial jitter.

Anyway, if you put everything that has to be timed accurately into the same sequence >then ITI jitter is no problem.

You all keep saying this but it's simply not true. OpenSesame is inserting a fairly large, random ITI into my experiments. Suppose I'm study habituation, and I don't know that OS is inserting this jitter (which I'm going to say is more than likely for most of your users). My results are flawed, period. There are many other areas of study that would similarly be effected by this. Hoping your users have a read a paragraph hidden away in your docs and are using something like the script Edwin described to compensate is really wishful thinking no? It would be interesting to do a poll of OS users to see, maybe I'm wrong and everyone knows about this behavior.

Well thank you for your responses. It's been very informative.

absurd

Hi rip,

Well, it is true what we are saying. If you put stimuli within one sequence they are accurately timed (all shorter jitter or delay occurring there is due to your hardware then. These delays occur no matter what presentation software you use (i.e. even with Presentation and the like. The only solution to this is getting better hardware).

A sequence is equivalent to a trial so what we wanted to say is, if you enlarge your trials in a way that everything is inside that needs to be accurately timed, then ITI jitter cannot be an issue, because you are doing a workaround. If this strategy is not what you want you can program your own solution by using inline or external scripts.

The 'load while presenting' strategy still needs a good memory management which I sadly experienced to be not as good, stable and accurate (at least in Presentation) as it should be. That might not be a problem of Presentation itself, but of ones hardware interacting with it.

It is true, jitter can flaw results. You should discern within-trial-jitter and ITI jitter.
In Opensesame and all other presentation software within-trial-jitter is very small and due to your specific hardware (i.e. refresh rate of your screen, onset speed of your sound card, etc.). Although you can reduce this by the right adjustments (e.g. enabling vsync, see documentation on timing). If this jitter is an issue, depends on how you want/need to design your experiment. If you want to have really exact millisecond timing, you should either buy a complete system (software+hardware from one manufacturer) as it is e.g. available from Neuroscan. These complete solutions are very expensive, though. Another solution would be -- if you have the knowledge or you can employ a technician and buy the appropriate measuring tools -- to carefully select and build up your own hardware.
ITI jitter is larger with all software. It is less large with software that uses some kind of preload strategy, though this is often not a perfect way out, as preloading again depends on your hardware, this reduces your overall ITI jitter but may introduce other hardware related jitter and may sometimes decrease overall stability of the experiment run.

That said, if you put all time sensitive procedures into the same sequence (as a workaround to the ITI jitter) you are left only with timing inaccuracies that are due to your hardware (if you did not do anything wrong elsewhere in your experiment programming, e.g. 'show if' vs. 'run if'). That workaround might be not very elegant with some designs, but as I said, you could program that part as an external script.

Hoping your users have a read a paragraph hidden away in your docs and are using something like the script Edwin described to compensate is really wishful thinking no?

Nothing is 'hidden' in the documentation there is a separate section on timing in which all timing related things are discussed and further information is linked. Timing is a technically very complicated issue. Opensesame does not hide this. I have worked with other presentation software that claimed to have millisecond precision where this just was not documented at all. When I compared timing of OS and other software on my hardware and with my experimental designs, I found out that OS was even more accurate. So, don't believe any 'millisecond precision' advertising, it is a) related to your hardware and b) related to your programming. Loops are bad for timing in any software, you have to work around it or your software has some built-in workaround. With the latter you have to trust and believe that software, the first is more transparent and you have the possibility to control it. And if you are in doubt, you can always discuss it in this forum.

Best, Johannes.