[open] Using Video as Stimuli: how to bind key responses and sound onset to specific frames

Marnix

Hi there,

First of all thanks for putting out this great program! Quick question: in PsychoPy it is possible to start recording keypresses at certain frames. My experiment uses a paradigm where I need a 'beep' played at a certain frame (the first after an edit) and the participant will then press a button, the RT time of which also needs to be recorded. How do I anchor the keypress, the recording, and the sound to a certain frame?

Regards,
Marnix

Marnix

So anyone used video as stimuli before in Opensesame? I was hoping to solve it in the PSychoPy builder but the latency is so big it is nigh unusable, so I'll need to rerun the experiment in OpenSesame anyhow. I am guessing it must be really easy to do, I just have 0 computer skills. You'd be really helping me out if you culd answer my question!

sebastiaan

Hi Marnix,

There are a few video-player plug-ins available for OpenSesame, the best developed of which is probably the media_player_gst plug-in by @dschreij. You can find the installation instructions in the readme, here:

• https://github.com/dschreij/media_player_gst

Assuming that you're using the media_player_gst, the easiest thing to do is probably to set the duration to 'keypress'. This way you can easily collect the response, response time, etc, from the start of the video. (I actually experience some issues with this on my Linux system, but they might be Linux specific. If you experience trouble as well, please post a detailed description here.)

To get the response time relative to a specific frame, you simply subtract a value (based on the frame number and duration) from the regular response time during the analysis. So you do this during data analysis, and not during the experiment itself.

Regarding the beep: The easiest solution would be to embed the sound in the video itself. Is that what you're planning to do? Otherwise, you'll have to do things in parallel, which makes the experiment slightly more complex (but it's possible too).

Cheers!
Sebastiaan

Marnix

Hi Sebastiaan,

Thank you for your response! I think your suggestions are great and I added them to the mix. I have however two small problems I kept running into (and by now, I have tried for weeks already fiddling with all the settings/possible solutions I could think of):

• The video does not run in full screen mode, and when it does, it is so laggy or . This is funny since in windowed mode it works fine (and thats what I do now, so its not a major issue).

• More importantly, I keep having a problem of having key response/time being duplicated. At first, the first and second response would apparently be exactly the same, while the rest of the responses would be perfect (i.e. 1280 ms, 1280 ms, 3245 ms, 5431 ms etc) whereas with some fiddling, this will change the order of which response is duplicated (for instance 1280ms, 1344 ms, 1344ms, 4356ms, 6785ms, etc.). Adding reset feedback buttons does nothing to prevent this. When I remade the experiment in the exact same manner it now doesn't register the first keyresponse! I am at a bit of a loss here since it seems quite random, and the sequences are exactly the same save the video file being used.

I can upload a picture if this would help with the output. Cheers and thank you very much!

sebastiaan

Hi Marnix,

The video does not run in full screen mode, and when it does, it is so laggy or . This is funny since in windowed mode it works fine (and thats what I do now, so its not a major issue).

This is because some back-ends use OpenGL when running in fullscreen mode, which can, in some cases, slow things down. You should be able to work around this by switching to the legacy back-end, or by setting "Use OpenGL" to no in the xpyriment back-end (under back-end settings).

More importantly, I keep having a problem of having key response/time being duplicated. At first, the first and second response would apparently be exactly the same, while the rest of the responses would be perfect (i.e. 1280 ms, 1280 ms, 3245 ms, 5431 ms etc) whereas with some fiddling, this will change the order of which response is duplicated (for instance 1280ms, 1344 ms, 1344ms, 4356ms, 6785ms, etc.).

Could you describe in a bit more detail how you are collecting responses, or post some informative screenshots? I.e. what does your trial sequence look like, how do you collect and log the responses, etc

Cheers!
Sebastiaan

Marnix

Hi Sebastiaan, Thank you for your Help!

The data response is the space bar on a wireless usb keyboard. The video is displayed on an External Monitor connected to my laptop, and the USB keyboard is also connected to the laptop. Of course, I am not using the keyboard during data gathering!

So, the first setup is this:

and here is the logger

This is the outcome:

As you can see, 2 and 3 are the same, and they are in every trial.

So I tried to remove the text in the beginning, and I get this:

it doesnt measure the 1st keypress.

I tried to make a completely new one:

and I get this Error:

What I noticed from the last time screwing around with it, is when I used the standard logger which automatically detects and logs the variables, is that it would log the text display as the first item logged, not the 1st keypress in the video. But I have no idea how to change that.

Thank you again! And I think by now I owe you a beer if you ever go to Amsterdam

sebastiaan

Hi Marnix,

It looks like the logger sometimes precedes the media_player_vlc item. Since OpenSesame works sequentially, this means that you log before collecting a response. This probably explains your duplication issue, because the response variable will still have the value from the previous response.

You also have multiple logger items. This is not necessary and leads to repetition of column headers in your data file.

So basically: Use the same logger item in all your sequences, and have it be the last item in every sequence.

Does that resolve your issue?

Cheers,
Sebastiaan