Building experiment with videos

Jo_ALV · September 2018

Hey everyone,

Following a recent mail exchange with Florian, I tranfer my issues to the forum to get some insight from the community.

For my first experiment I need to design an experiment using moving stimuli and I have to implement an expending or concentric movement on a circle (increase and decreasing the diameter of the circle in a way) and on annular gratings (Gabor Patch). So we conclude that it would be easier to just create videos of the stimuli and to present them.

So i went over the begginer's tutorial and tried to follow this example to create a very simple, 1 block, 4 stimuli experiment. I also follow the instruction about playing videos but I'm unsure on how to build and on how to code this efficiently. I coded the following experiment this morning but only return an error during the preloading of the video.

Anyway, I'll join what I coded so we can share thoughts and ways to improve things so that I can learn to efficiently code experiments in the future.

`import expyriment

exp = expyriment.design.Experiment(name="First Experiment")
expyriment.control.initialize(exp)

block_one = expyriment.design.Block(name="Block_1")

trial_one = expyriment.design.Trial()
trial_two = expyriment.design.Trial()
trial_three = expyriment.design.Trial()
trial_four = expyriment.design.Trial()

circle_s = expyriment.stimuli.Video('Circle_S.mp4')
circle_s.preload()
circle_s.play()
circle_s.present()
video_presentation_time = exp.clock.time
circle_s.wait_end()
circle_s.stop()

circle_e = expyriment.stimuli.Video('Circle_E.mp4')
circle_e.preload()
circle_e.play()
circle_e.present()
video_presentation_time = exp.clock.time
circle_e.wait_end()
circle_e.stop()

gabor_s = expyriment.stimuli.Video('Gabor_S.mp4')
gabor_s.preload()
gabor_s.play()
gabor_s.present()
video_presentation_time = exp.clock.time
gabor_s.wait_end()
gabor_s.stop()

gabor_e = expyriment.stimuli.Video('Gabor_E.mp4')
gabor_e.preloas()
gabor_e.play()
gabor_e.present()
video_presentation_time = exp.clock.time
gabor_e.wait_end()
gabor_e.stop()

trial_one.add_stimulus(circle_s)
trial_two.add_stimulus(circle_e)
trial_three.add_stimulus(gabor_s)
trial_four.add_stimulus(gabor_e)

block_one.add_trial(trial_one)
block_one.add_trial(trial_two)
block_one.add_trial(trial_three)
block_one.add_trial(trial_four)

exp.add_block(block_one)

expyriment.control.start()
for block in exp.blocks:
for trial in block.trials:
trial.stimuli[0].present()
exp.clock.wait(1000)

expyriment.control.end()`

Also; the error I get looks like :

Traceback (most recent call last): File "test.py", line 20, in <module> circle_s.preload() File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/expyriment/stimuli/_video.py", line 303, in preload videorenderfunc=self._update_surface) File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/mediadecoder/decoder.py", line 73, in __init__ if not self.load_media(mediafile, play_audio): File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/mediadecoder/decoder.py", line 162, in load_media self.clip = VideoFileClip(mediafile, audio=play_audio) File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/moviepy/video/io/VideoFileClip.py", line 91, in __init__ fps_source=fps_source) File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/moviepy/video/io/ffmpeg_reader.py", line 33, in __init__ fps_source) File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/moviepy/video/io/ffmpeg_reader.py", line 243, in ffmpeg_parse_infos is_GIF = filename.endswith('.gif') TypeError: endswith first arg must be bytes or a tuple of bytes, not str

thanks again for your time,

fladd · September 2018

The error is related to a bug. This has been fixed, but no official new version of Expyriment has been released yet.

You can either use the development snapshot directly from the repository (github.com/expyriment/expyriment/) or you can install a beta version of Expyriment (https://github.com/expyriment/expyriment/releases/tag/v0.9.1b2).

The new official release of Expyriment (0.9.1) will include this fix.

Jo_ALV · September 2018

Hi !

Thanks for that, switching to the beta version fixed the problem.
Now during the preparation of the experiment I get this error message :

Traceback (most recent call last):
  File "Pilot_test.py", line 24, in <module>
    circle_s.present()
  File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/expyriment/stimuli/_video.py", line 496, in present
    self.update()
  File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/expyriment/stimuli/_video.py", line 514, in update
    self._surface, quadDims=(1,1,1,1), position=self._position)
  File "/home/jrdnalvs/anaconda3/envs/expyriment/lib/python3.4/site-packages/expyriment/stimuli/_visual.py", line 62, in __init__
    surface_size = surface.get_size()
AttributeError: 'numpy.ndarray' object has no attribute 'get_size'

The video file is in .mp4, so the experiment video module should be able to launch it.
Is it because of a mistake in the experiment coding ? Should I specify the size of the screen/canvas to presents the video on ?

Thanks again,

fladd · September 2018

Ah, now you ironically ran into another bug. This bug, however, is also fixed already, but not in the beta yet. We will release a new beta with the fix soon, but for now, you can install Expyriment directly from the repository. That should work.

Jo_ALV · September 2018

Hi !
Is installing from the repository = git clone from the git hub page in my environment ? Because if yes it still really doesn't work (now I return the error that design isn't a module in expyriment). If not there's no real documentation on how to do that in the installation docs pages. Can you help/guide through it ?

fladd · September 2018

Hi,

easiest is to just download the zip (expyriment-master.zip) file. You install it then in the standard way to install Python packages: pip install expyriment-master.zip (assuming you are in the directory wherethat file is located).

Jo_ALV · September 2018

Hi ! Thanks for that !

It's now working, the videos are loading and are presented so I can go back to my experiment design.
One question now would be : how do I get a smooth rendering of the videos ? Because it keeps on dropping frames resulting in a non smooth video and poor quality.

fladd · September 2018

What resolution are the videos in? On the machines I tested it on (Windows, Mac, Linux) I could play full HD videos without dropping any frames.

How exactly are you showing the videos? Can you post your code?

Jo_ALV · September 2018

The videos were created in 4K res.

For now I just tested out the basic way of preload, play, present, stop & unload. Now I'm trying to create trials that will present 100+ the videos to present in one a block.

fladd · September 2018

Since the videos are played without hardware acceleration, 4K can indeed lead to dropped frames, I am not too surprised by this. It's a very high resolution. You might consider recoding to full HD (or try with a more powerful machine; i.e. with a more powerful CPU). Maybe changing the video format can also help (to something uncrompressed), but I am not sure about this.

Jo_ALV · September 2018

Hi Florian !

So I changed the stimuli and now everything run smoothly. I also found on a discussion here something some how similar to what I want to do so I tried to use it as a base to code something for me.
I don't get any error, the experiment seems to run as I get to the goodbye message but it won't present the videos.

Here's what I tried to code to loop and present the videos randomly multiple time :

import expyriment as xpy

exp = xpy.design.Experiment("First experiment")
xpy.control.initialize(exp)

videos = {"circle_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_S.mp4'),
          "circle_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_E.mp4'),
          "gabor_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_S.mp4'),
          "gabor_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_E.mp4')}
for video in videos:
    videos[video].preload()

canvas = xpy.stimuli.BlankScreen(colour=(255,255,255))
canvas.preload()

for motion_type, patterns in {"shrinking": ["circle_s", "gabor_s"],
                              "expanding": ["circle_e", "gabor_e"]}.items(): 
    block = xpy.design.Block()
    block.set_factor("MotionType", motion_type)
    for trial_repetition in range(100):
        trial = xpy.design.Trial()
        pattern = xpy.design.randomize.rand_element(patterns)
        trial.set_factor("Pattern", pattern)
        trial.add_stimulus(videos[pattern])
    exp.add_block(block)

########## START ##############
xpy.control.start()

canvas.present()

videos.present()
videos.play()
videos.wait_end()
videos.stop()

######## END EXPERIMENT ########
xpy.control.end(goodbye_text="Thank you for participating",
                       goodbye_delay=3000)

I also tried some variations of that such as :

xpy.control.start()

canvas.present()
for block in exp.blocks:
    for trials in block.trials:
        videos.present()
        videos.play()
        videos.wait_end()
        videos.stop()

Or also things along those lines :

xpy.control.start()

canvas.present()
for block in exp.blocks:
    for trials in block.trials:
        block.trials.stimuli.present()
        block.trials.stimuli.play()
        block.trials.stimuli.wait_end()
        block.trials.stimuli.stop()

But nothing seems to do the trick. Any advice on how to get this to play the videos ?
Thanks for the support !

fladd · September 2018

Hi,

Two things:

You forgot to add your trial to the block:

for trial_repetition in range(100):
    trial = xpy.design.Trial()
    pattern = xpy.design.randomize.rand_element(patterns)
    trial.set_factor("Pattern", pattern)
    trial.add_stimulus(videos[pattern])
    block.add_trial(trial)
exp.add_block(block)

You need to address a specific stimulus, not the entire stimulus list. Since you only add a single stimulus, it is the first element:

for block in exp.blocks:
    for trial in block.trials:
        trial.stimuli[0].play()
        trial.stimuli[0].present()  # only needed if you need the exact timing of this presentation
        trial.stimuli[0].wait_end()
        trial.stimuli[0].stop()

Jo_ALV · September 2018

Hi Florian !

Thanks for that ! I tried it and it runs all the trials. There's also some frame that are still droped even after I lowered the quality of the video but I guess this is just my personal settings that are not as powerful as I wished.
There's also this things, sometimes, the BlankScreen doesn't load and leave the back screen black and only the video playing in the middle. Besides my graphic settings, do you think of any possible other causes for that ?

Thanks again for everything,
Jordan.

fladd · September 2018

Well, in your code above you only present the blank screen once at the beginning of the experiment.Not sure how your current code looks like.

Jo_ALV · September 2018

It's pretty much the same as it is above, I just added your comments.
Should I implement a blankscreen before each stimuli presentation ?
The thing is, it's there sometimes but the pattern seems random. It just appears for 10+ stim and then it's not presented again etc..

fladd · September 2018

Given your code, the presentation of the blank screen cannot be random. In fact, it is only presented a single time, in the beginning. Afterwards, it is not presented anymore. If you are still seeing it afterwards, it might be related to a potential double buffer.
If you want to see it before each stimulus, then you have to present it explicitly before each stimulus.

Jo_ALV · September 2018

Okay !
I added it on the loop of stimuli presenting, ending up to something like :

xpy.control.start()

canvas.present()
for block in exp.blocks:
    for trial in block.trials:
        canvas.present()
        trial.stimuli[0].play()
        trial.stimuli[0].present()
        trial.stimuli[0].wait_end()
        trial.stimuli[0].stop()

If I want to link a different marker to each stimuli, should I embed the expyriment.io.MarkerOutput function in the stimulus or should I create another object and include it in the loop ?

fladd · October 2018

You only need one marker output object. On every call to its send method you can tell it what marker to send. Please also see: http://docs.expyriment.org/expyriment.io.MarkerOutput.html

Jo_ALV · October 2018

Hi !

So I tried things but it does seem to do what I want it to do. Also, I can't really seem to get the logic behind that because in my attempt here :

import expyriment as xpy
xpy.control.defaults.open_gl = False
samplerate=5000

exp = xpy.design.Experiment("First experiment")
xpy.control.initialize(exp)

videos = {"circle_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_S.mp4'),
          "circle_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_E.mp4'),
          "gabor_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_S.mp4'),
          "gabor_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_E.mp4')}

for video in videos:
    videos[video].preload()

markers = {'1':xpy.io.MarkerOutput(interface=xpy.io.SerialPort(port='name'),
                                   default_code=1, 
                                   default_duration=1000/samplerate),
           '2':xpy.io.MarkerOutput(interface=xpy.io.SerialPort(port='name'),
                                   default_code=2, 
                                   default_duration=1000/samplerate),
           '3':xpy.io.MarkerOutput(interface=xpy.io.SerialPort(port='name'),
                                   default_code=3, 
                                   default_duration=1000/samplerate),
           '4':xpy.io.MarkerOutput(interface=xpy.io.SerialPort(port='name'),
                                   default_code=4, 
                                   default_duration=1000/samplerate)}

canvas = xpy.stimuli.BlankScreen(colour=(255,255,255))
canvas.preload()

for motion_type, patterns in {"shrinking": ["circle_s", "gabor_s"],
                              "expanding": ["circle_e", "gabor_e"]}.items():   # use iteritems() in Python 2
    block = xpy.design.Block()
    block.set_factor("MotionType", motion_type)
    for trial_repetition in range(100):
        trial = xpy.design.Trial()
        pattern = xpy.design.randomize.rand_element(patterns)
        trial.set_factor("Pattern", pattern)
        trial.add_stimulus(videos[pattern])
        block.add_trial(trial)
    exp.add_block(block)

########## START ##############
xpy.control.start()

canvas.present()
for block in exp.blocks:
    for trial in block.trials:
        canvas.present()
        trial.stimuli[0].play()
        trial.stimuli[0].present()
        trial.stimuli[0].wait_end()
        trial.stimuli[0].stop()

######## END EXPERIMENT ########
xpy.control.end(goodbye_text="Thank you for participating",
                       goodbye_delay=3000)

what I present is randomized. So I'm afraid it will call any random marker when I want it to send a specific marker for a specific video, not to choose randomly in the markers.
So I'm kind of lost on how to actually code this. I went to the doc but I don't really get it either.

Thanks for helping
Jordan.

fladd · October 2018

In your code, you never send any marker. Also, you do not need to create a marker object for each marker.
Assuming, that you want to send a marker everytime a video is being presented, here is some code:

import expyriment as xpy
xpy.control.defaults.open_gl = False
samplerate=5000

exp = xpy.design.Experiment("First experiment")
xpy.control.initialize(exp)

videos = {"circle_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_S.mp4'),
          "circle_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Circle_E.mp4'),
          "gabor_s": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_S.mp4'),
          "gabor_e": xpy.stimuli.Video('/home/jrdnalvs/Documents/Pilot_exp/Stim/Gabor_E.mp4')}

for video in videos:
    videos[video].preload()

marker = xpy.io.MarkerOutput(interface=xpy.io.SerialPort(port='name'),  # don't forget to replace 'name' by the actual port name!
                             default_duration=1000/samplerate) # not sure why you need this, but I left it in
markers = {'circle_e': 1,
           'circle_s': 2,
           'gabor_s': 3,
           'gabor_e': 4}

canvas = xpy.stimuli.BlankScreen(colour=(255,255,255))
canvas.preload()

for motion_type, patterns in {"shrinking": ["circle_s", "gabor_s"],
                              "expanding": ["circle_e", "gabor_e"]}.items():   # use iteritems() in Python 2
    block = xpy.design.Block()
    block.set_factor("MotionType", motion_type)
    for trial_repetition in range(100):
        trial = xpy.design.Trial()
        pattern = xpy.design.randomize.rand_element(patterns)
        trial.set_factor("Pattern", pattern)
        trial.add_stimulus(videos[pattern])
        block.add_trial(trial)
    exp.add_block(block)

########## START ##############
xpy.control.start()

canvas.present()
for block in exp.blocks:
    for trial in block.trials:
        canvas.present()
        trial.stimuli[0].play()
        trial.stimuli[0].present()
        marker.send(markers[trial.get_factor("Pattern")])
        trial.stimuli[0].wait_end()
        trial.stimuli[0].stop()

######## END EXPERIMENT ########
xpy.control.end(goodbye_text="Thank you for participating",
                       goodbye_delay=3000)

Jo_ALV · October 2018

Okay ! Thanks a lot for the correction & the support
I'll get the name of the port name & test it out !

yoniastern · January 2019

Hey Florian
I am also trying to play a video in an experiment, and have run into the the same bugs described above.
I have updated to beta, when trying to downlaod the zip as described
"Hi,

easiest is to just download the zip (expyriment-master.zip) file. You install it then in the standard way to install Python packages: pip install expyriment-master.zip (assuming you are in the directory wherethat file is located)."
I receive the following error

I'm a newbie to python
Thanks a lot
Yoni

fladd · January 2019

Hi Yoni,

I actually realized that, too, yesterday. I fixed it, if you redownload the zip file it should work. Sorry for the confusion. We are preparing for the next release.

yoniastern · January 2019

Hey Florian,
Thanks for the help,
Now that I download the zip I get another error (attached below). I am running on windows 10, with python 3.7 (32 bit). I have tried uninstalling Expyriment but it hasn't helped either
Thanks

yoniastern · January 2019

also running it with python 3.6 runs into the same problem

fladd · January 2019

Difficult to see from the short snippet you posted, but it appears that you are installing the source via zip? In that case, did you update setuptools and wheel first?

What happens if you install it normally (i.e. via the setup.py)?

yoniastern · January 2019

Yes I am trying to install via the zip as posted above.
Sorry for the beginners question, but what do you mean by normally?
Pressing on it?
Typing in to the command line the setup.py location returns the following

Thanks again for all the help

yoniastern · January 2019

and trying to run setup.py script via pycharm returns the following:

fladd · January 2019

Make first sure that setuptools and wheel are up to date:

pip install -U setuptools
pip install -U wheel

Then from within the expyriment-master directory:

pip install expyriment-master.zip

An alternative (and what I was referring to with "normal" above) is this to run this from within the unzipped expyriment-master directory:

python setup.py install

Tanto · February 2019

Hi all,
I've a few question about using video as stimuli:
1). If this possible to collect reaction times while video runs?
2). I played video with frame height 364, frame weight 380, frame per second 11frames/second and lenght 6 second. The video runs but not smoothly, the video frame still dropped. For the first time I think my laptop that cause the dropping frame so I try with my PC with Ryzen 5, and 8GB ram and still the same. All expyriment package are up to date.
Great thanks for the reply

fladd · February 2019

Hi,

what kind of video is this (i.e. what format and specs of the encoding)?
I am assuming that you are using the new video implementation in Expyriment? With this I can play full HD videos here on a 2013 MacBook Pro without dropping frames. Can you maybe make the video available so I can check if it works here?

Howdy, Stranger!

Categories

Building experiment with videos

Comments

Howdy, Stranger!

Quick Links

Categories

Building experiment with videos

Comments