Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Response time during video display

Hi there,

I'm currently converting an old e-prime experiment to Opensesame and I'm struggling to handle response times during video presentations. I'm using the media_player_mpy plugin as suggested here. I found similar threads on the forum but I couldn't make it work.

The trial sequence is as follows: Fixation (keypress) - Video (1100ms; response with a keyrelease) - Blank screen (1000ms).

The video clips are exactly 1100 ms long and in each of these clips, one of two events (let's call them 'event A' and 'event B') can occur precisely at 500 ms. Participants are asked to release '1' or '2' in response to 'event A' or 'event B', respectively.

So, I'd like to record participants' response times to these events (with a timeout of 1500 ms from the 'event' onset); so responses should be possible as soon as the video starts with a timeout that overlaps with the blank screen (hope that's clear). I tried several things (e.g., setting the duration to keypress doesn't seem to record anything). Any hints on this ?

Then, I don't exactly know how to handle this 'keypress'/'keyrelease' issue ... I found this on an existing thread but I don't really know how to implement it properly in the experimental sequence ...

import pygame key = None start_time = self.time() while key == None: time = self.time() for event in pygame.event.get(): if event.type == pygame.KEYUP: key = event.key rt = time - start_time exp.set('keyreleaseRT', rt) print '%s was released after %d ms' % (key, rt)

Any advice would be greatly appreciated ! Thanks a lot !

CL

Comments

  • EDIT:

    Some progress have been made based on this thread

    By adding this in the python code section of the video object, responses are collected as long as they are given during the video playback.

    if event: key = event[1] if var.correct_response == key: correct=1 else: correct=0 responses.add(response=key, correct=correct, response_time=clock.time()-var.time_video) continue_playback = True

    The problem is that responses should also be possible once the video has stopped playing (during the 1000 ms blank screen). As it is, 'nothing' is collected and for those trials where responses are given after the end of the video, the same response times as the trial before are collected. Any ideas ?

    Thanks

  • Hi Cletess,

    Can you print the rest of the code that you use? In general, extending the response collection period, requires only to loop a little longer before proceding with the next trial. And optionally, change the stuff that is presented on the screen during this interval. Once I know how you present everything, I can probably also give you more specific suggestions how you could implement it.

    Eduard

  • edited February 2017

    Hi again and sorry for the delay !

    We made some changes on the paradigm and I took some time to read different threads on this forum. It appears (maybe I'm wrong) that for what I need, I'll have to use openCV to deal with the video presentation. Let me explain:

    • After fixation, we want the videos to be played whenever a key is pressed (key '1' or '2' of the numerical pad).
    • From video onset, we would like to record participants' response time to one of two possible events that can occur in the videos by releasing one of the two key that was used to initiate the video ('1' or '2').
    • Finally, whenever participants make a response in a specific condition (presented in a variable in the trial loop), we would like a colored cue to appear on top of the video. This seems to be possible using the pygame.draw.rect command in opencv.

    I saw that this code was suggested in a previous post:

        import cv2
        import numpy
        import pygame
    
        path = pool[self.get('video')]
        video = cv2.VideoCapture(path)
    'insert wait for keypress here?'
        while go:
            success, frame = video.read()
            if not success:
                break
            frame = numpy.rot90(frame)
            # The video uses BGR colors and PyGame needs RGB
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            # Create a PyGame surface
            surf = pygame.surfarray.make_surface(frame)
            # Now you can draw whatever you want onto the PyGame surface!
            pygame.draw.rect(surf, (255,0,0), (25, 25, 25, 25))
            # Show the PyGame surface!
            surface.blit(surf, (0, 0))
            pygame.display.flip()
    

    For now, the screen just goes black for each trial and I can't display the videos. No specific error message. It seems that the value of 'Success' is always False therefore breaking the loop for some reasons. Also, I'm not sure on how to initiate the loop with a key press and to record the reaction time until it is released (the videos must not be stopped once the key is released). I tried something but I'm not really sure it's even correct ...

    key = None
    while key == None:
       start_time = self.time()
        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                key = event.key
                go = True
    
    'Insert video playback code here'
    
    while key == event.key:
       end_time = self.time()
        for event in pygame.event.get():
            if event.type == pygame.KEYUP:
                key = None
    end_time = self.time()
    rt = end_time- start_time  
    exp.set('keyreleaseRT', rt)
    print '%s was released after %d ms' % (key, rt)
    

    Thanks a lot for your advice

  • Hi,

    First, I haven't worked much with videos and not at all with OpenCV, so I can't help you with that. However, I think some of your questions are more general in nature. So, let's see whether I can help you.

    Also, I'm not sure on how to initiate the loop with a key press and to record the reaction time until it is released

    This is rather simple. Basically, when a key is pressed you have something like video.start() which should also give a time stamp for when exactly this happened. If there is not time stamp sent automatically by calling 'video.start', you can add a line like start_time = clock.time() right before you start the video. Importantly, releasing the key shouldn't stop the video, which won't happen unless you write something like "if key is released, stop video playback". So, whenever the key is actually released, you can take a second time stamp. Later, when the trial is over (after all the loops), you can stop the video, by something like "video.stop()". Below the part of the code that would accomplish that (pretty much the same as what you already have had):

    # note, you don't check for pressing a particular key, any key will do in this example
    key = None
    while key == None:
        for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                start_time = self.time()
                key = event.key
                go = True # doesn't do anything in this code snippet
    
    #'Insert video playback code here'
    
    while key == event.key:
        for event in pygame.event.get():
            if event.type == pygame.KEYUP:
                key = None
    end_time = self.time()
    rt = end_time- start_time  
    
    exp.set('keyreleaseRT', rt)
    print '%s was released after %d ms' % (key, rt)
    
    # stop the video
    #'Insert video stopping here'
    

    The other issue, I don't know what is happening exactly. If you provide a working example (Opensesame file, incl. an example video), I'll gladly look into to it.

    Eduard

  • Hi Eduard,

    Many thanks for the prompt response ! This makes perfect sense. However, I noticed two things with this:

    • It requires a key to be pressed to initiate the video (which is what I want) but in order to get the release time, I have to press and release it again during the video playback otherwise I get the following error:

        Error while executing inline script
      
      item-stack: experiment[run].Block[run].trial[run].new_inline_script[run]
      exception type: AttributeError
      exception message: 'Event' object has no attribute 'key'
      item: new_inline_script
      time: Tue Feb 07 16:55:13 2017
      phase: run
      
      Traceback:
        File "C:\Program Files (x86)\OpenSesame\lib\site-packages\libopensesame\inline_script.py", line 102, in run
          self.experiment.python_workspace._exec(self.crun)
        File "C:\Program Files (x86)\OpenSesame\lib\site-packages\libopensesame\python_workspace.py", line 161, in _exec
          exec(bytecode, self._globals)
        File "<string>", line 42, in <module>
      AttributeError: 'Event' object has no attribute 'key'
      

    I'm assuming that the keydown command has to be within the video playback loop right ?

    • As for the second issue, I'm assuming that it might be due to the way my video clip are encoded as it seems to be dependent on the video clip i'm using (working for some videos, not for others. All in .avi). I've read something about ffmpeg being not properly installed or something. Would that be possible ?

    I'll look more into that !

  • EDIT:

    Ok, so it seems to be related to how certain videos are encoded. Using the Xvid video codec package (https://www.xvid.com/) seems to solve the problem. The thing now is to figure out how to adapt the frame rate (videos are way too quick). I've read in another thread that it could be possible to achieve something like that using the self.sleep() thingy somewhere in the loop.

    My question would be then: How accurate can we be in setting the frame rate using that option ? The videos I'm using have a framerate of exactly 20fps (I made them from different pictures using VirtualDub). This allows me to control the timing of specific events in the videos and to relate that to participants' response time. Any advice ?

  • EDIT 2:

    Well, after some violent battles with the script, I think I managed to write something that fulfills my needs.
    I think it could potentially be useful for others having similar problems so I post it here (any comments are greatly appreciated)
    This script deals with video presentation using opencv (as suggested elsewhere on this forum).

    This script will wait for a keypress before playing the video and record participants' reaction time when they release the key. Also, depending on participants' responses, we change the background color during video presentation (as a feedback). As the video playback script did not respect the video files 20Hz frame rate (they were way faster), we used the tick_busy_loop(20) to fix the frame rate at 20 fps. It seems to work that way !

        import cv2
        import numpy
        import pygame
    
        #define background colors
        red = (255,0,0)
        black = (0,0,0)
    
        #create the pygame clock object
        FPSCLOCK=pygame.time.Clock()
    
        #check whether the correct backend is being used
        if self.get('canvas_backend') == 'legacy':
            surface = exp.surface
        elif self.get('canvas_backend') == 'xpyriment':
            surface = exp.window
        else:
            raise Exception('This script requires legacy or xpyriment (no opengl)')
    
    
        # Path to the video file. Get the filename from the 'video' variable
        path = pool[self.get('video')]
        # Open the video and determine the video dimensions
        video = cv2.VideoCapture(path)
    
        #initiate the video with a keypress and start the clock
        key = None
        while key == None:
           for event in pygame.event.get():
              if event.type == pygame.KEYDOWN:
                  start_time = self.time()
                  key = event.key
                  go = True
    
        #initiate the video loop
        while go:
    
            #set the frame rate to 20fps and wait if too fast
            FPSCLOCK.tick_busy_loop(20)
    
            # Get a frame and break the loop if we failed, usually because the video
            # ended.
            success, frame = video.read()
            if not success:
                break
    
            #for some reasons the frames appeared inverted & flipped
            frame = numpy.fliplr(frame)
            frame = numpy.rot90(frame)
    
                # The video uses BGR colors and PyGame needs RGB
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    
                # Create a PyGame surface
            surf = pygame.surfarray.make_surface(frame)
    
            # Fill the background color of the PyGame surface whenever a specific key is released + collect key release time!
            for event in pygame.event.get():
                if event.type == pygame.KEYUP:
                    #get pygame key name
                    newKey=pygame.key.name(event.key)
    
                    if  newKey == self.get('contingent_action'):
                        end_time = self.time()
                        background_color = red
                        surface.fill(background_color)
                        pygame.display.update
                    else:
                        end_time = self.time()
                        background_color = black
                        surface.fill(background_color)
                        pygame.display.update
    
            # Show the PyGame surface!
            surface.blit(surf, ((surface.get_rect().centerx-150),(surface.get_rect().centery-100)))
            pygame.display.flip()
    
        #define the experimental variable
        rt = end_time-start_time
        exp.set('keyreleaseRT', rt)
        print '%s was released after %d ms' % (key, rt)
    

    Any comments are greatly appreciated (there might be errors :>) and I hope this will help others

  • Hi cletess,

    Thanks for sharing!
    Using video in an experiment can be (always is) very challenging. The website www.videohelp.com can be informative. And the tool XMedia Recode could be handy to convert all videos to the same format.

    Best,
    Jarik

Sign In or Register to comment.