Response time during video display

Comments

• cletess

  January 2017

  EDIT:

  Some progress have been made based on this thread

  By adding this in the python code section of the video object, responses are collected as long as they are given during the video playback.

  if event: key = event[1] if var.correct_response == key: correct=1 else: correct=0 responses.add(response=key, correct=correct, response_time=clock.time()-var.time_video) continue_playback = True

  The problem is that responses should also be possible once the video has stopped playing (during the 1000 ms blank screen). As it is, 'nothing' is collected and for those trials where responses are given after the end of the video, the same response times as the trial before are collected. Any ideas ?

  Thanks
• eduard

  January 2017

  Hi Cletess,

  Can you print the rest of the code that you use? In general, extending the response collection period, requires only to loop a little longer before proceding with the next trial. And optionally, change the stuff that is presented on the screen during this interval. Once I know how you present everything, I can probably also give you more specific suggestions how you could implement it.

  Eduard

  
• cletess

  February 2017 edited February 2017

  Hi again and sorry for the delay !

  We made some changes on the paradigm and I took some time to read different threads on this forum. It appears (maybe I'm wrong) that for what I need, I'll have to use openCV to deal with the video presentation. Let me explain:

  • After fixation, we want the videos to be played whenever a key is pressed (key '1' or '2' of the numerical pad).
  • From video onset, we would like to record participants' response time to one of two possible events that can occur in the videos by releasing one of the two key that was used to initiate the video ('1' or '2').
  • Finally, whenever participants make a response in a specific condition (presented in a variable in the trial loop), we would like a colored cue to appear on top of the video. This seems to be possible using the pygame.draw.rect command in opencv.

  I saw that this code was suggested in a previous post:

      import cv2
      import numpy
      import pygame
  
      path = pool[self.get('video')]
      video = cv2.VideoCapture(path)
  'insert wait for keypress here?'
      while go:
          success, frame = video.read()
          if not success:
              break
          frame = numpy.rot90(frame)
          # The video uses BGR colors and PyGame needs RGB
          frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
          # Create a PyGame surface
          surf = pygame.surfarray.make_surface(frame)
          # Now you can draw whatever you want onto the PyGame surface!
          pygame.draw.rect(surf, (255,0,0), (25, 25, 25, 25))
          # Show the PyGame surface!
          surface.blit(surf, (0, 0))
          pygame.display.flip()
  

  For now, the screen just goes black for each trial and I can't display the videos. No specific error message. It seems that the value of 'Success' is always False therefore breaking the loop for some reasons. Also, I'm not sure on how to initiate the loop with a key press and to record the reaction time until it is released (the videos must not be stopped once the key is released). I tried something but I'm not really sure it's even correct ...

  key = None
  while key == None:
     start_time = self.time()
      for event in pygame.event.get():
          if event.type == pygame.KEYDOWN:
              key = event.key
              go = True
  
  'Insert video playback code here'
  
  while key == event.key:
     end_time = self.time()
      for event in pygame.event.get():
          if event.type == pygame.KEYUP:
              key = None
  end_time = self.time()
  rt = end_time- start_time  
  exp.set('keyreleaseRT', rt)
  print '%s was released after %d ms' % (key, rt)
  

  Thanks a lot for your advice
• eduard

  February 2017

  Hi,

  First, I haven't worked much with videos and not at all with OpenCV, so I can't help you with that. However, I think some of your questions are more general in nature. So, let's see whether I can help you.

  Also, I'm not sure on how to initiate the loop with a key press and to record the reaction time until it is released

  This is rather simple. Basically, when a key is pressed you have something like video.start() which should also give a time stamp for when exactly this happened. If there is not time stamp sent automatically by calling 'video.start', you can add a line like start_time = clock.time() right before you start the video. Importantly, releasing the key shouldn't stop the video, which won't happen unless you write something like "if key is released, stop video playback". So, whenever the key is actually released, you can take a second time stamp. Later, when the trial is over (after all the loops), you can stop the video, by something like "video.stop()". Below the part of the code that would accomplish that (pretty much the same as what you already have had):

  # note, you don't check for pressing a particular key, any key will do in this example
  key = None
  while key == None:
      for event in pygame.event.get():
          if event.type == pygame.KEYDOWN:
              start_time = self.time()
              key = event.key
              go = True # doesn't do anything in this code snippet
  
  #'Insert video playback code here'
  
  while key == event.key:
      for event in pygame.event.get():
          if event.type == pygame.KEYUP:
              key = None
  end_time = self.time()
  rt = end_time- start_time  
  
  exp.set('keyreleaseRT', rt)
  print '%s was released after %d ms' % (key, rt)
  
  # stop the video
  #'Insert video stopping here'
  

  The other issue, I don't know what is happening exactly. If you provide a working example (Opensesame file, incl. an example video), I'll gladly look into to it.

  Eduard

  
• cletess

  February 2017

  Hi Eduard,

  Many thanks for the prompt response ! This makes perfect sense. However, I noticed two things with this:

  • It requires a key to be pressed to initiate the video (which is what I want) but in order to get the release time, I have to press and release it again during the video playback otherwise I get the following error:

      Error while executing inline script
    
    item-stack: experiment[run].Block[run].trial[run].new_inline_script[run]
    exception type: AttributeError
    exception message: 'Event' object has no attribute 'key'
    item: new_inline_script
    time: Tue Feb 07 16:55:13 2017
    phase: run
    
    Traceback:
      File "C:\Program Files (x86)\OpenSesame\lib\site-packages\libopensesame\inline_script.py", line 102, in run
        self.experiment.python_workspace._exec(self.crun)
      File "C:\Program Files (x86)\OpenSesame\lib\site-packages\libopensesame\python_workspace.py", line 161, in _exec
        exec(bytecode, self._globals)
      File "<string>", line 42, in <module>
    AttributeError: 'Event' object has no attribute 'key'
    

  I'm assuming that the keydown command has to be within the video playback loop right ?

  • As for the second issue, I'm assuming that it might be due to the way my video clip are encoded as it seems to be dependent on the video clip i'm using (working for some videos, not for others. All in .avi). I've read something about ffmpeg being not properly installed or something. Would that be possible ?

  I'll look more into that !
• cletess

  February 2017

  EDIT:

  Ok, so it seems to be related to how certain videos are encoded. Using the Xvid video codec package (https://www.xvid.com/) seems to solve the problem. The thing now is to figure out how to adapt the frame rate (videos are way too quick). I've read in another thread that it could be possible to achieve something like that using the self.sleep() thingy somewhere in the loop.

  My question would be then: How accurate can we be in setting the frame rate using that option ? The videos I'm using have a framerate of exactly 20fps (I made them from different pictures using VirtualDub). This allows me to control the timing of specific events in the videos and to relate that to participants' response time. Any advice ?
• cletess

  February 2017

  EDIT 2:

  Well, after some violent battles with the script, I think I managed to write something that fulfills my needs.
  I think it could potentially be useful for others having similar problems so I post it here (any comments are greatly appreciated)
  This script deals with video presentation using opencv (as suggested elsewhere on this forum).

  This script will wait for a keypress before playing the video and record participants' reaction time when they release the key. Also, depending on participants' responses, we change the background color during video presentation (as a feedback). As the video playback script did not respect the video files 20Hz frame rate (they were way faster), we used the tick_busy_loop(20) to fix the frame rate at 20 fps. It seems to work that way !

      import cv2
      import numpy
      import pygame
  
      #define background colors
      red = (255,0,0)
      black = (0,0,0)
  
      #create the pygame clock object
      FPSCLOCK=pygame.time.Clock()
  
      #check whether the correct backend is being used
      if self.get('canvas_backend') == 'legacy':
          surface = exp.surface
      elif self.get('canvas_backend') == 'xpyriment':
          surface = exp.window
      else:
          raise Exception('This script requires legacy or xpyriment (no opengl)')
  
  
      # Path to the video file. Get the filename from the 'video' variable
      path = pool[self.get('video')]
      # Open the video and determine the video dimensions
      video = cv2.VideoCapture(path)
  
      #initiate the video with a keypress and start the clock
      key = None
      while key == None:
         for event in pygame.event.get():
            if event.type == pygame.KEYDOWN:
                start_time = self.time()
                key = event.key
                go = True
  
      #initiate the video loop
      while go:
  
          #set the frame rate to 20fps and wait if too fast
          FPSCLOCK.tick_busy_loop(20)
  
          # Get a frame and break the loop if we failed, usually because the video
          # ended.
          success, frame = video.read()
          if not success:
              break
  
          #for some reasons the frames appeared inverted & flipped
          frame = numpy.fliplr(frame)
          frame = numpy.rot90(frame)
  
              # The video uses BGR colors and PyGame needs RGB
          frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
  
              # Create a PyGame surface
          surf = pygame.surfarray.make_surface(frame)
  
          # Fill the background color of the PyGame surface whenever a specific key is released + collect key release time!
          for event in pygame.event.get():
              if event.type == pygame.KEYUP:
                  #get pygame key name
                  newKey=pygame.key.name(event.key)
  
                  if  newKey == self.get('contingent_action'):
                      end_time = self.time()
                      background_color = red
                      surface.fill(background_color)
                      pygame.display.update
                  else:
                      end_time = self.time()
                      background_color = black
                      surface.fill(background_color)
                      pygame.display.update
  
          # Show the PyGame surface!
          surface.blit(surf, ((surface.get_rect().centerx-150),(surface.get_rect().centery-100)))
          pygame.display.flip()
  
      #define the experimental variable
      rt = end_time-start_time
      exp.set('keyreleaseRT', rt)
      print '%s was released after %d ms' % (key, rt)
  

  Any comments are greatly appreciated (there might be errors :>) and I hope this will help others
• Jarik

  February 2017

  Hi cletess,

  Thanks for sharing!
  Using video in an experiment can be (always is) very challenging. The website www.videohelp.com can be informative. And the tool XMedia Recode could be handy to convert all videos to the same format.

  Best,
  Jarik