[open] Visual Search and the Mouse

Comments

• SeaRabbits

  August 2013 edited August 2013

  I think Murphy's law is in effect.. every time I write something here (after much teeth gnashing) about 2 hours later it comes right, so here goes nothing

  Following the gaze cue experiment I have:

  draw image 0 0 "[images].png" scale=1 center=1 show_if="always"
  draw image [x_pos] [y_pos] "redapple.png" scale=1 center=1 show_if="always"
  

  What I am trying to do now is: Participant clicks on fixation dot. Image + corresponding red apple appear. They then move mouse to click red apple. Clicking the red apple is a correct response and so music.wav will play. The fixation point will reappear at end of music and everything repeats.

  What I can't seem to do is differentiate the redapple.png from [image].png. Clicking anywhere on the screen is a 'correct response.' Should I make the redapple.png into a.. widget??
• EoinTravers

  August 2013 edited August 2013

  Hi Fiona,
  You'll need to use a little bit of inline python scripting to anything complex with the mouse.
  In your case, I think you'll get what you want if you set the duration of Sketchpad item you've described to 0 (so that it is shown, and then moves straight to the next item), and then place an inline_script straight after it, looking something like this:

  # Set up a few things
  from openexp.mouse import mouse
  my_mouse = mouse(exp, visible=True)
  from openexp.sampler import sampler
  audio_file = exp.get_file('music.wav') 
  # or 'Materials/music.wav', where ever you have it saved to
  correct_music = sampler(exp, audio_file)
  
  
  # Get x_pos and y_pos of the target as python variables
  target_x = exp.get('x_pos')
  target_y = exp.get('y_pos')
  
  
  # Wait for a mouse click
  button, position, timestamp = my_mouse.get_click()
  x, y = position
  if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
      # Click was on the target
      exp.set('accuracy', 1)
      correct_music.play()
  else:
      # Click was anywhere else:
      exp.set('accuracy', 0)
  

  What this will do is wait for a click, and if it's within the bounds of the target (i.e. 176 pixels horizontally, or 128 vertically) set's the variable 'accuracy' to 1 and plays the music, and sets 'accuracy' to 0 otherwise.

  Alternatively, you could use this:

  from openexp.mouse import mouse
  my_mouse = mouse(exp, visible=True)
  from openexp.sampler import sampler
  audio_file = exp.get_file('music.wav') 
  # or 'Materials/music.wav', where ever you have it saved to
  correct_music = sampler(exp, audio_file)
  
  
  # Get x_pos and _pos of the target as python variables
  target_x = exp.get('x_pos')
  target_y = exp.get('y_pos')
  
  misclicks = 0
  while True:
      # Wait for a mouse click
      button, position, timestamp = my_mouse.get_click()
      x, y = position
      if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
          # Click was on the target
          exp.set('accuracy', 1)
          exp.set('misclicks ', misclicks)
          correct_music.play()
          break
      else:
          misclicks += 1
  

  ...which doesn't end until the participant does click on the target (and gives you a variable, [misclicks], counting how many times they clicked somewhere else.

  • Disclaimer: I'm not at a computer with OpenSesame installed. There more than likely will be syntax errors in this code, but the idea should be sound.

  Edit: Oh, I missed that you needed participants to click on the fixation before the trial starts. You can do this by placing the following in an inline_script BEFORE the sketchpad item:

  # Set up a few things
  from openexp.mouse import mouse
  my_mouse = mouse(exp, visible=True)
  from openexp.canvas import canvas
  my_canvas = canvas(exp)
  mx, my = my_canvas.xcenter(), my_canvas.ycenter()
  
  my_canvas.clear()
  my_canvas.fixdot()
  my_canvas.show()
  
  # Wait for a mouse click
  while True:
      button, position, timestamp = my_mouse.get_click()
      x, y = position
      if x > (mx - 8) and x < (mx+ 8) and y > (my - 8) and y < (my + 8):
          # Click was within 8 pixels of the center of the screen.
          break
  

  ...which shows a fixdot, and then waits for you to successfully click on it.

  Eoin
• SeaRabbits

  August 2013 edited August 2013

  Thanks so much for replying!

  I have inserted your script into my experiment but I am still experiencing problems. You mentioned it may be due to syntax problems but I doubt I would be able to tell if there is a problem. Python is a new language for me.

  Firstly, the fixation point works fine and my sound file plays when I click, so yay for that!

  However if I use the script with no misclicks, I'm still experiencing all clicks as valid. Is my understanding of this correct?

  I have a target object. It says it is 151x159. So in the script I have:

  if x > (target_x - 76) and x < (target_x + 76) and y > (target_y - 80) and y < (target_y + 80):
      # Click was on the target
  

  Which means when it chooses a random image (say setsize5c), the redapple.png appears at the corresponding coordinates 352x256; if it was setsize5d they would be different. This means that any clicks of 352 +/- 76 and 256 +/- 80 would be recognised as on target? - Sorry for the basic-ness of the question, I'm just trying to make sure I understand how it works.

  However since it is still thinking all clicks are correct I am probably missing something. You mention: exp.set('accuracy', 1) - Should I have somewhere, like under 'sequence,' that instead of 'always' it is 1?

  Secondly, if I use the script with misclicks (which is what I would prefer), all clicks are invalid (in a manner of speaking). If I am using setsize5d, the target appears at 96x256, however the click that is accepted is somewhere in the -544x-224 area (but only for that image, I have no idea where the other acceptable places are for the other images).

  Would you be able to explain any of these problems?

  Thanks,

  Fiona
• SeaRabbits

  August 2013 edited August 2013

  Ah! Figured it out! Well kinda, and not really. But: http://forum.cogsci.nl/index.php?p=/discussion/266/solved-mouse-response-images-and-shapes/p1

  PsychoPy indeed uses the notation as described earlier, but for the sake of consistency between backends, the psycho backend in OpenSesame then converts this to coordinates with their origin at the top left.

  This all works well, except when the actual resolution of the screen does not correspond to the resolution as specified in the experiment. When this happens, (0,0) is the top left of the "screen" in the real screen.

  So when I changed the x_pos and y_pos for my images to

  a) 100 100
  b) 200 200
  c) 300 300

  the target appeared consistently in a diagonal pattern from the centre of the screen progressing towards the bottom right. However the mouse click position started at the top left progressing down towards the centre.

  EDIT: Aha!

  # Get x_pos and y_pos of the target as python variables
  target_x = exp.get('x_pos') + mx
  target_y = exp.get('y_pos') + my
  

  It works! Whooooo!

  Now all that is left to do is figure out how to track the mouse/time coordinates from when a participant clicks the fixation point to when they click the target image.

  Any ideas for that last one?

  Thanks

  Fiona.
• EoinTravers

  August 2013 edited August 2013

  Hi

  Glad to hear you got sorted!

  I'm actually in the process of making a plugin for OS that does just that, but I won't have a chance to finish it for a while, I don't think.
  There was a pretty in depth discussion of mouse tracking in OpenSesame at http://forum.cogsci.nl/index.php?p=/discussion/351/open-mousetracking-and-visual-world-paradigm/p1, where you'll find the code you need, but a whole lot else besides, but I'll try and share a the simplified version saved on my office computer when I get a chance.

  Finally, it occurs to me, and I hope I'm not being rude in suggesting that people use something other than OpenSesame, that your experiment should be possible to create in Jon Freeman's excellent MouseTracker (http://www.dartmouth.edu/~freemanlab/mousetracker/).

  Eoin
• SeaRabbits

  August 2013 edited 9:03AM

  Hey

  I've seen the visual world paradigm discussion, but I'm unsure how to incorporate it into my experiment since I'm not using sampler files, so would just removing those parts screw things up?

  Heh, you're not being rude. I've never heard of the mousetracker software (nor did I think to search for one). One of my courses last semester involved learning how to use OpenSesame, so I just automatically used it to create this experiment.

  I think it would be best if I didn't switch programs, since I can kinda use it, and you're so helpful.

  I'll go play with the code and see what I can break :P

  Thanks

  Fiona
• EoinTravers

  August 2013 edited 9:03AM

  Finally...

  from openexp.mouse import mouse
  my_mouse = mouse(exp, visible=True)
  from openexp.sampler import sampler
  audio_file = exp.get_file('music.wav') 
  # or 'Materials/music.wav', where ever you have it saved to
  correct_music = sampler(exp, audio_file)
  mx, my = my_canvas.xcenter(), my_canvas.ycenter()
  sample_rate = 25
  xList, yList, tList = [], [], []
  
  # Get x_pos and _pos of the target as python variables
  target_x = exp.get('x_pos') + mx
  target_y = exp.get('y_pos') + my
  
  
  # Set up mouse tracking
  t0 = start = exp.time()
  t1 = t0 + sample_rate
  misclicks = 0
  while 1:
      position, timestamp = my_mouse.get_pos()
      if timestamp > t1:
          # This runs every 25ms (running constantly would overload memory)
          t1 += sample_rate
          t = timestamp - start
          x, y = position
          lclick = my_mouse.get_pressed()[0] # Change 0 to 1 for right clicks, I think.
          # Add the current x and y location, and time since start of trial, to lists.
          xList.append(x)
          yList.append(y)
          tList.append(t)
          if lclick:
              if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
                  correct_music.play()
                  break
              else:
                  misclicks += 1
  
  # Blank the screen after a response
  my_canvas.clear()
  my_canvas.show()
  
  # Set the variables (so that you can record them with a logger item)
  # Note: accuracy isn't been noted, because you're only letting people
  # proceed once they've given the right response
  exp.set('misclicks ', misclicks)
  exp.set("rt", t)
  exp.set("xTrajectory", str(xList))
  exp.set("yTrajectory", str(yList))
  exp.set("tTrajectory", str(tList))
  
  # Pause before the next trial
  self.sleep(1000)
  

  This is the bones of the code that I'm using in the majority of the experiments in my PhD. The one warning is that actually analyzing the mouse tracking data recorded this way isn't completely straightforward. Your data files will have columns of what are actually Python lists, looking like this:

  xTrajectory, yTrajectory, tTrajectory,

  [584, 283, 593], [584, 283, 593], [584, 283, 593],

  [584, 283, 593], [584, 283, 593], [584, 283, 593],

  only that there will be 120 numbers within each [] for a 3 second response.
  I will be sharing the analysis scripts I have written in python to handle this kind of data in the next few months (after I actually analyse my own data), but in the mean time:

  • ALWAYS log the width and height variables when you run an experiment, as the mouse position is given in pixels, not relative position
  • If you're feeling confident about python, to analyse the data you'll need to install some advanced features. Specifically, Pandas is needed to read the data from the .csv files in a useful format, Numpy and Scipy are needed for the analysis/maths, and Matplotlib is needed for visualization (and also because these are very useful, free tools in general). The easiest way to get all of this isn't with a scientific version of Python, such as WinPython (https://code.google.com/p/winpython/ ; doesn't need to be installed to run), or Enthought (https://www.enthought.com/ ; fancy MATLAB style software, free for academics)

  Hope this is of help to you, and anyone else looking to use OS for this kind of work!

  Eoin
• SeaRabbits

  August 2013 edited August 2013

  Hello again

  This should be the part where I say, everything works, yay, yay, yay!

  But sadly no

  Firstly, I added in

  from openexp.canvas import canvas
  my_canvas = canvas(exp)
  

  because it says that 'my_canvas' is not defined.

  Secondly when I try to run the experiment, it freezes on the "preparing experiment" loading screen (it's been 5mins+ and still nothing). So everything works up until the part when I added the tracking script.

  EDIT: OMG! ARGH. I'm an idiot!

  I wrote the script in the "prepare phase" not the "run phase." So the experiment is running!

  And now I have a duplicate/echo sound file playing each time I click the target.

  If it's not one thing, it's another >.>

  I'm going to assume it's from the first script to get the sound playing - duplicate scripts, duplicate sounds.

  I shall go search and destroy.

  EDIT: YES! Life is wonderful

  Apparently having duplicate scripts didn't matter. But still having the "sampler item" in my sequence did. I removed that and no more echo effects

  So experiment is pretty much complete. Probably just a couple more tweaks to do, but I'm sure I can manage that.

  Anyhow, thank you so so much for your help over the last couple of days! You've been a lifesaver. I would never have made it this far without you. ^:)^ ^:)^ ^:)^

  Until next time,

  Fiona