[open] Visual Search and the Mouse

SeaRabbits

Hello - just edited this to make it make sense.

I'm an amateur at OpenSesame and I was hoping for some advice on an experiment I am attempting to create.

Basically it's a visual search - 5 stimuli, 1 target, 4 distractors (set size will increase but I'm starting small to get the hang of OpenSesame). Participants start at a centre fixation point and click the target. Eventually I shall add in a mouse tracking script but I'm starting at the easier stuff >.>

What I have at the moment is I have created 4 images with transparent backgrounds, and I wish to add in an additional image at certain coordinates to each of them. This means that I can hopefully make music play when they click on the correct image fingers crossed.

I have:

draw image 0 0 "[images].png" scale=1 center=1 show_if="always"

What I can't seem to do is create something like:

draw image 352.0 256.0 "redapple.png" scale=1 center=1 show_if="setsize5a.png"

I have attempted using the variables, but I can get the basic background image (setsize5a) to show, but I am trying to add an additional image (redapple) at the coordinates 352.0 256.0 on that particular image. Both images have transparent backgrounds so they should show up together. However I think I'm missing how to merge the two together or something.

Fiona

SeaRabbits

I think Murphy's law is in effect.. every time I write something here (after much teeth gnashing) about 2 hours later it comes right, so here goes nothing

Following the gaze cue experiment I have:

draw image 0 0 "[images].png" scale=1 center=1 show_if="always"
draw image [x_pos] [y_pos] "redapple.png" scale=1 center=1 show_if="always"

What I am trying to do now is: Participant clicks on fixation dot. Image + corresponding red apple appear. They then move mouse to click red apple. Clicking the red apple is a correct response and so music.wav will play. The fixation point will reappear at end of music and everything repeats.

What I can't seem to do is differentiate the redapple.png from [image].png. Clicking anywhere on the screen is a 'correct response.' Should I make the redapple.png into a.. widget??

EoinTravers

Hi Fiona,
You'll need to use a little bit of inline python scripting to anything complex with the mouse.
In your case, I think you'll get what you want if you set the duration of Sketchpad item you've described to 0 (so that it is shown, and then moves straight to the next item), and then place an inline_script straight after it, looking something like this:

# Set up a few things
from openexp.mouse import mouse
my_mouse = mouse(exp, visible=True)
from openexp.sampler import sampler
audio_file = exp.get_file('music.wav') 
# or 'Materials/music.wav', where ever you have it saved to
correct_music = sampler(exp, audio_file)


# Get x_pos and y_pos of the target as python variables
target_x = exp.get('x_pos')
target_y = exp.get('y_pos')


# Wait for a mouse click
button, position, timestamp = my_mouse.get_click()
x, y = position
if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
    # Click was on the target
    exp.set('accuracy', 1)
    correct_music.play()
else:
    # Click was anywhere else:
    exp.set('accuracy', 0)

What this will do is wait for a click, and if it's within the bounds of the target (i.e. 176 pixels horizontally, or 128 vertically) set's the variable 'accuracy' to 1 and plays the music, and sets 'accuracy' to 0 otherwise.

Alternatively, you could use this:

from openexp.mouse import mouse
my_mouse = mouse(exp, visible=True)
from openexp.sampler import sampler
audio_file = exp.get_file('music.wav') 
# or 'Materials/music.wav', where ever you have it saved to
correct_music = sampler(exp, audio_file)


# Get x_pos and _pos of the target as python variables
target_x = exp.get('x_pos')
target_y = exp.get('y_pos')

misclicks = 0
while True:
    # Wait for a mouse click
    button, position, timestamp = my_mouse.get_click()
    x, y = position
    if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
        # Click was on the target
        exp.set('accuracy', 1)
        exp.set('misclicks ', misclicks)
        correct_music.play()
        break
    else:
        misclicks += 1

...which doesn't end until the participant does click on the target (and gives you a variable, [misclicks], counting how many times they clicked somewhere else.

• Disclaimer: I'm not at a computer with OpenSesame installed. There more than likely will be syntax errors in this code, but the idea should be sound.

Edit: Oh, I missed that you needed participants to click on the fixation before the trial starts. You can do this by placing the following in an inline_script BEFORE the sketchpad item:

# Set up a few things
from openexp.mouse import mouse
my_mouse = mouse(exp, visible=True)
from openexp.canvas import canvas
my_canvas = canvas(exp)
mx, my = my_canvas.xcenter(), my_canvas.ycenter()

my_canvas.clear()
my_canvas.fixdot()
my_canvas.show()

# Wait for a mouse click
while True:
    button, position, timestamp = my_mouse.get_click()
    x, y = position
    if x > (mx - 8) and x < (mx+ 8) and y > (my - 8) and y < (my + 8):
        # Click was within 8 pixels of the center of the screen.
        break

...which shows a fixdot, and then waits for you to successfully click on it.

Eoin

SeaRabbits

Thanks so much for replying!

I have inserted your script into my experiment but I am still experiencing problems. You mentioned it may be due to syntax problems but I doubt I would be able to tell if there is a problem. Python is a new language for me.

Firstly, the fixation point works fine and my sound file plays when I click, so yay for that!

However if I use the script with no misclicks, I'm still experiencing all clicks as valid. Is my understanding of this correct?

I have a target object. It says it is 151x159. So in the script I have:

if x > (target_x - 76) and x < (target_x + 76) and y > (target_y - 80) and y < (target_y + 80):
    # Click was on the target

Which means when it chooses a random image (say setsize5c), the redapple.png appears at the corresponding coordinates 352x256; if it was setsize5d they would be different. This means that any clicks of 352 +/- 76 and 256 +/- 80 would be recognised as on target? - Sorry for the basic-ness of the question, I'm just trying to make sure I understand how it works.

However since it is still thinking all clicks are correct I am probably missing something. You mention: exp.set('accuracy', 1) - Should I have somewhere, like under 'sequence,' that instead of 'always' it is 1?

Secondly, if I use the script with misclicks (which is what I would prefer), all clicks are invalid (in a manner of speaking). If I am using setsize5d, the target appears at 96x256, however the click that is accepted is somewhere in the -544x-224 area (but only for that image, I have no idea where the other acceptable places are for the other images).

Would you be able to explain any of these problems?

Thanks,

Fiona

SeaRabbits

Ah! Figured it out! Well kinda, and not really. But: http://forum.cogsci.nl/index.php?p=/discussion/266/solved-mouse-response-images-and-shapes/p1

PsychoPy indeed uses the notation as described earlier, but for the sake of consistency between backends, the psycho backend in OpenSesame then converts this to coordinates with their origin at the top left.

This all works well, except when the actual resolution of the screen does not correspond to the resolution as specified in the experiment. When this happens, (0,0) is the top left of the "screen" in the real screen.

So when I changed the x_pos and y_pos for my images to

a) 100 100
b) 200 200
c) 300 300

the target appeared consistently in a diagonal pattern from the centre of the screen progressing towards the bottom right. However the mouse click position started at the top left progressing down towards the centre.

EDIT: Aha!

# Get x_pos and y_pos of the target as python variables
target_x = exp.get('x_pos') + mx
target_y = exp.get('y_pos') + my

It works! Whooooo!

Now all that is left to do is figure out how to track the mouse/time coordinates from when a participant clicks the fixation point to when they click the target image.

Any ideas for that last one?

Thanks

Fiona.

EoinTravers

Hi

Glad to hear you got sorted!

I'm actually in the process of making a plugin for OS that does just that, but I won't have a chance to finish it for a while, I don't think.
There was a pretty in depth discussion of mouse tracking in OpenSesame at http://forum.cogsci.nl/index.php?p=/discussion/351/open-mousetracking-and-visual-world-paradigm/p1, where you'll find the code you need, but a whole lot else besides, but I'll try and share a the simplified version saved on my office computer when I get a chance.

Finally, it occurs to me, and I hope I'm not being rude in suggesting that people use something other than OpenSesame, that your experiment should be possible to create in Jon Freeman's excellent MouseTracker (http://www.dartmouth.edu/~freemanlab/mousetracker/).

Eoin

SeaRabbits

Hey

I've seen the visual world paradigm discussion, but I'm unsure how to incorporate it into my experiment since I'm not using sampler files, so would just removing those parts screw things up?

Heh, you're not being rude. I've never heard of the mousetracker software (nor did I think to search for one). One of my courses last semester involved learning how to use OpenSesame, so I just automatically used it to create this experiment.

I think it would be best if I didn't switch programs, since I can kinda use it, and you're so helpful.

I'll go play with the code and see what I can break :P

Thanks

Fiona

EoinTravers

Finally...

from openexp.mouse import mouse
my_mouse = mouse(exp, visible=True)
from openexp.sampler import sampler
audio_file = exp.get_file('music.wav') 
# or 'Materials/music.wav', where ever you have it saved to
correct_music = sampler(exp, audio_file)
mx, my = my_canvas.xcenter(), my_canvas.ycenter()
sample_rate = 25
xList, yList, tList = [], [], []

# Get x_pos and _pos of the target as python variables
target_x = exp.get('x_pos') + mx
target_y = exp.get('y_pos') + my


# Set up mouse tracking
t0 = start = exp.time()
t1 = t0 + sample_rate
misclicks = 0
while 1:
    position, timestamp = my_mouse.get_pos()
    if timestamp > t1:
        # This runs every 25ms (running constantly would overload memory)
        t1 += sample_rate
        t = timestamp - start
        x, y = position
        lclick = my_mouse.get_pressed()[0] # Change 0 to 1 for right clicks, I think.
        # Add the current x and y location, and time since start of trial, to lists.
        xList.append(x)
        yList.append(y)
        tList.append(t)
        if lclick:
            if x > (target_x - 176) and x < (target_x + 176) and y > (target_y - 128) and y < (target_y + 128):
                correct_music.play()
                break
            else:
                misclicks += 1

# Blank the screen after a response
my_canvas.clear()
my_canvas.show()

# Set the variables (so that you can record them with a logger item)
# Note: accuracy isn't been noted, because you're only letting people
# proceed once they've given the right response
exp.set('misclicks ', misclicks)
exp.set("rt", t)
exp.set("xTrajectory", str(xList))
exp.set("yTrajectory", str(yList))
exp.set("tTrajectory", str(tList))

# Pause before the next trial
self.sleep(1000)

This is the bones of the code that I'm using in the majority of the experiments in my PhD. The one warning is that actually analyzing the mouse tracking data recorded this way isn't completely straightforward. Your data files will have columns of what are actually Python lists, looking like this:

xTrajectory, yTrajectory, tTrajectory,

[584, 283, 593], [584, 283, 593], [584, 283, 593],

[584, 283, 593], [584, 283, 593], [584, 283, 593],

only that there will be 120 numbers within each [] for a 3 second response.
I will be sharing the analysis scripts I have written in python to handle this kind of data in the next few months (after I actually analyse my own data), but in the mean time:

• ALWAYS log the width and height variables when you run an experiment, as the mouse position is given in pixels, not relative position
• If you're feeling confident about python, to analyse the data you'll need to install some advanced features. Specifically, Pandas is needed to read the data from the .csv files in a useful format, Numpy and Scipy are needed for the analysis/maths, and Matplotlib is needed for visualization (and also because these are very useful, free tools in general). The easiest way to get all of this isn't with a scientific version of Python, such as WinPython (https://code.google.com/p/winpython/ ; doesn't need to be installed to run), or Enthought (https://www.enthought.com/ ; fancy MATLAB style software, free for academics)

Hope this is of help to you, and anyone else looking to use OS for this kind of work!

Eoin

SeaRabbits

Hello again

This should be the part where I say, everything works, yay, yay, yay!

But sadly no

Firstly, I added in

from openexp.canvas import canvas
my_canvas = canvas(exp)

because it says that 'my_canvas' is not defined.

Secondly when I try to run the experiment, it freezes on the "preparing experiment" loading screen (it's been 5mins+ and still nothing). So everything works up until the part when I added the tracking script.

EDIT: OMG! ARGH. I'm an idiot!

I wrote the script in the "prepare phase" not the "run phase." So the experiment is running!

And now I have a duplicate/echo sound file playing each time I click the target.

If it's not one thing, it's another >.>

I'm going to assume it's from the first script to get the sound playing - duplicate scripts, duplicate sounds.

I shall go search and destroy.

EDIT: YES! Life is wonderful

Apparently having duplicate scripts didn't matter. But still having the "sampler item" in my sequence did. I removed that and no more echo effects

So experiment is pretty much complete. Probably just a couple more tweaks to do, but I'm sure I can manage that.

Anyhow, thank you so so much for your help over the last couple of days! You've been a lifesaver. I would never have made it this far without you. ^:)^ ^:)^ ^:)^

Until next time,

Fiona