[open] Mousetracking and Visual World Paradigm

hunterh · March 2013

Good afternoon,

I am attempting to use Open Sesame for a mouse-tracking experiment in the visual world paradigm. The trial procedure is basically thus:

Participant sees a fixation dot for a fixed amount of time.
After the delays, a sound file plays while the fixation dot remains.
They click on the dot and three pictures appear, one each in the lower left, top right, and bottom right corners.
A sound file also plays immediately.
They move the mouse to click on one of the pictures.
End of trial. Repeat randomly with other stimuli.

Most of this is quite straightforward, though I am hitting a few problems I've been unable to solve so far. I will mention the biggest one first.

1) I can display three images in the right locations using the standard Sketchpad. Plus, I can make the fixation dot follow the mouse until a mouse-click occurs. I am using the inline code from http://osdoc.cogsci.nl/python-inline-code/mouse-functions/ for the mouse action. What I don't understand yet is how to integrate the two if possible. For instance, the "follow the mouse" inline code continually clears the canvas (I could remove that line if needed). I assume I need to draw the canvas inside the same inline code that handles the mouse-tracking, but my attempts to do that so far using the documented canvas functions are unsuccessful.

2) Displaying the mouse during a fixation cross. I'm guessing this one is trivial to the knowledgeable. The fixation dot appears. It remains until mouseclick. The purpose of that is so that the mouse is at the center when the pictures and sample are played. However, the mouse does not show during this fixation cross time, so the participant doesn't know where their mouse is. (It does just show with my tracking inline code.) I've been trying to use set.visible, but I'm apparently not placing it in the right location.

3) Finally, I'm having trouble getting it to write the continuous mouse positions (pos and time) to the log file. Advice here?

Apologies to have my first support request be so needy, but I am not solving these problems myself fast enough for a student who needs this up and going.

Thanks for any time you have for this,

Hunter

sebastiaan · March 2013

Hi Hunter,

Welcome to the forum!

I can display three images in the right locations using the standard Sketchpad. Plus, I can make the fixation dot follow the mouse until a mouse-click occurs. (...) What I don't understand yet is how to integrate the two if possible.

Generally speaking, if you do part of the drawing with an inline_script item, you have to do all of it that way (at least for things that happen simultaneously). So instead of a sketchpad item, you would need to use openexp.canvas, as described here:

http://osdoc.cogsci.nl/python-inline-code/canvas-functions/

Basically, instead of drawing just the fixation dot, you would also draw the three images to the canvas and omit the sketchpad altogether.

Incidentally, might I ask why you are drawing a mouse-contingent fixation dot, instead of using the regular mouse cursor? If the mouse cursor doesn't show (after calling mouse.set_visible()), this might be due to the back-end. Some back-end configurations prevent the mouse cursor from being shown (in fullscreen only), in which case you could try a different back-end.

However, the mouse does not show during this fixation cross time, so the participant doesn't know where their mouse is.

The easiest way is probably to insert a mouse_response after the item used to draw the fixation cross (a sketchpad, probably?) and tick the 'Visible mouse cursor' box. Make sure to set the duration of the preceding item to 0, so that it advances immediately to the mouse_response.

Finally, I'm having trouble getting it to write the continuous mouse positions (pos and time) to the log file. Advice here?

Polling the mouse position is done with mouse.get_pos(). However, logging a continuous signal into a row-per-trial logfile is problematic, because it's not clear how the data should be organized in that case. Ideally speaking, how would you like to organize your data file?

Good luck and cheers!

Sebastiaan

Erwin · March 2013

Hi Hunter

For a experiment, I did something similar to what you need to do. The problem I had using openexp.canvas is that there's a lot jitter in the cursor trajectory. So I prefered to use Pygame in inline script to present the mouse cursor.

EoinTravers · March 2013

I think I can help with the data logging here.
I've been building a template for something similar for touchscreens (so haven't needed to see the cursor).
What I've done is, after the stimuli have been presented (most conveniently with the sketchpad, although you can control this in the inline script for more flexibility), I create an inline object called "Tracker".
In the prepare phase of this, I put:

xList, yList, tList = [], [], []
global xList, yList, tList

which sets things up for logging.

In the run phase, use a variation of what's below.

start = self.time()
tracking = True
while tracking:
    position, timestamp = my_mouse.get_pos()
    x, y = position
    #print str(x)+', '+str(y)
    t = timestamp - start
    #uncomment this to print to the debug window.
    #print str(t)+', '+str(x)+', '+str(y)
    xList.append(x)
    yList.append(y)
    tList.append(t)
    if x < 350 and y < 175:
        resp = 1
        print 'left'
        tracking = False
        break
    if x > 930 and y < 175:
        resp = 2
        print 'right'
        tracking = False
        break
    #This dictates sampling rate. 50 hertz is more than enough
    self.sleep(20)
my_canvas.clear()
print 'answered'    
self.experiment.set("response", resp)
if resp == self.get('correct'):
    self.experiment.set("accuracy", 1)
else:
    self.experiment.set("accuracy", 0)
self.experiment.set("xTrajectory", str(xList))
self.experiment.set("yTrajectory", str(yList))
self.experiment.set("tTrajectory", str(tList))
self.experiment.set("position", position)

Because on the touchscreen, I just need the cursor to be over the response buttons in the two corners to register a response (touching the screen is one big click).
As of today, I know that you can place

from pygame.mouse import get_pressed
global get_pressed

at the start of your experiment somewhere, call it in the 'while tracking' loop above as

lclick, mclick, rclick =  get_pressed()

and change my:
if x < 350 and y < 175:
to
if x < 350 and y < 175 and (lclick == 1):

Hope this helps!
Eoin.

Edit: You obviously need to stick a logger item after this each time, and log xTrajectory, yTrajectory, tTrajectory, response, accuracy, and position.

hunterh · March 2013

Thanks, everyone, for your assistance and apologies for being so slow to get back. Basically, I'm able to work on research Saturday - Monday and then disappear into lectures Tuesday - Friday right now. It's Saturday here in New Zealand and so I'm back on this.

First, before I forget, Eoin, my collaborator and I have been working on an open source Android-platform that's under development (I've seen Open Sesame's work on this as well.) The project is located at https://github.com/Otago-PsyAn-Lab/Otago-PsyAn-Lab. Yell if you'd like to discuss projects. It's only at a state where it runs designed experiments but has no way to design them. The latter's currently being created. (And I hope mentioning that work here is not inappropriate.)

Sebastiaan, there's no reason actually to have the fixation dot follow the mouse. It simply did no harm and was ready-made inline code. If I remove that and just track the cursor, then the clearing of the canvas line is unneeded and that can help solve my presentation of pictures.

Yes, how to write the tracking information? Ideally, I believe it would be something like:
x-pos ; y-pos ; timestamp ; item number.

where item number is some identification of which item the participant is currently engaged with. One would then append these rows at an appropriate sample rate.

Which gets us to Eoin's code. Thanks very much indeed for this. Reading your code, am I correct in understanding that you are comparing the location of the mouse against a determined pixel number in order to categorise? I don't believe we will need that as we're just interested in raw location with the hope of comparing curvature towards pictures on the right as a result of the sentence played. (The picture on the left is a distractor for non-filler items.) Is there more going on in that section of the code that I am not following?

OK, I will attempt to implement the suggestions so far and report back.

EoinTravers · March 2013

Psy-An lab looks interesting. I'll be sure to take a look at it later in the week.
You're right about my code, I have a response button in the top left and top right corners (as in Freeman's MouseTracker program), and the trial ends and logs a response as soon as the cursor is over either button.
My code saves the x and y positions and the timestamp as 3 python lists (values in square brackets, separated by commas), so that they can be relatively opened an manipulated with SciPy/NumPy, although I haven't got as far as this yet. Because I actually wrote this code for a simple lexical decision task to show my supervisors OpenSesame's potential, on each trial I log the following:

LoggerCount (or similar) - chronological trial number.
stimulus
response
accuracy
RT
xTrajectory (list of values on x axis)
yTrajectory
tTrajectory (a misleading name: the timestamps corresponding to each entry on x and y trajectories).

Finally, I had a go at showing the cursor position last week.

    position, timestamp = my_mouse.get_pos()
    x, y = position
    my_canvas.clear()
    my_canvas.fixdot(x, y, color = 'red')
    ...
    #the rest of your canvas (response buttons, stimuli, etc)
    ...
    my_canvas.show()
    ...
    #the rest of your script
    ...
    self.sleep(5)

...worked fine on my desktop in the office (Intel 3.4GHz, 8GB), but I haven't tried it on the Nexus 7).
Eoin

hunterh · March 2013

Thanks Eoin and everyone,

I'm making a good bit of progress on this and I think it's going to work out when I have all the little details worked out. When the whole thing is ready, I will post the entire code here, because I think this is an experimental design others will be interested in.

The particular issue I'm dealing with at the moment is "flushing". Based upon Eoin's code I am using get.pressed() from pygame to register the mouse click. Everything goes as expected for the first trial. However, when I go through the second trial, the while loop ends immediately as if the mouse has already been pressed. I think I need some way to clear out the value of get_pressed from a previous trial.

The format is roughly: inline code 1 presents a fixation dot and waits for mouse click. (This is where my problem eventually comes.) After click, I have a couple samplers and a sketchpad using the standard OpenSesame building tools. Then there's the final Tracker inline code based largely on Eoin's sample. I've commented out a few things that may be deleted or implemented as I keep working.

Fixation Dot script (where I see the problem on trial 2)

#Use inline script to as to call the set_visible function
my_mouse.set_visible()
w,h = exp.resolution()
#print (get_pressed)

#a while loop that presents a dot until click
showdot = True
while showdot:
    my_canvas.clear()
    my_canvas.fixdot(w/2,h/2,color='black')
    my_canvas.show()
    lclick2, mclick2, rclick2 =  get_pressed()
    get()
    #print (lclick2)
    if lclick2 == 1:
        print 'Fixdot Clicked'
        showdot = False
        break

And the second Tracker script (still has clutter to be removed later)


start = self.time()
my_mouse.set_visible()
tracking = True
while tracking:
    position, timestamp = my_mouse.get_pos()
    x, y = position
    #print str(x)+', '+str(y)
    t = timestamp - start
    #uncomment this to print to the debug window.
    #print str(t)+', '+str(x)+', '+str(y)
    xList.append(x)
    yList.append(y)
    tList.append(t)
    lclick, mclick, rclick =  get_pressed()
    get()
    if lclick == 1:
        #resp = 1
        print 'Clicked'
        tracking = False
        break
    #if x > 930 and y < 175:
        #resp = 2
        #print 'right'
        #tracking = False
        #break
    #This dictates sampling rate. 50 hertz is more than enough
    self.sleep(20)
my_canvas.clear()
print 'answered'
#self.experiment.set("response", resp)
#if resp == self.get('correct'):
        #self.experiment.set("accuracy", 1)
#else:
        #self.experiment.set("accuracy", 0)
self.experiment.set("xTrajectory", str(xList))
self.experiment.set("yTrajectory", str(yList))
self.experiment.set("tTrajectory", str(tList))
self.experiment.set("position", position)
self.experiment.set("flush","yes")

Should add that it's possible that I have not diagnosed the error correctly. Both scripts are using get_pressed() and on trial 2, only the first script immediately advances as if the mouse has already been clicked. The second script (Tracker) correctly waits for the mouse click.

Edwin · March 2013

You're supposed to call pygame.event.get() before calling pygame.mouse.get_pressed(). You could try this and hope it works (makes for a little less unpredictable script). Apart from this, note that the print sometimes has the same effect.

By the way: might I suggest to try the playground version of OpenSesame? (github.com/smathot/OpenSesame then click to 'playground' branch). A get_pressed was added to the openexp mouse class very recently. Why not try this one?

Good luck!

hunterh · March 2013

Hi Edwin,

Thanks for catching the get() issue so quickly. Yep, that was it. I reordered when it was done and that problem went away. I've been making further modifications and I am 95% certain, we've pulled this off. I'm playing with alternate data logging a bit and then will post the final solution here for posterity in case someone else wants to do mouse tracking in Open Sesame. Thanks again.

hunterh · March 2013

I think we have the mouse tracking experiment working now. Thanks to everyone for their help again. I am going to document it here because I think this is a fairly common design. The experiment shown here is missing a training period and some more instructions, but the critical trial sequence is displayed in the image below. Below that, I copy in the necessary inline scripts. One odd thing in the current design is that we're both sending some things to logger and some things to a custom myLog file. The purpose of the latter is because it makes for pretty easy analysis in R later without heavy manipulation. However, some Logger items, such as the location and area of the final click are only appropriate in Logger. Clearly, one could remove the trajectories from logger since myLog is implemented.

Design:
The participant, during trials, is presented with a fixation dot. When they left click on the dot (and only there, or a square around it), a question is played as a sound file. After the question, three images are displayed and a further sentence is played through sampler. The unusual thing about the design here is that there are three possible destinations rather than the usual two. Mouse position and timestamps are recorded until they left-click on one of the three designated areas. There is a slight delay and then the next trial proceeds.

Script opening up MyLog. Run Phase only.

#initialise myLog

#note that this will append data to myLog, not overwrite it.
global myLog
myLog = open('T:/Supervision/Ryosuke/mylog.csv','a')

#Remove comment if you want a Header.
#Will add a single header for each run through experiment.
#Operating with a file that already has a header to prevent duplicate headers.
#myLog.write('x_coor' + ',' + 'y_coor' + ',' + 'timestamp' + ',' + 'count_seq' + ',' + 'participant' + ',' + 'condition + ',' + 'snumber' + chr(10))

Show Dot prepare phase

#Initialise mouse object
from openexp.mouse import mouse
global my_mouse
my_mouse = mouse(exp, visible = True)
my_mouse.flush()
my_mouse.set_visible()

#Initialise canvas
from openexp.canvas import canvas
global my_canvas
my_canvas = canvas(exp)

#initialise pygame.mouse
from pygame.mouse import get_pressed
global get_pressed

from pygame.event import get
global get

Show Dot Run Phase

self.experiment.set("flush","yes")

#Use inline script to as to call the set_visible function
my_mouse.set_visible()
#get screen size and set a border .02% around center
w,h = exp.resolution()
c_right = w/2+.02*w
c_left = w/2-.02*w
c_above = h/2+.02*w
c_below = h/2-.02*w

#print (get_pressed)

#a while loop that presents a dot until click
showdot = True
while showdot:
    #clear and display fixdot
    my_canvas.clear()
    my_canvas.fixdot(w/2,h/2,color='black')
    my_canvas.show()
    #get position of mouse to compare to window
    position, timestamp = my_mouse.get_pos()
    x1, y1 = position   
    get()
    lclick2, mclick2, rclick2 =  get_pressed()
    #If click is inside border, break   
    if x1 < c_right and x1 > c_left and y1 < c_above and y1 > c_below and lclick2 == 1:     
        print 'Fixdot Clicked'
        showdot = False
        break

Tracker Prepare Phase

xList, yList, tList = [], [], []
global xList, yList, tList

global myLog

#initialise sampler
from openexp.sampler import sampler
src = exp.get_file(self.get("sent") + ".wav")
my_sampler = sampler(exp,src)
global my_sampler

Tracker Run Phase

#initialise
start = self.time()
my_mouse.set_visible()

#initialise borders
w,h = exp.resolution()
#define top-right margins; 0,0 is top-left margin
tr_left = w-0.35*w
tr_below = 0+0.30*h
#define bottom-right margins; 0,0 is top-left margin
br_left = w-0.35*w
br_above = h-0.30*h
#define bottom-left margins; 0,0 is top-left margin
bl_right = 0+0.35*w
bl_above = h-0.30*h

#initialise logging variables; not necessary, but makes the write command cleaner.
part = self.get('subject_nr')
seq = self.get('count_sequence')
condition = self.get('cond')
snumber = self.get('sent_num')

#play sound - implemented here so that tracking starts with the sound, not after.
my_sampler.volume(1.0)
my_sampler.play()

#set a tracking loop
tracking = True

while tracking:
    position, timestamp = my_mouse.get_pos()
    x, y = position
    t = timestamp - start
    #uncomment this to print to the debug window.
    #print str(t)+', '+str(x)+', '+str(y)
    #write to myLog
    myLog.write(str(x) + ',' + str(y) + ',' + str(t) + ',' + str(seq) + ',' + str(part) + ',' + str(condition) + ',' + str(snumber) + chr(10))
    #assemble to write trajectories to OS logger (duplicate data and could be removed)
    xList.append(x)
    yList.append(y)
    tList.append(t)
    #set up mouse click
    lclick, mclick, rclick =  get_pressed()
    get()
    #three defined areas, one for each picture
    if x > tr_left and y < tr_below and lclick == 1:
        resp = 1
        mouse_rt = timestamp - start
        #print 'Clicked'
        #stop the sound if still playing
        if my_sampler.is_playing():
            my_sampler.stop()
        my_canvas.clear()
        tracking = False
        break
    if x > br_left and y > br_above and lclick == 1:
        resp = 2
        mouse_rt = timestamp - start
        #print 'Clicked'        
        #stop the sound if still playing
        if my_sampler.is_playing():
            my_sampler.stop()
        my_canvas.clear()
        tracking = False
        break
    if x < bl_right and y > bl_above and lclick == 1:
        resp = 3
        #print 'Clicked'
        mouse_rt = timestamp - start        
        #stop the sound if still playing
        if my_sampler.is_playing():
            my_sampler.stop()
        my_canvas.clear()
        tracking = False
        break
    #This dictates sampling rate. 50 hertz is more than enough
    self.sleep(20)
my_canvas.clear()
#print 'answered'
#if resp == self.get('correct'):
    #self.experiment.set("accuracy", 1)
#else:
    #self.experiment.set("accuracy", 0)
self.experiment.set("Mouse_RT", mouse_rt)
self.experiment.set("xTrajectory", str(xList))
self.experiment.set("yTrajectory", str(yList))
self.experiment.set("tTrajectory", str(tList))
self.experiment.set("position", position)
#response areas 1, 2, 3 are top right, bottom right, bottom left, respectively.
self.experiment.set("pic_response", resp)

Finally, the CloseLog Run phase


global myLog
myLog.close()

There's likely some clutter in there and simpler ways of doing things, but this appears functional.

Note on MyLog:

The data format is:
x_coord, y_coord, timestamp, sequence, participant, condition, sentence number
So it's essentially an indexed vertical list of x and y coordinates for the mouse over time.

Cheers.

parfal · April 2014

Hi,

Actually, I am also going to use mouse tracking inline code. However, because I am amateur I cannot manage it. In my experiment two pictures will be shown at top-left and top-right of the screen and the participant should select one of these two pictures. Consider that I have 30 trials, so I should have 30 choices at the end. Their choice is important for me. I have designed this part with a form-based object. Beside that I also would like to record their mouse trajectories and I faced with problem in this part.

By these changes, when I run the program the first trial starts, I even select the first picture and then suddenly the program ends and it gives an error related to MyLog code in the run phase of the tracker, I mean this line:

myLog.write(str(x) + ',' + str(y) + ',' + str(t) + ',' + str(seq) + ',' + str(part) + ',' + str(condition) + ',' + str(snumber) + chr(10))

Actually, I eliminated the line related to LOG file in the run phase of the tracked (I mean the above line) and just by eliminating that line, the program runs without error. However, as expected I do not get the data related to the mouse trajectory in my responses!

I really appreciate if you could help me in this regard and sorry, if the question is really basics. I have also explained my experiment here:

Regards,
Pa

parfal · April 2014

Hi
I just would like to inform you that I solved the problem and now, I get the mouse tracking data. However, it is not finalized and I am still working on that !:)
Bests,
PA

Howdy, Stranger!

Categories

[open] Mousetracking and Visual World Paradigm

Comments

Howdy, Stranger!

Quick Links

Categories

[open] Mousetracking and Visual World Paradigm

Comments