[open] 2 cameras at once?

jakubch

Hi Sebastiaan,

it's been a long time since my last post on cogsci.nl. I have used OpenSesame for my diploma thesis project (to my satisfaction) but then moved on to other projects. Alas, now I was searching for some movement tracker - and here you are again with a handy tool right for me Or at least possibly.

So here goes my question: Is it possible to use one instance of Mantra to track objects that are being filmed by 2 different cameras? Or, would it be possible to use 2 Mantras but sync them reliably so we could compare the txt outputs and compare the onsets and closures of movements of both objects? Context: In a synchrony experiment, we want to be able to measure movements from 2 subjects at the same time.

[EDIT:] Also, is there any way for Mantra to receive triggers from Matlab?

Thanks in adnvance!

Jakub

sebastiaan

Hi Jakub,

So here goes my question: Is it possible to use one instance of Mantra to track objects that are being filmed by 2 different cameras?

No, one instance is tied to one specific camera.

Or, would it be possible to use 2 Mantras but sync them reliably so we could compare the txt outputs and compare the onsets and closures of movements of both objects?

This should be possible, yes. The timestamps in the logfile are in milliseconds from the moment of initialization. By themselves, these are not suitable for synchronization, because each instance is initialized at a different moment. However, each log file starts with a timestamp, like this:

8664817 DATETIME Thu Apr 29 14:42:47 2010

With this, you can convert the timestamps to the actual time, and thus synchronize two recordings afterwards with second accuracy. If you want a higher degree of temporal precision, you would need to write a script (possibly in OpenSesame) that continuously polls the two Mantra servers and writes the data from both servers to a single log file. Does that make sense?

[EDIT:] Also, is there any way for Mantra to receive triggers from Matlab?

If Matlab supports UDP communication, and I suppose it must, then you can send and receive commands to Mantra this way. This is not very well documented, but you can see how this works in libmantra.py, which does exactly this (but in Python). Essentially you would need to re-implement libmantra.py in Matlab, which is easier than it sounds:

• https://github.com/smathot/mantra/blob/master/opensesame/mantra_connect/libmantra.py

Cheers!
Sebastiaan

jakubch

Thanks for the answers!

Could these 2 instances of mantra run on the same computer? And if more than one webcam is plugged into the PC via USB, would I be able to tell to each Mantra instance to receive input from a desired camera? And finally, could I do it all with only 1 computer?

As for the precision, I think the roughest unit of measurement for us would be 10ms.

sebastiaan

Could these 2 instances of mantra run on the same computer? And if more than one webcam is plugged into the PC via USB, would I be able to tell to each Mantra instance to receive input from a desired camera? And finally, could I do it all with only 1 computer?

Yes, you would have three separate programs running (two servers and one "synchronization script"), but that can easily be done on a single (reasonably modern) computer. Cameras have names such as /dev/video0, which you can use to select a specific camera.

As for the precision, I think the roughest unit of measurement for us would be 10ms.

Just to point out the obvious: Most regular webcams work with 30 fps, which means that you can only capture a frame every 33 ms. But of course you might be using a high-speed camera.

Cheers!

jakubch

Actually, sometimes it does not hurt to point out the obvious We are trying to combine EMG and object tracking here to measure subject's movements, and still working out the details.

Having higher frame-rate would solve the precision problem; however, I suspect that the computational load of 2 say 100 fps cameras could be too much even for a reasonably modern computer. On the other hand, we might be able to come up with a design that would allow us to count with lower precision for movement tracking.

One last question: Could Open Sesame receive triggers from Matlab? The idea is to have Matlab on the top of the hierarchy so that it can send triggers to the rest of the enssamble (EMG machine for recording, some kind of metronome that would play sounds to subjects and finally to Open Sesame which would in turn send triggers to both Mantras). Does this make sense?

sebastiaan

I suspect that the computational load of 2 say 100 fps cameras could be too much even for a reasonably modern computer.

I've never done this, but assuming that processing load scales up linearly with framerate, I actually think it should be possible. I did most of the Mantra validation stuff on a 1.6 Ghz intel atom netbook! The timecritical part of Mantra has been written in C, which is why it's pretty fast (and a hassle to install).

Could Open Sesame receive triggers from Matlab?

That should be possible. I'm not sure what kind of communication you want to set up, but in principle you can receive information through a parallel/ serial/ network port just as easily as you can send it.

jakubch

Hi,

a small update on our situation, we were testing Mantra in order to see whether we can use it and so far we are quite impressed In moderately good lighting conditions, it is very reliable and fast. Log files can be analysed very nicely in Matlab, too (although we haven't explored the possibility of Matlab->Mantra communication yet).

For our experiment, most likely we won't need OpenSesame. However, as I like to play with it, I wanted to try some example Mantra experiments - and I ran into trouble.

I had the exact same problem as described in Froukehe's post except updating mantra_connect didn't help (downloaded from github). Output as follows:

from libmantra import libmantra
m = libmantra()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "libmantra.py", line 43, in __init__
    self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
print m.sample()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'm' is not defined

Perhaps I should note I'm using mantra on Xubuntu 13.04 without GUI in order to (successfully) solve the issue described here. But I guess this issue is not related to the current one.

EDIT: Also, the simple experiment you provided seems to work. Yet, both "affordances" and "inline_code" examples give me the same error as in Froukehe's post.

Thanks for any input ;-)

Jakub

sebastiaan

we were testing Mantra in order to see whether we can use it and so far we are quite impressed In moderately good lighting conditions, it is very reliable and fast.

Good to hear!

With respect to the error, it means that the socket cannot be created, because it is already in use. But this is odd, because no port or address has been specified at this point. What comes to mind is that you are running this code from within OpenSesame, and libmantra has already been initialized by the mantra_connect plug-in. Could that be it?

jakubch

I don't know if I understand correctly. Let's see: I ran

from libmantra import libmantra
m = libmantra()
print m.sample()

from terminal using python interpreter (as suggested by you in Froukehe's tread). If at the same time OpenSesame was running with mantra plugin added as one element in the experiment tree, would that mean that the mantra_connect is running and thus preventing the socket from being created?

If so, I can check the above code's output with OpenSesame not running (or without mantra plugin used in the experiment). Would that be informative?

EDIT: ok, now I see perhaps I confused you a bit. The procedure was: Try affordances/inline code example with the typical "Failed to connect..." error (although the port is set correctly, UDP in use). "Simple experiment" however works. And when I try the above code in the python interpreter, I get the output as stated in my previous post.

sebastiaan

If at the same time OpenSesame was running with mantra plugin added as one element in the experiment tree, would that mean that the mantra_connect is running and thus preventing the socket from being created?

No, I don't think that should be a problem.

It seems that the error may not have anything to do with Mantra per se, but that Python simply fails to create a socket object. I don't see any reason why the line that causes the problem should fail, regardless of whether Mantra is functioning properly. And yet it does.

What happens if you run these two lines by themselves in a Python interpreter? This simply creates a UDP socket, and should not produce any error message.

import socket
socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

jakubch

Sebastiaan, I don't know what's changed, but suddenly things are working as they're supposed to. It must have been me

I'll post an update on our progress with Mantra when we get somewhere. Bye,

Jakub

jakubch

Hi there again,

I have just stumbled upon something strange. When I change the resolution in the mantrascript.py from 320, 240 to, say, 800, 600, the object definition window won't load and it gives me segmentation fault:

Fatal Python error: (pygame parachute) Segmentation Fault
Aborted (core dumped)

It is not critical for me but why is this happening?

sebastiaan

ITA I'll get back to you on this!

sebastiaan

Hi Jakub,

Basically, I don't know. It appears that the video4linux libraries occasionally triggers these segmentation faults. This didn't used to happen, back when I was developing Mantra on Ubuntu 9.10 and 10.04. But with more recent versions of Ubuntu, things started to break. See for example this discussion.

This is clearly a big problem, but it will require considerable development resources to address this properly (the mantrascript is basically a hack). So for now, I can only confirm that this indeed happens, in (for me) unpredictable circumstances.

Cheers,
Sebastiaan

jakubch

Thanks for the answer.

I think I will install 10.04 on some machine to be able to test everything properly and perhaps stick with it for the time needed to do the experiments. We're lucky Ubuntu is free and people keep old releases so that at least switching between OS versions does not have to be a pain.

Until next time! ;-)

Jakub

jakubch

Well, interestingly enough, it just gave me that same segmentation fault on Ubuntu 10.04 (after using the "Add object" thing in the menu). Any ideas?