screen_recorder plugin

Wouter

Hey Sebastiaan,

As you know, I've started trying to write a screen recorder plugin. This plugin would allow the user to write (either all or a selection of) image files to the disk, to capture what's on the canvas at certain times. At the end of the experiment (-- or actually, at any point), I want the plugin to offer the user the possibility of collecting all these images and generating a video file. (to me, this just screams openCV). I have a bunch of issues that I'd like to talk through before I fully start hacking:

---

To write a selection of 'screencaps' away to a file doesn't seem too difficult, as the openexp.canvas.surface object would be callable from anywhere, right? (as long as a canvas has been created)
Pygame itself aready contains functions to save images using surface objects (pygame.image.save). However, I'm thinking to maybe import openCV to do this, as this will offer more flexibility to the user in terms of output format or output geometry (e.g. shrink the surface; one will not always want to generate large fullscreen images). Would there be any objection to this?

One interesting detail would be that, as this operation would be uncoupled from 'openexp.canvas.show()', this would mean that there is a possibility that the user generates images that are not necessarily those that are on the screen at a given point. Do you feel this is a problem? Or does simply reporting this in the documentation seem sufficient?

---

Making a plugin that writes images everytime .show() is called, seems a bit more tricky, and maybe even unfeasible at this point. You suggested writing an extension to the canvas class which extends .show(), but that seems to be difficult because of precisely this stament which determines the way it morphs:

openexp/canvas.py, l. 46:

exec("openexp._canvas.%s.%s.init(self, experiment, bgcolor, fgcolor)" % (experiment.canvas_backend, experiment.canvas_backend))

=> which seems to imply that whatever extension is used, the way it is now, it needs to be in the openexp._canvas module;

I feel there are two solutions here:

1) I could write a canvas class which would need to be placed in _canvas . The object would then use similar morphing, and take on the form of whatever canvas class is used, and would implement it's own show()-function which also writes the image file.
The downside of this is, that then the plugin wouldn't be much of a 'plugin' the way they were intended: you can't simply download it, store it in plugins/ and expect it to work anymore; However, a screen_recorder item clearly seems to be one of those things that not all users want or need, and clearly a thing that one DOES want to work in a plugin fashion.

2)the canvas constructor could be extended to make it possible to have plugins define the desired canvas to be used, thus including a constractor statement that looks whether experiment.canvas_backend (?) is one of the 'normal' types, meaning it would look for its definition in openexp._canvas, or look for it elsewhere.
Maybe an even more elegant option would be that experiment would get an extra variable, e.g. 'canvas_definition_path' ; usually this would be set to openexp_canvas, and a plugin could set this varibale to plugins. before any canvas is created. In that case, if the plugin wants to modify the canvas definition, it could, in that folder, implement the new class definitions for every backend type (or whichever way, morph into an extended version of the appropriate backend canvas).

---

Extending the plugin to eventually form a movie file would then not be too difficult; python's openCV implementation has functions that are more than capable of doing this, so a simple item could be implemented that collects all images with a predetermined filename, sorts them using their timestamp and creates a movie file out of them.

One remaining issue: would you think it's necessary that the files generated by this plugin are automatically stored in the file_pool ? Or does that seem somewhat irrelevant?

I'm curious to hear your thoughts about this!

Wouter

sebastiaan

Hi Wouter!

Pygame itself aready contains functions to save images using surface objects (pygame.image.save).

This will work, but only for the legacy (=PyGame) backend. The xpyriment and psycho back-ends have analogous functions for saving the surface to a file though (see my comment below).

However, I'm thinking to maybe import openCV to do this, as this will offer more flexibility to the user in terms of output format or output geometry (e.g. shrink the surface; one will not always want to generate large fullscreen images). Would there be any objection to this?

OpenCV would be a good choice, certainly for the video capabilities. For manipulating (e.g. resizing) images you can also use PIL or pygame, it doesn't really matter.

The canvas constructor could be extended to make it possible to have plugins define the desired canvas to be used, thus including a constractor statement that looks whether experiment.canvas_backend (?) is one of the 'normal' types, meaning it would look for its definition in openexp._canvas, or look for it elsewhere.

Right, I was thinking about something similar. The easiest implementation would probably be to check whether the backend variables (canvas_backend etc.) point to a file, and if so use that file, and if not assume that the variable points towards an openexp module. This way you could say things like:

exp.set('canvas_backend', 'my_canvas_backend.py') # Use a file
exp.set('canvas_backend', 'xpyriment') # Use a standard backend

This should be doable. For now, maybe you could assume that this functionality already exists and just put your own canvas class in the openexp._canvas folder. Later on it can then be turned into a proper plug-in.

A downside of this method is that you will need to write a different custom backend to wrap around each of the existent backends. So, for example, you would create wouters_legacy that extends the legacy backend, wouters_psycho that extends the psycho backend, and wouters_expyriment that extends the xpyriment backend. Otherwise the screen recorder would only work with one of the backends (although initially that may be enough, of course).

One remaining issue: would you think it's necessary that the files generated by this plugin are automatically stored in the file_pool ? Or does that seem somewhat irrelevant?

That would be nice, I think, although there's the risk of creating a huge filepool that leads to slow saving etc. There a no real guidelines for this, whatever you think is most convenient.

Good luck and keep me posted!
Sebastiaan

Wouter

Ok great!

So now I'm thinking about a plugin design based on three items (I've actually aready made their logos -- first things first of course )

• A screen_record_snapshot plugin; to be inserted at every point in the experiment where one would like to take a snapshot:
• A screen_record_show plugin; inserted (usually as a first item in the main sequence) and from that point on generates snapshots automatically on every call to 'exp.canvas.show()'.
• A screen_record_video plugin; (inserted usually at the end of the experiment) which would wrap all images generated during the expt into a video file.

I can get started with '_snapshot' and '_video' right away; these don't have to mess with canvas and can simply use exp.canvas_backend to implement the appropriate way of saving images.

Right, I was thinking about something similar. The easiest implementation would probably be to check whether the backend variables (canvas_backend etc.) point to a file, and if so use that file, and if not assume that the variable points towards an openexp module. This way you could say things like:

exp.set('canvas_backend', 'my_canvas_backend.py') # Use a file

exp.set('canvas_backend', 'xpyriment') # Use a standard backend

I have been thinking about changing the variable 'canvas_backend' as well; however, did you see my brief remark that:

Maybe an even more elegant option would be that experiment would get an extra variable, e.g. 'canvas_definition_path' ; usually this would be set to openexp._canvas, and a plugin could set this variable to 'plugins/' before any canvas is created.

The reason I'm calling this 'maybe more elegant' is that then, 'exp.canvas_backend' would still correspond to the actual canvas backend that is used. This seems to me important for the following reasons:

1. We don't want to give the user the impression that its initial settings were overwritten and the backend was different than what was initially defined (e.g. when reading the logfile).
2. Other plugins, additions to Opensesame, or even inline/external scripts might make use of exp.canvas_backend and assume it corresponds to one of those values; since the backend hasn't changed.

I'm trying to think of the most generally applicable solution here;
for my case, it's clearly still the case that the backend that is used still corresponds to what is set initially; it's just that its current definition is in a different place.

Other scenarios, where the plugin actually does implement a different backend, which would 'deserve' that exp.canvas_backend gets changed (those are maybe actually best not classified as plugins at all, but actually should be included in _canvas to provide it as an option in the experiment-item), could nevertheless still use this setup and change both the canvas_definition_location AND the canvas_backend variable.

A downside of this method is that you will need to write a different custom backend to wrap around each of the existent backends.

There's no obvious way around this when writing a child-class: constructor will always need to know how to implement the member functions: however, couldn't I generate a class that 'morphs' just like the 'generic' canvas class does? dependent on the value of exp.canvas_backend it could both call a different constructor AND use a different .show() implementation;

One remaining issue: would you think it's necessary that the files generated by this plugin are automatically stored in the file_pool ? Or does that seem somewhat irrelevant?

That would be nice, I think, although there's the risk of creating a huge filepool that leads to slow saving etc. There a no real guidelines for this, whatever you think is most convenient.

Yes; the huge filepool was exactly why I asked: I'm thinking not using the filepool then. Instead, I'm thinking to store all the images in a different folder per session; otherwise the video plugin would get confused when merging different video files from different sessions/runs.

...but anyway, I will get started!
Wouter

sebastiaan

Hi Wouter,

We don't want to give the user the impression that its initial settings were overwritten and the backend was different than what was initially defined (e.g. when reading the logfile).

The back-end would be different. Your back-end would replace the original one, even though it would (ideally) not be very different from (or incompatible with) the one that was originally selected. I'm not really a fan of the idea of having a separate variable (e.g., canvas_definition_path), because that would mean that there are two variables that determine which back-end is loaded, whereas one variable will do (and is more transparent).

Other plugins, additions to Opensesame, or even inline/external scripts might make use of exp.canvas_backend and assume it corresponds to one of those values; since the backend hasn't changed.

Right, that's true. I think this will only apply to a few of cases, such as Daniel's media_player plug-in. But changing the back-end variable will throw the media_player off unnecessarily. Maybe a better idea would be to add an is_compatible_with() function to the back-end, so that plug-ins could do a check like this:

if my_canvas.is_compatible_with('psycho'):
    # do PsychoPy things

(...) couldn't I generate a class that 'morphs' just like the 'generic' canvas class does? dependent on the value of exp.canvas_backend it could both call a different constructor AND use a different .show() implementation;

I don't think that's possible. The morphing really changes one object into an object of another type, but I don't see how this allows you to somehow combine multiple classes into a single class, if you catch my drift. It might be possible in a very hackish way (the morphing is a pretty dodgy technique already though), but I don't think it's necessary. There's nothing wrong with writing three small classes for each new back-end, right?

Cheers!
Sebastiaan