Thinking ahead: Re-thinking some of OpenSesame's design principles

sebastiaan

Update: A rough list of all the areas of improvement that have been suggested:

• GUI: A better loop table
• GUI: Better drag-and-drop
• GUI: A better sketchpad
• Runtime: More sensible variable management
• Runtime: Consistent coordinates (i.e. always 0,0 as the display center)
• Runtime: A way to do things in parallel
• Runtime: Arithmetic in OpenSesame script
• Runtime: Fully integrate Gstreamer back-end

Hi fellow developers and interested users,

I'm quite happy with how the 2.8 series of OpenSesame is turning out, and how we're gradually working towards a bug-free utopia. Thank you to everyone who contributed, in whatever way! (The increased stability of OpenSesame is to a large part also due to improvements in Expyriment and PsychoPy, so credit to these guys as well.)

Nevertheless, OpenSesame has a number of suboptimal design choices that I would like to see improved in the future, potentially for the 2.9 series, or perhaps even later. Many of these design choices are deeply ingrained in the source code and workflow, and will require considerable changes and potentially introduce backwards incompatibility. That's why they haven't been addressed so far.

With this post I would like to pick your brain about what you currently consider to be problematic, and how you would like to see this fixed. (We're talking about fundamental changes here, so please keep minor bug-reports and inconveniences elsewhere, such as in this discussion.)

I'll start with what I think should be improved, and some thoughts on how. Feel free to comment on these points, suggest alternatives, and also raise new points. I'll update this discussion as necessary.

Managing experimental variables

Currently, as you know, you need to set experimental variables like so ...

exp.set('my_var', 'my_value')

... and get them like so ...

my_value = self.get('my_var')

This works, but it feels clumsy. Also, from a technical point of view, these variables are either stored as properties of the experiment object (for experiment-wide variables) or properties of the item object (for item-specific) variables. This often leads to conflicts, because, for example, a variable cannot have the same name as a function of the experiment object.

So how could this be improved? I'm thinking (but this is just a sketch in my mind) about something like the following. There is a var object that you can use to get and set experimental variables in a transparent way, like so:

var.my_var = 'my_value' # Like exp.set('my_var', 'my_value')
my_value = var.my_var # Like my_value = self.get('my_var')

This offers a transparent API that is easy to understand, and usually works as you would expect. However, this basic API ignores the fact that variables can live in the experiment object, or in an item object. The following API could offer a more flexible (but rarely used) way to get and set experiment variables:

# Set experiment variable
var.my_var = 'my_value'
var.experiment.my_var = 'my_value'
# Set variable to current item (usally an inline_script)
var.item.my_var = 'my_value'
# Set variable to item named 'my_sketchpad'
var.my_sketchpad.my_var = 'my_value'

# Get variable. First look in current item, fall back to experiment.
my_value = var.my_var
my_value = var.item.my_var
# Get experiment variable.
my_value = var.experiment.my_var
# Get variable. First look in item named my_sketchpad, fall back to experiment.
my_value = var.my_sketchpad.my_var

We also need to rethink how variable types are dealt with. Right now, variables are saved as int, float, or unicode, depending on which type fits best. Things such as lists are converted to their unicode representation. This can be confusing. My suggestion would be to simplify this to the following logic:

• Numeric values are saved as float.
• str and unicode values are saved as unicode, assuming utf-8 encoding for str objects.
• All other types trigger an osexception. So they are not silently converted to unicode anymore.

Finally, how can we implement (something like) the API above in a way that doesn't break backwards compatibility, and assures that current plug-ins continue working?

Top-left vs center-based coordinates

Coordinates in sketchpad objects (and elsewhere in the GUI) use 0,0 to indicate the center, whereas mouse coordinates and coordinates specified in inline_scripts use 0,0 to indicate the top-left. This is clearly inconsistent, and is a result of the way that OpenSesame gradually evolved. In principle, this can be changed relatively easily, but the obvious problem is that doing so will break a lot of existing experiments. I was thinking something like the following:

We add a new experimental variable, say unified_coordinates, that can be yes or no. When set to no, coordinates should interpreted as they do now (inconsistently), when set to yes all coordinates are interpreted relative to a 0,0 center (consistently). Then we change the templates such that they set unified_coordinates to yes. The result will be that old experiments continue to work, but new experiments all use a sensible and consistent coordinate system.

After some time, we can remove this hack (thus keeping the code clean) and only support center-based coordinates.

Easily doing things in parallel

Currently, it's difficult to do things in parallel in OpenSesame. The typical problem that users run into is that they want to collect responses while at the same time implementing some sequence of sketchpads. This requires scripting and is unnecessarily difficult, given that it's pretty basic functionality.

I had hoped that the parallel plug-in would solve this issue, but it turns out that using threads to do things in parallel is tricky and prone to crashes. So my question is pretty open: What would be a user-friendly way to do things in parallel? Could this be done by a plug-in, possible by improving/ fixing the current parallel plug-in? Technicalities aside, what would this functionality ideally look like?

Cheers!
Sebastiaan

psychofs

Hi Sebastiaan,
I'd totally agree to simplify the usage of variables. Which is in line with that which caused me some trouble, is when do I have to use "quotation marks" or [square brackets] or non of these... And also I got used to setting some default value variables in an inline script at the beginning of the end so that the variable already exists when preparing an Item. Maybe you could just also implement a kind of inline script in the background to create the variables when starting to run and to set some default values.

I'm looking forward to the changes.
Cheers.
Florian

nsriram

Before making widespread changes, we can document where the problems are and whether existing design patterns can be used.

I am quite happy with the scheme I am using (although my sketchpad contents are decided before the loop starts and not vary as a function of specific responses). One thing thats not clear to me is exp.set variable has some inconsistent behavior (that is not there when the variable is defined using global). If I do an exp.set inside a loop for the very first time, it seems hard to modify the variable in a subsequent inline script (I had to use globals for this). So it does seem that initialization using exp.set and global are somewhat different.

In my design pattern, the trialsequence object is used in all blocks. The only thing that is unique to each block is a blockinitialization script that sets values to various global vars and manipulates them. The loop that runs the trialsequence is also reused in each block. This presumes that the task designer has abstracted all the things necessary and converted those to variable settings in the block initialization.

So a non-programmer would only need to learn to set the variable values in the block init script (in my use case).

I guess in the parallel execution scenario you are talking about, the content of the sketchpad is to be modified as a result of some response. Say response k has an impact on sketchpad k+1. I thought the normal way to do this would be in the external loop controlling the prepare and run for the trialsequence? That would know what the response was (assume that is stored in a global) and the logic can be done in the loop.

If you want to reach non-programmers, then an alternative route might be declarative syntax (no GUI) that is something like Inquisit but it wont be able to do complicated things. But it can be quite wordy and repetitive because everything is concrete (values, not variables). So a declarative style syntax that allowed for variables could be handy and could cater to a large number of use cases in compact style.

btw, on windows I found dragging objects to change sequence to be quite hard. Also the screen gets corrupted when certain actions are done and a close and restart fixes it. Minor issues.

Edwin

Very good initiative! Some suggestions:

Variable Spaces

I think the issue with the variable definition is that it's unclear how OpenSesame handles different variables. As I see it, we have two variable workspaces: one for the GUI (let's call these OpenSesame variables), and one for inline_scripts (let's call these Python variables). The former can do with to types of variables: strings and floats. The latter requires all types of Python variables.

In my opinion, the functions to assign variables to any one of the workspaces should clearly reflect what they are doing. The advantage is that it's clearer what OpenSesame is doing with its variables. Also, I can't think of a situation in which an OpenSesame variable should ever be anything else than a number or a string (unless the OpenSesame syntax becomes more powerful, so it can access specific indices in lists). An example:

# create a variable that's local to an inline_script
my_var = "read this"

# create OpenSesame variable "feedbacktext" and assign
# to it the value of my_var
add_to_OpenSesame_workspace("feedbacktext", my_var)

# create a Python variable "my_var", to save it for use in
# later inline_scripts
add_to_Python_workspace("my_var", my_var)

In a feedback item displaying the text, you use [feedbacktext], but not [my_var].

Arithmetic in OpenSesame syntax

Currently, arithmetic e.g. to adjust stimulus locations in a sketchpad is not possible. This is not very intuitive. Would it be possible to add this in a straightforward way? Or would the parsing of OpenSesame syntax get out of hand?

Daniel

Great idea and I already Agree with @Edwin and you on your variable management points.

I have some suggested improvements of my own for existing OpenSesame procedures and interface widgets. These suggestions partly overlap with the other 'wishlist' thread (and thus might have been already mentioned there by me) and mainly concern improvements to the interface (I'm quite content with the new QProgEdit editor). I'll shortly summarize what I hope we can do differently in next versions:

More intuitive/classic tables in the loop items

My biggest gripe at the moment concerns the tables used in loop items. Compared to other 'spreadsheet-like' applications (or table editing in E-Prime), there are some crucial functionalities missing:

• changing order of rows and columns is impossible/hard (requires a lot of manual copy/pasting). It would for instance be nice if selected rows could simply be dragged to a position between other rows to position them there
• It is hard to insert new rows in between old ones, or 'ripple delete' existing rows: selecting rows and deleting them does not shift up the rows below them. It simply empties the cells. The user then has to manually move the contents of the lower cells upwards refill the empty cells and make the table consistently filled again.
• duplicating values in a column (or row, but this is less important) is really hard to do. This needs to be done now by simply copy/pasting the desired value to each cell. It would be nice if a cell could have a 'handle' at the bottom right (like in other spreadsheet applications), that can be dragged down and copies the values to the extent of which it is dragged

Summarizing: it would be nice if the handling and possibilities of tables in OS would resemble those of more classic spreadsheet applications more.

Dragging/dropping

Dragging and dropping could be made a lot smoother. I mainly say this because, when teaching OpenSesame, it can take students a long time to get the hang of the dragging and dropping possibilities, just because they are not very apparent to them and are hard to follow.

The nicest would be to use animations to make it clearer what is happening, but I don't know if this is easy to implement or possible at all. For instance, if I drag an item to the overview pane, I am always confused if the dragged item will appear above or below the item near which it is dropped. It might be nice if an items shifts away and makes space for the dragged item, so it is clear where it will end up in the tree.

The same holds for changing order of items in the sequence item. I know now, after working with OS for quite a while, that you have to drag an item on top of another item, in order to place the dragged item above it, but it would feel more intuitive and be easier to follow for newer users if the item, over which a dragged item hovers, shifts downward with an animation and thereby makes space for the dragged item.

Sketchpad operation

Shortly put: drawing with the sketchpad is really hard. Drawing intial figures is easy, but after this all alternations to drawn items need to be made in OpenSesame script or by right-clicking on the object. Would it be feasible to make a more 'InkScape'-style interface, with a toolbar, and draggable vertices for each item, in such a way that one can easily change color, shapes or line orientations? I know it might be a bridge too far for now and it would involve tons of work to make it like this, but I think it is a good idea to already start thinking about this and hear other peoples' opinions about this to see if there is a desire for such a massive overhaul of the sketchpad.

sebastiaan

Hi guys,

Thanks for all the feedback. I'm happy to hear your thoughts.

I've started to make things a bit more concrete. First off, I removed the playground branch and added two new branches:

• gibson is the currently stable 2.8 series. This will receive only bugfixes and incremental improvements.
• heisenberg is the currently unstable 2.9 series. So any major changes should go there. I already did some restructuring to clean up the codebase, so if you send me a pull request, please make sure that you're up-to-date with heisenberg.

There's clearly a lot to be done, but the GUI improvements are low-hanging fruit. These can be implemented relatively easily, and they don't affect the runtime, so we don't have to worry about backwards compatibility. I think Daniel spotted the main areas of improvement, so let's see how we can approach them.

Loop tables

My biggest gripe at the moment concerns the tables used in loop items. Compared to other 'spreadsheet-like' applications (or table editing in E-Prime), there are some crucial functionalities missing:

This is probably the easiest thing to improve, because it is more-or-less a standalone project. It comes down to extending and polishing libqtopensesame.widgets.good_looking_table, which is an extension of QtGui.QTableWidget. Possibly, this could become a small standalone project, because a good spreadsheet-like widget would be useful outside of OpenSesame as well.

Question: Daniel, interested in taking this on?

Drag'n'drop

Dragging and dropping could be made a lot smoother. I mainly say this because, when teaching OpenSesame, it can take students a long time to get the hang of the dragging and dropping possibilities, just because they are not very apparent to them and are hard to follow.

This is a little tricky, because drag'n'drop has been implemented in an ad-hoc way, and is spread throughout the code. The following modules are relevant:

• libtqopensesame.widgets.draggables contains sequence-related stuff
• libqtopensesame.widgets.tree_overview contains overview-area-related stuff

Sketchpad

Shortly put: drawing with the sketchpad is really hard. Drawing intial figures is easy, but after this all alternations to drawn items need to be made in OpenSesame script or by right-clicking on the object. Would it be feasible to make a more 'InkScape'-style interface, with a toolbar, and draggable vertices for each item, in such a way that one can easily change color, shapes or line orientations?

Yes, that's been on my todo list for some time. The sketchpad is a mess at the moment, and I want to rewrite it completely. I'll handle that part. But once the code is clean, we need a nice GUI on top, and I could use some inspiration.

Question: Any nice mock-ups for the sketchpad interface? An image will do, or something created using Qt Designer.

Cheers,
Sebastiaan

Daniel

Hi Sebastiaan,

thanks for the further input! I'd be happy to dive into enhancing the QTableWidget/good_looking_table functionalities. What is the best way to start this as a standalone project. Just start afresh while following conventions that make it easier to load the module in OpenSesame, or just 'extract' the good_looking_table class and build on from there? How did you do this for QProgEdit? I can't make any promises on the timely progress of this project though. I have quite a lot on my plate at the moment and really have to work on this on the spare moments I can find, but I will do my best.

Regarding the sketchpad interface. Is it a good idea to mimic some interface that people might already be familiar with, such as InkScape or Gimp? We can use that as a template and then make the necessary changes from there. A QT-Designer mockup is a good starting point. I can try to make something like this if you want, but maybe it's a good idea to do a requirements analysis first, to determine what the new sketchpad should be able to do? From there we can try to fit all these wishes into an intuitive interface.

sebastiaan

@dschreij

I'd be happy to dive into enhancing the QTableWidget/good_looking_table functionalities. What is the best way to start this as a standalone project. Just start afresh while following conventions that make it easier to load the module in OpenSesame, or just 'extract' the good_looking_table class and build on from there?

You can start from scratch if you think that's easiest. Basically, for integration with OpenSesame, we'll need only a few things:

• An API to load data into the table.
• An API to retrieve data from the table.
• A signal that is sent when the user made changes to the table.

Once that exists, it won't be too hard to include the table widget in OpenSesame.

Regarding the sketchpad interface. Is it a good idea to mimic some interface that people might already be familiar with, such as InkScape or Gimp? We can use that as a template and then make the necessary changes from there. A QT-Designer mockup is a good starting point. I can try to make something like this if you want, but maybe it's a good idea to do a requirements analysis first, to determine what the new sketchpad should be able to do? From there we can try to fit all these wishes into an intuitive interface.

A requirements analysis sounds very professional. How would you suggest we go about that?

sebastiaan

I finally got around to doing some serious work on OpenSesame. Most of it has been under the hood, but I also reworked the sketchpad. The current snapshot is in the heisenberg branch:

• https://github.com/smathot/OpenSesame/tree/heisenberg

Let me know what you think. It's still a work in progress, but it's already a lot better than the previous sketchpad. Some highlights:

• Move elements by dragging
• Select and change multiple elements (for example changing the font for a series of textline elements at once)
• Better use of screen real estate
• Possibility to raise and lower objects (i.e. a z_index property)

Daniel

I really like it! The ergonomics of this sketchpad are already far superior to the previous version.

sebastiaan

I pushed some more updates to the heisenberg branch. It's becoming a very major update! But beware, big improvements come with many bugs and regressions! So any testing is welcome. You can get an (early) pre-release Windows package from the link below, and I also built a package on the opensesame-next ppa.

• http://files.cogsci.nl/software/opensesame/pre-releases/
• https://launchpad.net/~smathot/+archive/ubuntu/opensesame-next

Drag and drop

Drag-and-drop support has improved enormously, and the sequence item has been redesigned as a part of that. Basically, you can now do everything you would expect through drag and drop, including dragging items from one instance of OpenSesame to another. You can also create shallow copies by dragging while holding control, and deep copies by dragging while holding control+shift.

Split view

Another nice feature is that you can now edit the script and the GUI controls simultaneously (i.e. split view), as you can see below:

This is not just a GUI enhancement, but is the result of reworking (and massively improving) the way that changes are applied under the hood. Because of this, you will also see that the overview area doesn't continuously unfold anymore, and things should feel more snappy in general.

(A persistent) Python workspace

A more subtle modification is the addition of a 'Python workspace', which is basically the Python environment in which inline_scripts are executed. The idea is that this workspace remains accessible through the debug window after the experiment has finished (or after it has crashed) to facilitate debugging. Right now, this only happens with the inprocess_runner, so @dschreij, we should take a look at how to implement this for the multiprocess_runner as well.

The end goal is to have a graphical browser of the Python workspace, which allows you to inspect all variables and functions that have been defined in inline_scripts. Ideally, you would also be able to insert breakpoints into your experiment. But for now, just having a persistent Python workspace that remains accessible through the debug window is a good start.

Quick open item

Finally, a small but nice feature is the 'quick open item' dialog (Meta+O), which allows you to quickly switch between items using the keyboard.

That's it for now!