Memory leaks in OSweb 1.4 (sketchpad/feedback/sampler)

Fab

Hi there,

While running an experiment with OSWeb 1.4, several participants reported the task crashing or sound stimuli getting distorted or stopping playing towards the end of the experiment. In that experiment (with just over 1200 trials, with 2 sketchpads, a feedback object and one sampler used in every trial), it affected about a third of the participants.

I noticed the problem on a couple of computers myself. While I initially thought it had to do with my experiment or the participants' computer, after some investigating, I realized that the problems appears to be memory leaks in OSWeb.

With basic sequences running just a single sketchpad, there is no problem (no leak). However, as soon as a sequence contains more than one sketchpad, a feedback object in addition to a sketchpad, or a sampler, every execution of the sequence leaves memory leaks. When many trials are used, this can result in the experiment malfunctioning or crashing.

Here's a screenshot taken using the browser's developer's tools to track memory usage. This is the result of 50 trials with a sequence involving two sketchpads, a feedback object and a sampler.

I've done tests with very basic tasks, on different computers. Here's a comparison for different sequences involving more or less objects:

I've tried replacing the sketchpads with Javascript code (canvas), but the result is the same. For the memory leak related to the audio, I managed to eliminate it by getting rid of the sampler altogether, and using Javascript code to play sounds, after loading the sounds I need into variables (vars) from URLs point to Github resources. Bypassing the sampler in this way seems to work as a temporary fix (out of 40 participants, one reported a crash (but was using Safari), the other 39 reported that the sound worked throughout the experiment). Still, the memory leaks from sketchpads/feedback objects are a concern to me as I'm worried problems will arise if I increase the number of trials. In the experiment I'm running now, I'm using 640 trials but I often require a lot more.

I noticed that memory leaks for images was an issue in an earlier version of OSWeb that seemed to have been addressed (https://osdoc.cogsci.nl/3.3/notes/336/), but it looks as if memory leaks remain present in OSWeb 1.4 (for sketchpads/feedback objects, but also for the sampler).

Just thought I'd flag that out as something to look into for the next update.

Many thanks!

Best,

Fabrice.

sebastiaan

Hi @Fab ,

Thanks for reporting this. I'm going to look into it! It does indeed sound similar to the earlier memory-leak issue.

— Sebastiaan

kiwifoxb612

Hello,

Our experiment also has this problem.

It can effect half of the results. Please let us know if you fix it @sebastiaan . Thanks!

HenkvanSteenbergen

A very similar case here. Even repeating the same sequence with sketchpads that do not contain images or sounds is increasing the consumed memory with several MB each time it is presented. In the example below (I removed the task itself) I have a questionnaire with a progress bar that is updated each 250 ms. Memory use is exploding and goes up to 1GB within a minute.

see file below (please change extension from .osexp_.txt to .osexp):

https://www.henkvansteenbergen.com/wp-content/uploads/2022/02/test_memoryleakage.osexp_.txt

sebastiaan

@Fab @kiwifoxb612 @HenkvanSteenbergen Thanks for chipping in. I filed an issue for this, and I'm hoping to find a solution soon. (Maybe not for inclusion OpenSesame 3.3.11 though, unless it turns out to be easier than I think it is.)

Fab

Many thanks @sebastiaan! Good luck with this! Would be awesome to solve it (I'm concerned that it might affect the reliability of response times in some paradigms).

All the best,

Fabrice.

HenkvanSteenbergen

Thank you so much @sebastiaan! Fingers crossed!

sebastiaan

@Fab @HenkvanSteenbergen In Henk's experiment, the culprit appears to be text rendering, which indeed results in this memory leak. Basically any experiment that presents a large number (hundreds) of displays with text appears to suffer from this. (But only on Windows and Chrome. Under Linux the exact same experiment runs fine.)

Before I assume that text rendering is the only issue: Fabrice, if you remove all text, do you still see this memory leak?

HenkvanSteenbergen

@sebastiaan Cool to hear there is some progress already! If you have a JavaScript snippet ready that I can insert to get rid of this issue, please let me know

sebastiaan

@HenkvanSteenbergen @Fab I think I found the culprit, at least if it's indeed purely text-related as I suspect. Could you test if upgrading OSWeb to this prerelease package resolves the issue?

!pip install https://github.com/open-cogsci/opensesame-extension-osweb/releases/download/prerelease%2F1.4.4.0a1/opensesame_extension_osweb-1.4.4.0a1-py2.py3-none-any.whl --upgrade

Fab

Hi @sebastiaan, (@HenkvanSteenbergen)

Thanks for looking into this!

The experiment I initially experienced problems with didn't actually present text within the trial sequence, only sketchpads presenting a picture, and a sampler playing a short tone.

To try it out again, I set up a basic flanker task from scratch, avoiding any text stimulus (not even for instruction screens):

Basic flanker.osexp

And I tested it with Edge and with Chrome on Windows 10 (first using OSWeb currently installed on my office computer: 1.4.2.1), interrupting the memory monitoring after 16 or 17 trials. If detached elements are an indication of memory leaks, it does look as if they are present even when text is not used.

With Edge:

With Chrome:

I then upgraded to OSWeb 1.4.4.0a1 and ran the same task with the same browsers, interrupting it after 16 trials.

Here is what the memory monitoring threw up:

With Edge:

With Chrome:

Updating to OSWeb 1.4.4.01a doesn't seem to have made a difference to the number of detached elements (so, assuming that they indicate a leak - I'm no expert -, it looks as if the problem persists). In my experience, the problem has been mostly visible through sound failure and in some cases, the task crashing (I assume that sounds use up more memory). I addressed it by replacing the sampler object with Javascript code to play the sounds (as a quick fix but I don't know if I can trust the timing accuracy of that method). The task then seemed to work better (students did not report crashes and the sound seemed to work fine), but the sketchpads still left detached objects and I'm unsure whether this affected the task (I ran two experiments in which I failed to obtain a fairly large effect I had obtained in 4 experiments with E-Prime with face-to-face testing or online using EPrimeGo; though I can't be sure whether it reflects a problem with my programming or if memory leaks caused inaccurate response time measurements).

Hope the above information is of help. If you'd like me to run some more specific diagnostic test or a specific task, do let me know.

Best,

Fabrice.

HenkvanSteenbergen

@sebastiaan thanks for the super quick action. The upgraded version does indeed solve the memory increase during the sketchpad with text and lines. However, when I included my trials again (each trial contains up to 10 sketchpads with images, circles and many fixdots (see example below for one) the memory increase still is dramatic quickly reaches 1GB.

set start_response_interval no
set duration 0
set description "Displays stimuli"
draw circle color="[myfb2col1]" fill=1 penwidth=1 r=220 show_if=always x=0 y=0 z_index=4
draw circle color="#000000" fill=1 penwidth=1 r=135 show_if=always x=0 y=0 z_index=3
draw image center=1 file="arc_[bgcolor]_10.png" scale=0.25 show_if=always x=0 y=0 z_index=1
draw textline center=1 color=white font_bold=no font_family=mono font_italic=no font_size=18 html=yes show_if=always text="..........." x=0 y=0 z_index=0
draw image center=1 file="FB_bg03.png" scale=1 show_if=always x=0 y=301 z_index=0
draw fixdot color="[accblock1_1_col]" show_if="[accblock1_1] != 99" style=default x="[accblock1_1_x]" y="[accblock1_1_y]" z_index=0
draw fixdot color="[accblock1_2_col]" show_if="[accblock1_2] != 99" style=default x="[accblock1_2_x]" y="[accblock1_2_y]" z_index=0
draw fixdot color="[accblock1_3_col]" show_if="[accblock1_3] != 99" style=default x="[accblock1_3_x]" y="[accblock1_3_y]" z_index=0
draw fixdot color="[accblock1_4_col]" show_if="[accblock1_4] != 99" style=default x="[accblock1_4_x]" y="[accblock1_4_y]" z_index=0
draw fixdot color="[accblock1_5_col]" show_if="[accblock1_5] != 99" style=default x="[accblock1_5_x]" y="[accblock1_5_y]" z_index=0
draw fixdot color="[accblock1_6_col]" show_if="[accblock1_6] != 99" style=default x="[accblock1_6_x]" y="[accblock1_6_y]" z_index=0
draw fixdot color="[accblock1_7_col]" show_if="[accblock1_7] != 99" style=default x="[accblock1_7_x]" y="[accblock1_7_y]" z_index=0
draw fixdot color="[accblock1_8_col]" show_if="[accblock1_8] != 99" style=default x="[accblock1_8_x]" y="[accblock1_8_y]" z_index=0
draw fixdot color="[accblock1_9_col]" show_if="[accblock1_9] != 99" style=default x="[accblock1_9_x]" y="[accblock1_9_y]" z_index=0
draw fixdot color="[accblock1_10_col]" show_if="[accblock1_10] != 99" style=default x="[accblock1_10_x]" y="[accblock1_10_y]" z_index=0
draw fixdot color="[accblock1_11_col]" show_if="[accblock1_11] != 99" style=default x="[accblock1_11_x]" y="[accblock1_11_y]" z_index=0
draw fixdot color="[accblock1_12_col]" show_if="[accblock1_12] != 99" style=default x="[accblock1_12_x]" y="[accblock1_12_y]" z_index=0
draw fixdot color="[accblock2_1_col]" show_if="[accblock2_1] != 99" style=default x="[accblock2_1_x]" y="[accblock2_1_y]" z_index=0
draw fixdot color="[accblock2_2_col]" show_if="[accblock2_2] != 99" style=default x="[accblock2_2_x]" y="[accblock2_2_y]" z_index=0
draw fixdot color="[accblock2_3_col]" show_if="[accblock2_3] != 99" style=default x="[accblock2_3_x]" y="[accblock2_3_y]" z_index=0
draw fixdot color="[accblock2_4_col]" show_if="[accblock2_4] != 99" style=default x="[accblock2_4_x]" y="[accblock2_4_y]" z_index=0
draw fixdot color="[accblock2_5_col]" show_if="[accblock2_5] != 99" style=default x="[accblock2_5_x]" y="[accblock2_5_y]" z_index=0
draw fixdot color="[accblock2_6_col]" show_if="[accblock2_6] != 99" style=default x="[accblock2_6_x]" y="[accblock2_6_y]" z_index=0
draw fixdot color="[accblock2_7_col]" show_if="[accblock2_7] != 99" style=default x="[accblock2_7_x]" y="[accblock2_7_y]" z_index=0
draw fixdot color="[accblock2_8_col]" show_if="[accblock2_8] != 99" style=default x="[accblock2_8_x]" y="[accblock2_8_y]" z_index=0
draw fixdot color="[accblock2_9_col]" show_if="[accblock2_9] != 99" style=default x="[accblock2_9_x]" y="[accblock2_9_y]" z_index=0
draw fixdot color="[accblock2_10_col]" show_if="[accblock2_10] != 99" style=default x="[accblock2_10_x]" y="[accblock2_10_y]" z_index=0
draw fixdot color="[accblock2_11_col]" show_if="[accblock2_11] != 99" style=default x="[accblock2_11_x]" y="[accblock2_11_y]" z_index=0
draw fixdot color="[accblock2_12_col]" show_if="[accblock2_12] != 99" style=default x="[accblock2_12_x]" y="[accblock2_12_y]" z_index=0
draw fixdot color="[accblock3_1_col]" show_if="[accblock3_1] != 99" style=default x="[accblock3_1_x]" y="[accblock3_1_y]" z_index=0
draw fixdot color="[accblock3_2_col]" show_if="[accblock3_2] != 99" style=default x="[accblock3_2_x]" y="[accblock3_2_y]" z_index=0
draw fixdot color="[accblock3_3_col]" show_if="[accblock3_3] != 99" style=default x="[accblock3_3_x]" y="[accblock3_3_y]" z_index=0
draw fixdot color="[accblock3_4_col]" show_if="[accblock3_4] != 99" style=default x="[accblock3_4_x]" y="[accblock3_4_y]" z_index=0
draw fixdot color="[accblock3_5_col]" show_if="[accblock3_5] != 99" style=default x="[accblock3_5_x]" y="[accblock3_5_y]" z_index=0
draw fixdot color="[accblock3_6_col]" show_if="[accblock3_6] != 99" style=default x="[accblock3_6_x]" y="[accblock3_6_y]" z_index=0
draw fixdot color="[accblock3_7_col]" show_if="[accblock3_7] != 99" style=default x="[accblock3_7_x]" y="[accblock3_7_y]" z_index=0
draw fixdot color="[accblock3_8_col]" show_if="[accblock3_8] != 99" style=default x="[accblock3_8_x]" y="[accblock3_8_y]" z_index=0
draw fixdot color="[accblock3_9_col]" show_if="[accblock3_9] != 99" style=default x="[accblock3_9_x]" y="[accblock3_9_y]" z_index=0
draw fixdot color="[accblock3_10_col]" show_if="[accblock3_10] != 99" style=default x="[accblock3_10_x]" y="[accblock3_10_y]" z_index=0
draw fixdot color="[accblock3_11_col]" show_if="[accblock3_11] != 99" style=default x="[accblock3_11_x]" y="[accblock3_11_y]" z_index=0
draw fixdot color="[accblock3_12_col]" show_if="[accblock3_12] != 99" style=default x="[accblock3_12_x]" y="[accblock3_12_y]" z_index=0
draw fixdot color="[accblock4_1_col]" show_if="[accblock4_1] != 99" style=default x="[accblock4_1_x]" y="[accblock4_1_y]" z_index=0
draw fixdot color="[accblock4_2_col]" show_if="[accblock4_2] != 99" style=default x="[accblock4_2_x]" y="[accblock4_2_y]" z_index=0
draw fixdot color="[accblock4_3_col]" show_if="[accblock4_3] != 99" style=default x="[accblock4_3_x]" y="[accblock4_3_y]" z_index=0
draw fixdot color="[accblock4_4_col]" show_if="[accblock4_4] != 99" style=default x="[accblock4_4_x]" y="[accblock4_4_y]" z_index=0
draw fixdot color="[accblock4_5_col]" show_if="[accblock4_5] != 99" style=default x="[accblock4_5_x]" y="[accblock4_5_y]" z_index=0
draw fixdot color="[accblock4_6_col]" show_if="[accblock4_6] != 99" style=default x="[accblock4_6_x]" y="[accblock4_6_y]" z_index=0
draw fixdot color="[accblock4_7_col]" show_if="[accblock4_7] != 99" style=default x="[accblock4_7_x]" y="[accblock4_7_y]" z_index=0
draw fixdot color="[accblock4_8_col]" show_if="[accblock4_8] != 99" style=default x="[accblock4_8_x]" y="[accblock4_8_y]" z_index=0
draw fixdot color="[accblock4_9_col]" show_if="[accblock4_9] != 99" style=default x="[accblock4_9_x]" y="[accblock4_9_y]" z_index=0
draw fixdot color="[accblock4_10_col]" show_if="[accblock4_10] != 99" style=default x="[accblock4_10_x]" y="[accblock4_10_y]" z_index=0
draw fixdot color="[accblock4_11_col]" show_if="[accblock4_11] != 99" style=default x="[accblock4_11_x]" y="[accblock4_11_y]" z_index=0
draw fixdot color="[accblock4_12_col]" show_if="[accblock4_12] != 99" style=default x="[accblock4_12_x]" y="[accblock4_12_y]" z_index=0
draw line color="[accblock1_col]" penwidth=5 show_if="[accblock1] != 99" x1="[accblock1_x1]" x2="[accblock1_x2]" y1="[accblock1_y]" y2="[accblock1_y]" z_index=0
draw line color="[accblock2_col]" penwidth=5 show_if="[accblock2] != 99" x1="[accblock2_x1]" x2="[accblock2_x2]" y1="[accblock2_y]" y2="[accblock2_y]" z_index=0
draw line color="[accblock3_col]" penwidth=5 show_if="[accblock3] != 99" x1="[accblock3_x1]" x2="[accblock3_x2]" y1="[accblock3_y]" y2="[accblock3_y]" z_index=0
draw line color="[accblock4_col]" penwidth=5 show_if="[accblock4] != 99" x1="[accblock4_x1]" x2="[accblock4_x2]" y1="[accblock4_y]" y2="[accblock4_y]" z_index=0

sebastiaan

@Fab Detached elements are, in itself, normal. They are elements that are not currently embedded in the page (the DOM tree), but that do exist in the JavaScript workspace. With OpenSesame this happens a lot, for example when a sketchpad is being prepared but not yet shown.

So to check whether there's a memory leak, you need to use a Task Manager that shows how much memory is actually being used. Values around 200 - 500 Mb seem normal for a single process of Chrome. If you see these values spiraling out of control (as happened with text), then there may be a memory leak.

If I run your flanker task, all seems fine.

@HenkvanSteenbergen You say that memory consumption goes up to about 1 Gb. Does it keep increasing, or does it stay relatively constant at that value? If the latter, then it's not a memory leak, which is what I want to focus on right now. (Although of course it's still pretty excessive consumption, and can probably be optimized both in your experiment and in OSWeb.)

HenkvanSteenbergen

@sebastiaan It keeps increasing, I stopped the experiment after ~2 minutes. The total experiment lasts about 40 minutes so I am sure it will crash on computers with low RAM

sebastiaan

@HenkvanSteenbergen I looked at the full experiment. I see memory going up to about 600 Mb (for a single Chrome process. Chrome as an app will consume more) and then staying more or less constant. I.e. no memory leak as far as I can see, and while memory consumption is pretty intense, it's also a very complex experiment with a large number of sketchpads!

This is with Chrome 97.0.4692.99 on Windows 11, using the updated OSWeb that I linked to above. Can you double-check whether you've indeed tried this with the updated version of OSWeb (1.4.4a1). If so, what system do you test this on?

Fab

Hi @sebastiaan,

Many thanks for your clarification about detached elements! I guess I ended up looking at these because I couldn't work out what was going on with my task and why a third of participants reported sound distortion. I still don't know what caused it (the trials did not involve presenting text), though getting rid of the sampler and using Javascrip code to play the sound appears to have improved things a lot (could there be an issue with the sampler when the task contains many trials?). I'll have to explore it further (I did 5 experiments, three of which generated sound distortion or failed to replicate previous results). Anyway, going back to OSWeb 1.4.4.01a...

I went back to one of the tasks I've been running online (which does not involve sound but does involve text and pictures on every trial), looked at memory usage with the Task Manager, and ran it on two computers: a desktop with OSWeb 1.4.4.01a, and my laptop running OSWeb 1.4.0.0. Both with Chrome.

Here's the memory usage with each at three points in time:

OSWeb 1.4.4.0a:

Beginning: 890 MB

Trial 500: 1302 MB

Trial 1216 (last): 933 MB

OSWeb 1.4.0.0:

Beginning: 780 MB

Trial 500: 1626 MB

Trial 1216 (last): 2151 MB

So, the change you made to OSWeb does make a difference with the task I tried!

When I have a moment, I'll try the flanker task with sound with a lot of trials, comparing the two versions of OSWeb, just to see what the memory usage looks like.

Best,

Fabrice.

HenkvanSteenbergen

@sebastiaan, my apologies. You are right! I was too quick. The memory used does not exceed 600 MB even when the task runs for a long time. Thank you so much for providing the solution so quickly!

Fab

Hi @sebastiaan,

I've done another comparison using a version of a task that generated problems for me in the past (sound distortion or task crashing for many participants). I ran it on my desktop with OSWeb 1.4.4.01a and on my laptop with OSWeb 1.4.0.0, both in Chrome, and wrote down the memory usage from the task manager at different stages of the task:

The task contains some 10 practice trials + 4 x 300 test trials of a Flanker-type task involving no text stimuli but sketchpads with pictures and a sampler:

There are also 12 trials of a different type that do include some text on sketchpads (3 trials at the end of each block of Flanker trials).

The memory usage does seem to go up to a greater extent with OSWeb 1.4.0.0 than with OSWeb 1.4.4.01a, but with both versions, sound distortion started to appear in Block 2 (trials 301-600), becoming severe and constant in both versions. Because the ISI is constant, the presentation of the sounds on the two computers should have been in phase, but I noticed clear variations, suggesting that the timing of events was not stable (though I can't tell whether variations occurred on both computers or just on one).

I don't know whether the sound distortion and variation in timing is due to memory problems, but if it is not, I'm at a loss as to what the issue might be.

Best,

Fabrice.

sebastiaan

Hi @Fab ,

Thanks! From your description, it seems that the sound distortion is related to something accumulating internally (why else would it take time to emerge?), but is not a memory leak as such. I'll see if I can reproduce it and then file a separate issue for it.

— Sebastiaan

sebastiaan

Hi @Fab ,

The sampler issue was relatively easy to fix. Unfortunately I didn't include it with OpenSesame 3.3.11 (because I thought it would take some figuring out), but you can update OSWeb separately to 1.4.5 to get this fix. For me OSWeb 1.4.5 resolves the stuttering sound, which indeed emerged after several hundred calls to the sampler. Can you let me know if this works for you too?

— Sebastiaan

Fab

Hi @sebastiaan,

Many thanks for looking into this and taking the time to modify OSWeb!

I've just done a test with the same task as last time on my PC after updating OSWeb to version 1.4.5.0, and the sound distortion is gone! 😀

The memory usage (measured from the task manager) is as follows:

Compared to the same task on the same computer with version 1.4.4.01a, the increase in memory consumption is greater with version 1.4.5.0 (356 vs 68 MB), at least based on my two tests, but, crucially, the sound played perfectly throughout the task.

Many thanks for your help with this!!

Best,

Fabrice.