jatos.service: Failed with result 'oom-kill'

SilenceThisNoise

Hello,

I have recently upgraded my JATOS installation to the latest version. However, since then I have had problems with the JATOS service becoming unavailable, and the server logs are telling me the following: jatos.service:

Failed with result 'oom-kill

I am running on a digital ocean instance of a Ubuntu LAMP server. Further to this, I can see that there are periodic CPU spikes to 100%, which could be related to the crashes.

The specs of the VPS have always been sufficient for my JATOS usage (4 years with no issues) and it is not the case that the server is being bombarded with users - it happens when no-one is running a study, in the middle of the night, or when I am logged in and in the admin section just checking on stuff. That is to say, it does not seem to be related to usage of JATOS per se.

The error itself is written in the log of the JATOS service that is started by systemmd. Sometimes the stoppages are accompanied by the following text:

 line 10: 18571 Killed         env GENERATED_SECRET="$secret" "$dir/bin/jatos" "${args[@]}" -J-server 2> >(tee -a "$dir/logs/loader.log")

Any help would be appreciated!

kri

Hi!

Apparently this 'oom-kill' stands for "out of memory" and Linux decided to kill JATOS because the machine run out of memory. This line 10: 18571 Killed... is just the line of the loader.sh where JATOS gets started.

Now the question seems to be why your 4-year old JATOS uses less memory than the latest version? This is difficult to say. JATOS uses Java and Java is known for gobbling up memory as long as its memory consumption is not limited. Did you maybe limit JATOS' (Java's) memory consumption in the old installation but not in the new one? Did you add the parameter -J-Xmx to your systemd service file?

And out of interest: how much memory does your machine have?

Best,

Kristian

SilenceThisNoise

Thanks again for your response!

I realised that I should have made clearer another difference between the new and old installations: previously I used a docker container (the standard digitalocean installation from JATOS website).

When I upgraded to the new version of JATOS, I wanted to use a standard LAMP server. This could introduce additional variables. However, I have followed all of the installation info on JATOS website, so there should be no surprises.

I will try adding this parameter to the systemd file. Does it need to be added on the line where loader.sh start is called?

The VPS is 1GB RAM with 1CPU and NVME SSD - this has been sufficient previously, so I feel like it shouldn't be the bottleneck...

Your help is much appreciated.

kri

previously I used a docker container

Maybe you limited the memory consumption with the docker container then?

When I upgraded to the new version of JATOS, I wanted to use a standard LAMP server.

I'm confused, LAMP stands for Linux, Apache, MySQL, and PHP, doesn't it? Do you want to use Apache and MySQL and PHP? For JATOS you definitely don't need the PHP part, MySQL is optional (by default JATOS uses an embedded H2 database). That leaves Apache as an reverse proxy, but this I'd install on a normal Linux distribution.

But now that you are mentioning LAMP, did you switch from H2 to MySQL by any chance? This could explain your out-of-memory issues. MySQL uses more memory than the embedded H2. Your 1GB mem is already pretty tight, but with an additional MySQL it wont be enough. I would give the machine at least 2 GB and limit JATOS memory consumption to 1GB (50%) with -J-Xmx1G to leave MySQL some memory.

EDIT:

To use the -J-Xmx parameter with systemd have a look at: https://www.jatos.org/JATOS-on-a-server.html#optional-auto-start-jatos-via-systemd. It's adding it to the call of loader.sh, e.g.

ExecStart=/my/path/to/jatos/loader.sh start -J-Xmx1G

Best,

K.

SilenceThisNoise

Hi, thanks again for the response.

When I say I switched to a LAMP server, this is a default stack for setting up the server for general web use. I don't use PHP or MySQL for JATOS, they are just available on the server if needed.

Thanks for the info re: memory usage. I'm not using SQL, but I might upgrade to 2GB anyway to be safe, and limit the JATOS memory accordingly, to stop OOM errors.

On the plus side, since I last posted, I spun up a new VPS with the same configuration and restored my JATOS installation from a backup of the server that was having problems. It has been running for 12 hours with no issues, whereas the server with the issues has crashed once in that timeframe with an OOM error. Strange.

Anyway, it is good to know more about JVM memory usage and specs needed to JATOS.

Much appreciated, as always!

kri

When I say I switched to a LAMP server, this is a default stack for setting up the server for general web use. I don't use PHP or MySQL for JATOS, they are just available on the server if needed.

But if you don't use them then the LAMP stack is not necessary, isn't it? You could at least turn off the MySQL since it uses memory and you don't need it.

I wish you good luck!

Kristian