JATOS on server shut down after two hours
Dear all,
First of all, many thanks to the developers of JATOS for this wonderful tool! I have been using it for a while and it is key to my experimental workflow.
I am writing here as I have a problem I have no idea how to solve. I have JATOS installed on a server (a VPS at OVH which is entirely devoted to JATOS). When I start JATOS, it functions for two hours or so and then it shut down for no reasons. I had this in the past, but then JATOS has also functioned for several months without any issue.
Here is the application-log from yesterday:
2023-02-24 17:49:47,630 [INFO] - a.e.s.Slf4jLogger - Slf4jLogger started
2023-02-24 17:49:48,124 [INFO] - n.s.e.p.s.f.AnnotationSizeOfFilter - Using regular expression provided through VM argument net.sf.ehcache.pool.sizeof.ignore.pattern for IgnoreSizeOf annotation : ^.*cache\..*IgnoreSizeOf$
2023-02-24 17:49:48,448 [INFO] - n.s.e.p.s.JvmInformation - Detected JVM data model settings of: 64-Bit OpenJDK JVM with Compressed OOPs
2023-02-24 17:49:48,582 [INFO] - n.s.e.p.s.AgentLoader - Failed to attach to VM and load the agent: class java.io.IOException: Can not attach to current VM
2023-02-24 17:49:48,593 [INFO] - n.s.e.p.i.DefaultSizeOfEngine - using Unsafe sizeof engine
2023-02-24 17:49:48,775 [INFO] - p.a.d.DefaultDBApi - Database [default] initialized
2023-02-24 17:49:48,783 [INFO] - application - Creating Pool for datasource 'default'
2023-02-24 17:49:48,809 [INFO] - c.z.h.HikariDataSource - HikariPool-1 - Starting...
2023-02-24 17:49:48,852 [INFO] - c.z.h.HikariDataSource - HikariPool-1 - Start completed.
2023-02-24 17:49:48,952 [INFO] - p.a.d.HikariCPConnectionPool - datasource [default] bound to JNDI as DefaultDS
2023-02-24 17:49:49,510 [INFO] - o.h.j.i.u.LogHelper - HHH000204: Processing PersistenceUnitInfo [name: mysqlPersistenceUnit]
2023-02-24 17:49:49,766 [INFO] - o.h.Version - HHH000412: Hibernate Core {5.4.2.Final}
2023-02-24 17:49:50,185 [INFO] - o.h.a.c.Version - HCANN000001: Hibernate Commons Annotations {5.1.0.Final}
2023-02-24 17:49:50,551 [INFO] - o.h.d.Dialect - HHH000400: Using dialect: org.hibernate.dialect.MySQL57Dialect
2023-02-24 17:49:51,579 [INFO] - o.h.v.i.u.Version - HV000001: Hibernate Validator 6.0.16.Final
2023-02-24 17:49:53,440 [INFO] - o.h.e.t.j.p.i.JtaPlatformInitiator - HHH000490: Using JtaPlatform implementation: [org.hibernate.engine.transaction.jta.platform.internal.NoJtaPlatform]
2023-02-24 17:49:54,261 [INFO] - g.c.Common - Path to study assets directory is [masked]
2023-02-24 17:49:54,261 [INFO] - g.c.Common - Path to study logs directory is [masked]
2023-02-24 17:49:54,261 [INFO] - g.c.Common - Path to uploads directory is [masked]
2023-02-24 17:49:54,262 [WARN] - g.c.Common - User session validation is switched off. This decreases security.
2023-02-24 17:49:55,284 [INFO] - g.OnStartStop - JATOS started
2023-02-24 17:49:56,330 [INFO] - p.a.Play - Application started (Prod) (no global state)
2023-02-24 17:49:57,329 [INFO] - p.c.s.AkkaHttpServer - Listening for HTTP on [masked]
2023-02-24 20:02:02,343 [INFO] - p.c.s.AkkaHttpServer - Stopping server...
2023-02-24 20:02:02,425 [INFO] - g.OnStartStop - JATOS shutdown
2023-02-24 20:02:02,449 [INFO] - application - Shutting down connection pool.
2023-02-24 20:02:02,451 [INFO] - c.z.h.HikariDataSource - HikariPool-1 - Shutdown initiated...
2023-02-24 20:02:02,465 [INFO] - c.z.h.HikariDataSource - HikariPool-1 - Shutdown completed.
Here is my current configuration:
JATOS configuration
Version: 3.6.1
URL host: [masked]
URL basepath: /
URL host + basepath: [masked]
Local basepath: [masked]
Study assets root path: [masked]
Result data max size: 5.0 MB
Result uploads allowed: true
Result uploads path: [masked]
Result uploads max file size: 30.0 MB
Result uploads limit per study run: 50.0 MB
Study logs allowed: true
Study logs path: [masked]
Uses in-memory DB: false
User session validation: false
User session timeout: 1440
User session inactivity: 60
DB URL: jdbc:h2:./database/jatos;MODE=MYSQL;DATABASE_TO_UPPER=FALSE;IGNORECASE=TRUE;DEFAULT_LOCK_TIMEOUT=10000;SELECT_FOR_UPDATE_MVCC=FALSE
DB driver: org.h2.Driver
JPA unit: mysqlPersistenceUnit
Google OAuth allowed: false
LDAP URL:
LDAP base DN:
LDAP timeout: 5000
Max results DB query size: 10
OS info
OS name: Linux
OS version: 4.19.0-23-cloud-amd64
System load average: 0.0
Available processors: 1
JVM info
Uptime: 5m 30.546s
Name: [masked]
PID: 2379
Java name: OpenJDK 64-Bit Server VM
Java version: 11.0.18
Thread count: 26
Peak thread count: 43
Total memory: 66.1 MB
Free memory: 21.3 MB
Used memory: 44.8 MB
Max memory: 494.7 MB
Heap memory used: 42 MB
Non-heap memory used: 114 MB
I would be extremely glad for any clue on how to stop having JATOS shutting down after two hours.
Best,
Josh
Comments
Hi Josh,
I'm really sorry for the late reply. Sometimes I do not get a notification email for new posts in this forum and I only stumble over them coincidentally.
From what I see your installation looks fine but something shuts down your JATOS.
My first guess was low memory but then one usually get an OutOfMemoryException somewhere in the log. But anyway how much memory does your host have? Can you somehow monitor your host, especially the memory and disk space? It would be interesting to see if something drains them.
Best,
Kristian
Hi Kristian,
Thank you very much for your response, and of course no problems for your late reply! It's already very nice of you to help everyone out.
I am using a VPS from OVH with 2 GB Memory and 40 GB Storage. Here are the current storage infos:
Filesystem Size Used Avail Use% Mounted on
udev 967M 0 967M 0% /dev
tmpfs 195M 5.3M 190M 3% /run
/dev/sda1 40G 3.2G 35G 9% /
tmpfs 975M 0 975M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 975M 0 975M 0% /sys/fs/cgroup
Here are the memory infos without JATOS running:
total used free shared buff/cache available
Mem: 1996480 113788 1305552 5352 577140 1732224
Swap: 0 0 0
And here the memory infos with JATOS running:
total used free shared buff/cache available
Mem: 1996480 338688 1080560 5352 577232 1507320
Swap: 0 0 0
So it seems ok with my current 2GB memory setting (this VPS is only dedicated to JATOS).
Do you see a clue of something problematic in these infos?
I can upgrade the VPS to 4GB memory, but given the report above it should not have a direct effect.
Thanks again for your help with this!
Best,
Josh
Hi Josh,
No, I can't see anything problematic with your stats, disk nearly empty and memory half unused. But I have two suggestions. First upgrade to 3.7.6 (or wait a couple of days until 3.8.1 is out). The recent versions have some performance and memory improvements. My guess is still that your JATOS is running out of memory. Java has the unpleasant property to gobble up memory where ever it can (even if it could run with less). Usually Java (or better the JVM) has some default maximum that it is allowed to use but this changes from version to version and JVM implementation to implementation. So I'd recommend to set the
Xmxparameter to limit the max memory consumption to 50%, in your case 1GB. That leaves the host's system 1G. The parameter is-J-Xmx1G.Hope this helps!
Best,
Kristian
EDIT: Fix the parameter example and some spelling
Hi Kristian,
Many thanks for these tips! I have updated to JATOS 3.7.6. We'll see if it shuts down after a few hours. If it does, I'll try out with the
-J-Xmx1Goption.I'll report if it works.
Thanks again!
Best,
Josh