"Cannot read result data." after long "Working"

Kian

Hi everyone,

We have about 1500 results and we can't inspect them through the JATOS administrator dashboard.

We have a rather powerful virtual machine and a good internet connection, and we didn't have this issue with a smaller amount of results.

Anyone having a suggestion on how to download the results?

kri

Hi Kian,

I've seen this before. I guess it has to do with your MySQL database setup. By default it limits the amount of data that can be loaded at once. JATOS' docs has a chapter about it:

http://www.jatos.org/JATOS-with-MySQL.html#optional-increase-max_allowed_packet-size-in-older-mysqls

Basically you (or your admin) have to adjust the max_allowed_packet to a higher value in your mysql.cnf. E.g. to allow 512MB use:

max_allowed_packet=512MB

I also try to find a solution for this problem in JATOS but this will take a couple of days.

Best,

Kristian

Kian

@kri

Thank you for the quick reply Kristian.

We did all the steps mentioned by you (and our MySQL is not older than 8.xx) but didn't work.

We ended up downloading the results directly from the database.

Now we are experiencing the same issue with 800 responses.

We will go through the documentation again to check if we missed something while setting up the server and the database, but wanted to reply here in case you have a solution for it.

Thanks again for your time.

kri

I'm sorry to hear you had to download your data directly from the database. This is annoying and JATOS is intended to make this unnecessary. May I ask how big your result dataset is expected to be altogether? I'm asking because there is an additional issue in MySQL that it by default loads all data into memory at once. If you have very large datasets then even a setting of max_allowed_packet won't help because it has a maximum of 1GB.

It's planned for JATOS to change the way it queries MySQL so it does not load all data at once into memory. But I can't say yet when this will be released.

Best,

Kristian

Kian

@kri

Hi Kristian,

For a study that we have with 871 participants, the data size we are getting from the table ComponentResult is only 522 MB (we are only using the ComponentResult because this is where our data is. We know that normally JATOS is also querying some of the other tables as well thus trying to show in the dashboard more than 522MB).

We see that JATOS has pagination implemented in mind when we try to see the result via the drop-down menu "Show [10, 25, 50, All] entries", but it doesn't look like it is fetching the selected amount from the Database, and then when you go to the next page, the next amount and so on, but it appears that JATOS is trying to SELECT everything from the Database and then trying to visualize it on the admin dashboard.

Thank you very much for the hints. We hope to see that it is coming in near future.

kri

Hi Kian,

Sorry for the late reply - some holidays came in the way.

For a study that we have with 871 participants, the data size we are getting from the table ComponentResult is only 522 MB (we are only using the ComponentResult because this is where our data is. We know that normally JATOS is also querying some of the other tables as well thus trying to show in the dashboard more than 522MB).

522MB is big for a database query, but should be possible with enough memory in the server and the receiving computer. Did you try to set max_allowed_packet to 1GB?

We see that JATOS has pagination implemented in mind when we try to see the result via the drop-down menu "Show [10, 25, 50, All] entries", but it doesn't look like it is fetching the selected amount from the Database, and then when you go to the next page, the next amount and so on, but it appears that JATOS is trying to SELECT everything from the Database and then trying to visualize it on the admin dashboard.

You are correct, the pagination setting changes only the number of results shown at once in the UI - but does not change the way JATOS queries the database. JATOS queries all result data at once which is actually a bug in Hibernate (a library used by JATOS; more about the bug). But I'm thinking about fetching the results in a different way which would circumvent the bug. This would also be paginated. Then the results are send via chunked HTTP to the client side and displayed. It's working beautifully in my dev version. But I can't yet say when this will be finished and go into a release.

Best,

Kristian

Kian

Thanks Kristian for the reply.

Yes, we also tried the max_allowed_packet to be 1GB.

Great, looking forward to having it as a release :)

Is there any chance that you could share your dev version with us?

Thanks again for all the support,

kri

Hi Kian,

The updated result pagination changes are currently mixed in together with other stuff that not yet ready for release. But as soon as I extracted the result pagination changes I'll prepare a release and let you know. But the current Corona lockdown in the country were I live makes it difficult to work for me. So it might take a couple weeks.

Best,

Kristian

kesslery

Hi,

I'm experiencing a similar problem, but with only 63 responses. Doesn't make sense that memory is an issue. Any idea?

Thanks!

Yoav

kesslery

Solved by resizing the DigitalOcean droplet.

Kian

Thank you, Kristian,

We will happily be waiting for the new release.

Thanks again for all the support.

Best regards,

Kian

NatMac16303

Hi Guys,

I have inherited looking a system with version 3.5.8 of Jatos using postgres and I have a user with this issue. I have tried increasing the shared buffers and work mem but that hasn't helped so far. Any tips on what else to try or is there possibly a configuration change required in Jatos itself?

Thanks,

Nathan

NatMac16303

Hi @kri,

I did manage to get the system working with the buffer increases and increasing the number of connections, not sure which fixed the issue exactly or if it was both. I now have the same user with the problem again, and I am wondering if you can offer any tips or advice as the users number of results is much lower than those mentioned above (around 400).

Thanks,

Nathan

NatMac16303

So by increasing the connections and buffers, plus restarting postgres and jatos, it is working again. Unfortunately I didn't think to just try restarting the services first. The dataset for this study is about 1.8GB, the same user was having an issue with other smaller datasets so I'm wondering if a restart would have fixed that.

NatMac16303

Stopped working for this user again, would allow me to load results if I restarted Jatos, but then a non-descript error when trying to export all.

kri

Hi Nathan,

First a hint, questions are more visible if you post each new question in its own forum thread. Don't add something to a question in 2024 that is from 2021.

Second,  3.5.8 is pretty old. I know it can be challenging to update a running system, but you should consider it anyways.

Third, JATOS isn't compatible with Postgres. You sure you aren't using a MySQL (or MariaDB)? Or the embedded H2 database.

Fourth, you just write "I have inherited looking a system with version 3.5.8 of Jatos using postgres and I have a user with this issue.". Can you please describe your issue again. It's also my time and going through the whole thread here and what has been written 3 years ago, is quite a drag.

Best,

Kristian