OSWeb-Jatos: 'Failed to transfer data'
I am facing an issue in collecting online data. More than 30% of participants are receiving the following message at the end of the experiment: "Transferring your data, please wait a few moments.", after a long time the message "A problem occurred: Failed to transfer data" appears. Data are displayed in Jatos as "FAIL - Failed to transfer data." but, the data are there in full, without any missing trial. This is a problem for us because the participants are not redirected to the subsequent part of the study. Furthermore, I am worried about the fact that the process could potentially lead to an incomplete dataset in some cases.
One participant received a slightly different message at the end of exp ("JATOS isn't allowed to run a study with the study result ID 327.") but his results had the same FAIL label on Jatos.
I found a similar problem issued on this page but it was closed as it was solved with OSWeb 1.3.7 (I am using 1.3.11).
I would like to understand where the problem could lie.
When participants are running the experiment the Server uses a small % of its capacity. I am using a Digital Ocean server to host Jatos - Server specs: Droplet with 2 Dedicated vCPUs / 4 TB Transfer / 16 GB Memory / 25 GB Disk / LON1 - Docker 19.03.12 on Ubuntu 18.04.
188.8.131.52 OSWeb; OpenSesame 3.3.6; Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 21:48:41) [MSC v.1916 64 bit (AMD64)]; datamatrix 0.11.0; qdatamatrix 0.1.29; pseudorandom 0.2.2; fileinspector 1.0.2; QNotifications 2.0.6; QOpenScienceFramework 1.3.1; opencv 4.2.0; expyriment 0.10.0+opensesame2; IPython 7.12.0; numpy 1.18.1; scipy 1.3.1; PIL/ PILLOW [version unknown]; psychopy 2020.2.8; pygame 1.9.6; pygaze 0.7.2a2; pyglet 1.5.0; PyQt 5.12.3; serial 3.4; markdown 3.2.1; yaml 5.3.
Thank you for your help,
I have no clue, but maybe @elisa or @kri have?
Your JATOS sever specs look good, only the 25 GB disk space might cause problems if you run studies with large result data or with many participants. But I don't think this is the case here.
It's difficult to say what caused 30% of your study runs to "Fail" (also not really fail since the result data are still there). Can you post me the logs from your JATOS around the time when you run the study? Maybe I can see something there.
thank you very much!
Right now I am not able to retrieve the server' logs older than two days, I am not sure why (I changed the "limit" parameter but it doesn't change the number of rows I get). However, I am running more participants, so, as soon as it happens again I will post what you asked.
the error appeared again, I attach the log file and an image of what I think is the error description line in the log. The ID of the participant is 396.
Thanks for any insight you may provide!
I can't see anything new in the logs apart from that you have quite a few EntityStreamException.
Would it be possible for you to share your study with me (give me the jzip)? I want to run it on my JATOS and look at the traffic to the server. My idea is that you overload something (not necessarily the JATOS server) with the many appendResultData calls.
sure, I attach the opensesame and the zip file of the experiment.
Thank you very much for your help!
p.s. I am thinking about logging only a subset of all the variables. I will let you know if this will change something.
I looked at your study. I completely failed at the task seeing those Ts or Ls in this short glimpse :D
But to the problem at hand:
Hope this helps. Best.
thank you very much @kri! Your really useful advises helped us solving issues we were facing in running experiments on Jatos!
Apparently, settings of MySQL (binary log) were a main problem, and they were causing the server disk to run out of space.
Unfortunately we didn't solve the "failing in transferring data" at the end of experiment, that is still there in roughly 10% of cases.
Regarding your first comment, it seems not feasible for us to change the frequency of the logging because every few seconds we have a different answer coming in from the participants and we need to store it, otherwise it is overwritten by the subsequent answer(?).
The real problem could be the last jatos.appendResultData, the one that is apparently transferring all the data from a local copy to the server at once, overwriting the previous data on the server. Does it make sense? And if so, is there any way for preventing OSWeb to call the last jatos.appendResultData? Maybe @sebastiaan knows more about that?
Hi @DamianoG ,
jatos.appendResultData()every time that a
loggeritem is executed. This ensures that the data is logged to server even when something goes wrong before the experiment successfully completes. That's why you have the data.
Then, at the end of the experiment, OSWeb calls
jatos.submitResultData(). This function writes all the data that was collected during the experiment, including some additional (context) information, to the server at once. When successful, this overwrites the previous results that were sent with
jatos.appendResultData(), and adds a 'richer' set of data. This is were things go wrong in your case.
So that's a bit of background, but (based on a quick scan of this discussion) I think the solution in your case is very simple: disable the option to automatically detect and log all variables in the
logger, and instead select only the variables that you're interested in. This massively reduce the amount of data that needs to be logged! (And will be recommended by the compatibility check with the next update to OSWeb.)