OSWeb-Jatos: 'Failed to transfer data'

Comments

• eduard

  March 2021

  Hi Damiano,

  I have no clue, but maybe @elisa or @kri have?

  Good luck,

  Eduard

  
• kri

  March 2021

  Hi Damiano!

  Your JATOS sever specs look good, only the 25 GB disk space might cause problems if you run studies with large result data or with many participants. But I don't think this is the case here.

  It's difficult to say what caused 30% of your study runs to "Fail" (also not really fail since the result data are still there). Can you post me the logs from your JATOS around the time when you run the study? Maybe I can see something there.

  Best,

  Kristian
• DamianoG

  March 2021

  Hi Kristian,

  thank you very much!

  Right now I am not able to retrieve the server' logs older than two days, I am not sure why (I changed the "limit" parameter but it doesn't change the number of rows I get). However, I am running more participants, so, as soon as it happens again I will post what you asked.

  Best,

  Damiano
• DamianoG

  March 2021

  Hi @kri,

  the error appeared again, I attach the log file and an image of what I think is the error description line in the log. The ID of the participant is 396.

  Error_Jatos_fail_log.txt

  
  

  

  Thanks for any insight you may provide!

  Best,

  Damiano
• kri

  March 2021

  Hi Damiano,

  I can't see anything new in the logs apart from that you have quite a few EntityStreamException.

  Would it be possible for you to share your study with me (give me the jzip)? I want to run it on my JATOS and look at the traffic to the server. My idea is that you overload something (not necessarily the JATOS server) with the many appendResultData calls.

  Best,

  Kristian
• DamianoG

  March 2021

  Hi @kri,

  sure, I attach the opensesame and the zip file of the experiment.

  O_648trials_72X9_500msfeedback.osexp

  O_648trials_72X9_500msfeedback.osexp.zip

  Thank you very much for your help!

  p.s. I am thinking about logging only a subset of all the variables. I will let you know if this will change something.

  best,

  Damiano
• kri

  March 2021

  Hi Damiano,

  I looked at your study. I completely failed at the task seeing those Ts or Ls in this short glimpse :D

  But to the problem at hand:

  • You (or OSWeb) use jatos.appendResultData to send every 2-5 s around 5 kB result data to the JATOS server. That's a bit too often - but your server is big and as you said the CPU uses only a small percentage of its capacity. So I don't think (but can't rule out) that this is the problem. I'd recommend to send result data less often. As far as I know it's called 'loggers' in OpenSesame: your experiment needs less loggers.
  • Your server's disk space with 25 GB is a bit low. I'm not sure about your setup but e.g. the database can take up a lot of space. How much space is left from your disk?
  • Which database are you using: MySQL or H2 (default)? I'm not sure about the performance of H2 when it comes to many parallel accesses. I'd try with MySQL and make sure to turn off the binary log.

  Hope this helps. Best.

  Kristian
• DamianoG

  April 2021

  Hi all,

  thank you very much @kri! Your really useful advises helped us solving issues we were facing in running experiments on Jatos!

  Apparently, settings of MySQL (binary log) were a main problem, and they were causing the server disk to run out of space.

  Unfortunately we didn't solve the "failing in transferring data" at the end of experiment, that is still there in roughly 10% of cases.

  Regarding your first comment, it seems not feasible for us to change the frequency of the logging because every few seconds we have a different answer coming in from the participants and we need to store it, otherwise it is overwritten by the subsequent answer(?).

  The real problem could be the last jatos.appendResultData, the one that is apparently transferring all the data from a local copy to the server at once, overwriting the previous data on the server. Does it make sense? And if so, is there any way for preventing OSWeb to call the last jatos.appendResultData? Maybe @sebastiaan knows more about that?

  Best,

  Damiano
• sebastiaan

  April 2021

  Hi @DamianoG ,

  OSWeb calls jatos.appendResultData() every time that a logger item is executed. This ensures that the data is logged to server even when something goes wrong before the experiment successfully completes. That's why you have the data.

  Then, at the end of the experiment, OSWeb calls jatos.submitResultData() . This function writes all the data that was collected during the experiment, including some additional (context) information, to the server at once. When successful, this overwrites the previous results that were sent with jatos.appendResultData() , and adds a 'richer' set of data. This is were things go wrong in your case.

  So that's a bit of background, but (based on a quick scan of this discussion) I think the solution in your case is very simple: disable the option to automatically detect and log all variables in the logger , and instead select only the variables that you're interested in. This massively reduce the amount of data that needs to be logged! (And will be recommended by the compatibility check with the next update to OSWeb.)

  — Sebastiaan

  
  
  Check out SigmundAI.eu for our OpenSesame AI assistant!