Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

OSWeb-Jatos: 'Failed to transfer data'

edited March 2 in OSWeb

Hi all,

I am facing an issue in collecting online data. More than 30% of participants are receiving the following message at the end of the experiment: "Transferring your data, please wait a few moments.", after a long time the message "A problem occurred: Failed to transfer data" appears. Data are displayed in Jatos as "FAIL - Failed to transfer data." but, the data are there in full, without any missing trial. This is a problem for us because the participants are not redirected to the subsequent part of the study. Furthermore, I am worried about the fact that the process could potentially lead to an incomplete dataset in some cases.

One participant received a slightly different message at the end of exp ("JATOS isn't allowed to run a study with the study result ID 327.") but his results had the same FAIL label on Jatos.

I found a similar problem issued on this page but it was closed as it was solved with OSWeb 1.3.7 (I am using 1.3.11).

I would like to understand where the problem could lie.

System information:

When participants are running the experiment the Server uses a small % of its capacity. I am using a Digital Ocean server to host Jatos - Server specs: Droplet with 2 Dedicated vCPUs / 4 TB Transfer / 16 GB Memory / 25 GB Disk / LON1 - Docker 19.03.12 on Ubuntu 18.04.

1.3.11.0 OSWeb; OpenSesame 3.3.6; Python 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 21:48:41) [MSC v.1916 64 bit (AMD64)]; datamatrix 0.11.0; qdatamatrix 0.1.29; pseudorandom 0.2.2; fileinspector 1.0.2; QNotifications 2.0.6; QOpenScienceFramework 1.3.1; opencv 4.2.0; expyriment 0.10.0+opensesame2; IPython 7.12.0; numpy 1.18.1; scipy 1.3.1; PIL/ PILLOW [version unknown]; psychopy 2020.2.8; pygame 1.9.6; pygaze 0.7.2a2; pyglet 1.5.0; PyQt 5.12.3; serial 3.4; markdown 3.2.1; yaml 5.3.

Thank you for your help,

Damiano

Comments

  • Hi Damiano,

    I have no clue, but maybe @elisa or @kri have?

    Good luck,

    Eduard

    Buy Me A Coffee

  • Hi Damiano!

    Your JATOS sever specs look good, only the 25 GB disk space might cause problems if you run studies with large result data or with many participants. But I don't think this is the case here.

    It's difficult to say what caused 30% of your study runs to "Fail" (also not really fail since the result data are still there). Can you post me the logs from your JATOS around the time when you run the study? Maybe I can see something there.

    Best,

    Kristian

    Buy Me A Coffee

  • Hi Kristian,

    thank you very much!

    Right now I am not able to retrieve the server' logs older than two days, I am not sure why (I changed the "limit" parameter but it doesn't change the number of rows I get). However, I am running more participants, so, as soon as it happens again I will post what you asked.

    Best,

    Damiano

  • Hi @kri,

    the error appeared again, I attach the log file and an image of what I think is the error description line in the log. The ID of the participant is 396.


    Thanks for any insight you may provide!

    Best,

    Damiano

  • Hi Damiano,

    I can't see anything new in the logs apart from that you have quite a few EntityStreamException.

    Would it be possible for you to share your study with me (give me the jzip)? I want to run it on my JATOS and look at the traffic to the server. My idea is that you overload something (not necessarily the JATOS server) with the many appendResultData calls.

    Best,

    Kristian

    Buy Me A Coffee

  • Hi @kri,

    sure, I attach the opensesame and the zip file of the experiment.

    Thank you very much for your help!

    p.s. I am thinking about logging only a subset of all the variables. I will let you know if this will change something.

    best,

    Damiano

  • Hi Damiano,

    I looked at your study. I completely failed at the task seeing those Ts or Ls in this short glimpse :D

    But to the problem at hand:

    • You (or OSWeb) use jatos.appendResultData to send every 2-5 s around 5 kB result data to the JATOS server. That's a bit too often - but your server is big and as you said the CPU uses only a small percentage of its capacity. So I don't think (but can't rule out) that this is the problem. I'd recommend to send result data less often. As far as I know it's called 'loggers' in OpenSesame: your experiment needs less loggers.
    • Your server's disk space with 25 GB is a bit low. I'm not sure about your setup but e.g. the database can take up a lot of space. How much space is left from your disk?
    • Which database are you using: MySQL or H2 (default)? I'm not sure about the performance of H2 when it comes to many parallel accesses. I'd try with MySQL and make sure to turn off the binary log.

    Hope this helps. Best.

    Kristian

    Buy Me A Coffee

  • Hi all,

    thank you very much @kri! Your really useful advises helped us solving issues we were facing in running experiments on Jatos!

    Apparently, settings of MySQL (binary log) were a main problem, and they were causing the server disk to run out of space.

    Unfortunately we didn't solve the "failing in transferring data" at the end of experiment, that is still there in roughly 10% of cases.

    Regarding your first comment, it seems not feasible for us to change the frequency of the logging because every few seconds we have a different answer coming in from the participants and we need to store it, otherwise it is overwritten by the subsequent answer(?).

    The real problem could be the last jatos.appendResultData, the one that is apparently transferring all the data from a local copy to the server at once, overwriting the previous data on the server. Does it make sense? And if so, is there any way for preventing OSWeb to call the last jatos.appendResultData? Maybe @sebastiaan knows more about that?

    Best,

    Damiano

  • Hi @DamianoG ,

    OSWeb calls jatos.appendResultData() every time that a logger item is executed. This ensures that the data is logged to server even when something goes wrong before the experiment successfully completes. That's why you have the data.

    Then, at the end of the experiment, OSWeb calls jatos.submitResultData() . This function writes all the data that was collected during the experiment, including some additional (context) information, to the server at once. When successful, this overwrites the previous results that were sent with jatos.appendResultData() , and adds a 'richer' set of data. This is were things go wrong in your case.

    So that's a bit of background, but (based on a quick scan of this discussion) I think the solution in your case is very simple: disable the option to automatically detect and log all variables in the logger , and instead select only the variables that you're interested in. This massively reduce the amount of data that needs to be logged! (And will be recommended by the compatibility check with the next update to OSWeb.)

    — Sebastiaan

    Buy Me A Coffee

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games