Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

"Failed to Transfer Data" Error & Incomplete Data

Hi everyone,

We have a problem related to the "failed to transfer data" error. Many participants report that they completed the experiment but got this error at the end. And their data are incomplete (nearly half of the trials were saved, out of 180). So we cannot use any of these participants' data.

In fact, we had encountered the same error from time to time in our previous projects, but there was no problem with data at that time - all were full. Now I cannot figure out what makes this difference and what should I do to prevent this error.

Our current experiment includes many pictures and the experiment file is much larger. Do you think this might cause this problem? Or do you have any other comments or suggestions for this situation?

Thank you very much for your help,

Pinar

Comments

  • Hi Pinar!

    Yes, it sounds like you temporarily overloaded your JATOS server. Do you have JATOS' log around the time when this happened? And what are the specifications of your JATOS server (memory, CPU, disk space)?

    It might be a good idea to spread out your participants so they do not run all in parallel. Another possibility is to reduce the load your experiment is imposing on the system, most importantly reduce the amount of result data and how often you send it.

    Best,

    Kristian

  • Hi Kristian,

    Thank you very much! Currently I don't have the access to the information you asked, but I will write you again after I learn. I don't think many participants run the experiment in parallel right now, because we sent the links only a few people for now.

    Reducing the load seems to be a great suggestion, thanks! What would you suggest for reducing the amount of result data? I checked "log all variables (recommended)" option for all the loggers in the OpenSesame experiment. If I manually selected only the variables I need, do you think this could help or should I make some other settings?

    And, do you think making the size of the images in the file pool smaller (shrinking the sizes) could also help, or the file pool is just not related to this situation?

    Thank you so much for your support.

    Best,

    Pınar

  • Is it possible that you reduce your logger points? I'm not super familiar with OpenSesame and I only now that a logger is the point when it send the data to JATOS. I know from other OS experiments that they often have much too many loggers. It's important to find a balance between logging and server resources. Many just put loggers as often as possible to be on the safe side and always have the latest result data in JATOS. The other extreme is to log only once in the end of the experiment: this reduces the server load but if your participant doesn't finish the experiment you won't have any results. My rule of thumb is once a minute and not less then every 30 s. And if you don't need partial data than logging only once in the end is actually fine. You can see how often you send data in JATOS' log: search for submitResultData and appendResultData.

    Then about the amount of result data. Again I've seen in other OpenSesame experiments that their result data can be huge (at least huge for an online study). I've never looked at it in detail but I suspect that there are a lot of unnecessary/redundant data. So try to reduce your data to the fields that you actually need. And don't send them all with each logger: general info can be sent only once in the beginning of the study. Do you know the size of your result data for one study run?

    Another point that comes to my mind: what database do you use H2 or MySQL. If it's MySQL than I recommend to deactivate the binary log. This can eat a lot of server resources and you don't need it (in a normal setup).

    And, do you think making the size of the images in the file pool smaller (shrinking the sizes) could also help, or the file pool is just not related to this situation?

    Here I'm not sure what you mean with 'file pool'. Are those files loaded from the JATOS server? Then yes, reducing the size can help here. But if you load them from an external source, then it wouldn't change.

    Best,

    Kristian

  • Hi Kristian!

    Thank you so much! I will definetely follow all your suggestions!


    I have just learned the server specifications that you asked:

    Server specs:

    This is an AWS t3.medium instance

    2 CPU

    4 GB RAM

    Disk/storage 30 GB EBS volume

    Linux version 4.14.219-164.354.amzn2.x86_64

    JATOS version 3.5.2


    And you are right, the experiment has many loggers! So I will reduce the number of them and try to send only those information that are related to that specific part of the experiment. I will follow your other suggestions, too.

    I hope all these will work! I really appreciate your help, thank you very much again!

    Best,

    Pınar

  • I don't have a direct solution for @pinarke and it seems their problem is fixed -- but ran into a similar issue; failing to transfer and eventual incomplete data files... For me the solution was in this:

    Then about the amount of result data. Again I've seen in other OpenSesame experiments that their result data can be huge (at least huge for an online study). I've never looked at it in detail but I suspect that there are a lot of unnecessary/redundant data. So try to reduce your data to the fields that you actually need

    It could be that I simply didn't look in the right places, but this might be worth highlighting somewhere in the docs of either JATOS (or OSWeb @sebastiaan ?) -- before running into this issue I actually had no idea what ballpark 'huge for on online study' would be.

    The issue can arise pretty quickly when approaching it from a novice perspective (in this case my own): in particular when

    • An experiment deals with a long list, such as a list of all stimulus files.
    • The most straightforward way in OSweb to pass this list around is to put it in the 'vars' object
    • Loggers by default are set to 'automatically log and detect all variables

    The result is that the list of filenames is written into the logged row on each trial...

    I feel this is a scenario that's bound to reoccur , and might be worth explicitly warning users about (and keep jatos servers happy)

    Thanks!

    W

  • It could be that I simply didn't look in the right places, but this might be worth highlighting somewhere in the docs of either JATOS

    @kri and I were actually discussing this. I decided to make the compatibility check emit a warning when the 'Automatically log and detect all variables' option is enabled. Not to forbid it, but just to tell the user that this is probably not necessary.

  • Sounds like a perfect solution :)

  • Another thing that might help here: in 3.5.9 we added visual feedback for the participants if there are still data being sent in the background in the end of a component. jatos.js has a background worker called httpLoop.js: one can use jatos.submitResultData or jatos.appendResultData as often as one want and the corresponding HTTP requests are just queued in the httpLoop.js worker and send one by one. That is very handy since the study's JavaScript can just continue with the experiment and doesn't have to care or wait for the request to finish. But it can get problematic if Internet is slow, the result data large or many are sent in quick succession: the requests queue up in the worker and if the component ends they are still being sent in the background and prevent the study from going to the next component (or finish the study) - all this in the background without the participant knowing what's going on, maybe with an apparently 'frozen' screen. So from 3.5.9 we at least show a message as an overlay on the screen 'Still sending data - Please wait' with an spinning wheel.

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games