Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

Randomize conditions between unknown number of workers

Hi,

I have an experiment which has 5 parts set up in Jatos: one introduction, three subtest and one final remark.

I want to have the three subtests distributed evenly across participants whereas the introduction and the final remark should be in the same order for everyone.

I thought I got the solution in the thread 90% of prolific workers "returned" for OSWEB experiment — Forum (cogsci.nl) with @myrte_v's answer, but unfortunately it didn't help me all the way.

I'm gonna copy some of the things I also need from the above mentioned solution by @myrte_v :

I used the Randomize Workers Across Tasks template to distribute workers, but I ran into a problem: this template imposes a Max total workers on the batch even if no max is specified in the batch properties. This is by virtue of its array of conditions, which shortens with every incoming worker regardless of whether they successfully complete the study or not

(...)

The solution to this is to make the array of conditions in Batch Session Data much larger, so there's no risk of running out before Prolific reaches its max number of workers. But I want an equal number of participants per condition - and if workers are randomly assigned, I might not get that. So I wrote the following code snippet to distribute workers over conditions sequentially (i.e. ABCDABCDABCD) instead.

My study would rather be in the format ABCDE for one participant, ACDBE for another, ADCBE for a third and so on.

I don't use Prolific, but I want to distribute the link (General Single Worker) to as many students as possible where everyone gets one of the three different subtests.

I use Jatos Version 3.5.5 by the way.

I hope my description makes sense if not feel free to ask.

Best,

Martin

Comments

  • Hi Martin,

    Sorry for the delay, we're really busy these days.One more question, and I'll try to reply quicker. How exactly is your study, do you have 5 components in Jatos, or just 3, each of which corresponds to one of the orders you want?

  • Hi Elisa,

    Right now, the study in Jatos looks like this:

    1. Instructions - A
    2. Part 1 of a working memory test - B
    3. Part 2 of a working memory test - C
    4. Part 3 of a working memory test - D
    5. Goodbye - E

    It is the three different parts that I want to have each participant to complete in more or less random order. E.g. Participant 1 gets the study order ABCDE, participant 2 gets ACDBE, particpant 3 gets ADBCE and so on.

    Does it make sense?

  • Hi, Yes, it does make sense, but my question is how exactly it's implemented. The reason I'm asking is because the exact answer depends on how your study is structured.

    Do you have a single component (HTML script) that is split into all those parts? Or 5 different HTML files? Or maybe just 3, one for each order?

    Best

    Elisa

  • edited February 22

    Okay, now I get it. :)

    A, B, C and E are open sesame components with separate html files. 4 in total.

    D is a jsPsych element including HTML file and plugins.

    And then I actually also have the sortCondition.html file which is placed after the instructions component (A).

    Best,

    Martin

  • Hi Martin,

    I replied yesterday, but fir some reason the post didn't appear here, maybe because it had an attachment. Let me try again and sorry for the -this time unnecessary- delay).

    In your case you could try one of our contributed examples (which I just realized is linked to the wrong study on our docs, we'll fix it. In the meantime, try downloading it from here: https://www.dropbox.com/s/d6f3qclht813omg/randomize_task_order_between_workers.jzip?dl=0).

    Here you set the maximum number of participants assigned to each component order in the Batch session data. Once it reached the maximum, no more participants will be allowed to continue.

    And you set the possible orders of your conditions in the first HTML file, sortStartingCondition.html

    switch (nextCondition) {
                case "A":
                    order=[2,3,4];
                    break;
                case "B":
                    order=[3,4,2];
                    break;
                case "C":
                    order=[4,2,3];
                    break;
            }
    


    Note that the number of participants assigned to each condition is based on people who started the task, not those that actually finished it. So you might get a correct and balanced assignment of conditions without actually getting the same number of valid study results. You will always need to do some kind of manual checking of the result data and potentially allow some more people to run the study based on the quality/finished states of the result data.


    Again sorry for the delay, let me know if this doesn't solve your problem.

    Elisa

  • edited February 23

    Hi Elisa,

    No worries with the delayed feedback. I'm just happy that you are responding and helping me so much. :)

    Edit: Now, I got it to work!

    Thanks a lot for your great help! I really appreciate your responses. This is what makes this forum and the programs or server etc so great.

    I might get back to you if I catch some problems again in the future. :)

    Best,

    Martin

  • Okay. It was in the near future I caught some more problems..

    Now, the problem is for the OpenSesame-html files that I see the following on one side of the screen and the OpenSesame study on the other.

    <p>You are running task <b>A</b>.</p>
        Your order is <p id='ord'></p> 
        Your lucky number is <p id='rndnum'></p>
        <button id="continueButton" class="pure-button" disabled>Continue</button>
    

    Is it possible to have the script running in the background without the participants seeing it?

    I have implemented sortStartingCondition after the </body> element as you can see in the attached file index1 or index2 which are the html files corresponding to the OpenSesame files.

    Best,

    Martin


  • edited February 24

    Hi Martin,

    I didn't fully understand your last message (and I can't run the study easily, it would be easier if you could create a .jzip by exporting your study from JATOS), but it sounds like you got it to mostly work. Or did I misunderstand?

    The answer is yes, you can definitely get the variables filled with the correct values without displaying them on the webpage that participants see. All you have to do is comment out (or delete) any lines that have the .innerHTML property set,

    e.g.

    document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];
    
  • edited February 24

    Hi again,

    I found out that my problem is that I don't know where to place the script from the taskA, taskB or taskC in the OpenSesame html-files.

    If I place it underneath the </body> I get the situation described above.

    If I place it subsequently to the  <body onload="onPageLoad()">then I get an error saying "Uncaught TypeError: $ is not a function"

    Do you have a suggestion on how to implement it or solve the issue with the $ sign?

    Best,

    Martin

    Edit: I tried to comment out the following lines:

    //document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];

    and

    //document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2];

    But I keep getting the"Uncaught TypeError: $ is not a function".

    Unfortunately, I cannot attach my experiment as a file here. But you should be able to get it here: https://www.dropbox.com/s/oainpus2lbnen0f/eksperiment.jzip?dl=0

  • edited March 1

    Ah, I see. That's something that the OSWeb people will be better able to answer actually. Your question falls somewhere between JATOS and OSWeb, so it's a bit hard, but I am not familiar with the structure of the OSWeb HTML files. I suspect you'll have to add that code as inline JavaScript on OSWeb directly.

    I wonder if @lvanderlinden can help better and faster than I can at this point.

    Best

    Elisa

  • Yeah it is right between the two functionalities. That would be great if @lvanderlinden could help.

    Then I also hope that the addition to the OSWeb script can be easily implemented into the jsPsych part of my study.

    Thanks a lot for your help though. I really appreciate it. :)

    Best,

    Martin

  • I found out where the dog was buried (as you say in Danish), @elisa !

    It was something as simple as remembering to include the head:

    <head>
        <script src="libs/jquery-3.1.1.min.js"></script>
        <script src="jatos.js"></script>
        <link rel="stylesheet" href="css/pure-release-0.6.0/pure-min.css">
        <link rel="stylesheet" href="css/sort.css">
    </head> 
    

    in all of the three different parts of the study as well as having the specific files in the study_assets_root folder.

    That was the reason why I kept getting the error messages concerning the uncaught error of the $ sign I think. The jquery script was simply not included in my html files.

    Best,

    Martin

  • Hi again,

    I tried out the study just to be sure that everyone gets a working link and the sorting of conditions is working as intended.

    Unfortunately, it does not work properly. I will try to describe the problem underneath.

    When I get the case "C", defined in the sortStartingCondition.html, and get assigned to the order [5,3,4], I only get to do the part 5, which is the jsPsych-element. Then I get the final message which corresponds to part 6 of the complete study leaving part 3 and 4 behind.

    What could be wrong? I have a link to the complete study as I get an error while trying to upload my .zip file here: https://we.tl/t-yMQyKvQn5q

    I have tried to implement the following snippet into the on_finish: function() in jsPsych without any luck:

    on_finish: function() {
          var resultCSV = jsPsych.data.get().filter([{trial_type:'spatial-span-recall'}, {trial_type:'symmetry-judgement-task'}]).csv();
          jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
    

    I also have the snippet in the jatos.onLoad(function() part of the code.

    Maybe you can help again @elisa or someone else who has an idea on how to solve the issue?

    Best,

    Martin

  • Hi Martin,

    It will take me a few days to find the time to debug your study, but what happens if you put a breakpoint on that line in the console?

    jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
    

    and evaluate the value in

    order[jatos.studySessionData["taskIndex"]]
    

    ?

    Does it have the value you expect it to have?

  • Hi Elisa,

    First, there was no value such as "order" defined in the on.finish-part. But then I added the line:

    let order=jatos.studySessionData["order"];
    

    and called the debugger function just before the line:

    debugger;
    jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
    

    It actually does have the value 5 (with the scope [5, 3, 4]) as I expect it to have.

    Therefore, it might be at some point in the sortStartingCondition.html where it doesn't work as expected.

    Unfortunately, my knowledge of JavaScript is quite limited so I don't really know what could be formulated differently in the script.

    I'll upload my full experiment in case you find some time to debug it: https://we.tl/t-H17MT6nCei

    Thanks a lot for pointing me in the right direction.

    Best,

    Martin

  • edited March 3

    Hi Martin,

    I think I found the problem, in the call:

    jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
    

    You are giving a parameter that jatos.submitResultData does not expect (the message). Have a look at the docs: https://www.jatos.org/jatos.js-Reference.html#jatossubmitresultdata

    When I changed the code to:

    jatos.submitResultData(resultCSV, () => { jatos.startComponentByPos(2) });
    

    (I hard-coded the 2 just to be sure) it worked.

    I know that you have the same function call with text in other places of the jsPsych component, but maybe you never get there and you didn't notice the bug?

    Also, you said (and I also saw) that when you debug your code, you should be getting redirected to the component in position 5. But that is the same component that you're running. Shouldn't that value be 3?

    Hope this helps

    Elisa

  • Ah, also, I noticed that you send the message 'everything ok' on every trial to the server. This should be fine because it's just a small message but exchanging information with the server so often might lead to trouble, especially if too many people are running the study at the same time.

    Maybe you just have this for debugging/diagnostic purposes but if not, I'd recommend removing any message or data sending to the server that's not really necessary. Sending information on every trial generally leads to more problems than solutions.

    Best

  • edited March 4


  • edited March 4

    I don't know why it sent the comment three times..

  • Hi Elisa,

    Thanks a lot for getting back to me.

    It is indeed a problem when the jatos.submitResultData parameter does not expect the text input. I simply copy-pasted the code from your attached file above so I must admit that I haven't thought a lot about that part.

    As for the 'everything ok' message, it was just for troubleshooting purposes. But thanks for letting me know the impact on the server.

    With your help, I now think I might know where the trouble lies:

          $('#continueButton').click(function () {
              jatos.studySessionData["taskIndex"] += 1;
              if (jatos.studySessionData["taskIndex"] == 3) {
                  jatos.submitResultData(resultCSV,jatos.startNextComponent);
              }
              else {
                  //let order=jatos.studySessonData['order'];
                  debugger;
                  jatos.submitResultData(resultCSV, () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
                    }
                });
                $('#continueButton').prop('disabled', false);
    

    In the above code, the jQuery function .click is called without having a real button to click on...

    The problem is that I don't know how to change it to .keypress or something similar which might work.

    My knowledge of jQuery is extremely limited so I don't know how to solve it.

    Do you have any ideas?

    I link to the files once again here: https://we.tl/t-AYUt7Afoiv

    Best,

    Martin

  • edited March 4

    Hi, sorry, I don't understand the problem. For me, the modified line worked (and you did have a button to click on).

    But if you want to have a callback on a keypress anywhere on the document (and at any time) you could change your code to something like this

    $(document).on('keypress', function(e) {
            doSomething();
    });
    

    (As a hint, StackOverflow is extremely helpful for HTML/JS/CSS, writing JS is mostly copy-pasting from SO: https://www.goodreads.com/book/show/29437996-copying-and-pasting-from-stack-overflow :)

  • Uh that sounds like a most-read. I'll definitely check it out. Thanks for the recommendation.

    And yes StackOverflow is really useful, but most often I get quite overwhelmed by the (very) specific questions and elaborate answers. I find it hard to pin point my problem to get some useful tips from SO, but I guess you get better at that with experience. :)

    What if you get past the SortStartingCondition-component and the deltest3 (which has the HTML-title 'Mønstre')? Do you go on with deltest1 then?

    I just get a blank page without anything happening and I don't even get to the debugger state.

  • Hi,

    Ok, I see. The problem was not that you didn't have a button to press, but really that you weren't updating the field taskIndex within jatos.studySessionData (this is the index that iterates over your jatos.studySessionData.order vector, to tell JATOS which component it should jump to.

    you can download a modified version of your HTML file here https://www.dropbox.com/s/kasfoxtmrwsktj3/symmetry_span_task_da.html?dl=0

    which works in that it will jump to the corresponding OSWeb component after pressing Enter.

    For future reference, I moved the part where you update the taskIndex from the callback on a button press to jsPsych's on_finish:

    jatos.onLoad(function () {
        //document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];
        let order = jatos.studySessionData["order"];
        //document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2]; 
    
    
        $('#continueButton').click(function () {});
        $('#continueButton').prop('disabled', false);
      },
        jsPsych.init({
          timeline: timeline,
          use_webaudio: false,
          preload_audio: all_audio_files,
          on_finish: function () {
    
    
            jatos.studySessionData["taskIndex"] += 1;
          if (jatos.studySessionData["taskIndex"] == 3) {
            jatos.submitResultData("Task complete, exp. complete", jatos.endStudy);
          }
          else {
            //let order=jatos.studySessonData['order'];
            jatos.submitResultData("Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
          }
    
    
            var resultCSV = jsPsych.data.get().filter([{ trial_type: 'spatial-span-recall' }, { trial_type: 'symmetry-judgement-task' }]).csv();
            let order = jatos.studySessionData["order"];
            debugger;
            jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
          }
        }));
    

    Now, because I'm not familiar with OSWeb's code editing, I don't know exactly how to modify your other, OSWeb components, to do the same. Best to ask in their forum if you still need to modify that.

    Elisa

  • Hi Elisa,

    Wow! Thank you so much. It works with the jsPsych part of the study now. That is really great.

    I see that you're keeping the quotes in the jatos.submitResultData component. Are there any specific reasons for that?

    Hm. I don't know what could be done in the OSWeb parts so I might have to ask them in the forum.

    Do you know whether it is possible to manually change the order of the different parts of the experiment on the Jatos server?

    Then I could just take a look every now and then and change the order so it is kind of counterbalanced. I know it is not optimal, but it could be an easy workaround if I can't get it to work with the OSWeb parts.

    Best,

    Martin

  • You're right! I forgot to remove the quotes. (The reason is that I made a mistake: I did it rather quickly and just copy-pasted your original code from one place to another). Now that you say this, I think there are a couple of errors in the code: There are two calls to submitResultData, and the one that is actually used, as it stands, does not submit your actual result data but the text in quotes instead. All your results will say "Task complete, moving to the next". So there are some more changes to make to that code. I can try it out later this week, but maybe you check that the experiment works correctly in that the correct data really do get sent. If you find what goes wrong, it will be easier for me to change it. Sorry for the Verschlimbesserung, as they say here.

    To answer your second question: yes, it is possible to change the order of components. You just drag and drop them to a new position (you'll see a 4-pointed cursor if you hover with your mouse over the component order number on the left). However this isn't great because it makes it hard to track how many participants did which task in which order. I would recommend you make sure you can unambiguously figure this out at the point of data analysis before choosing to make manual changes.

  • Hi Martin, I fixed the HTML. Now with order [5,3,4] it will redirect to the correct OSWeb component and also store the data appropriately. Sorry for the previous messy solution.

    Download it here

    https://www.dropbox.com/s/kasfoxtmrwsktj3/symmetry_span_task_da.html?dl=0

  • Hi Elisa,

    Thanks a lot for fixing the HTML after the Verschlimbesserung. :-)

    Now, the problem is still how I get the index-HTMLs of the Open Sesame files to work with the code which corresponds to part 3 and 4 in my study.

    I have placed the following script underneath the  </head> in the two HTML files:

    <script type="text/javascript">
            jatos.onLoad(function () {
                //document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];
                let order=jatos.studySessionData["order"];
                //document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2]; 
                $('#continueButton').click(function () {
                    jatos.studySessionData["taskIndex"] += 1;
                    if (jatos.studySessionData["taskIndex"] == 3) {
                        jatos.submitResultData("Task complete, exp. complete",jatos.startNextComponent);
                    }
                    else {
                        //let order=jatos.studySessonData['order'];
                        jatos.submitResultData(() => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
                    }
                });
                $('#continueButton').prop('disabled', false);
            });
            </script>
    

    I have tried to ask the question here: Randomize parts between unknown number of workers — Forum (cogsci.nl) this Friday.

    Do you know anything about how to implement it or who could know it instead?

    Best,

    Martin

  • That’s best to ask in the OSWeb forum

    you’re welcome, I feel bad it took so long.

  • I'll try that. Hopefully somebody has the answer in there.

    That's alright. I had a lot of other stuff to do meanwhile so don't feel bad about that. :)

    Best,

    Martin

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games