Randomize conditions between unknown number of workers

marbh · February 2021

Hi,

I have an experiment which has 5 parts set up in Jatos: one introduction, three subtest and one final remark.

I want to have the three subtests distributed evenly across participants whereas the introduction and the final remark should be in the same order for everyone.

I thought I got the solution in the thread 90% of prolific workers "returned" for OSWEB experiment — Forum (cogsci.nl) with @myrte_v's answer, but unfortunately it didn't help me all the way.

I'm gonna copy some of the things I also need from the above mentioned solution by @myrte_v :

I used the Randomize Workers Across Tasks template to distribute workers, but I ran into a problem: this template imposes a Max total workers on the batch even if no max is specified in the batch properties. This is by virtue of its array of conditions, which shortens with every incoming worker regardless of whether they successfully complete the study or not

(...)

The solution to this is to make the array of conditions in Batch Session Data much larger, so there's no risk of running out before Prolific reaches its max number of workers. But I want an equal number of participants per condition - and if workers are randomly assigned, I might not get that. So I wrote the following code snippet to distribute workers over conditions sequentially (i.e. ABCDABCDABCD) instead.

My study would rather be in the format ABCDE for one participant, ACDBE for another, ADCBE for a third and so on.

I don't use Prolific, but I want to distribute the link (General Single Worker) to as many students as possible where everyone gets one of the three different subtests.

I use Jatos Version 3.5.5 by the way.

I hope my description makes sense if not feel free to ask.

Best,

Martin

elisa · February 2021

Hi Martin,

Sorry for the delay, we're really busy these days.One more question, and I'll try to reply quicker. How exactly is your study, do you have 5 components in Jatos, or just 3, each of which corresponds to one of the orders you want?

marbh · February 2021

Hi Elisa,

Right now, the study in Jatos looks like this:

Instructions - A
Part 1 of a working memory test - B
Part 2 of a working memory test - C
Part 3 of a working memory test - D
Goodbye - E

It is the three different parts that I want to have each participant to complete in more or less random order. E.g. Participant 1 gets the study order ABCDE, participant 2 gets ACDBE, particpant 3 gets ADBCE and so on.

Does it make sense?

elisa · February 2021

Hi, Yes, it does make sense, but my question is how exactly it's implemented. The reason I'm asking is because the exact answer depends on how your study is structured.

Do you have a single component (HTML script) that is split into all those parts? Or 5 different HTML files? Or maybe just 3, one for each order?

Best

Elisa

marbh · February 2021

Okay, now I get it. :)

A, B, C and E are open sesame components with separate html files. 4 in total.

D is a jsPsych element including HTML file and plugins.

And then I actually also have the sortCondition.html file which is placed after the instructions component (A).

Best,

Martin

elisa · February 2021

Hi Martin,

I replied yesterday, but fir some reason the post didn't appear here, maybe because it had an attachment. Let me try again and sorry for the -this time unnecessary- delay).

In your case you could try one of our contributed examples (which I just realized is linked to the wrong study on our docs, we'll fix it. In the meantime, try downloading it from here: https://www.dropbox.com/s/d6f3qclht813omg/randomize_task_order_between_workers.jzip?dl=0).

Here you set the maximum number of participants assigned to each component order in the Batch session data. Once it reached the maximum, no more participants will be allowed to continue.

And you set the possible orders of your conditions in the first HTML file, sortStartingCondition.html

switch (nextCondition) {
            case "A":
                order=[2,3,4];
                break;
            case "B":
                order=[3,4,2];
                break;
            case "C":
                order=[4,2,3];
                break;
        }

Note that the number of participants assigned to each condition is based on people who started the task, not those that actually finished it. So you might get a correct and balanced assignment of conditions without actually getting the same number of valid study results. You will always need to do some kind of manual checking of the result data and potentially allow some more people to run the study based on the quality/finished states of the result data.

Again sorry for the delay, let me know if this doesn't solve your problem.

Elisa

marbh · February 2021

Hi Elisa,

No worries with the delayed feedback. I'm just happy that you are responding and helping me so much. :)

Edit: Now, I got it to work!

Thanks a lot for your great help! I really appreciate your responses. This is what makes this forum and the programs or server etc so great.

I might get back to you if I catch some problems again in the future. :)

Best,

Martin

marbh · February 2021

Okay. It was in the near future I caught some more problems..

Now, the problem is for the OpenSesame-html files that I see the following on one side of the screen and the OpenSesame study on the other.

<p>You are running task <b>A</b>.</p>
    Your order is <p id='ord'></p> 
    Your lucky number is <p id='rndnum'></p>
    <button id="continueButton" class="pure-button" disabled>Continue</button>

Is it possible to have the script running in the background without the participants seeing it?

I have implemented sortStartingCondition after the </body> element as you can see in the attached file index1 or index2 which are the html files corresponding to the OpenSesame files.

Best,

Martin

Working memory study.zip

elisa · February 2021

Hi Martin,

I didn't fully understand your last message (and I can't run the study easily, it would be easier if you could create a .jzip by exporting your study from JATOS), but it sounds like you got it to mostly work. Or did I misunderstand?

The answer is yes, you can definitely get the variables filled with the correct values without displaying them on the webpage that participants see. All you have to do is comment out (or delete) any lines that have the .innerHTML property set,

e.g.

document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];

marbh · February 2021

Hi again,

I found out that my problem is that I don't know where to place the script from the taskA, taskB or taskC in the OpenSesame html-files.

If I place it underneath the </body> I get the situation described above.

If I place it subsequently to the <body onload="onPageLoad()">then I get an error saying "Uncaught TypeError: $ is not a function"

Do you have a suggestion on how to implement it or solve the issue with the $ sign?

Best,

Martin

Edit: I tried to comment out the following lines:

//document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];

and

//document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2];

But I keep getting the"Uncaught TypeError: $ is not a function".

Unfortunately, I cannot attach my experiment as a file here. But you should be able to get it here: https://www.dropbox.com/s/oainpus2lbnen0f/eksperiment.jzip?dl=0

elisa · February 2021

Ah, I see. That's something that the OSWeb people will be better able to answer actually. Your question falls somewhere between JATOS and OSWeb, so it's a bit hard, but I am not familiar with the structure of the OSWeb HTML files. I suspect you'll have to add that code as inline JavaScript on OSWeb directly.

I wonder if @lvanderlinden can help better and faster than I can at this point.

Best

Elisa

marbh · February 2021

Yeah it is right between the two functionalities. That would be great if @lvanderlinden could help.

Then I also hope that the addition to the OSWeb script can be easily implemented into the jsPsych part of my study.

Thanks a lot for your help though. I really appreciate it. :)

Best,

Martin

marbh · March 2021

I found out where the dog was buried (as you say in Danish), @elisa !

It was something as simple as remembering to include the head:

<head>
    <script src="libs/jquery-3.1.1.min.js"></script>
    <script src="jatos.js"></script>
    <link rel="stylesheet" href="css/pure-release-0.6.0/pure-min.css">
    <link rel="stylesheet" href="css/sort.css">
</head>

in all of the three different parts of the study as well as having the specific files in the study_assets_root folder.

That was the reason why I kept getting the error messages concerning the uncaught error of the $ sign I think. The jquery script was simply not included in my html files.

Best,

Martin

marbh · March 2021

Hi again,

I tried out the study just to be sure that everyone gets a working link and the sorting of conditions is working as intended.

Unfortunately, it does not work properly. I will try to describe the problem underneath.

When I get the case "C", defined in the sortStartingCondition.html, and get assigned to the order [5,3,4], I only get to do the part 5, which is the jsPsych-element. Then I get the final message which corresponds to part 6 of the complete study leaving part 3 and 4 behind.

What could be wrong? I have a link to the complete study as I get an error while trying to upload my .zip file here: https://we.tl/t-yMQyKvQn5q

I have tried to implement the following snippet into the on_finish: function() in jsPsych without any luck:

on_finish: function() {
      var resultCSV = jsPsych.data.get().filter([{trial_type:'spatial-span-recall'}, {trial_type:'symmetry-judgement-task'}]).csv();
      jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });

I also have the snippet in the jatos.onLoad(function() part of the code.

Maybe you can help again @elisa or someone else who has an idea on how to solve the issue?

Best,

Martin

elisa · March 2021

Hi Martin,

It will take me a few days to find the time to debug your study, but what happens if you put a breakpoint on that line in the console?

jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });

and evaluate the value in

order[jatos.studySessionData["taskIndex"]]

?

Does it have the value you expect it to have?

marbh · March 2021

Hi Elisa,

First, there was no value such as "order" defined in the on.finish-part. But then I added the line:

let order=jatos.studySessionData["order"];

and called the debugger function just before the line:

debugger;
jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });

It actually does have the value 5 (with the scope [5, 3, 4]) as I expect it to have.

Therefore, it might be at some point in the sortStartingCondition.html where it doesn't work as expected.

Unfortunately, my knowledge of JavaScript is quite limited so I don't really know what could be formulated differently in the script.

I'll upload my full experiment in case you find some time to debug it: https://we.tl/t-H17MT6nCei

Thanks a lot for pointing me in the right direction.

Best,

Martin

elisa · March 2021

Hi Martin,

I think I found the problem, in the call:

jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });

You are giving a parameter that jatos.submitResultData does not expect (the message). Have a look at the docs: https://www.jatos.org/jatos.js-Reference.html#jatossubmitresultdata

When I changed the code to:

jatos.submitResultData(resultCSV, () => { jatos.startComponentByPos(2) });

(I hard-coded the 2 just to be sure) it worked.

I know that you have the same function call with text in other places of the jsPsych component, but maybe you never get there and you didn't notice the bug?

Also, you said (and I also saw) that when you debug your code, you should be getting redirected to the component in position 5. But that is the same component that you're running. Shouldn't that value be 3?

Hope this helps

Elisa

elisa · March 2021

Ah, also, I noticed that you send the message 'everything ok' on every trial to the server. This should be fine because it's just a small message but exchanging information with the server so often might lead to trouble, especially if too many people are running the study at the same time.

Maybe you just have this for debugging/diagnostic purposes but if not, I'd recommend removing any message or data sending to the server that's not really necessary. Sending information on every trial generally leads to more problems than solutions.

Best

marbh · March 2021

marbh · March 2021

I don't know why it sent the comment three times..

marbh · March 2021

Hi Elisa,

Thanks a lot for getting back to me.

It is indeed a problem when the jatos.submitResultData parameter does not expect the text input. I simply copy-pasted the code from your attached file above so I must admit that I haven't thought a lot about that part.

As for the 'everything ok' message, it was just for troubleshooting purposes. But thanks for letting me know the impact on the server.

With your help, I now think I might know where the trouble lies:

      $('#continueButton').click(function () {
          jatos.studySessionData["taskIndex"] += 1;
          if (jatos.studySessionData["taskIndex"] == 3) {
              jatos.submitResultData(resultCSV,jatos.startNextComponent);
          }
          else {
              //let order=jatos.studySessonData['order'];
              debugger;
              jatos.submitResultData(resultCSV, () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
                }
            });
            $('#continueButton').prop('disabled', false);

In the above code, the jQuery function .click is called without having a real button to click on...

The problem is that I don't know how to change it to .keypress or something similar which might work.

My knowledge of jQuery is extremely limited so I don't know how to solve it.

Do you have any ideas?

I link to the files once again here: https://we.tl/t-AYUt7Afoiv

Best,

Martin

elisa · March 2021

Hi, sorry, I don't understand the problem. For me, the modified line worked (and you did have a button to click on).

But if you want to have a callback on a keypress anywhere on the document (and at any time) you could change your code to something like this

$(document).on('keypress', function(e) {
        doSomething();
});

(As a hint, StackOverflow is extremely helpful for HTML/JS/CSS, writing JS is mostly copy-pasting from SO: https://www.goodreads.com/book/show/29437996-copying-and-pasting-from-stack-overflow :)

marbh · March 2021

Uh that sounds like a most-read. I'll definitely check it out. Thanks for the recommendation.

And yes StackOverflow is really useful, but most often I get quite overwhelmed by the (very) specific questions and elaborate answers. I find it hard to pin point my problem to get some useful tips from SO, but I guess you get better at that with experience. :)

What if you get past the SortStartingCondition-component and the deltest3 (which has the HTML-title 'Mønstre')? Do you go on with deltest1 then?

I just get a blank page without anything happening and I don't even get to the debugger state.

elisa · March 2021

Hi,

Ok, I see. The problem was not that you didn't have a button to press, but really that you weren't updating the field taskIndex within jatos.studySessionData (this is the index that iterates over your jatos.studySessionData.order vector, to tell JATOS which component it should jump to.

you can download a modified version of your HTML file here https://www.dropbox.com/s/kasfoxtmrwsktj3/symmetry_span_task_da.html?dl=0

which works in that it will jump to the corresponding OSWeb component after pressing Enter.

For future reference, I moved the part where you update the taskIndex from the callback on a button press to jsPsych's on_finish:

jatos.onLoad(function () {
    //document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];
    let order = jatos.studySessionData["order"];
    //document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2]; 


    $('#continueButton').click(function () {});
    $('#continueButton').prop('disabled', false);
  },
    jsPsych.init({
      timeline: timeline,
      use_webaudio: false,
      preload_audio: all_audio_files,
      on_finish: function () {


        jatos.studySessionData["taskIndex"] += 1;
      if (jatos.studySessionData["taskIndex"] == 3) {
        jatos.submitResultData("Task complete, exp. complete", jatos.endStudy);
      }
      else {
        //let order=jatos.studySessonData['order'];
        jatos.submitResultData("Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
      }


        var resultCSV = jsPsych.data.get().filter([{ trial_type: 'spatial-span-recall' }, { trial_type: 'symmetry-judgement-task' }]).csv();
        let order = jatos.studySessionData["order"];
        debugger;
        jatos.submitResultData(resultCSV, "Task complete, moving to the next", () => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
      }
    }));

Now, because I'm not familiar with OSWeb's code editing, I don't know exactly how to modify your other, OSWeb components, to do the same. Best to ask in their forum if you still need to modify that.

Elisa

marbh · March 2021

Hi Elisa,

Wow! Thank you so much. It works with the jsPsych part of the study now. That is really great.

I see that you're keeping the quotes in the jatos.submitResultData component. Are there any specific reasons for that?

Hm. I don't know what could be done in the OSWeb parts so I might have to ask them in the forum.

Do you know whether it is possible to manually change the order of the different parts of the experiment on the Jatos server?

Then I could just take a look every now and then and change the order so it is kind of counterbalanced. I know it is not optimal, but it could be an easy workaround if I can't get it to work with the OSWeb parts.

Best,

Martin

elisa · March 2021

You're right! I forgot to remove the quotes. (The reason is that I made a mistake: I did it rather quickly and just copy-pasted your original code from one place to another). Now that you say this, I think there are a couple of errors in the code: There are two calls to submitResultData, and the one that is actually used, as it stands, does not submit your actual result data but the text in quotes instead. All your results will say "Task complete, moving to the next". So there are some more changes to make to that code. I can try it out later this week, but maybe you check that the experiment works correctly in that the correct data really do get sent. If you find what goes wrong, it will be easier for me to change it. Sorry for the Verschlimbesserung, as they say here.

To answer your second question: yes, it is possible to change the order of components. You just drag and drop them to a new position (you'll see a 4-pointed cursor if you hover with your mouse over the component order number on the left). However this isn't great because it makes it hard to track how many participants did which task in which order. I would recommend you make sure you can unambiguously figure this out at the point of data analysis before choosing to make manual changes.

elisa · March 2021

Hi Martin, I fixed the HTML. Now with order [5,3,4] it will redirect to the correct OSWeb component and also store the data appropriately. Sorry for the previous messy solution.

Download it here

https://www.dropbox.com/s/kasfoxtmrwsktj3/symmetry_span_task_da.html?dl=0

marbh · March 2021

Hi Elisa,

Thanks a lot for fixing the HTML after the Verschlimbesserung. :-)

Now, the problem is still how I get the index-HTMLs of the Open Sesame files to work with the code which corresponds to part 3 and 4 in my study.

I have placed the following script underneath the </head> in the two HTML files:

<script type="text/javascript">
        jatos.onLoad(function () {
            //document.getElementById('rndnum').innerHTML = jatos.studySessionData['luckyNumber'];
            let order=jatos.studySessionData["order"];
            //document.getElementById('ord').innerHTML = order[0] + "," + order[1] + "," + order[2]; 
            $('#continueButton').click(function () {
                jatos.studySessionData["taskIndex"] += 1;
                if (jatos.studySessionData["taskIndex"] == 3) {
                    jatos.submitResultData("Task complete, exp. complete",jatos.startNextComponent);
                }
                else {
                    //let order=jatos.studySessonData['order'];
                    jatos.submitResultData(() => { jatos.startComponentByPos(order[jatos.studySessionData["taskIndex"]]) });
                }
            });
            $('#continueButton').prop('disabled', false);
        });
        </script>

I have tried to ask the question here: Randomize parts between unknown number of workers — Forum (cogsci.nl) this Friday.

Do you know anything about how to implement it or who could know it instead?

Best,

Martin

elisa · March 2021

That’s best to ask in the OSWeb forum

you’re welcome, I feel bad it took so long.

marbh · March 2021

I'll try that. Hopefully somebody has the answer in there.

That's alright. I had a lot of other stuff to do meanwhile so don't feel bad about that. :)

Best,

Martin

Howdy, Stranger!

Categories

Randomize conditions between unknown number of workers

Comments

Howdy, Stranger!

Quick Links

Categories

Randomize conditions between unknown number of workers

Comments