90% of prolific workers "returned" for OSWEB experiment

Comments

• kri

  June 2020

  Hi Mark,

  Where in JATOS did you set the 'maximum number of 8 links'? Is it the 'Max total workers' in 'Batch Properties'? And what was the error message your participants got? I'm asking because I want to figure out if this is a problem on JATOS' or Prolific's side. JATOS gives a different error message when the batch is 'full': 'For the batch (ID: batchId) the maximum number of workers is already reached.'.

  Best,

  Kristian
• MarkNieuwenstein

  June 2020 edited June 2020

  Thanks for looking into this!

  Correct, I had set the "max total workers" option in the batch properties to 8. Most participants just indicated that the study was "full", but one of them indeed literally pointed out the error message you wrote:

  "I clicked the link to begin and it is telling me this error message : For the batch (ID: 392) the maximum number of workers is already reached."

  The same participant was able to do the study once I removed the max number, 2 minutes later:

  "Never mind - has been resolved"

  I guess my main question is how it could be that this already started with the third participant, given that that was still well below the max number of 8.

  PS: I will collect data for another pilot study tomorrow, and will then let you know if the issue occurs again if I do not use the max number option in jatos.
• MarkNieuwenstein

  June 2020

  Collected data for another pilot this morning, without setting a max in the batch options in jatos, and it went very smoothly:

  

  
  
• kri

  June 2020

  That is good to see!

  But this leaves the questions for me why it was so weird with JATOS' Max total workers in Batch Properties. Let me try this out on my JATOS.
• kri

  June 2020

  So I tried it on my JATOS and it seemed to work as expected. I set the Max total workers in Batch Properties and tried to run the study with General Multiple Worker and to be sure with Personal Single too. I could run the study as many times as I specified in the Max total workers. But I noticed an easy fallacy: The Max total workers specify the total number including the runs that already happened in this batch - not only the future ones. E.g. I had already a couple of runs in my batch and go the error message right away (which surprised me in the beginning). Maybe this was the issue with your batch too?
• myrte_v

  June 2020

  Hi,

  I recently had a similar issue, and I wanted to share my solution in case it's useful to anyone else.

  I'm running a study for which I want 120 participants, distributed evenly over 4 conditions. I used the Randomize Workers Across Tasks template to distribute workers, but I ran into a problem: this template imposes a Max total workers on the batch even if no max is specified in the batch properties. This is by virtue of its array of conditions, which shortens with every incoming worker regardless of whether they successfully complete the study or not. Prolific, of course, doesn't know this and keeps funneling new workers to JATOS, who are then disappointed and confused because they get the max number of workers reached error message.

  The solution to this is to make the array of conditions in Batch Session Data much larger, so there's no risk of running out before Prolific reaches its max number of workers. But I want an equal number of participants per condition - and if workers are randomly assigned, I might not get that. So I wrote the following code snippet to distribute workers over conditions sequentially (i.e. ABCDABCDABCD) instead. (No doubt there's a more elegant solution, but I'm a Javascript novice, so bear with me.)

  \\ after fillArrayWithValues

  const distinctValues = [...new Set(conditions)].sort();

  var valueDictionary = {};

  var maxValueCount = 0;

  
  

   for (var i = 0; i < distinctValues.length; i++) {

       const allValuesMatchingThisType = conditions.filter(value => value === distinctValues[i]);

       if (allValuesMatchingThisType.length > maxValueCount) {

         maxValueCount = allValuesMatchingThisType.length;}

       valueDictionary[distinctValues[i]] = allValuesMatchingThisType;}

  var outputArray = [];

     for (var i = 0; i < maxValueCount; i++) {

       for (var j = 0; j < distinctValues.length; j++) {

         if (valueDictionary[distinctValues[j]].length > i) {

           outputArray.push(valueDictionary[distinctValues[j]][i]);}}}

  And then adjust variable names etc. in the rest of the HTML file accordingly.

  Hope this helps someone else!
• marbh

  February 2021 edited February 2021

  Hi @myrte_v

  Thanks a lot for uploading your solution which is just what I need for my study as well.

  However, I run into the following problem when I try to implement your code:

  start?srid=215:43 Uncaught ReferenceError: conditions is not defined
    at fillArrayWithValues (start?srid=215:43)
    at initBatchConditions (start?srid=215:33)
    at start?srid=215:18
    at readyForOnLoad (jatos.js:527)
    at jatos.js:313
    at c (jquery-3.5.1.min.js:2)
    at fireWith (jquery-3.5.1.min.js:2)
    at e (jquery-3.5.1.min.js:2)
    at t (jquery-3.5.1.min.js:2)
    at jquery-3.5.1.min.js:2
  

  I'm a real novice so please bear with me becuase I don't know how to define the conditions.

  I have tried to implement your code in the sortCondition.html file as you see here:

  function fillArrayWithValues() {
  		const distinctValues = [...new Set(conditions)].sort();
          	var valueDictionary = {};
          	var maxValueCount = 0;
          	for (var i = 0; i < distinctValues.length; i++) {
               		const allValuesMatchingThisType = conditions.filter(value => value === distinctValues[i]);
               		if (allValuesMatchingThisType.length > maxValueCount) {
         	     		maxValueCount = allValuesMatchingThisType.length;
  			}
       			valueDictionary[distinctValues[i]] = allValuesMatchingThisType;
  			}
  		var outputArray = [];
     		for (var i = 0; i < maxValueCount; i++) {
       			for (var j = 0; j < distinctValues.length; j++) {
         			if (valueDictionary[distinctValues[j]].length > i) {
           		outputArray.push(valueDictionary[distinctValues[j]][i]);
  			}}
  		}
  	}
  

  I've attached the html-file (as a .txt-file) that I've tried to rewrite as well.

  Maybe you can help me figuring out where the problem lies?

  sortConditionRW.txt

  
  

  Thanks a lot in advance!

  Best,

  Martin
• marbh

  February 2021

  Hi again,

  In a second thought, I think the problem has something to do with the JSON input written in JATOS.

  I don't really know what to put there with the above mentioned implementation.

  At the moment, I just have the same written as in the Randomize Workers Across Tasks template, but this gives me the error mentioned above.

  Maybe you can help figuring it out, @kri or @elisa?

  Thanks a lot in advance.

  Martin
• elisa

  February 2021

  Hi Martin,

  This question is getting a bit confusing and hard to follow, sorry.

  I agree with you: you modified the HTML file from the study example and included the line

  const distinctValues = [...new Set(conditions)].sort();
  

  Which requires conditions to be defined. If you haven't defined it, it will not work. I am lost as to what exactly you want to achieve with this (I think you didn't explain that in the thread). So, a quick solution:

  You could include in your JSON input the value for conditions (not sure what that value should be) like this:

  {
    "conditionCounts": {
      "A": 30,
      "B": 60,
      "C": 90
    },
    "conditions" : 4
  }
  

  Then, in your HTML, set

  var conditions = jatos.componentJsonInput.conditions
  

  If that doesn't work, creates more mess, or you get confused, please start a new thread, let us know what you want to do and we'll try to help

  Best

  Elisa
• marbh

  February 2021

  Hi Elisa,

  Thanks for getting back to me. I just start a new thread where I can formulate my problem better.

  Best,

  Martin
• elisa

  February 2021

  Ok, then maybe one of the contributed examples will help (attached). Here you can set the maximum number of participants per starting condition, by changing the Properties of the component called Sort Starting Condition. Here there are just three dummy HTML pages (A,B,C) that exchange information (the 'lucky number') but hopefully you'll just be able to replace ABC with your three components.

  If you reach the maximum number of participants/condition, you'll get an error saying that no more conditions are available.

  Note that this is based on people starting the study, not actual complete, healthy study results. It might well be that people never finished the study and there will always be some degree of manual checking and adjusting to make sure you get the expected number of complete study results that you're after.

  Let me know if this doesn't help.

  Elisa

  randomize_task_order_between_workers.zip

  
  
• cormac

  December 2023

  Hi,

  Thanks for sharing this example, it helped me get my experiment up and running.

  I tried to overestimate the number of participants in each condition to allow for dropouts on prolific etc., however I have still managed to under-estimate it. I was wondering if its possible to somehow retroactively increase the max number. I tried updating the properties of Sort Starting Condition for the study (which i paused on prolific), but for whatever reason it is still maxed out for new participants. Is there a workaround for this or will I have to start a new prolific study?
• elisa

  December 2023

  Hi,

  Sorry for the delay (Christmas is to blame ;)

  Have you figured this out yet? I would have to run it and check it in more detail to be sure, but I suspect that the reason why it's not updating is that (e.g.) the function fillArrayWithValues only gets called if the Batch session is empty.

  So my question (if you haven't solved it yet) is which properties did you change in the Batch session?