Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

90% of prolific workers "returned" for OSWEB experiment

I had my first experience with deploying a study online through prolific. The experiment is hosted on https://jatos.cogsci.nl/

I aimed for 8 participants as a pilot dataset, and set the required sample size to 8 on prolific. I used the "general multiple worker" option in Jatos, and there too I set the maximum number of links to 8.

Once the experiment was deployed many workers "started", and most got "returned". When they got back to me about this, they told me that they received a message that the experiment was already full once they clicked on the link.

I then thought this might have been due to the maximum number of 8 links set in jatos, and I removed this maximum. The problem persisted with 6 out of 10 participants still receiving the same "study is full" message.

All in all, about 80 people started my experiment through prolific, of whom 90% got the "study is full" message. Looking back at the timeline of when people started on prolific, it seemed unlikely that the study could already have been full when participants "returned" after getting the "study is full" message, see below:


Comments

  • Hi Mark,

    Where in JATOS did you set the 'maximum number of 8 links'? Is it the 'Max total workers' in 'Batch Properties'? And what was the error message your participants got? I'm asking because I want to figure out if this is a problem on JATOS' or Prolific's side. JATOS gives a different error message when the batch is 'full': 'For the batch (ID: batchId) the maximum number of workers is already reached.'.

    Best,

    Kristian

  • edited June 2020

    Thanks for looking into this!

    Correct, I had set the "max total workers" option in the batch properties to 8. Most participants just indicated that the study was "full", but one of them indeed literally pointed out the error message you wrote:

    "I clicked the link to begin and it is telling me this error message : For the batch (ID: 392) the maximum number of workers is already reached."

    The same participant was able to do the study once I removed the max number, 2 minutes later:

    "Never mind - has been resolved"

    I guess my main question is how it could be that this already started with the third participant, given that that was still well below the max number of 8.

    PS: I will collect data for another pilot study tomorrow, and will then let you know if the issue occurs again if I do not use the max number option in jatos.

  • Collected data for another pilot this morning, without setting a max in the batch options in jatos, and it went very smoothly:


  • That is good to see!

    But this leaves the questions for me why it was so weird with JATOS' Max total workers in Batch Properties. Let me try this out on my JATOS.

  • So I tried it on my JATOS and it seemed to work as expected. I set the Max total workers in Batch Properties and tried to run the study with General Multiple Worker and to be sure with Personal Single too. I could run the study as many times as I specified in the Max total workers. But I noticed an easy fallacy: The Max total workers specify the total number including the runs that already happened in this batch - not only the future ones. E.g. I had already a couple of runs in my batch and go the error message right away (which surprised me in the beginning). Maybe this was the issue with your batch too?

  • Hi,

    I recently had a similar issue, and I wanted to share my solution in case it's useful to anyone else.

    I'm running a study for which I want 120 participants, distributed evenly over 4 conditions. I used the Randomize Workers Across Tasks template to distribute workers, but I ran into a problem: this template imposes a Max total workers on the batch even if no max is specified in the batch properties. This is by virtue of its array of conditions, which shortens with every incoming worker regardless of whether they successfully complete the study or not. Prolific, of course, doesn't know this and keeps funneling new workers to JATOS, who are then disappointed and confused because they get the max number of workers reached error message.

    The solution to this is to make the array of conditions in Batch Session Data much larger, so there's no risk of running out before Prolific reaches its max number of workers. But I want an equal number of participants per condition - and if workers are randomly assigned, I might not get that. So I wrote the following code snippet to distribute workers over conditions sequentially (i.e. ABCDABCDABCD) instead. (No doubt there's a more elegant solution, but I'm a Javascript novice, so bear with me.)

    \\ after fillArrayWithValues

    const distinctValues = [...new Set(conditions)].sort();

    var valueDictionary = {};

    var maxValueCount = 0;


     for (var i = 0; i < distinctValues.length; i++) {

         const allValuesMatchingThisType = conditions.filter(value => value === distinctValues[i]);

         if (allValuesMatchingThisType.length > maxValueCount) {

           maxValueCount = allValuesMatchingThisType.length;}

         valueDictionary[distinctValues[i]] = allValuesMatchingThisType;}

    var outputArray = [];

       for (var i = 0; i < maxValueCount; i++) {

         for (var j = 0; j < distinctValues.length; j++) {

           if (valueDictionary[distinctValues[j]].length > i) {

             outputArray.push(valueDictionary[distinctValues[j]][i]);}}}

    And then adjust variable names etc. in the rest of the HTML file accordingly.

    Hope this helps someone else!

  • edited February 2021

    Hi @myrte_v

    Thanks a lot for uploading your solution which is just what I need for my study as well.

    However, I run into the following problem when I try to implement your code:

    start?srid=215:43 Uncaught ReferenceError: conditions is not defined
      at fillArrayWithValues (start?srid=215:43)
      at initBatchConditions (start?srid=215:33)
      at start?srid=215:18
      at readyForOnLoad (jatos.js:527)
      at jatos.js:313
      at c (jquery-3.5.1.min.js:2)
      at fireWith (jquery-3.5.1.min.js:2)
      at e (jquery-3.5.1.min.js:2)
      at t (jquery-3.5.1.min.js:2)
      at jquery-3.5.1.min.js:2
    

    I'm a real novice so please bear with me becuase I don't know how to define the conditions.

    I have tried to implement your code in the sortCondition.html file as you see here:

    function fillArrayWithValues() {
    		const distinctValues = [...new Set(conditions)].sort();
            	var valueDictionary = {};
            	var maxValueCount = 0;
            	for (var i = 0; i < distinctValues.length; i++) {
                 		const allValuesMatchingThisType = conditions.filter(value => value === distinctValues[i]);
                 		if (allValuesMatchingThisType.length > maxValueCount) {
           	     		maxValueCount = allValuesMatchingThisType.length;
    			}
         			valueDictionary[distinctValues[i]] = allValuesMatchingThisType;
    			}
    		var outputArray = [];
       		for (var i = 0; i < maxValueCount; i++) {
         			for (var j = 0; j < distinctValues.length; j++) {
           			if (valueDictionary[distinctValues[j]].length > i) {
             		outputArray.push(valueDictionary[distinctValues[j]][i]);
    			}}
    		}
    	}
    

    I've attached the html-file (as a .txt-file) that I've tried to rewrite as well.

    Maybe you can help me figuring out where the problem lies?


    Thanks a lot in advance!

    Best,

    Martin

  • Hi again,

    In a second thought, I think the problem has something to do with the JSON input written in JATOS.

    I don't really know what to put there with the above mentioned implementation.

    At the moment, I just have the same written as in the Randomize Workers Across Tasks template, but this gives me the error mentioned above.

    Maybe you can help figuring it out, @kri or @elisa?

    Thanks a lot in advance.

    Martin

  • Hi Martin,

    This question is getting a bit confusing and hard to follow, sorry.

    I agree with you: you modified the HTML file from the study example and included the line

    const distinctValues = [...new Set(conditions)].sort();
    

    Which requires conditions to be defined. If you haven't defined it, it will not work. I am lost as to what exactly you want to achieve with this (I think you didn't explain that in the thread). So, a quick solution:

    You could include in your JSON input the value for conditions (not sure what that value should be) like this:

    {
      "conditionCounts": {
        "A": 30,
        "B": 60,
        "C": 90
      },
      "conditions" : 4
    }
    

    Then, in your HTML, set

    var conditions = jatos.componentJsonInput.conditions
    

    If that doesn't work, creates more mess, or you get confused, please start a new thread, let us know what you want to do and we'll try to help

    Best

    Elisa

  • Hi Elisa,

    Thanks for getting back to me. I just start a new thread where I can formulate my problem better.

    Best,

    Martin

  • Ok, then maybe one of the contributed examples will help (attached). Here you can set the maximum number of participants per starting condition, by changing the Properties of the component called Sort Starting Condition. Here there are just three dummy HTML pages (A,B,C) that exchange information (the 'lucky number') but hopefully you'll just be able to replace ABC with your three components.

    If you reach the maximum number of participants/condition, you'll get an error saying that no more conditions are available.

    Note that this is based on people starting the study, not actual complete, healthy study results. It might well be that people never finished the study and there will always be some degree of manual checking and adjusting to make sure you get the expected number of complete study results that you're after.

    Let me know if this doesn't help.

    Elisa


  • Hi,

    Thanks for sharing this example, it helped me get my experiment up and running.

    I tried to overestimate the number of participants in each condition to allow for dropouts on prolific etc., however I have still managed to under-estimate it. I was wondering if its possible to somehow retroactively increase the max number. I tried updating the properties of Sort Starting Condition for the study (which i paused on prolific), but for whatever reason it is still maxed out for new participants. Is there a workaround for this or will I have to start a new prolific study?

  • Hi,

    Sorry for the delay (Christmas is to blame ;)

    Have you figured this out yet? I would have to run it and check it in more detail to be sure, but I suspect that the reason why it's not updating is that (e.g.) the function fillArrayWithValues only gets called if the Batch session is empty.

    So my question (if you haven't solved it yet) is which properties did you change in the Batch session?

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games