Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Supported by

90% of prolific workers "returned" for OSWEB experiment

I had my first experience with deploying a study online through prolific. The experiment is hosted on https://jatos.cogsci.nl/

I aimed for 8 participants as a pilot dataset, and set the required sample size to 8 on prolific. I used the "general multiple worker" option in Jatos, and there too I set the maximum number of links to 8.

Once the experiment was deployed many workers "started", and most got "returned". When they got back to me about this, they told me that they received a message that the experiment was already full once they clicked on the link.

I then thought this might have been due to the maximum number of 8 links set in jatos, and I removed this maximum. The problem persisted with 6 out of 10 participants still receiving the same "study is full" message.

All in all, about 80 people started my experiment through prolific, of whom 90% got the "study is full" message. Looking back at the timeline of when people started on prolific, it seemed unlikely that the study could already have been full when participants "returned" after getting the "study is full" message, see below:


Comments

  • Hi Mark,

    Where in JATOS did you set the 'maximum number of 8 links'? Is it the 'Max total workers' in 'Batch Properties'? And what was the error message your participants got? I'm asking because I want to figure out if this is a problem on JATOS' or Prolific's side. JATOS gives a different error message when the batch is 'full': 'For the batch (ID: batchId) the maximum number of workers is already reached.'.

    Best,

    Kristian

    Buy Me A Coffee

  • edited June 15

    Thanks for looking into this!

    Correct, I had set the "max total workers" option in the batch properties to 8. Most participants just indicated that the study was "full", but one of them indeed literally pointed out the error message you wrote:

    "I clicked the link to begin and it is telling me this error message : For the batch (ID: 392) the maximum number of workers is already reached."

    The same participant was able to do the study once I removed the max number, 2 minutes later:

    "Never mind - has been resolved"

    I guess my main question is how it could be that this already started with the third participant, given that that was still well below the max number of 8.

    PS: I will collect data for another pilot study tomorrow, and will then let you know if the issue occurs again if I do not use the max number option in jatos.

  • Collected data for another pilot this morning, without setting a max in the batch options in jatos, and it went very smoothly:


  • That is good to see!

    But this leaves the questions for me why it was so weird with JATOS' Max total workers in Batch Properties. Let me try this out on my JATOS.

    Buy Me A Coffee

  • So I tried it on my JATOS and it seemed to work as expected. I set the Max total workers in Batch Properties and tried to run the study with General Multiple Worker and to be sure with Personal Single too. I could run the study as many times as I specified in the Max total workers. But I noticed an easy fallacy: The Max total workers specify the total number including the runs that already happened in this batch - not only the future ones. E.g. I had already a couple of runs in my batch and go the error message right away (which surprised me in the beginning). Maybe this was the issue with your batch too?

    Buy Me A Coffee

  • Hi,

    I recently had a similar issue, and I wanted to share my solution in case it's useful to anyone else.

    I'm running a study for which I want 120 participants, distributed evenly over 4 conditions. I used the Randomize Workers Across Tasks template to distribute workers, but I ran into a problem: this template imposes a Max total workers on the batch even if no max is specified in the batch properties. This is by virtue of its array of conditions, which shortens with every incoming worker regardless of whether they successfully complete the study or not. Prolific, of course, doesn't know this and keeps funneling new workers to JATOS, who are then disappointed and confused because they get the max number of workers reached error message.

    The solution to this is to make the array of conditions in Batch Session Data much larger, so there's no risk of running out before Prolific reaches its max number of workers. But I want an equal number of participants per condition - and if workers are randomly assigned, I might not get that. So I wrote the following code snippet to distribute workers over conditions sequentially (i.e. ABCDABCDABCD) instead. (No doubt there's a more elegant solution, but I'm a Javascript novice, so bear with me.)

    \\ after fillArrayWithValues

    const distinctValues = [...new Set(conditions)].sort();

    var valueDictionary = {};

    var maxValueCount = 0;


     for (var i = 0; i < distinctValues.length; i++) {

         const allValuesMatchingThisType = conditions.filter(value => value === distinctValues[i]);

         if (allValuesMatchingThisType.length > maxValueCount) {

           maxValueCount = allValuesMatchingThisType.length;}

         valueDictionary[distinctValues[i]] = allValuesMatchingThisType;}

    var outputArray = [];

       for (var i = 0; i < maxValueCount; i++) {

         for (var j = 0; j < distinctValues.length; j++) {

           if (valueDictionary[distinctValues[j]].length > i) {

             outputArray.push(valueDictionary[distinctValues[j]][i]);}}}

    And then adjust variable names etc. in the rest of the HTML file accordingly.

    Hope this helps someone else!

Sign In or Register to comment.

agen judi bola , sportbook, casino, togel, number game, singapore, tangkas, basket, slot, poker, dominoqq, agen bola. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 50.000 ,- bonus cashback hingga 10% , diskon togel hingga 66% bisa bermain di android dan IOS kapanpun dan dimana pun. poker , bandarq , aduq, domino qq , dominobet. Semua permainan bisa dimainkan hanya dengan 1 ID. minimal deposit 10.000 ,- bonus turnover 0.5% dan bonus referral 20%. Bonus - bonus yang dihadirkan bisa terbilang cukup tinggi dan memuaskan, anda hanya perlu memasang pada situs yang memberikan bursa pasaran terbaik yaitu http://45.77.173.118/ Bola168. Situs penyedia segala jenis permainan poker online kini semakin banyak ditemukan di Internet, salah satunya TahunQQ merupakan situs Agen Judi Domino66 Dan BandarQ Terpercaya yang mampu memberikan banyak provit bagi bettornya. Permainan Yang Di Sediakan Dewi365 Juga sangat banyak Dan menarik dan Peluang untuk memenangkan Taruhan Judi online ini juga sangat mudah . Mainkan Segera Taruhan Sportbook anda bersama Agen Judi Bola Bersama Dewi365 Kemenangan Anda Berapa pun akan Terbayarkan. Tersedia 9 macam permainan seru yang bisa kamu mainkan hanya di dalam 1 ID saja. Permainan seru yang tersedia seperti Poker, Domino QQ Dan juga BandarQ Online. Semuanya tersedia lengkap hanya di ABGQQ. Situs ABGQQ sangat mudah dimenangkan, kamu juga akan mendapatkan mega bonus dan setiap pemain berhak mendapatkan cashback mingguan. ABGQQ juga telah diakui sebagai Bandar Domino Online yang menjamin sistem FAIR PLAY disetiap permainan yang bisa dimainkan dengan deposit minimal hanya Rp.25.000. DEWI365 adalah Bandar Judi Bola Terpercaya & resmi dan terpercaya di indonesia. Situs judi bola ini menyediakan fasilitas bagi anda untuk dapat bermain memainkan permainan judi bola. Didalam situs ini memiliki berbagai permainan taruhan bola terlengkap seperti Sbobet, yang membuat DEWI365 menjadi situs judi bola terbaik dan terpercaya di Indonesia. Tentunya sebagai situs yang bertugas sebagai Bandar Poker Online pastinya akan berusaha untuk menjaga semua informasi dan keamanan yang terdapat di POKERQQ13. Kotakqq adalah situs Judi Poker Online Terpercayayang menyediakan 9 jenis permainan sakong online, dominoqq, domino99, bandarq, bandar ceme, aduq, poker online, bandar poker, balak66, perang baccarat, dan capsa susun. Dengan minimal deposit withdraw 15.000 Anda sudah bisa memainkan semua permaina pkv games di situs kami. Jackpot besar,Win rate tinggi, Fair play, PKV Games