Pseudorandomization of Tasks within an experiment

Comments

• DahmSF

  December 2021

  Hi,

  try this in a javascript:

  if (vars.subject_nr%2 == 0) {
  do this and that / create a list to make a clear order
  }
  else if (vars.subject_nr%2 == 1) {
  do other stuff / create a different list
  }
  

  Or use in the 'if function of the sequence': e.g. [subject_nr]%2 = 0

  Best,

  Stephan
• reversemoonwalk

  December 2021 edited December 2021

  Hi Stephan,

  Thank you for your answer. I created an experiment file to get started. So in this experiment, I have created 3 java scripts:

  1) order_semanticandLDT : this should choose the order of the tasks, semantic categorization or lexical decision. However, each participant should take both tasks although with different order.

  2) order_semanticversions: this script should determine the order of semantic categorization

  experiment versions. Again, each participant should take the both versions - only the order should change from participant to participant.

  3) order_LDTversions: same the the "order_semanticversions"

  
  

  I'm not really sure about the code I should use to accomplish this. Also, if I use participant number as my method of choice, then I won't be able to create enough variation for the order. If the participant number is even, for example, then the participant will begin with a certain task- which is fine. However, since the order of versions will also be dependent on the participant number, then all participants taking this version will start with the same version of the task.

  
  

  Thanks in advance.

  SEM_LDT_randomizeorder.osexp

  
  
• Fab

  December 2021 edited December 2021

  Hi @reversemoonwalk,

  I had a quick look at your description and the task you attached.

  What you want to achieve is feasible without code, using nested loops, but a little cumbersome. So, I'd recommend a little coding (though minimal).

  Your task contains a lot of redundant objects and sequences. Since your trial sequence is the same for all the loops, there is no need to have multiple versions of everything, as it makes the program unnecessarily complicated, it makes it tricky to modify bits (forces you to go make the same changes in multiple places, which easily leads to errors), and using several loggers can make a mess of the logged data.

  Here's what I suggest:

  I used the stimuli you used in your example (even though one trial has "list2" as a target, which I assume is an error).

  Here's the structure of the whole task:

  

  Four loops (one per task) nested under a higher level loop that runs 4 times (4 cycles), with the higher level loop (Tasks) containing the order in which each task should be run based on the subject number (subject_nr) randomly allocated to the participant (in this case, I set it to vary between 0 and 3).

  

  Here's where I set the subject number to vary randomly between 0 and 3 (note that when you run the task through OSweb, the program picks that number randomly and so you may end up with different number of participants being allocated each number):

  

  
  

  Now, in the Tasks loop, you can see 4 variables called: sp_ldt_pos, en_ldt_pos, sp_sct_pos, en_sct_pos, (these variables code for the position of each task in the 4 cycles), and which contain other variables as values.: [sp_ldt], [en_ldt] etc.

  These values are set based using Javascript and based on the subject number. See the set_task_order inline_javascript object:

  // Determines the order based on the subject number picked by OSWeb (0-3)
  if (vars.subject_nr==0){
  vars.sp_ldt=1
  vars.en_ldt=2
  vars.sp_sct=3
  vars.en_sct=4
  vars.task=["Spanish LDT","English LDT","Spanish SCT","English SCT"]
  }
  
  if (vars.subject_nr==1){
  vars.sp_ldt=2
  vars.en_ldt=1
  vars.sp_sct=4
  vars.en_sct=3
  vars.task=["English LDT","Spanish LDT","English SCT","Spanish SCT"]
  }
  
  if (vars.subject_nr==2){
  vars.sp_ldt=3
  vars.en_ldt=4
  vars.sp_sct=1
  vars.en_sct=2
  vars.task=["Spanish SCT","English SCT","Spanish LDT","English LDT"]
  }
  
  if (vars.subject_nr==3){
  vars.sp_ldt=4
  vars.en_ldt=3
  vars.sp_sct=2
  vars.en_sct=1
  vars.task=["English SCT","Spanish SCT","English LDT","Spanish LDT"]
  }
  
  // sets up task counter
  vars.taskcounter=0
  

  You can see that depending on the subject number, the position if each tasks varies, such that half the subject will do the LDT tasks first, followed by the SCT tasks, and that within each half of the subjects, the order of the English and Spanish versions of either task is also counterbalanced.

  I created an array that contains a label of the task to display on the screen at the onset of each task, just for monitoring, and so that you can also save that to the logger (can be handy afterward to separate the data from the different tasks).

  The code ends with the setting up of a variable that will be incremented every time the tasks loop runs.

  Within the tasks_sequence, some Javascript is used to increment the task counter (under the "prepare" tab; that's very important as this has to run ahead of OSWeb setting up what is going to be presented in the sequence).

  

  Note that this code also sets up a variable (vars.task_info) that will contain the name of the task (as retrieved from the vars.task array we set up earlier. Since arrays' first element is in position 0 and we set up the task counter to run from 1 to 4, we use vars.taskcounter - 1 to address the correct element in the array.

  After that, it's just a matter of using the "Run if" property of the tasks_sequence to make sure that only the appropriate task runs in each of the loop's 4 cycles.

  

  As you can see, no need to create 4 independent copies of the task sequences and their objects. I created one sequence in one of the task loops, and used a linked copy in the others. That way: (1) anything you edit in that sequence is automatically applied to all tasks, and (2) you won't end up with a mess in terms of logged data.

  Talking about logged data, if you're planning to run this in a browser, you must check that the compatibility check for OSWeb 1.4 does not flag any issues. One of the things OSWeb does not allow is to set the loggers to log all variables (that takes a lot of bandwidth). So it's important that you manually set the logger to include the key variables you'll need. I included some in my example, but you should always double check this bit before running any experiment (as there is no going back once your data is collected).

  I attach my example so that you can explore it. I think that it should allow you to adapt your task and complete your experiment once you've tweaked it and added the bits and bobs you need.

  pseudorandomize_order_loops.osexp

  Hope this helps!

  Fabrice.

  PS: you might want to add some code to counterbalance the response keys across participants.

  
• reversemoonwalk

  December 2021 edited December 2021

  Hello @Fab !

  Thank you so much for your detailed answer and design. First, I will add a code to counterbalance the response keys according to your advice. Your current design works really well, but I just need one more feature.

  I will use 8 different lists, and each task should use one of these lists without repeating the lists already used in the same experiment. Is there a way to do that? Alternatively, I can create a single list and each task can choose certain row ranges pseudo-randomly (rows 1-50 in the first task, 150-200 in the second task ...etc. without repeating the same row ranges in the same experiment).

  
  

  

  
  

  Thanks you again!
• Fab

  December 2021

  Hi @reversemoonwalk,

  Not entirely sure I understand your description. Are you saying that you have a set of 8 lists and the experiment should pick 4 of these at random and allocate one to each task? I'll assume that's the case...

  Here's one solution that I think is quite economical.

  SELECTING 4 CSV FILES FROM A SET OF 8

  (1) We're gonna create a list of 4 values taken randomly between 1 and 8 without repetition and store it in an array. We do that by adding the following code in the set_task_order inline_javascript object:

  // function to pick length number between min and max
  function generateRandomArr(length, max, min) {
  const resultsArr = [];
  for (let i = 0; i < length; i++) {
  const newNumber = Math.floor(Math.random() * (max - min)) + min;
  resultsArr.includes(newNumber) ? length += 1 : resultsArr.push(newNumber);
  }
  return resultsArr;
  }
  
  // picks 4 digits between 1 and 8 and stores them in an array
  vars.listsarray = generateRandomArr(4, 8, 1);
  
  console.log ("List of 4 digits picked: "+vars.listsarray)
  

  The 4 digits are stored in vars.listsarray. I output them to the console just for checking what the 4 digits are. Later on, will will pick the elements from that array as the task runs to indicate to OSWeb which csv file to pick. (I created 8 basic csv file and coded the target as List1_1, List1_2... to List8_1, List8_2 etc., just so that I can check that the task is selecting the correct csv files. Note that the correct_response variable has disappeared from the table (I'll get back to that later).

  

  (2) We add some code to the increment_taskcounter inline_javascript object to retrieve the csv file we want for the current task.

  // allocate csv files to the current task, basd on previously randomy picked list
  console.log(vars.listsarray)
  vars.task_list="list"+vars.listsarray[vars.taskcounter-1]+".csv"
  console.log ("List loaded up for this condition is: "+vars.task_list)
  

  Note that a bit of the code is just for outputting stuff to the console in order to check what is going on as I'm programming; you can remove those line if you wish. The critical one is:

  vars.task_list="list"+vars.listsarray[vars.taskcounter-1]+".csv"
  

  That's where we're defining the csv file by appending "list" + the element from the array we want based on what task we're currently doing (if this is the first task, it'll pick the first element, etc.) + ".csv". We store the result in vars.task_list. Then all we need to do is to make sure to tell OSWeb to use that variable select the csv file:

  

  You need to do that for each of the four task loops.

  That solves the issue of picking up the files randomly from a set of 8 without repetition.

  
  

  COUNTERBALANCING RESPONSE KEYS

  On to the next issue: counterbalancing the response keys across subjects. If you want to do that in a manner that is orthogonal to the order of the tasks ((LDT to SCT vs SCT to LDT) x (Spanish to English vs English to Spanish), you end up with 8 combinations. I'll assume you'd refer to use a fully orthogonal arrangement rather than randomly allocating the response keys for each participant.

  (1) First, let's take care of that orthogonal counterbalancing.

  We'll need to edit the set_task_order inline_javascript, where we already specified tasks order depending on the subject's number. This time, to create vars.correctword and vars.correctnonword variables. With a fully orthogonal design, that part of the code now looks like this:

  // Determines the order based on the subject number picked by OSWeb (0-3)
  if (vars.subject_nr==0){
  vars.sp_ldt=1
  vars.en_ldt=2
  vars.sp_sct=3
  vars.en_sct=4
  vars.task=["Spanish LDT","English LDT","Spanish SCT","English SCT"]
  vars.correctword="z"
  vars.correctnonword="m"
  }
  
  if (vars.subject_nr==1){
  vars.sp_ldt=2
  vars.en_ldt=1
  vars.sp_sct=4
  vars.en_sct=3
  vars.task=["English LDT","Spanish LDT","English SCT","Spanish SCT"]
  vars.correctword="z"
  vars.correctnonword="m"
  }
  
  if (vars.subject_nr==2){
  vars.sp_ldt=3
  vars.en_ldt=4
  vars.sp_sct=1
  vars.en_sct=2
  vars.task=["Spanish SCT","English SCT","Spanish LDT","English LDT"]
  vars.correctword="z"
  vars.correctnonword="m"
  }
  
  if (vars.subject_nr==3){
  vars.sp_ldt=4
  vars.en_ldt=3
  vars.sp_sct=2
  vars.en_sct=1
  vars.task=["English SCT","Spanish SCT","English LDT","Spanish LDT"]
  vars.correctword="z"
  vars.correctnonword="m"
  }
  
  if (vars.subject_nr==4){
  vars.sp_ldt=1
  vars.en_ldt=2
  vars.sp_sct=3
  vars.en_sct=4
  vars.task=["Spanish LDT","English LDT","Spanish SCT","English SCT"]
  vars.correctword="m"
  vars.correctnonword="z"
  }
  
  if (vars.subject_nr==5){
  vars.sp_ldt=2
  vars.en_ldt=1
  vars.sp_sct=4
  vars.en_sct=3
  vars.task=["English LDT","Spanish LDT","English SCT","Spanish SCT"]
  vars.correctword="m"
  vars.correctnonword="z"
  }
  
  if (vars.subject_nr==6){
  vars.sp_ldt=3
  vars.en_ldt=4
  vars.sp_sct=1
  vars.en_sct=2
  vars.task=["Spanish SCT","English SCT","Spanish LDT","English LDT"]
  vars.correctword="m"
  vars.correctnonword="z"
  }
  
  if (vars.subject_nr==7){
  vars.sp_ldt=4
  vars.en_ldt=3
  vars.sp_sct=2
  vars.en_sct=1
  vars.task=["English SCT","Spanish SCT","English LDT","Spanish LDT"]
  vars.correctword="m"
  vars.correctnonword="z"
  }
  

  Important: You'll notice that the whole code in set_task_order under the prepare tab (no longer under the run tab). This is so that I can dynamically use the correctword and correctnonword variables on a sketchpad right after. If you put the code under the "run" tab, it'll prepare the sketchpad before the variables are defined and then of course an error will occur (OSWeb will say that these variables are undefined).

  (2) We need to make sure that OSWeb will pick subject numbers randomly between 0 and 7:

  

  
  

  (3) We'll need to define the correct response dynamically as the trials run. That means we no longer need to have a correct_response column in the individual csv files (which is why I removed that column in each file).

  To define the correct response dynamically, I inserted some Javascript code at the onset of the the trial_sequence:

  if (vars.category=="word"){
  vars.correct_response=vars.correctword
  } else {
  vars.correct_response=vars.correctnonword
  }
  

  Et voilà! Bob's your uncle!

  I think this should take care of both issues nicely for you, or that you should be able to complete your task.

  Here's the modified task:

  pseudorandomize_order_loops.osexp

  Hope you'll my help useful.

  Fabrice.

  
• reversemoonwalk

  December 2021

  Hello again @Fab ,

  I cannot thank you enough for all the help you have provided! You are amazing! This version will do nicely, and I have learned a lot thanks to your detailed explanations. One more small question, if I am allowed to take a little more of your time :)

  The randomization of lists is based on subject numbers, and if I'm not mistaken, these numbers are randomly generated at the beginning of the experiment. This means that the final data for each list might be unbalanced: List 1 might have been taken 80 times, while this number for List 5 might be 30. Is there a way to counterbalance this? Maybe I can assign a subject number to each participant, and they can enter this number in a box at the beginning of the experiment.

  
  

  Thanks a million! :)
• Fab

  December 2021

  Hi @reversemoonwalk,

  You're right, OSWeb picks the subject_nr at random and so you may not end up with the same number of participants in each condition.

  There are potentially three solutions to this.

  (1) There is apparently a way to use session variables on the JATOS server, though I haven't tried it. You can find more information in this thread: https://forum.cogsci.nl/discussion/7067/avoid-race-conditions-when-working-with-batch-session-data

  (2) If you have the option to instruct participants to enter a number at the onset of the experiment, then all that's needed is to get that information at the onset and overwrite subject_nr with the subject's response (and to do it in a way that ensures that they can't enter anything else than a number between 0 and 7).

  (3) if you're planning to contact participants individually or send invitation links to 8 different groups of participants, you could also include some code to interact with JATOS and retrieve a parameter from the URL used by the subject.

  I'll describe solutions (2) and (3) below.

  Solution 2

  Here, we're gonna ask the subject to key in a number between 0 and 7 as received from the experimenter. Rather than using a single keyboard input object, it is better to include that in a loop that will run 20 times unless the subject keys in a number between 0 and 7. That way, you make sure that if they key in, say , 8, they'll just have to key something else. Otherwise, without this, the task would go on but produce problems.

  Here's the loop. Note the "Break if" condition: the loop will stop when a correct response has been registered.

  

  To assess the subject's response and overwrite vars.subject_nr with it, we'll use this bit of code right after the get_response keyboard object:

  if (vars.response>=0 && vars.response<=7) {
  vars.correct=1
  vars.subject_nr=vars.response
  console.log("New subject_nr: "+vars.subject_nr)
  console.log("Response: "+vars.response)
  }
  

  This runs under the "Run" tab.

  You can see that there, I define vars.correct as 1 if the response is a digit between 0 and 7, and I overwrite the subject_nr value with that response.

  Now, another change is that we now include all the code from the set_task_order code under its "run" tab instead of its "prepare" tab. That ensures that the code will take into account the new subject_nr value I also changed the next screen from a sketchpad to a feedback object (welcome), so that its content also takes the new value into account (sketchpad are defined ahead of their execution, so if you want to display a variable whose content changes dynamically, it is better to use a feedback object.

  I attach the update version here:

  pseudorandomize_order_loops - overwrite subject_nr.osexp

  Solution 3

  For this solution, I'm starting from the above version but we no longer need the loop where we're asking the subject to key in a number, and will introduce some code to interact with JATOS. Note that therefore this solution will only work when you test the task as a JATOS experiment running from the server you're planning to use. So, you won't be able to try it from OS through your browser, you'll have to export your task as a JATOS experiment and set it up on your JATOS server or whatever JATOS server you're using (e.g., the freely available JATOS service from https://mindprobe.eu/).

  With this solution, your subject won't have to key in anything because that information will be contained in the link you send them.

  For this to work, you need to include the following code at the beginning of the task (under the "prepare" tab):

  

  Here's the code:

  if (typeof jatos !== "undefined") {
  // obtain information from JATOS Url
  var cond = jatos.urlQueryParameters.cond;
  vars.cond = cond ? cond : "empty";
  }
  else {
  cond = "empty";
  }
  
  // overwrites subject_nr
  vars.subject_nr=cond
  

  Once your experiment is set up on JATOS, get a link to it, and add parameter "cond=" and a number between 0 and 7.

  I uploaded this one to mindprobe. the link for a General Multiple Worker is https://jatos.mindprobe.eu/publix/3244/start?batchId=3686&generalMultiple. But you run it as such, it won't work, we need to add "&cond=1", or "=cond=2", etc.:

  https://jatos.mindprobe.eu/publix/3244/start?batchId=3686&generalMultiple&cond=1

  Then try replacing "cond=1" by "cond=2", etc... up to "cond=7". You'll see that these links open the task and set subject_nr to 0, 1,... to 7.

  That way, subjects do not need to input anything.

  I attach the modified task:

  pseudorandomize_order_loops - subject_nr through URL.osexp

  
  

  Hope this helps!

  Fabrice.

  
• reversemoonwalk

  December 2021

  Hello @Fab !

  Both of your solutions are perfect, but I preferred the second method as I might also need the participant numbers associated with each experiment data. I made slight modifications according to the one of the posts in this form (Collecting multiple-character input in OSWeb (e.g. student numbers) — Forum (cogsci.nl))

  
  

  1) Participants are asked to enter their participant ID, which will be provided in e-mails along with the experiment links. The last number of this participant ID is taken as the subject_nr variable and determines the lists and their order, thanks to your well-designed code.

  console.log(vars.response)
  if (vars.response in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) {
  vars.student_number += vars.response.toString()
  } else if (vars.response === 'backspace') {
  vars.student_number = vars.student_number.toString().slice(0, vars.student_number.toString().length - 1)
  }
  vars.lastDigit= vars.student_number % 10
  
  vars.subject_nr=vars.lastDigit
  
  console.log("New subject_nr: "+vars.subject_nr)
  console.log("Response: "+vars.lastDigit)
  

  
  

  2) To avoid potential errors, I just disabled numbers 8 and 9 in the input screen. This also means that I will avoid using these two numbers while generating participant IDs.

  

  Thank you again! I hope the final version of my experiment is okay, and other people can benefit from it too!

  pseudorandomize_order_loops - overwrite subject_nr_with_part_ID.osexp

  
  
• Fab

  December 2021

  Hi @reversemoonwalk,

  Looks great!

  Just a couple of things: avoid giving students an id number starting with 0 as this 0 will be ignored (this is because OS treats any string containing only integers as a numerical variable (and so "0123" will become "123"). There is a way around this issue (https://forum.cogsci.nl/discussion/comment/24540#Comment_24540) but not really worth the headache if you can give your subjects id numbers that do not start with 0.

  If you're not using digits 8 and 9, you can actually edit the Javascript code to:

  if (vars.response in [0, 1, 2, 3, 4, 5, 6, 7]) {
  vars.student_number += vars.response.toString()
  } else if (vars.response === 'backspace') {
  vars.student_number = vars.student_number.toString().slice(0, vars.student_number.toString().length - 1)
  }
  vars.lastDigit= vars.student_number % 10
  

  (where you take 8 and 9 out of the array of values you compare var.response to).

  Also, I noticed that if the subject makes a mistake, you're inviting them to escape the task and start over. This will work (though some subjects might not be sure what to press between "ok" and "cancel" after pressing ESC). An alternative is to nest the get_student_number_loop into a higher level loop set to be repeated a number of times and if be broken out of under a certain condition.

  New bits:

  

  In the validate_snumber javascript object, I just set an exit_snumber variable to 1:

  if (vars.response=="s") {
  vars.exit_snumber=1
  }
  

  ...and set the "break if" property of the enter_subjectid_loop accordingly:

  

  The advantage of this method is that now subjects can re-enter their id without any need to exit the start and start over.

  Here the modified version:

  pseudorandomize_order_loops - overwrite subject_nr_with_part_ID v2.osexp

  Hope this helps.

  Good luck with your experiment!

  Fabrice.

  
• reversemoonwalk

  December 2021

  It helped a lot, thanks @Fab ! One more detail about the experiments: They are going to include masked primes, so the 8 lists we will use are actually two different versions of 4 lists -varying only in the prime types. It is like this:

  List 1 - List 5 (Identical targets- only the masked primes will vary)

  List 2- List 6 (Identical targets- only the masked primes will vary)

  List 3 - List 7 (Identical targets- only the masked primes will vary)

  List 4- List 8 (Identical targets- only the masked primes will vary)

  A problem emerges when a participant takes identical lists (e.g. List 1 and List5); therefore, I must find a way to eliminate the identical list as well when a list is taken (e.g. When List 1 is selected, List 5 must also be excluded from the available lists too). Is there a way to accomplish this?

  Alternatively, I might group lists into 2 broad groups (List 1,2,3 and 4 in group 1; List 5,6,7,8, in group 2), and participants might choose one of these groups based on their participant numbers and carry out the experiments with the lists in these groups only.

  Thanks in advance!
• reversemoonwalk

  December 2021

  I have another idea! Maybe I can create only 4 lists with two prime columns (prime1 and prime2). Participants will be assigned to one of these two prime lists according to their participant number. I think this will be the simplest solution.
• reversemoonwalk

  December 2021 edited December 2021

  I have managed to accomplish pseudo-randomizing by the following code:

  vars.prime= '' 
  if (vars.subject_nr==0){
  vars.prime="[prime1]"}
  if (vars.subject_nr==1){
  vars.prime="[prime2]"}
  if (vars.subject_nr==2){
  vars.prime="[prime1]"}
  if (vars.subject_nr==3){
  vars.prime="[prime2]"}
  

  But when I delete the lists 5,6,7 and 8, the tasks do not start. How can I decrease the number of the lists in the experiment? I tried to edit the following code, but it did not work:

  // function to pick length number between min and max
  function generateRandomArr(length, max, min) {
  const resultsArr = [];
  for (let i = 0; i < length; i++) {
  const newNumber = Math.floor(Math.random() * (max - min)) + min;
  resultsArr.includes(newNumber) ? length += 1 : resultsArr.push(newNumber);
  }
  return resultsArr;
  }
  
  // picks 4 digits between 1 and 4 and stores them in an array
  vars.listsarray = generateRandomArr(4, 4, 1);
  
  console.log ("List of 4 digits picked: "+vars.listsarray)
  

  Edit: I managed to make it work by changing the following line:

  vars.listsarray = generateRandomArr(4, 4, 1);
  

  to

  vars.listsarray = generateRandomArr(4, 5, 1);
  

  Here is the final version of my experiment:

  pseudorandomize_order_loops - overwrite subject_nr_with_part_ID_alterantingprimeversions.osexp

  
  
• pbe

  June 2023

  An example is available at https://pablobernabeu.github.io/2023/assigning-participant-specific-parameters-automatically-in-opensesame