Looking for some recommendation to write adaptive testing code

Comments

• eduard

  July 2020

  Hi,

  Can you provide more information in how far this does not work? The code looks fine to me. Maybe the positioning in the experimental sequence is wrong, but without more detail, it is hard to pinpoint what is happening.

  
  

  Eduard

  
• Tzlim

  July 2020

  

  Hi Eduard,

  Thanks for your reply. My first trial of the experiment must be the item of middle level, and I have listed all of the trials into a particular level in the Experiment loop (var.item_level). However, I have tried a couple of times, but the first trial of the experiment sequence may not always be the particular item level that I wanted. Secondly, I have tried to insert either correct or incorrect responses; however, the next trial was not the particular level's item. e.g., I have inserted the correct response in the level 3' item, but the upcoming trial was not the item from level 4. It was randomly assigned from other item_level. So I have no clue that what I should do next, while these two settings were quite essential to my research design.

  I wonder these descriptions would add more information for you to know what I'm wanted to convey. Thanks again! I'm looking forward to your reply.
• Tzlim

  July 2020 edited July 2020

  Make it easier to know. My experiment design procedure would be like :

  My participants will see a fixation dot, and then stimuli sound on (sound_level [which is according to my var.item_level] start from level 3), then participants answer what they heard. If the participants insert the correct answer, then the next stimuli sound would be from level 4, else would be from level 2.

  #After fixation point, then sound on (sound should be starting from level 3[var.item_level])

  soundP.play() #This might be the part that I'm stuck
  target_blank_canvas.show()
  clock.sleep(1000)
  

  The figure I posted was not clear enough, so I re-post my code down here. Sorry for the inconvenience.

  #start from level 3
  var.item_level == 3
  
  #show fixation point to aware the participants
  fixdot_canvas.show()
  clock.sleep(495)
  
  #After fixation point, then sound on (#sound should draw according to var.item_level)
  soundP.play()
  target_blank_canvas.show()
  clock.sleep(1000)
  
  #then allow participant to answer the question
  t0 = target_answer_canvas.show()
  resp = "" #here we store the response
  
  while True:
  #Get a keyboard response
  key, time = kb.get_key()
  
  #if press return, then the loop should be exited.
  if kb.to_chr(key) == "return":
  break
  
  #handle backspace by removing the last character
  if kb.to_chr(key) == "backspace":
  resp = resp[:-1]
  else:
  resp += kb.to_chr(key)
  
  #save response time
  var.response_time = time - t0
  
  #save all the response key
  self.experiment.set("response", resp)
  
  if resp == var.correct_response: 
  var.correct = 1
  else:
  var.correct = 0 
  
  if var.correct == 1: #if correct = 1, then go to harder level
  var.item_level = var.item_level + 1
  else: #if correct = 0, then go to easier level
  var.item_level = var.item_level - 1
  
  if var.item_level == 7: #when reach top level and have correct response, remains in the same level
  var.item_level = 6
  if var.item_level == 0: #when reach lowest level and have incorrect response, remains in the same level.
  var.item_level = 1
  

  Thanks again! I'm looking forward to your reply.

  TZ
• eduard

  July 2020

  HI TZ,

  For a starter, your first line var.item_level == 3 only checks whether the current item level is 3. It doesn't do anything in the sequence, if you want to set the item level to 3, you need to use var.item_level = 3.

  However, if you want the current level to be +1/-1 relative to the previous trial, you can't reset it in the beginning of this block of code. Otherwise, everything that you set on the previous trial (the last lines in your script), will be undone by the first lines in the beginning of the next trial. So ,instead you probably want the item to be set only in the first trial, or even before the trial loop starts. Does that make sense?

  Seeing the experimental loops and variables in there would be helpful to know what is going on. If you say that the level is random on every trial and not incremented from the previous trial, I suspect you have the item level set in the experimental loop and on every trial it is pulling it from that table, rather than using the one you computed. Is that possible?

  Eduard

  
• Tzlim

  July 2020 edited July 2020

  Hi Eduard,

  I think I know what makes my works stuck! My experiment procedure will be like (1) fixation (2) stimuli sound (word sound) (3) Response (see the figure below).

  

  According to the script I wrote, the response step will note all of the responses that participants inserted, which is perfect! However, my stimuli sound won't start from the middle level at the beginning (first trail), which is level 3, for example. Should i add an inline_script var.item_level = 3 in before the experiment loop? or Experiment_procedure? Or somewhere else?

  

  Then, the sound playing script in the prepare and the run page was different, you can see as below.

  #prepare page

  path = pool(var.sound_experiment)
  soundP = sampler(path)
  

  #Run page

  soundP.play() #sound should draw according to var.item_level
  target_blank_canvas.show()
  clock.sleep(1000)
  

  I think I know the reason why my upcoming trial couldn't run the script I wrote, is it because I didn't define the sound stimulus' level? The code was telling it to run the sound besides telling it to retrieve according to the stimuli sound's level. If yes, how do I modify the code I wrote to satisfy my needs? Any recommendations?

  Appreciate your bits of help! 

  TZ
• eduard

  July 2020

  In your loop table, every line corresponds to one trial. So every time that you start a new iteration, item_level is set to the value that you see in the loop table. The adaptive character will be lost. You probably have to remove this columns altogether.

  >Should i add an inline_script var.item_level = 3 in before the experiment loop?

  That is what you should do I think. So after the start_sketchpad and before the ExperimentLoop. Then, the procedure will always start with the level set to 3, and during the loop it will dynamically be adapted depending on the performance (provided that you take out the column).

  If the variable item level should have an effect on a stimulus you'll have to set it of course. For example, if the loudness of the sound depends on the item level, you need to program it, otherwise, you won't see (or hear) any level-specific differences.

  Hope this clears things up.

  
  

  Eudard

  
• Tzlim

  August 2020

  Hi Eduard,

  Thanks for your advice! My experiment was nearly done!

  Now I have some couple question, which are

  (1) I wanted to show the responses of the participants have inserted; however, I have no idea how to write the code after the target_answer_canvas. Any recommendation?

  #show fixation point to aware the participants
  fixdot_canvas.show()
  clock.sleep(495)
  
  #After fixation point, then sound on (#start from level 3)
  soundP.play()
  target_blank_canvas.show()
  clock.sleep(1000)
  
  #then allow participant to answer the question
  t0 = target_answer_canvas.show()
  resp = "" #here we store the response
  while True:
  #Get a keyboard response
  key, time = kb.get_key()
  #if press return, then the loop should be exited.
  if kb.to_chr(key) == "return":
  break
  #handle backspace by removing the last character
  if kb.to_chr(key) == "backspace":
  resp = resp[:-1]
  else:
  resp += kb.to_chr(key)
  

  (2) As the figure below, my stimuli were repeatedly drawn after the break. For example, when my participants heard the sound (stimuli A) from the list of level 1. If he/she answered wrongly, then he/she will still in the level 1 loop; however, there will be a chance to draw exactly the same stimuli A from the previous one.  What modification do I need to do to solve this situation, because I want my ExpLoop not to draw the item/stimuli that have been used in the previous trials?

  

  Appreciate your bits of help! 

  TZ
• eduard

  August 2020

  Hi,

  I have not tested the code, but does this make sense? Basically include the drawing and showing a canvas inside the while loop.

  target_answer_canvas = Canvas()
  resp = "" #here we store the response
  target_answer_canvas.text = Text(resp) 
  t0 = clock.time()
  while True:
      target_answer_canvas.show()
      #Get a keyboard response
      key, time = kb.get_key()
      #if press return, then the loop should be exited.
      if kb.to_chr(key) == "return":
          break
      #handle backspace by removing the last character
      elif kb.to_chr(key) == "backspace":
          resp = resp[:-1]
      else:
          resp += kb.to_chr(key)
          target_answer_canvas.text += kb.to_chr(key)
  

  As for your other question, I unfortunately don't understand the structure of the experiment, the desired behaviour of it, and the problem that you are facing now. Could you elaborate?

  Thanks,

  Eduard

  
• Tzlim

  September 2020 edited September 2020

  Hi Eduard,

  Sorry for the late reply! I have tried the code, however, there was an error occur by showing unsupported operand type(s) for +=: 'RichText' and 'str'.

  target_answer_canvas.text += kb.to_chr(key)
  

  The purpose of the present study is to use adaptive testing to shorten the present hearing assessment tasks. We have done some previous studies to ensure that the difficulties of items and the clinical cut-off value. Now, we wanted to examine whether this kind of adaptive assessment tool has any differences with the traditional assessment, so we need to do some pilot testing. The procedure was shown as below.

  

  However, we face some difficulties while establishing the pilot experiment.

  The most important issue that must be solved was the chance of repeating the same sound stimuli. For example, when the first trial item was 'Apes' from level 3, no matter participants inserted correct or incorrect response, if the following stimuli were drawn from level 3, I don't want to run 'Apes' again. However, in the present setting, the same stimulus may have a chance to show more than 1 time during the experimental procedures. Any recommendations? I have been stuck for a couple of months, it kills me!

  

  

  Code:

  Prepare

  #creates fixation dot canvas
  fixdot_canvas = canvas()
  fixdot_canvas.fixdot()
  
  #create target_answer canvas
  target_answer_canvas = canvas(font_size = 50)
  #target_answer_canvas.text('請輸入您的答案', x = 0 , y = 0)
  
  #create target_blank canvas
  target_blank_canvas = canvas(color = 'black')
  target_blank_canvas.fixdot()
  
  #create keyboard response
  kb = keyboard()
  
  #create sound
  path = pool[var.sound_experiment]
  soundP = sampler(path)
  

  Run

  #show fixation point to aware the participants
  fixdot_canvas.show()
  clock.sleep(495)
  
  #After fixation point, then sound on (#start from level 3)
  soundP.play()
  target_blank_canvas.show()
  clock.sleep(1000)
  
  #then allow participant to answer the question
  target_answer_canvas = Canvas()
  resp = "" #here we store the response
  target_answer_canvas.text = Text(resp)
  t0 = clock.time()
  while True:
  target_answer_canvas.show()
  #Get a keyboard response
  key, time = kb.get_key()
  #if press return, then the loop should be exited.
  if kb.to_chr(key) == "return":
  break
  #handle backspace by removing the last character
  elif kb.to_chr(key) == "backspace":
  resp = resp[:-1]
  else:
  resp += kb.to_chr(key)
  target_answer_canvas.text += kb.to_chr(key)
  
  #save response time
  var.response_time = time - t0
  
  #save all the response key
  self.experiment.set("response", resp)
  
  #correct response
  if resp == var.correct_response:
  var.correct = 1
  else:
  var.correct = 0
  
  #switching block by following the rule
  if var.correct == 1:
  var.item_level = var.item_level + 1
  else:
  var.item_level = var.item_level - 1
  
  if var.item_level == 7: # reached highest level
  var.item_level = 6
  
  if var.item_level == 0: #reached lowest level
  var.item_level = 1
  

  Appreciate your bits of help! 

  TZ
• Tzlim

  September 2020 edited September 2020

  Hi all,

  It's great to come here again! thank to @eduard for your kindly responses! I had solved the question recently.

  However, I'm still have a small question that why I inserted correct answer, but still count as incorrect? For example, the correct response of the trial is 103, but when I insert 103, it doesn't count that the response as correct. See the figure below. What can I do now?

  

  
  

  At last, appreciate for the bit of helps.

  TZ
• Tzlim

  September 2020

  Hi all,

  I solved the question by writing in the inline script.

  if var.response == 103:
     var.correct = 1
  

  Thanks.

  TZ
• eduard

  September 2020

  Hi TZ,

  Great that you figured it out by yourself! (and sorry for not getting back to you earlier)

  Eduard