Detecting voice onsets (voicekey)

sebastiaan

Lately, there have been a few requests for a script to detect voice onsets (i.e. a voicekey script). The original example script is no longer available, so below is a new script, written for OpenSesame 3.0.

You can insert this script into the run phase of an inline_script, and use it like a keyboard_response. It creates the following variables:

• response which is either 'timeout' (when a timeout occurred) or 'detected' (when an onset was detected)
• response_time which is the response time of the voice onset, which is equal to the timeout when a timeout occurred
• loudness which is the loudness of the voice onset that triggered the response, or None when a timeout occurred.

You need to tweak (at least) the following properties in the script:

• sound_threshold, which determines how sensitive the detection is. The appropriate setting varies a lot from system to system.
• timeout is the maximum response time, after which a timeout is triggered.

Important disclaimer: This code should work, but is based on a very crude algorithm. In most cases, it's probably better to record the voices, and detect the onsets afterwards, rather than doing this online with a voicekey. Also, the code uses pyaudio, which is sometimes a bit unstable.

import pyaudio
import struct
import math 

# A low threshold increases sensitivity, a high threshold
# reduces it.
sound_threshold = 0.5
# Maximum response time
timeout = 5000

FORMAT = pyaudio.paInt16 
SHORT_NORMALIZE = (1.0/32768.0)
CHANNELS = 2
RATE = 44100  
INPUT_BLOCK_TIME = 0.01
INPUT_FRAMES_PER_BLOCK = int(RATE*INPUT_BLOCK_TIME)
chunk=1024

def get_rms(block):

    """Get root mean square as a measure of loudness"""

    count = len(block)/2
    format = "%dh" % (count)
    shorts = struct.unpack( format, block )
    sum_squares = 0.0
    for sample in shorts:
        n = sample * SHORT_NORMALIZE
        sum_squares += n*n
    return math.sqrt( sum_squares / count )

# Open the mic
stream = pyaudio.PyAudio().open(
    format=FORMAT,
    channels=CHANNELS,
    rate=RATE,
    input=True,
    input_device_index=0,
    frames_per_buffer=INPUT_FRAMES_PER_BLOCK
    )

# Listen for sounds until a sound is detected or a timeout occurs.
start_time = clock.time()
while True:
    if clock.time() - start_time >= timeout:
        response_time = timeout
        response = u'timeout'
        var.loudness = None
        break   
    try:
        block = stream.read(chunk)
    except IOError as e:
        print e
    loudness = get_rms(block)
    if loudness > sound_threshold:
        response_time = clock.time() - start_time
        response = u'detected'
        var.loudness = loudness
        break

# Process the response  
set_response(response=response, response_time=response_time)

# Close the audio stream
stream.close()
pyaudio.PyAudio().terminate()

Topher

Hello, I was excited to find this, as what you describe as its functions are exactly what I am trying to do. I am looking to find out how long it takes someone to name an image on screen, and so I need those vocal reaction times.

My problem is that whenever I run the experiment, it seems like it is not detecting any audio input. The response always times out and the loudness variable never reports back any value. I am using a standard headset which is set to the default recording device in windows 10. I'm hoping that I'm just missing something obvious, but any help would be greatly appreciated. I'm very new to OpenSesame and Python in general.

sebastiaan

Hi,

You can print out the loudness values to the debug window by adding a simple print() statement below the call to get_rms():

# ...
loudness = get_rms(block)
print(loudness)
# ...

If you see that they fluctuate, but simply have really low values, then you can reduce sound_threshold accordingly. If you see the values are always 0, then no sound is being recorded at all. In that case, you probably have to specify a different audio device through the input_device_index keyword.

Cheers!
Sebastiaan

jpg446

Hi,
Is pyaudio not included in the standalone OpenSesame distribution? I get an Import Error when running the above script.

Daniel

Hi jpg446,
Can you tell us which OpenSesame version you are using and on which platform?
PyAudio should be included in the default package but it could be that we forgot it this time (we are revamping the packaging process so this might be a result of the transition). The good thing of this new packaging process is that it is really easy to install extra modules yourself at any time, see http://osdoc.cogsci.nl/3.1/manual/environment/.

Simply running

import pip
pip.main(['install', 'pyaudio'])

once in the debug window should do the trick (although maybe not for pyaudio as it depends on C libraries and is thus kind of a 'difficult' package).

jpg446

I was using 3.1.2 for Windows.

I just tried using pip and it worked! I didn't realise the installed Win version supported pip; thank you for adding this!

Daniel

@sebastiaan? Shouldn't this post be an announcement so it stays on top? Or are you planning on integrating it in the general documentation at some point?

sebastiaan

Good point. Once I'm back I'll add this snippet to the documentation.

lauren77

Hello.
My experiment is meant to

1. Play audio.wav & Show a picture at the same time [solved]
2. Detect participant's voice onset time [no timeout]

I only need the voice onset time. And I copied the script above to the inline script.
I got 'loudness' result in numbers.

However, I didn't get 'response' right. The response represents the key-board response that I pressed.
I don't understand Sebatian wrote as

You can insert this script into the run phase of an inline_script, and use it like a keyboard_response. It creates the following variables:

What do you mean by 'use it like a keyboard_response' ?

My loop contains below.

• sequence
• audio
• picture
• keyboard response
• logger.

I'm totally new and don't know how to make script alone..

Please please help me.

Thanks in advance.

cf. I don't know what to write in the blanks in the keyboard response..

sebastiaan

Hi Lauren,

Both the keyboard_response and the script create a variable called response. Because the keyboard_response comes last, that's the one that you see in the log file.

What you can do is change this line in the script:

set_response(response=response, response_time=response_time)

To:

responses.add(response=response, response_time=response_time, item='voice_key')

By specifying an item keyword, you tell OpenSesame that it should also log all variables with _[item_name] appended. So you'll get response_time (which is not unique and overwritten by the keyboard_response) but also response_time_voice_key (which is probably unique and therefore not overwritten). Do you see the logic?

See also:

• http://osdoc.cogsci.nl/3.1/manual/python/responses/

---

I'm totally new and don't know how to make script alone..

and

cf. I don't know what to write in the blanks in the keyboard response..

The beginner and intermediate tutorials are good places to get started:

• http://osdoc.cogsci.nl/3.1/tutorials/beginner/
• http://osdoc.cogsci.nl/3.1/tutorials/intermediate/

Cheers!
Sebastiaan

lauren77

Thanks alot. I did what you said, and it worked!

However, the value of "response_voice_time" is same to every item.

In other words, I present Audio(question) + Picture at the same time, and my participants should answer to the questions while watching the picture. I have about 40 items in total.

But, the value of 'response_voice_time' and 'loudness' is all the same to every item..

Please help..

cf) of course I studied all the manuals and tutorials.. the problem is.. I don't get it.. idiot.. ..

Sorry, and Thanx, .

Best regards.

Lauren.

sebastiaan

But, the value of 'response_voice_time' and 'loudness' is all the same to every item..

The variable response_voice_time should be 'detected' when a voice onset is detected, and 'timeout' otherwise. There's no voice recognition that will tell you what the participant said, if that's what you were expecting‽ That loudness is also the same is a bit odd though. What about the response_time_voice_time? Does that vary?

It would be helpful if you upload an example datafile here, so that I can see whether the response values make sense

eleonoragalletti

Hi guys,

I have opensesame 3.2 and I've tried the inline script that Sebastiaan uploaded up on this page but the program doesn't work. I am new with open sesame and I have to build a simple stroop test where I need to calculate reaction times from the stimulus presentation to the voice onset.
Each session I would have :
-sketchpad (fixation dot)
-tone (corresponding to the stimulus presentation)
-stimulus
-inline script
Now I don't know what to do to make the inline script work.

Do you have some idea of what I did wrong?

Thank you so much guys.

eduard

Hi,

If you provide more detail on what exactly doesn't work, we might be able to help. Could you post your error message? And did you follow all steps described in this discussion, e.g. installing pyaudio?

Eduard

RoyT

Hi there,

I need the exact same type of experiment as eleonoragalletti two comments up - a simple stroop test which relies on voice response, measures the time from the stimulus presentation to the voice onset, and records the participants' responses. I have absolutely no programming background, so I wondered if there are any new tutorials on the topic? I tried to follow the instructions in this thread but to no avail ("Failed to compile inline script") - I am sure that what I did wrong was absolutely basic, but since I am so inexperienced an example could be way more efficient than trying to correct my mistakes.

Thanks!

Roy

Daniel

For the people who are using psychopy for the backend: they also have a dedicated module for this task: https://psychopy.org/api/voicekey.html

RoyT

Thanks, although it seems as if this solution still requires certain programming skills.

A different approach - I suppose I can use the "sound recording" option and just calculate the RTs manually. My only concern now is that I need to run this experiment online, so if I do try to do that, how do I get the sound files?

Thanks!

Daniel

Hi @RoyT. Currently it is not possible to perform sound recordings online with osweb, and this is very difficult in general because of many browsers' security models. If you require to perform these kinds of sound recordings for you experiment, I think collecting data online currently is not an option.

RoyT

Thanks for the quick response. Assuming I can give up on the sound recordings and stay just with RT measurement, should the voicekey script Sebastian uploaded work online? Would it require the participants to download anything? (I haven't yet used OpenSesame online)

Daniel

Hi @RoyT,

Currently sound recording is only possible offline! This is really hard to implement online because of all the security measures that browsers pose (and also is different per browsers). I don't expect this to be available soon.