voice key recording

Katharina · April 2020

Dear all,

I need help implementing a voice key as response. Does this function still exist?

Many thanks!

Katharina

eduard · April 2020

Hi Katherina,

There is no "official" voice key function in Opensesame, so you probably have to some coding yourself. Nevertheless, there are a couple of versions floating around the forum. How about you give it a browse?

For example, this one here might be a good starting point: https://forum.cogsci.nl/discussion/5397/issue-with-response-times-and-loudness-in-a-reading-aloud-task-with-voice-key

Eduard

Daniel · April 2020

Hi Katherina,

If you use psychopy for the backend, you can use its official voice recording module:

https://www.psychopy.org/api/voicekey.html

You will need to have to do some scripting then though, and you need a very recent version of OpenSesame as this functionality has only recently been added to psychopy too.

Katharina · May 2020

Thank you very much for the answers. I use Expyriment as the backend.

I remember there was a sund_start_recording plugin and even an element for that.

https://osdoc.cogsci.nl/3.2/manual/response/soundrecording/

I did not find it anymore.

Katharina · May 2020

@eduard I am sorry, I did not see your reply from March.

In fact, what I need is just set an event (reply given) by recording a sound, so that I can anylize the data response locked. Is there a simpler way to do it maybe without having to code a huge inline? Thanks

Katharina · May 2020

I have now browsed some examples on this forum, also the suggested one, but unfortunately, I did not help. I must confess, I am not skilled at programming though. Therefore I have several questions:

Do I have to install something for the voice recording?
Where exactly is the inline, assuming that this is the code?

all=[]
var.A_Response_time = timeout
start_time = clock.time()

# Open the mic
stream = pyaudio.PyAudio().open(
	format=FORMAT,
	channels=CHANNELS,
	rate=RATE,
	input=True,
	input_device_index=0,
	frames_per_buffer=INPUT_FRAMES_PER_BLOCK
	)


# Listen for sounds until a sound is detected or a timeout occurs.
while True:
	if clock.time() - start_time >= timeout:
#		var.loudness = None
		break   
	try:
		block = stream.read(chunk)
		all.append(block)
		
	except IOError as e:
		print e
	loudness = get_rms(block)
	
	if loudness > sound_threshold:
		var.A_Response_time = clock.time() - start_time
		var.A_loudness = loudness
		var.A_clocktime = clock.time()
		var.A_starttime = start_time

		# clean the canvas
		canvas.clear()
		canvas.show()
		
		#break

# Close the audio stream
stream.close()
pyaudio.PyAudio().terminate()


# write data to WAVE file
data = ''.join(all)
wf = wave.open(WAVE_OUTPUT_FILENAME, 'wb')
wf.setnchannels(CHANNELS)
wf.setsampwidth(pyaudio.PyAudio().get_sample_size(FORMAT))
wf.setframerate(RATE)
wf.writeframes(data)
wf.close()

3.As far as I understand it "listens" to the sound and if its louder than the threshhold it records.

if loudness > sound_threshold:
		var.A_Response_time = clock.time() - start_time

I don't know how this would work in my setup: I present a sound (50 ms) every 2 seconds and participants should say a word right after they hear a sound. Theoretically, they can already start saying while the sound is playing. I need the voice response onset and offset times rather then the reaction times. I measure their grip force while they do this contineously. The onset time will be used to lock the grip force data to it in the analysis (similar as in EEG for example). I cant see how this would work. Maybe I do not understand something.

I really appreciate your help and guidance.

Katharina · May 2020

You also wrote that its is better to record speech with some other device. Could you suggest some? Is this http://www.cog.brown.edu/people/mertus/BlissDownLoad.shtml something you meant?

Thanks a lot!

eduard · May 2020

Hi Katherina,

If timing is important, voice key could be a problem as it is not particular reliable/accurate (just imagine all the unrelated sounds/soft voices/ coughs/ etc that could cause misses and false alarms). So, maybe it might even to make more sense to use some other response measure. I haven't tried it, but the code that you posted looks like something that can be used to measure response onset (you understand correctly, the response is triggered if the volume exceeds a threshold). You have to put it in an inline_script item probably in the run phase and place this inline_script in the sequence after the item that you want your participants to respond to.

Hope this helps a little,

Eduard

Katharina · May 2020

Thank you for your quick reply. I do need to use a verbal response.

I was suggested to you this device as voice key https://cedrus.com/support/sv1/index.htm

and apparantly it can be integrated using python code

https://github.com/cedrus-opensource/pyxid

but I have no experience of doing this. Do you have any suggestings where to start?

Many thanks.

Katharina · May 2020

Thank you for your quick reply. I do need to use a verbal response.

I was suggested to you this device as voice key https://cedrus.com/support/sv1/index.htm

and apparantly it can be integrated using python code

https://github.com/cedrus-opensource/pyxid

but I have no experience of doing this. Do you have any suggestings where to start?

Many thanks.

eduard · June 2020

Hi Katharina,

Given that you want to run the study online, it might be better to start with javascript right away, rather than Python. But anyway, on Cedrus, they publish example python code how to connect the response box. YOu can find it here: https://github.com/cedrus-opensource/pyxid

A starting point would be to try to understand and reproduce the code and eventually to adapt it to your needs.

As for javascript, there doesn't seem to be code published by them, but maybe it is possible to access the protocols that they list with javascript, but unfortunately, I wouldn't know how.

Eduard

Katharina · June 2020

Thank you for your quick reply. I do need to use a verbal response.

I was suggested to you this device as voice key https://cedrus.com/support/sv1/index.htm

and apparantly it can be integrated using python code

https://github.com/cedrus-opensource/pyxid

but I have no experience of doing this. Do you have any suggestings where to start?

Many thanks.

Katharina · June 2020

Thank you, @eduard

This is an other study I want to run online, this one will be normal offline desktop study.

I will try the code posted here.

https://forum.cogsci.nl/index.php?p=/discussion/1772/

Katharina · June 2020

Sorry to bother you again.

Can you explain when the clock.time starts to count in this setup?

# Listen for sounds until a sound is detected or a timeout occurs.
start_time = clock.time()
while True:
	if clock.time() - start_time >= timeout:
		response_time = timeout
		response = u'timeout'
		var.loudness = None
		break  
	try:
		block = stream.read(chunk)
	except IOError as e:
		print e
	loudness = get_rms(block)
	if loudness > sound_threshold:
		response_time = clock.time() - start_time
		response = u'detected'
		var.loudness = loudness
		break

# Process the response  
set_response(response=response, response_time=response_time)

What I need is actually the start_time (and Rt as well but start_time is more important)

many thanks!

Katharina · June 2020

in parallel, I would also like to save the voice data recordings in a wav file. Itried to import wave but it didnt work.

eduard · June 2020

Hi Katharina,

Sorry for the late reply.

Start time of what? The trial or the voice? if you want to measure if from the start of the trial, that the way you do it, looks good to me. Provided that before the line start_time=clock.time() there is nothing else and the code is put in the run phase of the inline_script, start_time would correspond to the moment Opensesame finishes with the sampler item. Depending on the settings of its duration parameter this can either be after the sound finished playing (sound), or after initializing playing the sound.

Aside of that, voice keys are not really good RT measures (a lot of background noise, etc. ), so don't put to much hope to get as accurate results as with keyboards, let alone button boxes.

What packages do you use to record the voice? Pyaudio? Information on how to export recorded sounds are probably explained there. This link looks also promising.

Hope this helps,

Eduard

Katharina · June 2020

Thank you for your answer.

I have now tried

import sounddevice as sd

from scipy.io.wavfile import write

fs = 44100 # Sample rate

seconds = 1 # Duration of recording

myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)

sd.wait() # Wait until recording is finished

write('output.wav', fs, myrecording) # Save as WAV file

in the beginning of the inline with voice key

import pyaudio

import struct

import math

import sounddevice as sd

from scipy.io.wavfile import write

fs = 44100 # Sample rate

seconds = 1 # Duration of recording

myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)

sd.wait() # Wait until recording is finished

write('output.wav', fs, myrecording) # Save as WAV file

# A low threshold increases sensitivity, a high threshold

# reduces it. We need to play around with it.

sound_threshold = 0.05

# Maximum response time

timeout = 945

but I get an error

IOError: [Errno 13] Permission denied: 'output.wav'

What do you think this could be? I use Expyriment as background

eduard · June 2020

Hi Katherina,

You don't seem to have permission to write files where you are trying to write the files to. Can you specify a full pah to your desktop?

I don't know what exactly it is (also depending on your operating system, but something like: C://katherina/Desktop/output.wav?

Eduard

Katharina · June 2020

thanks for a speedy reply @eduard

It's

C:\Users\Katharina Kühne\Desktop

eduard · June 2020

So? Does it work if you specify that path?

Katharina · June 2020

Still get an error

IOError: [Errno 2] No such file or directory: 'C:\\Users\\Katharina Kuehne\\Desktop\\output.wav'

Maybe I placed it in a wrong way

import pyaudio
import struct
import math 

import sounddevice as sd
from scipy.io.wavfile import write

fs = 44100 # Sample rate
seconds = 3 # Duration of recording


myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('C:\Users\Katharina Kuehne\Desktop\output.wav', fs, myrecording) # Save as WAV file 


# A low threshold increases sensitivity, a high threshold
# reduces it. We need to play around with it. 
sound_threshold = 0.05
# Maximum response time 
timeout = 945

FORMAT = pyaudio.paInt16 
SHORT_NORMALIZE = (1.0/32768.0)
CHANNELS = 2
RATE = 44100  
INPUT_BLOCK_TIME = 0.01
INPUT_FRAMES_PER_BLOCK = int(RATE*INPUT_BLOCK_TIME)
chunk=1024



def get_rms(block):

	"""Get root mean square as a measure of loudness"""

	count = len(block)/2
	format = "%dh" % (count)
	shorts = struct.unpack( format, block )
	sum_squares = 0.0
	for sample in shorts:
		n = sample * SHORT_NORMALIZE
		sum_squares += n*n
	return math.sqrt( sum_squares / count )


# Open the mic
stream = pyaudio.PyAudio().open(
	format=FORMAT,
	channels=CHANNELS,
	rate=RATE,
	input=True,
	input_device_index=0,
	frames_per_buffer=INPUT_FRAMES_PER_BLOCK
	)

# Listen for sounds until a sound is detected or a timeout occurs.
start_time = clock.time()
while True:
	if clock.time() - start_time >= timeout:
		response_time = timeout
		response = u'timeout'
		var.loudness = None
		break  
	try:
		block = stream.read(chunk)
	except IOError as e:
		print e
	loudness = get_rms(block)
	if loudness > sound_threshold:
		response_time = clock.time() - start_time
		response = u'detected'
		var.loudness = loudness
		var.voice_onset = clock.time()
		var.starttime = start_time
		break

# Process the response  
set_response(response=response, response_time=response_time)




# Close the audio stream
stream.close()
pyaudio.PyAudio().terminate()

eduard · June 2020

Not sure, but I think a way is by using the soundFile library. In any case, you can check out this documentation here: https://python-sounddevice.readthedocs.io/en/0.3.12/examples.html

in particular (recording with arbitrary duration).

It probably also works with plain python methods, but I wouldn't know fromthe top of my head, what the problem is with your code.

Eduard

Katharina · June 2020

I would be all right when it just records 1 s, the responses are around 500 ms, I can cut the rest if I want but it's more of a document WHAT they say not how.

I believe I am placing that in a wrong place

import sounddevice as sd
from scipy.io.wavfile import write

fs = 44100 # Sample rate
seconds = 3 # Duration of recording


myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('C:\Users\Katharina Kuehne\Desktop\output.wav', fs, myrecording) # Save as WAV file

sorry for the naive questions

eduard · July 2020

Hi Katherina,

I just checked the code and it works fine. It writes a file after a file is recorded for 3 seconds. However, it does not produce a voice key, so there is nothing happening once you start speaking.

For that you would indeed need a voice key plugin, like the one provided in psychopy (which you can import in Opensesame and use).

Sorry for asking again, but can you summarize again, what exactly it is that you need, and what it is that doesn't work.

a voice key, that is, marking a response, every time a participant says something?
a recorded wave file that you can analyse offline?
recorded sound that you can do operations on during the experiment?

As far as I understand, you want in any case a wave file. That should be easily possible with the sounddevice code. If it doesn't work, then I need more information on what the problem is. Are there error messages? Is a file produced?

Thanks,

Eduard

Katharina · July 2020

Thanks a lot!

I always get

IOError: [Errno 13] Permission denied: 'output.wav'

I specified the path and everything but still. Having reponse a s voice key I finally managed :) thanks to you all

but now it would be nice to have a wave file.

Do you have anything special installed on your machine?

Many thanks!

eduard · July 2020

aha, okay. So, only the write file issue left.

The error message is probably misleading. You probably have the rights to write files to your desktop, but the problem is related to the wrong paths or folders that don't exist. It is hard to really clearly tell you what to do. You can check some online resources that might help addressing the error: for example:

I hope this helps,

Eduard

Katharina · July 2020

Thanks a lot! I realized that this very error was because of my account name with ü, changing the account did not change the path. I recorded it on a neutral USB stick and it worked. But! It only records the very last trial.

Moreover, it somehow enormly enlarged the presentation time - from 2 to 4 ms!

Thank you for your patience and help.

Katharina · July 2020

I mean, seconds of course :)

eduard · July 2020

Yeah, Umlaute (as well as white space) is not ideal for file names and directories. I'd change that to something simpler (e.g. katkue?). But nice that it is fixed.

The delay might come from running it on the USB. However, a doubling of the presentation time is unlikely to come from a device lag. Probably, something else is happening. Can you try to change the presentation time to other values (0.5, 1, 4 seconds) and check whether the actual times are still double each time? If so, then somewhere in your code there is probably some accidental doubling happening, e.g. calling one thing twice. If you share your code or experiment, I could have a look then.

Eduard

Katharina · July 2020

Many thanks!

RNG_test.osexp

import pyaudio
import struct
import math 

# A low threshold increases sensitivity, a high threshold
# reduces it. We need to play around with it. 
sound_threshold = 0.02
# Maximum response time 
timeout = 990

FORMAT = pyaudio.paInt16 
SHORT_NORMALIZE = (1.0/32768.0)
CHANNELS = 2
RATE = 44100  
INPUT_BLOCK_TIME = 0.01
INPUT_FRAMES_PER_BLOCK = int(RATE*INPUT_BLOCK_TIME)
chunk=1024

import sounddevice as sd
from scipy.io.wavfile import write

fs = 44100 # Sample rate
seconds = 1 # Duration of recording

myrecording = sd.rec(int(seconds * fs), samplerate=fs, channels=2)
sd.wait() # Wait until recording is finished
write('D:\myselftest\output.wav', fs, myrecording) # Save as WAV file 

def get_rms(block):

	"""Get root mean square as a measure of loudness"""

	count = len(block)/2
	format = "%dh" % (count)
	shorts = struct.unpack( format, block )
	sum_squares = 0.0
	for sample in shorts:
		n = sample * SHORT_NORMALIZE
		sum_squares += n*n
	return math.sqrt( sum_squares / count )


# Open the mic
stream = pyaudio.PyAudio().open(
	format=FORMAT,
	channels=CHANNELS,
	rate=RATE,
	input=True,
	input_device_index=0,
	frames_per_buffer=INPUT_FRAMES_PER_BLOCK
	)

# Listen for sounds until a sound is detected or a timeout occurs.
start_time = clock.time()
while True:
	if clock.time() - start_time >= timeout:
		response_time = timeout
		response = u'timeout'
		var.loudness = None
		break  
	try:
		block = stream.read(chunk)
	except IOError as e:
		print e
	loudness = get_rms(block)
	if loudness > sound_threshold:
		response_time = clock.time() - start_time + prepare_duration
		response = u'detected'
		var.loudness = loudness
		var.voice_onset = clock.time()
		var.starttime = start_time
		break

# Process the response  
set_response(response=response, response_time=response_time)

# Close the audio stream
stream.close()
pyaudio.PyAudio().terminate()

The file is included.

It's actually a mega easy experiment, with a mentronome sound coming every (ideally) 2 seconds, and participants having to say smth after the sound. This sould be recorded as voice_onset time and the wav.file

eduard · July 2020

have you checked the consistency of the delay as I suggested?

Howdy, Stranger!

Categories

voice key recording

Comments

Howdy, Stranger!

Quick Links

Categories

voice key recording

Comments