Audio recording in an online experiment

Kruthi

Hello everyone,

I am reaching out ask for help from anyone who has experience/knowledge of recording participant audio in a browser-based experiment. I am new to online experiments and therefore to javascript and would love any guidance on the (additional) steps I'd need to take in order to record participants' vocal responses. Any helpful links, guidelines, or sample codes are highly appreciated!

SigmundAI

Great question! Recording participants’ voices in a browser-based OpenSesame experiment (OSWeb) is possible using the Web Audio/MediaRecorder APIs via JavaScript. Below is a practical roadmap, including what works online, what to avoid, and a minimal working pattern you can adapt.

Key points up front

• Desktop vs. online: The Audio Low Latency plugins are excellent for desktop Python experiments, but they do not work online. For OSWeb, use inline_javascript with getUserMedia + MediaRecorder.
• Permissions: Audio capture requires a user gesture before the browser will allow the microphone (e.g., a click on a welcome screen or start button). Keep the OSWeb welcome screen enabled.
• File format: MediaRecorder typically produces webm/opus in Chrome/Edge and ogg/opus in Firefox. You’ll log a URL or upload the blob to JATOS as a result file.
• Data storage: When running on JATOS, save blobs via jatos.uploadResultFile. When running locally from OpenSesame, you can’t save blobs to disk automatically; you can expose a download link as a fallback.

Minimal workflow outline

1. Include an instruction/welcome screen to ensure a user gesture occurs before requesting the microphone.
2. In an inline_javascript “prepare” phase: set up variables and helper functions, but don’t request the mic yet.
3. In an inline_javascript “run” phase: after a user gesture, request the mic with navigator.mediaDevices.getUserMedia({ audio: true }), prepare MediaRecorder, and start recording right before your trial prompt.
4. Stop the recording when the participant responds (e.g., keypress) or after a timeout. Collect resulting audio chunks, assemble into a Blob.
5. Store:
   • On JATOS: upload with jatos.uploadResultFile and log the returned filename/path in your LOGGER.
   • Locally: createObjectURL for the blob and show a download link at the end (for testing).

1. Log metadata: onset time, stop time, duration, and any trial identifiers so you can link audio files to trials.

Example: OSWeb inline_javascript snippets I added a minimal template to the workspace that:

• Prepares mic/recorder helpers in Prepare.
• Starts and stops recording in Run.
• Uploads to JATOS if available; otherwise exposes a download link for testing.
• Logs the generated filename so your LOGGER can capture it.

// Inline_javascript — RUN phase
// Typical sequence around your stimulus/response collection.

trial_tag = 'trial_' + (typeof count_trial_sequence !== 'undefined' ? count_trial_sequence : 'NA')

// Step 1: Ensure mic permission (after user gesture earlier in the flow)
ensureMicPermissionThen(function() {
    // Step 2: Prepare the recorder
    prepareRecorder()

    // Step 3: Show your stimulus on a Canvas and START recording
    stim = Canvas()
    stim.text({ text: 'Say the word now!', x: 0, y: 0, center: true })
    stim.show()

    startRecording()

    // Step 4: Stop recording after a fixed window or on keypress
    // Example: stop after 2000 ms; alternatively, connect to a keyboard_response item and
    // stopRecording right after response is received in a subsequent inline_javascript.
    setTimeout(function() {
        stopRecording()
    }, 2000)
})

// You can log these variables with a LOGGER:
// audio_filename, audio_duration_ms, audio_saved, mic_error, trial_tag

How to integrate with your sequence

• Before first trial: Keep OSWeb’s welcome screen enabled, or present a clickable “Start” button. After the click, call ensureMicPermissionThen to prompt the browser once, so later trials don’t re-prompt.
• During each trial:
  • Inline_javascript (prepare): define trial_tag and make sure helpers exist.
  • Inline_javascript (run): show stimulus and startRecording.
  • keyboard_response (GUI) or custom response: after response, in another inline_javascript (run), call stopRecording and then proceed to LOGGER.
  • LOGGER: log audio_filename, audio_duration_ms, response, response_time, correct, etc.

• On JATOS: The upload runs asynchronously but typically finishes before the next trial. If you want to be strict, you can await upload before continuing by sequencing logic around jatos.uploadResultFile’s promise.

Tips and caveats

• Browser differences: Chrome/Edge produce webm/opus, Firefox often ogg/opus. Safari supports MediaRecorder only in recent versions and may vary. Test on your target browsers.
• Clipping/silence detection: The snippet doesn’t do VAD (voice activity detection). If you need auto-stop on speech, consider integrating a lightweight VAD library client-side, or record a fixed window and trim server-side.
• File sizes: Short prompts (1–3 s) keep files small. For longer recordings, consider compressing on the fly (default opus is good).
• Privacy: Inform participants that the mic will be used, and do not start capturing before consent.
• Debugging: Run locally with OSWeb mode and open the browser console (F12) to spot permission or MIME errors.

Want me to adapt this to your task?

• Tell me your trial flow: when should recording start/stop (keypress, button, fixed window)?
• Are you running on JATOS/MindProbe?
• Which browsers do you need to support?

I can then tailor the inline_javascript items and LOGGER variables precisely to your experiment.

Workspace:

// Inline_javascript — PREPARE phase
// Note: Use non-strict mode; do not use let/const/var in OSWeb inline_javascript.

mic_stream = null
media_recorder = null
audio_chunks = []
recording_started_at = null
recording_stopped_at = null
audio_filename = ''
trial_tag = ''  // set per trial in your loop, e.g., "trial_003_item_A"

function ensureMicPermissionThen(cb) {
    if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
        console.warn('getUserMedia not supported')
        mic_error = 'unsupported'
        return
    }
    // Request mic if not yet acquired
    if (!mic_stream) {
        navigator.mediaDevices.getUserMedia({ audio: true }).then(function(stream) {
            mic_stream = stream
            cb()
        }).catch(function(err) {
            console.error('Mic permission error:', err)
            mic_error = 'denied'
        })
    } else {
        cb()
    }
}

function prepareRecorder() {
    if (!mic_stream) {
        console.warn('No mic stream available')
        return
    }
    audio_chunks = []
    try {
        media_recorder = new MediaRecorder(mic_stream)
    } catch (e) {
        console.error('MediaRecorder error:', e)
        mic_error = 'recorder_error'
        return
    }
    media_recorder.ondataavailable = function(e) {
        if (e.data && e.data.size > 0) {
            audio_chunks.push(e.data)
        }
    }
    media_recorder.onstop = function() {
        // Build blob and save
        blob = new Blob(audio_chunks, { type: media_recorder.mimeType || 'audio/webm' })
        // Compose filename using subject_nr/trial_tag/timestamp
        ts = Date.now()
        fn_base = 'audio_' + (typeof subject_nr !== 'undefined' ? subject_nr : 'NA') + '_' + (trial_tag || 'trial') + '_' + ts
        ext = (blob.type.includes('ogg') ? '.ogg' : '.webm')
        audio_filename = fn_base + ext

        // If in JATOS, upload blob; otherwise create a download link for local testing
        if (typeof jatos !== 'undefined' && jatos.uploadResultFile) {
            jatos.uploadResultFile(blob, audio_filename).then(function() {
                console.log('Uploaded audio:', audio_filename)
                audio_saved = 1
            }).catch(function(err) {
                console.error('Upload failed:', err)
                audio_saved = 0
            })
        } else {
            // Local/OSWeb test: provide a download URL (optional: store link for a debrief form)
            url = URL.createObjectURL(blob)
            audio_download_url = url
            audio_saved = 0
        }

        // Compute duration metadata
        if (recording_started_at && recording_stopped_at) {
            audio_duration_ms = recording_stopped_at - recording_started_at
        } else {
            audio_duration_ms = -1
        }
    }
}

function startRecording() {
    if (!media_recorder) return
    try {
        media_recorder.start()
        recording_started_at = Date.now()
    } catch(e) {
        console.error('startRecording error:', e)
    }
}

function stopRecording() {
    if (!media_recorder) return
    try {
        recording_stopped_at = Date.now()
        media_recorder.stop()
    } catch(e) {
        console.error('stopRecording error:', e)
    }
}

SigmundAI

Great question! Recording participants’ voices in a browser-based OpenSesame experiment (OSWeb) is possible using the Web Audio/MediaRecorder APIs via JavaScript. Below is a practical roadmap, including what works online, what to avoid, and a minimal working pattern you can adapt.

Key points up front

• Desktop vs. online: The Audio Low Latency plugins are excellent for desktop Python experiments, but they do not work online. For OSWeb, use inline_javascript with getUserMedia + MediaRecorder.
• Permissions: Audio capture requires a user gesture before the browser will allow the microphone (e.g., a click on a welcome screen or start button). Keep the OSWeb welcome screen enabled.
• File format: MediaRecorder typically produces webm/opus in Chrome/Edge and ogg/opus in Firefox. You’ll log a URL or upload the blob to JATOS as a result file.
• Data storage: When running on JATOS, save blobs via jatos.uploadResultFile. When running locally from OpenSesame, you can’t save blobs to disk automatically; you can expose a download link as a fallback.

Minimal workflow outline

1. Include an instruction/welcome screen to ensure a user gesture occurs before requesting the microphone.
2. In an inline_javascript “prepare” phase: set up variables and helper functions, but don’t request the mic yet.
3. In an inline_javascript “run” phase: after a user gesture, request the mic with navigator.mediaDevices.getUserMedia({ audio: true }), prepare MediaRecorder, and start recording right before your trial prompt.
4. Stop the recording when the participant responds (e.g., keypress) or after a timeout. Collect resulting audio chunks, assemble into a Blob.
5. Store:
   • On JATOS: upload with jatos.uploadResultFile and log the returned filename/path in your LOGGER.
   • Locally: createObjectURL for the blob and show a download link at the end (for testing).

1. Log metadata: onset time, stop time, duration, and any trial identifiers so you can link audio files to trials.

Example: OSWeb inline_javascript snippets I added a minimal template to the workspace that:

• Prepares mic/recorder helpers in Prepare.
• Starts and stops recording in Run.
• Uploads to JATOS if available; otherwise exposes a download link for testing.
• Logs the generated filename so your LOGGER can capture it.

// Inline_javascript — RUN phase
// Typical sequence around your stimulus/response collection.

trial_tag = 'trial_' + (typeof count_trial_sequence !== 'undefined' ? count_trial_sequence : 'NA')

// Step 1: Ensure mic permission (after user gesture earlier in the flow)
ensureMicPermissionThen(function() {
    // Step 2: Prepare the recorder
    prepareRecorder()

    // Step 3: Show your stimulus on a Canvas and START recording
    stim = Canvas()
    stim.text({ text: 'Say the word now!', x: 0, y: 0, center: true })
    stim.show()

    startRecording()

    // Step 4: Stop recording after a fixed window or on keypress
    // Example: stop after 2000 ms; alternatively, connect to a keyboard_response item and
    // stopRecording right after response is received in a subsequent inline_javascript.
    setTimeout(function() {
        stopRecording()
    }, 2000)
})

// You can log these variables with a LOGGER:
// audio_filename, audio_duration_ms, audio_saved, mic_error, trial_tag

How to integrate with your sequence

• Before first trial: Keep OSWeb’s welcome screen enabled, or present a clickable “Start” button. After the click, call ensureMicPermissionThen to prompt the browser once, so later trials don’t re-prompt.
• During each trial:
  • Inline_javascript (prepare): define trial_tag and make sure helpers exist.
  • Inline_javascript (run): show stimulus and startRecording.
  • keyboard_response (GUI) or custom response: after response, in another inline_javascript (run), call stopRecording and then proceed to LOGGER.
  • LOGGER: log audio_filename, audio_duration_ms, response, response_time, correct, etc.

• On JATOS: The upload runs asynchronously but typically finishes before the next trial. If you want to be strict, you can await upload before continuing by sequencing logic around jatos.uploadResultFile’s promise.

Tips and caveats

• Browser differences: Chrome/Edge produce webm/opus, Firefox often ogg/opus. Safari supports MediaRecorder only in recent versions and may vary. Test on your target browsers.
• Clipping/silence detection: The snippet doesn’t do VAD (voice activity detection). If you need auto-stop on speech, consider integrating a lightweight VAD library client-side, or record a fixed window and trim server-side.
• File sizes: Short prompts (1–3 s) keep files small. For longer recordings, consider compressing on the fly (default opus is good).
• Privacy: Inform participants that the mic will be used, and do not start capturing before consent.
• Debugging: Run locally with OSWeb mode and open the browser console (F12) to spot permission or MIME errors.

Want me to adapt this to your task?

• Tell me your trial flow: when should recording start/stop (keypress, button, fixed window)?
• Are you running on JATOS/MindProbe?
• Which browsers do you need to support?

I can then tailor the inline_javascript items and LOGGER variables precisely to your experiment.

Workspace:

// Inline_javascript — PREPARE phase
// Note: Use non-strict mode; do not use let/const/var in OSWeb inline_javascript.

mic_stream = null
media_recorder = null
audio_chunks = []
recording_started_at = null
recording_stopped_at = null
audio_filename = ''
trial_tag = ''  // set per trial in your loop, e.g., "trial_003_item_A"

function ensureMicPermissionThen(cb) {
    if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
        console.warn('getUserMedia not supported')
        mic_error = 'unsupported'
        return
    }
    // Request mic if not yet acquired
    if (!mic_stream) {
        navigator.mediaDevices.getUserMedia({ audio: true }).then(function(stream) {
            mic_stream = stream
            cb()
        }).catch(function(err) {
            console.error('Mic permission error:', err)
            mic_error = 'denied'
        })
    } else {
        cb()
    }
}

function prepareRecorder() {
    if (!mic_stream) {
        console.warn('No mic stream available')
        return
    }
    audio_chunks = []
    try {
        media_recorder = new MediaRecorder(mic_stream)
    } catch (e) {
        console.error('MediaRecorder error:', e)
        mic_error = 'recorder_error'
        return
    }
    media_recorder.ondataavailable = function(e) {
        if (e.data && e.data.size > 0) {
            audio_chunks.push(e.data)
        }
    }
    media_recorder.onstop = function() {
        // Build blob and save
        blob = new Blob(audio_chunks, { type: media_recorder.mimeType || 'audio/webm' })
        // Compose filename using subject_nr/trial_tag/timestamp
        ts = Date.now()
        fn_base = 'audio_' + (typeof subject_nr !== 'undefined' ? subject_nr : 'NA') + '_' + (trial_tag || 'trial') + '_' + ts
        ext = (blob.type.includes('ogg') ? '.ogg' : '.webm')
        audio_filename = fn_base + ext

        // If in JATOS, upload blob; otherwise create a download link for local testing
        if (typeof jatos !== 'undefined' && jatos.uploadResultFile) {
            jatos.uploadResultFile(blob, audio_filename).then(function() {
                console.log('Uploaded audio:', audio_filename)
                audio_saved = 1
            }).catch(function(err) {
                console.error('Upload failed:', err)
                audio_saved = 0
            })
        } else {
            // Local/OSWeb test: provide a download URL (optional: store link for a debrief form)
            url = URL.createObjectURL(blob)
            audio_download_url = url
            audio_saved = 0
        }

        // Compute duration metadata
        if (recording_started_at && recording_stopped_at) {
            audio_duration_ms = recording_stopped_at - recording_started_at
        } else {
            audio_duration_ms = -1
        }
    }
}

function startRecording() {
    if (!media_recorder) return
    try {
        media_recorder.start()
        recording_started_at = Date.now()
    } catch(e) {
        console.error('startRecording error:', e)
    }
}

function stopRecording() {
    if (!media_recorder) return
    try {
        recording_stopped_at = Date.now()
        media_recorder.stop()
    } catch(e) {
        console.error('stopRecording error:', e)
    }
}

sebastiaan

@Kruthi This is possible, but it's a bit involved if you're not familiar with JavaScript. I asked @SigmundAI to chip in to give you the general picture.