Messaging SDK

Audio

Receive raw audio bytes, run your own STT, and send back a transcript
View as Markdown

Audio flows agent-side as raw bytes. The messaging system does no STT, transcoding, or VAD.

StepDirectionMessageNotes
1Sidecar → agentAgentResponse.audio_configFormat (encoding, sample rate, channels).
2Sidecar → agentAgentResponse.audio_chunkRaw bytes. done marks end-of-utterance.
3Agent → sidecarAgentResponse.transcriptSTT result; platform replaces the placeholder.

AudioStreamConfig

FieldTypeRequiredNotes
encodingstringyesOne of the AudioEncoding values below.
sampleRatenumberyesHz: 8000 (telephony), 16000 (speech), 48000 (browser).
channelsnumberyes1 = mono (speech default), 2 = stereo.
languagestringnoBCP-47 hint for STT (e.g. en-US).
conversationIdstringyesLinks audio to a conversation.
sourcestringnoOrigin: browser, twilio, vonage, mobile, upload.
userIdstringnoSpeaking user’s identity.

AudioChunk

FieldTypeRequiredNotes
dataBuffer | Uint8ArrayyesRaw audio bytes. Empty when done: true.
sequencenumbernoMonotonic ordering counter.
donebooleannotrue = end of segment, run STT now.

AudioEncoding

ValueUse
LINEAR16PCM signed 16-bit LE. Universal baseline.
MULAWG.711 mu-law. Twilio / telephony (8 kHz).
OPUSRaw Opus frames. Low-latency codec.
MP3MP3. Batch uploads, pre-recorded.
WEBM_OPUSWebM/Opus. Browser MediaRecorder default.
OGG_OPUSOGG/Opus. Firefox MediaRecorder.
FLACFLAC lossless. High-quality uploads.
AACAAC. iOS native recording.

Use audioAsReadable() to feed Mastra’s voice.listen():

import { audioEncodingToFiletype } from '@astropods/messaging';
conversation.on('audioConfig', async (config) => {
  const audioStream = conversation.audioAsReadable();
  const filetype = audioEncodingToFiletype(config.encoding);  // 'webm' for WEBM_OPUS
  const transcript = await agent.voice.listen(audioStream, { filetype });
  conversation.sendTranscript(config.conversationId, transcript);
});

Next steps