Implement text analysis solutions
Speech Solutions for Agents
ImportantConvert speech to text and text to speech for agents, add voice as an agent modality with Voice Live and custom speech, reason over audio with audio-capable models, and translate speech with Azure Speech and language models.
Aligned to the AI-103 study guide with skills measured as of April 16, 2026; guide and product behavior verified October 12, 2026.
Why this matters
Voice agents must hear accurately, respond quickly, and sound natural. AI-103 tests which transcription mode fits a latency or volume need, when custom speech improves recognition, how Voice Live and realtime models remove pipeline plumbing, and how speech translation differs from text translation.
Must Know
- Azure Speech in Foundry Tools offers real-time speech to text for streaming audio, fast transcription for synchronous file transcription faster than real time, and batch transcription for large asynchronous volumes stored in Azure Storage.
- Fast transcription accepts files under five hours and 500 MB, supports diarization, and can use a phrase list; batch transcription writes results to storage and can take hours at peak.
- Custom speech improves recognition for domain vocabulary and acoustic conditions by training on text and audio with transcripts. A custom model needs a deployed endpoint for real-time use, while batch transcription can use it without one.
- Text to speech uses neural voices, and SSML controls voice, speaking style, rate, pitch, pauses, and pronunciation. Custom voice is a separate, limited-access capability for a branded voice.
- Voice Live is a fully managed speech-to-speech API that combines recognition, a language model, and synthesis over a WebSocket, with noise suppression, echo cancellation, interruption handling, end-of-turn detection, and support for custom speech and custom voice.
- Realtime audio models (gpt-realtime family) process speech input and output directly with low latency over WebRTC, WebSocket, or SIP, and voice-based prompt agents (preview) in Foundry Agent Service connect through Voice Live.
- Speech translation in Azure Speech translates spoken audio in real time into text or synthesized speech in target languages, and LLM speech (preview) can transcribe or translate prerecorded audio with prompting.
- Audio-capable models and Content Understanding audio analyzers (such as prebuilt-callCenter) turn recordings into transcripts, summaries, sentiment, and extracted fields for reasoning.
Compare and Distinguish
- Real-time versus fast versus batch transcription: live streaming versus synchronous file results versus large asynchronous jobs.
- Custom speech versus custom voice: better recognition of input speech versus a branded synthesized output voice.
- Cascaded pipeline versus speech-to-speech: separate recognition, model, and synthesis stages with more control versus one realtime model with lower latency.
- Speech translation versus text translation: audio in with text or speech out versus text in and text out.
Scenario examples
- Scenario: A pharmacy voice agent keeps mishearing drug names. Think: a custom speech model trained with domain text, or a phrase list for a quick improvement.
- Scenario: A call center wants yesterday's 40,000 recordings transcribed overnight into storage. Think: batch transcription.
- Scenario: A kiosk needs a natural spoken assistant with barge-in and noise suppression without building the audio pipeline. Think: Voice Live or a voice-based prompt agent.
Exam traps
- Batch transcription is not a real-time option; it suits large backlogs, not live calls.
- Custom voice does not improve recognition accuracy; custom speech does.
- A custom speech model used for real-time recognition must be deployed to an endpoint.
- Foundry guardrails on a chat deployment do not screen Speech transcription; Speech has its own profanity filtering, and transcripts still need a separate check for other harms before reuse.
Key takeaways
- Choose the transcription mode from latency and volume.
- Improve recognition with custom speech and phrase lists, and shape output with SSML.
- Use Voice Live or realtime models for low-latency voice agents.
How it works
- Voice Live accepts audio frames, detects the end of the user's turn, sends the recognized input to the model, and streams synthesized audio back.
- Batch transcription reads audio from storage URLs and writes JSON results back to a container.
Objects and administrative surfaces
- Speech SDK, Speech CLI, and the fast and batch transcription REST APIs.
- Custom speech projects with training data, test data, and endpoints.
- Voice Live WebSocket sessions and voice-based prompt agents.
- Realtime API sessions with turn detection settings.
When to use it
- Use fast transcription for single files that need results now.
- Use a cascaded pipeline when you need a specific text model, explicit transcripts, or custom voices at each step.
Security and governance implications
- Obtain consent and follow limited-access terms for custom and personal voices.
- Protect stored recordings and transcripts with storage access controls.
Troubleshooting signals
- If a voice agent cuts users off, tune end-of-turn detection; if it talks over users, enable interruption handling.
- If recognition is poor only in a noisy environment, train custom speech with audio from that environment.
More detail
- Implement speech to text and text to speech for agentic interactions.
- Integrate speech as an agent modality, including custom speech models.
- Enable multimodal reasoning from audio inputs.
- Translate speech into other languages with language models and Foundry Tools.
Ready for the quiz?
- Which transcription mode returns results synchronously for a recorded file?
- What training data does custom speech use?
- Which Voice Live features handle noisy rooms and interruptions?
- How does speech translation differ from transcribing and then translating text?
Related objectives
- D4.2.S1 — Implement workflows to convert speech to text and text to speech for agentic interactions
- D4.2.S2 — Integrate speech as an agent modality, including custom speech models
- D4.2.S3 — Enable multimodal reasoning from audio inputs
- D4.2.S4 — Translate speech into other languages by using language models and Foundry Tools