Voice Agents in Microsoft Foundry: Inside the Realtime Speech-to-Speech Architecture
- Authors
- Name
- Nino
- Occupation
- Senior Tech Editor
Every chat-based agent you have built so far has relied on the request/response boundary. A user sends a message, the agent processes it, perhaps calls a tool, and returns an answer. Voice, however, breaks this contract entirely. In a natural conversation, users interrupt, change their minds, and expect responses within milliseconds. Silence longer than 300ms is often perceived as a system failure. n1n.ai recognizes that voice is not just chat with an audio codec—it is a distinct runtime model.
The Voice Agent Taxonomy in Foundry
Microsoft Foundry introduces Voice Agents as a first-class citizen within the project_client.agents management surface. Unlike traditional text agents, these are designed for persistent WebSocket connections. By leveraging n1n.ai for your API management, you ensure that your infrastructure remains stable as you scale these complex voice sessions.
The Deferred-Response Pattern
In text agents, tool calling is sequential. In a real-time voice session, the model is continuously receiving events. If you attempt to send a tool output while a function-call response is still in flight, the service will return a concurrent-response error. You must implement a deferred-response pattern:
- Receive
response.function_call_arguments.done. - Collect the arguments but do not send the result yet.
- Wait for
response.done. - Submit the tool output and request the next response.
This ensures that you are not racing the server’s internal bookkeeping.
Server-Side VAD and Turn Detection
Foundry handles Voice Activity Detection (VAD) server-side. Key parameters include:
silence_duration_ms: The threshold for when the system considers a turn "done."prefix_padding_ms: Captures audio slightly before detection to prevent clipping.
Centralizing this in the platform ensures consistent behavior regardless of client-side implementation. For developers, this means fewer "reinvented wheels" and more time focusing on business logic.
Tool Execution: Client vs. Service
Architecting your tools requires a clear trust boundary:
| Tool Type | Executes Where | Best For |
|---|---|---|
| Function | Client Process | Local state, UI, or auth-dependent actions |
| MCP/Toolbox | Foundry Service | Backend data, enterprise systems, high-reliability logic |
| System | Platform-native | Call control (end, transfer, mute) |
Using n1n.ai to aggregate your LLM providers allows you to maintain consistent tool definitions across these boundaries, ensuring that your backend services remain decoupled from the telephony bridge.
Production Considerations
When deploying, remember that store=True creates a privacy-sensitive data trail. Always implement retention policies and consent flows. Furthermore, test your agents against adversarial speech—prompt injection is just as dangerous in a voice stream as it is in a text prompt.
Conclusion
Voice agents require a paradigm shift in how you handle state and latency. By moving away from stateless HTTP requests toward managed WebSocket sessions, you can build systems that feel truly conversational.
Get a free API key at n1n.ai