A
Overview
grok-voice/realtime is maintained by xai. grok-voice/realtime is an xAI Grok audio-to-audio model exposed through fal as fal-ai/grok-voice. It accepts a user speech recording, applies optional system instructions and tool configuration, and returns the agent’s spoken response as an audio file plus transcript text and duration. It is best suited to voice assistants, phone agents, and interactive voice systems; the supplied schema describes file-based audio input and output, while the model description also identifies bidirectional WebSocket streaming as the intended real-time architecture.Best use cases
- Voice assistants: provide an audio recording and optional persona or conversation-context instructions, then receive spoken output and its transcript.
- Phone agents: use the speech-response workflow for conversational call experiences, subject to your own telephony integration.
- Interactive voice systems: configure web search, X search, or MCP servers when the agent needs external information or tools.
- Prototypes for real-time voice applications: the model description specifically targets bidirectional audio and text streaming over WebSocket.
Limitations
- The confirmed input schema accepts an audio URL rather than documenting client-side microphone frames or a complete WebSocket message protocol.
- Input audio is converted to 16-bit, 24 kHz, mono PCM before being sent to Grok.
- Audio input is limited to 10 minutes and 50 MB; common audio formats are supported, but the schema does not enumerate them.
- The prompt is optional and limited to 20,000 characters. If omitted, Grok’s default persona is used.
- Tool use is opt-in for web search and X search. MCP connections can forward authorization values and headers to xAI, so configure them carefully.
- The supplied materials do not specify pricing, latency, availability, retention, safety behavior, or a license.
How it compares
grok-text-to-speech — grok-text-to-speech is a text-to-speech model with five voices, 20 languages, expressive speech tags, and MP3, WAV, and telephony output. Choose it when you already have text and need speech controls; choose grok-voice/realtime when the input is user speech and the agent must generate a conversational response.
grok-speech-to-text — grok-speech-to-text focuses on transcription and supports 25 languages, word-level timestamps, speaker diarization, multichannel audio, and files up to 500 MB. Choose it for transcription analysis; choose grok-voice/realtime for an agent that listens and replies with speech.
grok-imagine-video/text-to-video — grok-imagine-video/text-to-video generates videos with audio from text. It is not a voice-conversation endpoint, so grok-voice/realtime is the more direct choice for spoken-agent interactions.
grok-imagine-video/image-to-video — grok-imagine-video/image-to-video generates videos with audio from images. It targets visual content generation rather than speech-driven dialogue.
vibevoice — vibevoice is Microsoft’s text-to-speech model for long-form speech from text with sample voices. It is a speech-generation alternative, while grok-voice/realtime accepts speech and returns an agent response with audio and transcript.
Technical specifications
Creator: xAI. Platform: fal. Category tag: Audio-to-Audio. Endpoint ID: fal-ai/grok-voice. The model description specifies bidirectional audio and text streaming via WebSocket. The confirmed file-input path converts incoming audio to 16-bit, 24 kHz mono PCM. Maximum input duration is 10 minutes and maximum file size is 50 MB.
Model inputs and outputs
Inputs: required audio_url, a string containing the user-speech audio URL. Supported common audio formats are converted to 16-bit, 24 kHz mono PCM; maximum duration is 10 minutes and maximum size is 50 MB. Optional prompt accepts a string or null and has a maximum length of 20,000 characters. Optional tools contains x_search, web_search, and mcp_servers configurations. x_search is disabled by default and can filter by ISO dates and up to 20 allowed or excluded X handles; image and video understanding are optional. web_search is disabled by default and supports optional image understanding, location, and up to five allowed or excluded domains. MCP entries are enabled by default, require a server_url and server_label, and can include descriptions, authorization, headers, and allowed tools. Outputs: required audio object and duration number; audio includes a required downloadable url plus optional content_type, file_name, and file_size. duration is generated-audio duration in seconds. text is returned as a string transcript of the spoken response, defaulting to an empty string when Grok provides none.
Getting started
Code:
import { fal } from "@fal-ai/client";
const result = await fal.subscribe("fal-ai/grok-voice", {
input: {
audio_url: "https://v3b.fal.media/files/b/0a8dd5a2/hS140ygvRuxn-eY_qPhlv_assets_test_input_service.wav",
prompt: "You are a friendly assistant. Answer briefly and concretely."
},
logs: true,
onQueueUpdate: (update) => {
if (update.status === "IN_PROGRESS") {
update.logs?.forEach((log) => console.log(log.message));
}
});
console.log(result.data.text);
console.log(result.data.audio.url);
console.log(result.data.duration);Related research
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming — Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming is relevant because its supplied description concerns models that listen, think, and speak concurrently for real-time interactive conversations. That connection matches this model’s stated bidirectional audio-and-text streaming use case, but the supplied materials do not establish that grok-voice/realtime uses Mini-Omni or its architecture.
Frequently asked questions
Q: What does this model return?
A: It returns a generated audio file, the generated-audio duration in seconds, and a text transcript when Grok provides one.
Q: Is the prompt required?
A: No. The prompt is optional. If omitted, the agent uses Grok’s default persona.
Q: What audio input limits apply?
A: The audio URL must reference audio no larger than 50 MB and no longer than 10 minutes. The audio is converted to 16-bit, 24 kHz mono PCM.
Q: Does the model support external tools?
A: Yes. The schema supports optional web search, X search, and MCP server configurations. Web search and X search are disabled by default; MCP entries default to enabled.
Q: Can I restrict search results?
A: Yes. X search supports date bounds and up to 20 allowed or excluded handles. Web search supports up to five allowed or excluded domains and optional country, city, region, and timezone localization.
Q: Does the endpoint support real-time streaming?
A: The model description says it streams audio and text bidirectionally via WebSocket. The confirmed input schema also documents a file-based audio URL workflow, but it does not provide the WebSocket message schema.
Q: Can I use it for transcription only?
A: The endpoint is designed to have the Grok agent listen and reply with speech. For transcription-focused work, grok-speech-to-text is the related model specifically described for transcription.
This is a simplified guide to an AI model called grok-voice/realtime maintained by xai. If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter.