Documentation

Overview

Most teams build voice agents by renting every stage from an API — Deepgram for STT, OpenAI/Groq for the LLM, ElevenLabs/Cartesia for TTS. That works, but it means per-minute bills, a network hop (and a third party's queue) on every turn, and audio leaving your infrastructure. fusion-runtime runs all three stages in one process with open-weight models, so none of that is required in the default path.

Quickstart

Requires Python 3.11+.

git clone https://github.com/<org>/fusion-runtime.git
cd fusion-runtime
pip install -e ".[talk]"

frun models pull          # ~0.9 GB: Whisper tiny, Qwen2.5 0.5B, Kokoro, Silero VAD
frun doctor                # checks libraries, GPU, models and audio, and says how to fix problemsCopy

Start the server:

frun upCopy

Then open http://localhost:8000 and click Talk. That page is served by the runtime itself — no build step, nothing else to install. Talk over the reply to interrupt it.

Or talk from a second terminal instead:

frun talkCopy

On macOS, allow microphone access for your browser (or, for frun talk, your terminal app) under System Settings → Privacy & Security → Microphone. With headphones, frun talk --no-aec turns echo cancellation off.

Browser client

The runtime serves the browser client it uses itself, so putting a voice agent on your own page takes two script tags and no build step:

<script src="https://your-server/fusion-runtime.js"></script>
<button id="talk"></button>
<script>FusionRuntime.attach({ button: "#talk" });</script>Copy

With no url, it connects back to the server the script came from. Everything else is optional:

const session = FusionRuntime.attach({ button: "#talk", token: sessionToken });

session.on("transcript", msg => { if (msg.is_final) show("You: " + msg.text); });
session.on("response",   msg => { if (msg.is_final) show(msg.text); });
session.on("trace",      msg => console.log(msg.summary));   // TTFA, tokens/s, per-stage times
session.on("error",      e   => show(e.message));Copy

FusionRuntime.connect(options) returns the same session without binding a button, for a page with its own controls (session.start(), session.stop(), session.interrupt()).

The client uses the browser's own echo canceller, resamples the microphone in an AudioWorklet so a busy page can't stutter the audio, and schedules replies slightly ahead of real time so network jitter doesn't leave gaps. Interruptions are decided on the server, which hears clean audio: when it says the caller interrupted, queued audio is dropped immediately — including the tail of a sentence already synthesized.

In your own Python

frun up is a thin wrapper around a pipeline you can run yourself. Use the library directly when the voice turn belongs inside something you already run — a queue worker, a test, a batch job over recorded calls — and you don't want a server in the middle.

import asyncio, wave
from fusion_runtime import Agent, LLM, STT, TTS, PipelineOrchestrator, run_single_turn

agent = Agent(
    prompt="You are the order line for ShopKart. Answer in one short sentence.",
    stt=STT("whisper-tiny.en"),
    llm=LLM("qwen2.5-0.5b-q4", max_tokens=60),
    tts=TTS("kokoro-v1.0", voice="af_heart"),
)

async def main():
    orchestrator = PipelineOrchestrator(agent.config())
    await orchestrator.initialize()          # loads three models; reuse it across turns
    with wave.open("caller.wav", "rb") as w:
        audio = w.readframes(w.getnframes())
    reply = await run_single_turn(orchestrator, audio, system_prompt=agent.prompt)
    print(f"{len(reply) / 2 / 24000:.2f}s of speech")   # 24 kHz mono 16-bit PCM
    await orchestrator.shutdown()

asyncio.run(main())Copy

What the pieces are

Audio in is 16 kHz mono 16-bit PCM; audio out is the same format at 24 kHz, which is what Kokoro produces. examples/sdk_example.py in the repository runs both paths against a real recording and writes the reply to a WAV file you can listen to.

Authentication

Generate a key and put it where the server runs:

frun key new
FUSION_ACCEPTED_KEYS=web:frun_kR7m...        # .env, or the environmentCopy

Without keys the server answers on localhost only, and frun up --host 0.0.0.0 refuses to start. There is no flag to switch that off: a runtime reachable from elsewhere with no authentication is someone else's GPU, on your bill.

Two kinds of client, because only one of them can keep a secret:

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"
# {"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}Copy

A token works once and expires in about a minute, so a leaked URL, screenshot or log line is worthless by the time anyone reads it. frun token prints a console URL with one in it for your own machine.

Several keys

One variable, comma separated — not one variable per key, which would simply overwrite itself. The label in front is optional and grants nothing; it exists so frun keys list and your logs can say which key is busy or failing without the key itself ever appearing.

FUSION_ACCEPTED_KEYS=web:frun_kR7m...,mobile:frun_9xQ2...,partner:frun_Lm4v...Copy

With more than one key configured, the per-key session cap changes on its own: instead of "the whole server", each key gets all but one slot, so no single key can lock the others out. Set FUSION_MAX_SESSIONS_PER_KEY for a different split.

Past a handful, put them in a file — one per line, # comments allowed — which also makes them reloadable:

FUSION_ACCEPTED_KEYS_FILE=/etc/fusion/keysCopy

Rotating and revoking

The key list takes several, so rotation is: add the new key, move clients across, drop the old one. Point FUSION_ACCEPTED_KEYS_FILE at a file and kill -HUP re-reads it without a restart — which matters on a GPU, where restarting reloads the models. Removing a key is complete: its tokens are dropped and its conversations are closed.

Limits

Authentication says who may use the server; these say how much of it one caller may take.

Origins and proxies

FUSION_ALLOWED_ORIGINS lists the websites whose pages may open a socket. Unset means only pages this server itself serves: browsers do not stop one site connecting to another, so the server checks. Clients that aren't browsers send no Origin and are unaffected.

Behind a proxy that terminates TLS — RunPod's, nginx, Cloudflare — name it with FUSION_TRUSTED_PROXY=10.0.0.0/8. Until you do, its X-Forwarded-* headers are ignored, because anyone who can reach the port can write those headers. The forwarded address is used for counting only, never to decide who may in.

The frun CLI

fusion-runtime works as an alias for frun. If the command isn't on your PATH, use python3 -m fusion_runtime.cli.

Models

Every download is pinned to an exact Hugging Face commit and checked by file size.

Where models are stored, first match wins: FUSION_MODEL_DIR if set, a models/ folder next to the source (source checkouts), or ~/.cache/fusion-runtime/models. Silero VAD is the exception — it's cached by PyTorch in ~/.cache/torch/hub.

Using your own models

Those five are the defaults, not the limit. A model reference can be a catalog ID, a path on disk, a Hugging Face repo as hf:owner/repo, or the URL of an OpenAI-compatible server. Nothing is detected from a model's name — the runtime is chosen from the file format, and the model's own metadata does the rest.

frun models pull hf:Systran/faster-whisper-smallCopy

Then name it in the agent file. Download first — resolving a model never touches the network:

agent = Agent(
    stt=STT("hf:Systran/faster-whisper-small"),
    llm=LLM("hf:bartowski/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf"),
    tts=TTS("kokoro-v1.0", voice="af_heart"),
)Copy

A repo holding several quantizations needs the file named, as above. Safetensors models aren't run in process — serve one with vLLM and give the agent its URL.

How it works

Everything runs in one Python process today. The local client removes the bot's own voice from the microphone (echo cancellation), so the server hears clean audio and can decide when you're actually interrupting it versus hearing its own playback.

Configuration

Choose a profile with frun up --config (or FUSION_CONFIG if you start the server another way):

frun up --config productionCopy

LLM from any OpenAI-compatible endpoint

Speech stays local; the LLM comes from vLLM, llama-server, Ollama or a hosted API:

frun up --llm-url http://localhost:8080/v1 --llm-model my-model

export GROQ_API_KEY=...
frun up --llm-url https://api.groq.com/openai/v1 --llm-model <model> --llm-api-key-env GROQ_API_KEYCopy

API keys are never accepted in config — only the name of the environment variable holding one. frun doctor checks that the key is set, the endpoint answers, and it serves the model.

Turn detection: when the agent answers

By default the agent answers after 500 ms of silence, and stops speaking once the caller has talked over it for 300 ms:

frun up --turn-wait-ms 800          # more patient with pauses mid-sentence
frun up --interrupt-after-ms 500    # ignore coughs and "mm-hm" in noisy placesCopy

Speaking again right after a pause continues the same turn: if the caller resumes within 1.5s of their turn ending, both parts reach the LLM as one message. A turn-detector model can plug in to shorten or stretch the wait based on how likely the caller is done — silence always confirms the end of a turn, and a detector that's slow or fails just leaves the default wait.

frun up --turn-detector my_package.turns:MyDetectorCopy

Languages

Set what callers speak with STTConfig(language="hi") (omit it to detect per turn; English-only Whisper models like tiny.en refuse other languages at startup). Kokoro speaks its voice's language, or set TTSConfig(language=...). Reply text is split into sentences for speech correctly across scripts. The catalog currently ships English models only.

Server API

Seven endpoints, and it's clearest by who calls them.

A browser

The socket takes raw 16 kHz mono PCM as binary messages, and two JSON control messages: {"type":"playback","playing":true|false} while the agent is audible, and {"type":"interrupt"} from a client that stops itself. It sends 24 kHz PCM back as binary, and these as JSON:

Your backend

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"Copy

{"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}Copy

One call per visitor, on page load. The token works once and expires in about a minute, and a connected page is handed its next one over the socket — so reconnecting costs no round trip. This is the only endpoint a page's backend needs.

A script, or a test

curl -X POST https://your-server/v1/voice/chat -H "Authorization: Bearer $FUSION_API_KEY" \
  -d '{"audio_base64":"..."}'Copy

{"audio_base64":"...", "transcript":"...", "response_text":"...", "latency_ms":612,
 "turns":[{"user":"...","agent":"...","outcome":"completed","metrics":{...}}]}Copy

The whole conversation in one request, no socket. Useful for regression tests, for comparing models on the same audio, and for anything that isn't a live microphone.

Monitoring

Everything except /health, / and /fusion-runtime.js needs a key — or, on the WebSocket, a session token. See Authentication.

On an NVIDIA GPU

torch on Linux already carries CUDA — its PyPI wheels bundle the NVIDIA libraries — so a normal install is the GPU one. Two packages need help, and both fail quietly if they don't get it.

pip install -e ".[cuda]"
CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86" pip install llama-cpp-pythonCopy

llama-cpp-python is published as source only and builds without CUDA unless told otherwise; the prebuilt CUDA wheels people link to stopped at 0.2.66. Set CMAKE_CUDA_ARCHITECTURES to your card — 86 for a 3090 or A10, 89 for an L4 or 4090, 80 for an A100 — or it compiles kernels for every architecture and takes about an hour.

pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/Copy

export LD_LIBRARY_PATH="$(python -c "import site,glob,os; p=site.getsitepackages()[0]; print(':'.join(sorted(glob.glob(os.path.join(p,'nvidia','*','lib')))))"):$LD_LIBRARY_PATH"Copy

Check both before trusting a measurement: the log must show vad model.loaded and no Failed to create CUDAExecutionProvider. Either one missing means you are measuring a CPU.

Deploying

Any machine with an NVIDIA GPU runs this; there is nothing provider-specific in the runtime. The docs are written against RunPod because a pod has no request timeout and its proxy gives you https:// and wss:// without handling a certificate — which browsers require before granting a microphone.

With Docker

docker run --env-file .env -p 8000:8000 -v /models:/models fusion-runtimeCopy

--env-file reads the file on the host and sets real environment variables in the container. Never COPY .env into an image: that writes your keys into a layer anyone who pulls it can read, even if a later layer deletes the file.

Without Docker

[Service]
EnvironmentFile=/etc/fusion/runtime.env
ExecStart=/opt/fusion/bin/frun up /etc/fusion/agent.py --host 0.0.0.0Copy

It is a Python package, so a GPU VM with a systemd unit works as well as a container. Keep that environment file root:root and mode 0600, written at boot from Secrets Manager, SSM Parameter Store or equivalent — not baked into an image.

On a cloud that doesn't terminate TLS for you, put a load balancer or reverse proxy in front with a certificate, and set FUSION_TRUSTED_PROXY to its address range — until you do, its forwarded headers are ignored and token minting refuses, because a token in a URL over plain HTTP is readable by every hop.

Models belong on a volume rather than in the image: a rebuild then doesn't re-download several gigabytes, and two machines can share one copy. Point FUSION_MODEL_DIR at it.

Observability

Every stage emits structured events with a timestamp, session id, turn id, stage, model and duration, so a deployed agent never runs blind.

20:47:49.441  95b134d2 t1   vad      speech_start       audio_offset_ms=192 probability=0.88
20:47:52.162  95b134d2 t1   turn     end_detected       reason=silence detector=silence threshold_ms=500 wait_ms=569
20:47:52.334  95b134d2 t1   llm      first_token        171ms runtime=llama_cpp model=qwen2.5-0.5b-instruct-q4_k_m.gguf
20:47:52.670  95b134d2 t1   tts      first_chunk        326ms audio_ms=2525
20:47:52.671  95b134d2 t1   audio    first_sent         response_ms=509 ttfa_ms=778Copy

Logs: frun up --log-format pretty|json, --log-level debug|info|warning|error. Errors include a stable code, whether retrying can help, a suggested fix, and the stack trace (server logs only).

Private by default: logs record text lengths, not what people said. --log-content includes transcripts and replies. API keys are always redacted.

Per turn: time to first audio, end-of-turn wait, transcription delay, STT time, LLM time-to-first-token and tokens/s, TTS time-to-first-audio and real-time factor, playback start, interruptions and how fast generation stopped.

Development

pip install -e ".[dev,talk]"     # or: uv sync --extra dev --extra talk
pytest tests/Copy

Tested on Python 3.11, 3.12 and 3.13.

Package layout: cli/ (frun commands), catalog/ (model catalog, installs, downloads), config.py (settings, profiles, model directory), server.py (FastAPI + WebSocket), contract/ (the interface every model runtime implements), resolver.py (model reference → runtime), registry.py (runtime names and plugins → classes), runtimes/ (one adapter per engine: llama_cpp, ctranslate2, onnx, openai_http), engine/ (orchestrator, scheduler, streaming, barge-in), web/ (the console and the browser client), vad/, telemetry/, testing/ (conformance kit and fakes), audio/ (echo cancellation, duplex).

License

Apache-2.0. Embed fusion-runtime in a commercial product, rebrand it, and ship it closed. What Apache-2.0 asks in return is that you keep the copyright notice and the NOTICE file in what you distribute, and that you don't use the project's name in a way that implies it endorses your product. There is no CLA and no copyright assignment for contributors.

The models pulled by default are permissive too — Whisper MIT, Silero VAD MIT, Qwen2.5 Apache-2.0, Kokoro Apache-2.0 — so the whole default path is clear for commercial use, weights included. A model you point it at yourself carries its own licence, and some open-weight models restrict commercial use or rebranding. Check that one before you ship it.