Documentation

Overview

Most teams build voice agents by renting every stage from an API — Deepgram for STT, OpenAI/Groq for the LLM, ElevenLabs/Cartesia for TTS. That works, but it means per-minute bills, a network hop (and a third party's queue) on every turn, and audio leaving your infrastructure. fusion-runtime runs all three stages on your own machine with open-weight models, so none of that is required in the default path.

Quickstart

Requires Python 3.11 to 3.13.

pip install fusion-runtimeCopy

The first install can sit quiet for a few minutes: where there's no ready-made llama.cpp for your machine, pip builds it. That's normal.

An agent is one file. This is the whole thing:

# agent.py
from fusion_runtime import Agent, LLM, STT, TTS, Turns

agent = Agent(
    name="shopkart-orders",
    prompt="You are the order line for ShopKart. Keep answers to one short sentence.",
    greeting="Hi, this is ShopKart. How can I help with your order?",   # said as soon as a caller connects
    stt=STT("whisper-tiny.en"),          # or "whisper-small" for better accuracy
    llm=LLM("qwen2.5-0.5b-q4", max_tokens=256),
    tts=TTS("kokoro-v1.0", voice="af_heart"),
    turns=Turns(wait_ms=500, interrupt_after_ms=300),
)Copy

frun models pull agent.py     # exactly the models it names, nothing else
frun up agent.py              # add --reload to restart on every editCopy

Then talk to it from a second terminal:

pip install "fusion-runtime[talk]"
frun talkCopy

With a greeting, the agent speaks first, the moment a caller connects; without one it waits for them. It's spoken like any reply: talking over it stops it, and the model is told it has already said hello. FUSION_GREETING changes it without editing the file. greeting_interruptible=False makes sure it's heard in full (say, "this call may be recorded"): what the caller says over it is still heard, and answered once it ends.

That is the whole loop: one file, two commands, a conversation. frun up with no file runs a default agent if you just want to hear it work, and frun doctor checks libraries, GPU, models and audio and says how to fix what it finds.

In a browser instead

The runtime serves a browser client at http://localhost:8000 — the same one you would embed in your own page.

With no keys configured, open it and click Talk. With keys configured (FUSION_ACCEPTED_KEYS), a page can't hold a secret, so it needs a short-lived session token:

frun token        # prints a URL with a token in it — open thatCopy

Tokens are single-use and expire in about a minute. The page is handed its next one over the socket it already has, so a conversation keeps going without asking again. If you open the bare URL on a server with keys, the connection closes and the page says the token wasn't accepted.

On macOS, allow microphone access for your browser (or, for frun talk, your terminal app) under System Settings → Privacy & Security → Microphone. With headphones, frun talk --no-aec turns echo cancellation off.

Browser client

The runtime serves the browser client it uses itself, so putting a voice agent on your own page takes two script tags and no build step:

<script src="https://your-server/fusion-runtime.js"></script>
<button id="talk"></button>
<script>FusionRuntime.attach({ button: "#talk" });</script>Copy

With no url, it connects back to the server the script came from. Everything else is optional:

const session = FusionRuntime.attach({ button: "#talk", token: sessionToken });

session.on("transcript", msg => { if (msg.is_final) show("You: " + msg.text); });
session.on("response",   msg => { if (msg.is_final) show(msg.text); });
session.on("trace",      msg => console.log(msg.summary));   // TTFA, tokens/s, per-stage times
session.on("error",      e   => show(e.message));Copy

FusionRuntime.connect(options) returns the same session without binding a button, for a page with its own controls (session.start(), session.stop(), session.interrupt()).

The client uses the browser's own echo canceller, resamples the microphone in an AudioWorklet so a busy page can't stutter the audio, and schedules replies slightly ahead of real time so network jitter doesn't leave gaps. Interruptions are decided on the server, which hears clean audio: when it says the caller interrupted, queued audio is dropped immediately — including the tail of a sentence already synthesized.

Authentication

Generate a key and put it where the server runs:

frun key new
FUSION_ACCEPTED_KEYS=web:frun_kR7m...        # .env, or the environmentCopy

Without keys the server answers on localhost only, and frun up --host 0.0.0.0 refuses to start. There is no flag to switch that off: a runtime reachable from elsewhere with no authentication is someone else's GPU, on your bill.

Two kinds of client, because only one of them can keep a secret:

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"
# {"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}Copy

A token works once and expires in about a minute, so a leaked URL, screenshot or log line is worthless by the time anyone reads it. frun token prints a console URL with one in it for your own machine.

Several keys

One variable, comma separated — not one variable per key, which would simply overwrite itself. The label in front is optional and grants nothing; it exists so frun keys list and your logs can say which key is busy or failing without the key itself ever appearing.

FUSION_ACCEPTED_KEYS=web:frun_kR7m...,mobile:frun_9xQ2...,partner:frun_Lm4v...Copy

With more than one key configured, the per-key session cap changes on its own: instead of "the whole server", each key gets all but one slot, so no single key can lock the others out. Set FUSION_MAX_SESSIONS_PER_KEY for a different split.

Past a handful, put them in a file — one per line, # comments allowed — which also makes them reloadable:

FUSION_ACCEPTED_KEYS_FILE=/etc/fusion/keysCopy

Rotating and revoking

The key list takes several, so rotation is: add the new key, move clients across, drop the old one. Point FUSION_ACCEPTED_KEYS_FILE at a file and kill -HUP re-reads it without a restart — which matters on a GPU, where restarting reloads the models. Removing a key is complete: its tokens are dropped and its conversations are closed.

Limits

Authentication says who may use the server; these say how much of it one caller may take.

Origins and proxies

FUSION_ALLOWED_ORIGINS lists the websites whose pages may open a socket. Unset means only pages this server itself serves: browsers do not stop one site connecting to another, so the server checks. Clients that aren't browsers send no Origin and are unaffected.

Behind a proxy that terminates TLS — RunPod's, nginx, Cloudflare — name it with FUSION_TRUSTED_PROXY=10.0.0.0/8. Until you do, its X-Forwarded-* headers are ignored, because anyone who can reach the port can write those headers. The forwarded address is used for counting only, never to decide who may in.

The frun CLI

fusion-runtime works as an alias for frun. If the command isn't on your PATH, use python3 -m fusion_runtime.cli.

Using it as a library

Most people never need this. frun up agent.py covers running an agent. The same pipeline can run inside a process you already have — a queue worker, a test, a batch job over recorded calls — where a server in the middle makes no sense.

from fusion_runtime import PipelineOrchestrator, run_single_turn, load_agent

agent = load_agent("agent.py")               # the same file frun up reads
orchestrator = PipelineOrchestrator(agent.config())
await orchestrator.initialize()              # loads the models once; reuse it across turns
reply = await run_single_turn(orchestrator, audio, system_prompt=agent.prompt)Copy

Audio in is 16 kHz mono 16-bit PCM; audio out is the same at 24 kHz. initialize() is the expensive call — hold the orchestrator rather than building one per turn. For audio as it is produced, which is what makes barge-in possible, use orchestrator.run_pipeline(chunks, prompt), an async iterator of PCM chunks.

Models

Every download is pinned to an exact Hugging Face commit and checked by file size.

Where models are stored, first match wins: FUSION_MODEL_DIR if set, a models/ folder next to the source (source checkouts), or ~/.cache/fusion-runtime/models. Silero VAD is the exception — it's cached by PyTorch in ~/.cache/torch/hub.

Using your own models

Those five are the defaults, not the limit. A model reference can be a catalog ID, a path on disk, a Hugging Face repo as hf:owner/repo, or the URL of an OpenAI-compatible server. Nothing is detected from a model's name — the runtime is chosen from the file format, and the model's own metadata does the rest.

frun models pull hf:Systran/faster-whisper-large-v3Copy

Then name it in the agent file. Download first — resolving a model never touches the network:

agent = Agent(
    stt=STT("hf:Systran/faster-whisper-large-v3"),
    llm=LLM("hf:bartowski/Mistral-7B-Instruct-v0.3-GGUF/Mistral-7B-Instruct-v0.3-Q4_K_M.gguf"),
    tts=TTS("kokoro-v1.0", voice="af_heart"),
)Copy

A repo holding several quantizations needs the file named, as above. Safetensors models aren't run in process — name one as vllm:hf:owner/repo and frun up serves it with vLLM.

Tools

A tool is a function. The model reads its name, docstring and type hints, calls it when it needs to, and answers with what it returned:

from fusion_runtime import Agent, LLM, tool

@tool
async def order_status(order_id: str) -> dict:
    """Look up where an order is and when it will arrive.

    Args:
        order_id: The order number, as the caller reads it out.
    """
    return await orders.lookup(order_id)

agent = Agent(prompt="...", llm=LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ"), tools=[order_status])Copy

• What the model says before calling ("Let me check.") is spoken while the tool runs.
• Each call has a timeout — @tool(timeout_s=...), 10 s by default. Ordinary (non-async) functions run on a worker thread, so a blocking client library is fine.
• A tool that fails tells the model what went wrong rather than ending the call. Results are cut at 8000 characters.
• Talking over the wait stops the answer. A lookup that finishes within 2 seconds is still kept in the conversation, so the caller's next question is answered with its result, not a guess; a slower one is cancelled.
• After max_tool_rounds calls in one turn (4 by default) the model has to answer.
• Lookups stay in the conversation, so a later question knows an earlier answer came from a lookup. The model may answer a repeated question from that instead of looking again; if your data can change during a call, say so in the prompt.

A server is named with it in front. frun up starts vLLM and SGLang; llama-server you start yourself:

LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ")       # frun up starts it on 127.0.0.1:8002
LLM("sglang:hf:Qwen/Qwen2.5-7B-Instruct-AWQ")     # ...on 127.0.0.1:30000
LLM("llama_server:qwen2.5-7b-instruct")           # yours, on 127.0.0.1:8080
LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ", url="http://gpu-box:8002/v1")   # yours, elsewhereCopy

A runnable version is examples/tools_agent.py.

Answering from your own documents

Retrieval is a tool like any other: a function that takes the caller's question and returns a few short passages, which the model answers from.

@tool(timeout_s=3)
def search_help(question: str) -> dict:
    """Search the shop's help articles: returns, refunds, delivery, cancelling orders.

    Args:
        question: What the caller wants to know, in their words.
    """
    return {"passages": my_search(question, limit=3)}Copy

my_search is yours: a vector database, Elasticsearch, your help centre's API. examples/knowledge_agent.py is a runnable version with a small keyword search over the Markdown files in examples/knowledge/, in plain Python with nothing to install.

• A lookup costs a second model round before the caller hears anything. If your knowledge fits in a page or two, put it in the prompt instead: that's faster than any search.
• Keep passages short — a few sentences each, a few per search. They go into the prompt, and long ones slow the model's first word while the caller waits.
• Tell the model to answer only from what the search returns, and to say so when it finds nothing, rather than answering from memory.

Phone calls

Keep your own number and your provider account (and its KYC). The provider streams each call to fusion, and the agent answers it, with the same limits, interruption, tools and logs as a browser conversation.

Every call is checked before it's taken: Twilio's webhook signature, the key in Exotel's URL, Plivo's signature on the stream. Behind a proxy, set FUSION_PUBLIC_URL to the address the provider calls, since that's the address they sign; on a Runpod pod frun up sets it for you.

• The greeting is ready before the call. It's made when the server starts, so a caller hears it within about 50 ms.
• Interrupting works like in the browser. The moment the caller talks over the agent, fusion tells the provider to drop the audio it hasn't played yet. The provider's playback markers tell fusion what the caller has actually heard.
• A full server says so. Callers hear "all our lines are busy" instead of a dropped call.
• A dead line is noticed. On a phone, a dead audio path is faint hiss, not silence: nothing from the caller above FUSION_PHONE_FLOOR_RMS (40) for FUSION_DEAD_LINE_S (10 s) is logged as a quiet line. It's only logged, since a caller listening in a quiet room can sit under it too. Every call's session.end records its loudest moment and whether the caller was heard at all.
• In the logs, session.start carries transport=twilio (or exotel, plivo) and the call's id.

Try it without a number or an account. The phone simulator in the repo plays the provider's side from your machine, with your microphone, at phone quality:

export FUSION_TWILIO_AUTH_TOKEN=local-test-token
frun up agent.py
python3 tools/phone_sim/phone_sim.py          # in another terminal; --provider exotel or plivo tooCopy

Testing on Twilio: call a number of your own. The console's test call on a trial account comes from a shared sandbox number that neither signed its requests nor streamed the call's audio to fusion in our tests, and a newly upgraded account may hold calls until Twilio's KYC (a Trust Hub customer profile) is approved. A refused webhook is logged with the address fusion checked, which tells a wrong address from a wrong token.

Calls that cross a border can lose audio on the way, before it reaches fusion. In our tests, calls between India and a US number dropped parts of words, and calls Twilio placed to India reported "answered" while the phone was still ringing, then stayed silent. Test with a caller in the number's own country, or call it from a second number with Twilio's Dev Phone.

Phone audio is narrower than a microphone's, and Whisper small mishears more on it: "order 1042" came through as "RDEF 1042" for one caller. For callers, a larger model and a longer pause help, for about half a second more before each reply:

stt=STT("hf:deepdml/faster-whisper-large-v3-turbo-ct2"),   # frun models pull it first
turns=Turns(wait_ms=800),   # a caller who pauses mid-sentence isn't answered halfwayCopy

How it works

Speech runs in one Python process; the LLM runs there too, or on vLLM or SGLang beside it (see vLLM and SGLang). The local client removes the bot's own voice from the microphone (echo cancellation), so the server hears clean audio and can decide when you're actually interrupting it versus hearing its own playback.

Audio that is exactly zero (a muted microphone or a dead connection) is never sent to speech-to-text, even when the voice detector isn't running: recognisers can write fluent sentences out of digital silence. Real room tone, however quiet, isn't exactly zero, so nothing a caller says is dropped.

fusion also notices when it can't hear the caller at all: no audio arriving for 3 seconds, or only exact zeros (a muted mic). The client gets an audio_problem message ("Your microphone is sending only silence. Is it muted?") and the log an audio.problem warning; an audio_ok follows when it comes back. Because it runs from the moment a call connects, a muted mic is caught while the greeting is still playing.

Caller audio must be raw 16-bit little-endian mono PCM at the agreed rate (16 kHz). A message with an odd number of bytes is realigned rather than turned into noise; a WAV file in the right format has its header removed (with a warning to the client); a WAV in another format, or MP3 with a tag, Ogg, WebM or FLAC, ends the call with a bad_audio_format error naming what arrived and how to convert it. A wrong sample rate in raw PCM can't be told from the bytes, so make sure it matches.

Configuration

Choose a profile with frun up --config (or FUSION_CONFIG if you start the server another way):

The names are about the machine, not the stage of your project: development runs on a laptop's CPU, production on an NVIDIA GPU. A laptop with an NVIDIA GPU can use either.

frun up --config productionCopy

LLM from any OpenAI-compatible endpoint

Speech stays local; the LLM comes from vLLM, SGLang, llama-server, Ollama or a hosted API. In an agent file, name the server (see Tools); from the command line:

frun up --llm-url http://localhost:8080/v1 --llm-model my-model

export GROQ_API_KEY=...
frun up --llm-url https://api.groq.com/openai/v1 --llm-model <model> --llm-api-key-env GROQ_API_KEYCopy

API keys are never accepted in config — only the name of the environment variable holding one. frun doctor checks that the key is set, the endpoint answers, and it serves the model.

Turn detection: when the agent answers

By default the agent answers after 500 ms of silence, and stops speaking once the caller has talked over it for 300 ms. Turns(interruptible=False) turns interruptions off: the agent always finishes what it's saying, and answers what the caller said after:

frun up --turn-wait-ms 800          # more patient with pauses mid-sentence
frun up --interrupt-after-ms 500    # ignore coughs and "mm-hm" in noisy placesCopy

Speaking again right after a pause continues the same turn: if the caller resumes within 1.5s of their turn ending, both parts reach the LLM as one message. A turn-detector model can plug in to shorten or stretch the wait based on how likely the caller is done — silence always confirms the end of a turn, and a detector that's slow or fails just leaves the default wait.

frun up --turn-detector my_package.turns:MyDetectorCopy

Languages

Set what callers speak with STTConfig(language="hi"), or omit it to detect per turn. The production default is whisper-small, which is multilingual, so changing the language a deployment listens in is a config change and nothing else. The development default is tiny.en, English-only, and it refuses another language at startup rather than quietly transcribing nonsense.

Server API

Seven endpoints, and it's clearest by who calls them.

A browser

The socket takes raw 16 kHz mono PCM as binary messages, and two JSON control messages: {"type":"playback","playing":true|false} while the agent is audible, and {"type":"interrupt"} from a client that stops itself. It sends 24 kHz PCM back as binary, and these as JSON:

Your backend

curl -X POST https://your-server/v1/sessions -H "Authorization: Bearer $FUSION_API_KEY"Copy

{"token":"...","expires_in":60,"ws_url":"wss://your-server/v1/voice/ws?token=..."}Copy

One call per visitor, on page load. The token works once and expires in about a minute, and a connected page is handed its next one over the socket — so reconnecting costs no round trip. This is the only endpoint a page's backend needs.

A script, or a test

curl -X POST https://your-server/v1/voice/chat -H "Authorization: Bearer $FUSION_API_KEY" \
  -d '{"audio_base64":"..."}'Copy

{"audio_base64":"...", "transcript":"...", "response_text":"...", "latency_ms":612,
 "turns":[{"user":"...","agent":"...","outcome":"completed","metrics":{...}}]}Copy

The whole conversation in one request, no socket. Useful for regression tests, for comparing models on the same audio, and for anything that isn't a live microphone.

Monitoring

Everything except /health, / and /fusion-runtime.js needs a key — or, on the WebSocket, a session token. See Authentication.

On an NVIDIA GPU

torch on Linux already carries CUDA — its PyPI wheels bundle the NVIDIA libraries — so a normal install is the GPU one. Two packages need help, and both fail quietly if they don't get it.

pip install -e ".[cuda]"
CMAKE_ARGS="-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=86" pip install llama-cpp-pythonCopy

llama-cpp-python is published as source only and builds without CUDA unless told otherwise; the prebuilt CUDA wheels people link to stopped at 0.2.66. Set CMAKE_CUDA_ARCHITECTURES to your card — 86 for a 3090 or A10, 89 for an L4 or 4090, 80 for an A100 — or it compiles kernels for every architecture and takes about an hour.

pip install onnxruntime-gpu --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/Copy

export LD_LIBRARY_PATH="$(python -c "import site,glob,os; p=site.getsitepackages()[0]; print(':'.join(sorted(glob.glob(os.path.join(p,'nvidia','*','lib')))))"):$LD_LIBRARY_PATH"Copy

Check both before trusting a measurement: the log must show vad model.loaded and no Failed to create CUDAExecutionProvider. Either one missing means you are measuring a CPU.

Several callers

With the model in-process, one llama.cpp context decodes one reply at a time: four callers measure the same as one, and past that they queue. Move the model to vLLM or SGLang on the same GPU and leave speech where it is. Measured on one RTX 3090, three turns per caller, server response (turn end to first audio):

The model's first token stays at 36–72 ms from one caller to sixteen on both servers. What grows is speech, which isn't batched across callers yet: Kokoro's first audio reaches 1.5–2 s at sixteen callers, and Whisper grows with how long people talk.

To move the model onto vLLM or SGLang, name it in the agent — LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ") — and frun up starts the server for you, sized to share a 24 GB card with speech. See vLLM and SGLang.

Measure your own: with the server running, frun bench --callers 1,4,8,12 streams real callers at it and prints the median per stage, so you can see which one starts to queue. --audio order asks about an order (for tool agents), --target-ms 1500 adds "up to N callers stayed under 1500 ms", and --json results.json saves the numbers.

vLLM and SGLang

Name the server in front of the model and frun up runs it: it starts the server, waits until it answers, then loads speech-to-text and text-to-speech, so the server takes its share of GPU memory first. If the server exits, fusion says so and starts it again, waiting longer after each failure (up to a minute). If it keeps running but stops answering for 60 seconds (hung), fusion restarts it the same way: hung_after_s= changes that. Ctrl+C stops both.

from fusion_runtime import Agent, LLM

agent = Agent(
    prompt="...",
    llm=LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ"),     # or "sglang:hf:..."
    profile="production",                                # speech on the GPU too
)Copy

Installing the engine

vLLM and SGLang each pin their own torch and CUDA, so they can't share fusion's Python environment. Install the one you want into its own, once, and tell fusion where it is. fusion never installs it for you: it's several gigabytes and depends on your driver.

python3 -m venv ~/vllm-env && ~/vllm-env/bin/pip install vllm
export FUSION_LLM_ENGINE_ENV=~/vllm-env

# or SGLang
python3 -m venv ~/sglang-env && ~/sglang-env/bin/pip install "sglang[all]"
export FUSION_LLM_ENGINE_ENV=~/sglang-envCopy

engine_env="~/vllm-env" in the agent does the same. Without either, fusion uses a vllm command on your PATH (for SGLang, its own Python, if SGLang is installed there). If neither is there, frun up stops before loading anything and prints these commands. The engine's folder is put first on its PATH, because both compile kernels on their first start and call ninja from it.

Settings

Four settings have fusion names, translated to each engine's own flag. Everything else goes through extra_args.

LLM("vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ",
    gpu_memory=0.6,          # the server's share of the card; speech uses the rest
    max_model_len=4096,      # a voice call's context; longer reserves memory calls never use
    max_callers=16,          # replies decoded at once; more wait their turn
    extra_args="--kv-cache-dtype fp8")Copy

GPU memory. 0.6 of a 24 GB card is about 14 GB: a 4-bit 7B model's weights plus every caller's context, with the rest left for Whisper and Kokoro. vLLM's own default is 0.9, which leaves speech nothing. A 16-bit 7B model needs about 15 GB for its weights alone, so on a 24 GB card use a 4-bit one (AWQ or GPTQ).

Tool calls. The engine has to know the model's tool-call format. The default (hermes on vLLM, qwen25 on SGLang) is Qwen 2.5's, and fusion says at startup which it chose. For another model family, set tool_parser= to the name in the engine's docs: llama3_json for Llama 3.1 on vLLM, for example. An agent without tools gets no tool flags.

Anything else is passed to the engine unchanged, as a string or a list:

# half the memory per caller's context, so more callers fit
extra_args="--kv-cache-dtype fp8"

# speculative decoding that copies from the conversation: no extra model
extra_args='--speculative-config {"method":"ngram","num_speculative_tokens":4,"prompt_lookup_max":4}'

# a small draft model; the engine downloads it on first start
extra_args='--speculative-config {"model":"Qwen/Qwen2.5-0.5B-Instruct","num_speculative_tokens":4}'Copy

Speculative decoding speeds up the words after the first, not the first itself, and can slow things down when many callers keep the GPU busy. ngram costs no memory and suits replies that repeat the conversation, like a tool's result read back. Measure it at your number of callers before keeping it.

A setting fusion doesn't know is refused at startup with the likely one, so a typo can't start a server with the wrong memory share. An engine flag given by its own name (gpu_memory_utilization=) points you to the fusion name or extra_args.

GPU memory, checked before starting

The engine reserves its share of the card's total memory when it starts, and fails a few minutes in if that much isn't free. Before starting it, frun up asks nvidia-smi: if the share isn't free it stops at once and lists what holds the memory (usually an earlier engine or frun up still running), and if the share leaves too little for Whisper and Kokoro it says so, with a gpu_memory that would fit.

Error: vLLM needs 14.4 GB of GPU 0 (NVIDIA GeForce RTX 3090): gpu_memory=0.6 of 24.0 GB, but only 9.0 GB is free. Using it now:
    pid 4242  /root/vllm-env/bin/python3  14.5 GB
  Stop what you don't need (an earlier vLLM or frun up is the usual one: kill <pid>), or lower gpu_memory.

Warm before the first caller

A model server does some work on its first request: SGLang compiles, which took about 70 s on an RTX 4090, long enough for the first caller's request to time out. So fusion sends it one tiny request while loading, before "Models ready", and again after restarting a server that crashed. The log shows llm.warmed with how long it took; a warm-up that fails is reported as llm.warmup_failed and loading carries on.

On by default for vllm:, sglang: and llama_server:; off for a model given as a URL, which may be a paid API. warmup=False turns it off, warmup_timeout_s= (default 300) sets how long it may take.

Models and downloads

The engine loads the model from Hugging Face's cache, HF_HOME (~/.cache/huggingface unless set); on a cloud machine put that on a persistent volume, or every restart downloads again. frun models pull agent.py fetches the agent's LLM into that cache, with progress, along with its speech models (frun models pull vllm:hf:owner/repo for one by name). Skip it and the engine downloads the model during its first start, which frun up warns about: with kernel compiling that can take 5–15 minutes. fusion waits up to 15 (start_timeout_s= to change it) and says it's still starting every minute. A draft model is still the engine's to download. Gated models need HF_TOKEN, which fusion passes through.

A model on Hugging Face is served under its repo id; a revision goes after @: vllm:hf:Qwen/Qwen2.5-7B-Instruct-AWQ@<commit>.

Logs

By default every line the engine prints goes to fusion's terminal, marked with its name, so a container platform collects it with everything else. With --log-format json each line is a JSON object, with "event": "llm_server.log".

   vllm > starting vLLM
   vllm | INFO 10-01 17:11:51 [api_server.py] Starting vLLM API server on http://127.0.0.1:8002
   vllm > vLLM ready in 74 s

vLLM prints its throughput every few seconds, and a first start prints hundreds of lines. To keep them out of the terminal, give a file:

frun up agent.py --llm-log ~/vllm.log       # or FUSION_LLM_LOG=~/vllm.logCopy

The file then gets every line, and the terminal only the start, warnings and errors. If the server fails to start, its last 20 lines are printed either way.

Watching it while calls run

Every few seconds fusion asks the server for its own numbers: replies it's decoding, replies waiting, and how full its KV cache is. A queue that grows while the cache is full means more callers than the card can hold.

• GET /health, asked with a key, has an llm_server entry: {"engine": "vllm", "state": "up", "running": 7, "waiting": 3, "kv_cache_used": 0.62}
• GET /metrics adds fusion_llm_server_up, _requests_running, _requests_waiting and _kv_cache_used.
• Each turn's llm.request log line carries llm_server_waiting and kv_cache_used, so a slow reply shows whether it queued.
• The log says llm_server.down when the server stops answering and llm_server.up when it's back.

This works for a server fusion didn't start too, as long as the agent names it (vllm:, sglang:). SGLang publishes these only with --enable-metrics, which fusion adds when it starts it. For SGLang the check is that page answering, not its /health, which runs a one-token generation each time and would keep the GPU busy every few seconds.

When fusion doesn't start it

• A server already answers at the address with the same model: fusion uses it as it is. If it serves a different model, frun up names that model and stops, rather than starting a second server that can't get the port.
• url= points at another machine, or FUSION_LLM_URL is set: the server is elsewhere.
• launch=False in the agent: you start it yourself, with whatever flags you like.
• The model is given as a URL, or as llama_server:.

An in-process model is never moved to a server on its own. On a machine with an NVIDIA GPU, frun up says once that a server would take more callers.

Ports and safety

vLLM goes on 8002 rather than its usual 8000, which is frun up's (and not 8001, which some hosts, like Runpod's pod images, already use); SGLang on its usual 30000. A local url= moves it: url="http://127.0.0.1:9000/v1". The server listens on 127.0.0.1 only. It has no keys of its own, so it's never reachable from another machine: callers reach the agent, and the agent reaches it. fusion's own keys are left out of its environment.

On Linux, if fusion is killed outright (kill -9, out of memory), the server goes with it rather than holding the GPU.

Deploying

Any machine with an NVIDIA GPU runs this; there is nothing provider-specific in the runtime. The docs are written against RunPod because a pod has no request timeout and its proxy gives you https:// and wss:// without handling a certificate — which browsers require before granting a microphone.

With Docker

Two images, for two kinds of machine:

Both are at ghcr.io/samarthurs18/. On a laptop:

git clone https://github.com/SamarthUrs18/fusion-runtime && cd fusion-runtime/docker
docker compose run --rm fusion key new          # put it in docker/.env as FUSION_ACCEPTED_KEYS=<key>
docker compose run --rm fusion models pull      # once, about 1.5 GB into docker/models
docker compose up                               # http://localhost:8000Copy

On a GPU server, with the NVIDIA Container Toolkit installed:

cd fusion-runtime/docker
docker compose -f docker-compose.gpu.yml run --rm --no-deps fusion key new
docker compose -f docker-compose.gpu.yml run --rm --no-deps fusion models pull /app/agent.py
docker compose -f docker-compose.gpu.yml upCopy

vLLM takes 60% of the card and fusion's speech the rest; vLLM is reachable only from the fusion container, since it has no keys of its own. The first start downloads the LLM (about 5 GB, into docker/hf) and compiles kernels, and fusion waits for it. The GPU image's default agent (docker/agent.py) uses that vLLM; mount your own over /app/agent.py. To change the model, change --model in the compose file and the agent's LLM to match.

The GPU image runs only on an NVIDIA GPU. Started without one, it says so and points to the CPU image. Its llama.cpp is built for the CPU: on a GPU the LLM belongs on vLLM. If you want llama.cpp on the GPU instead, build it yourself with docker build --build-arg LLAMA_CUDA=1 -f docker/Dockerfile . (it compiles for about an hour).

--env-file reads the file on the host and sets real environment variables in the container. Never COPY .env into an image: that writes your keys into a layer anyone who pulls it can read, even if a later layer deletes the file.

Without Docker

[Service]
EnvironmentFile=/etc/fusion/runtime.env
ExecStart=/opt/fusion/bin/frun up /etc/fusion/agent.py --host 0.0.0.0Copy

It is a Python package, so a GPU VM with a systemd unit works as well as a container. Keep that environment file root:root and mode 0600, written at boot from Secrets Manager, SSM Parameter Store or equivalent — not baked into an image.

On a cloud that doesn't terminate TLS for you, put a load balancer or reverse proxy in front with a certificate, and set FUSION_TRUSTED_PROXY to its address range — until you do, its forwarded headers are ignored and token minting refuses, because a token in a URL over plain HTTP is readable by every hop.

HTTPS with Caddy

On a plain VM (AWS, GCP, your own server), Caddy gets a certificate for your domain on its own and passes WebSockets through. Point the domain's DNS at the machine, open ports 80 and 443, and write a Caddyfile:

voice.example.com {
    reverse_proxy 127.0.0.1:8000
}Copy

# .env next to the agent
FUSION_ACCEPTED_KEYS=<from frun key new>
FUSION_TRUSTED_PROXY=127.0.0.1                      # Caddy, on this machine
FUSION_ALLOWED_ORIGINS=https://your-site.example    # pages on other sites that open a socket
FUSION_PUBLIC_URL=https://voice.example.com          # the address frun up shows for the browser

frun up agent.py                                     # 127.0.0.1:8000: only Caddy can reach it
caddy run                                            # in the Caddyfile's folderCopy

fusion stays on 127.0.0.1, so nothing reaches it except through Caddy. Browsers then use https://voice.example.com and wss://voice.example.com/v1/voice/ws, and frun token --url http://127.0.0.1:8000 --public-url https://voice.example.com prints a link to try it.

Models belong on a volume rather than in the image: a rebuild then doesn't re-download several gigabytes, and two machines can share one copy. Point FUSION_MODEL_DIR at it.

Observability

Every stage emits structured events with a timestamp, session id, turn id, stage, model and duration, so a deployed agent never runs blind.

20:47:49.441  95b134d2 t1   vad.speech_start            audio_offset_ms=192 probability=0.88
20:47:52.162  95b134d2 t1   turn.end_detected           reason=silence detector=silence threshold_ms=500 wait_ms=569
20:47:52.334  95b134d2 t1   llm.first_token             171ms runtime=llama_cpp model=qwen2.5-0.5b-instruct-q4_k_m.gguf
20:47:52.670  95b134d2 t1   tts.first_chunk             326ms audio_ms=2525
20:47:52.671  95b134d2 t1   audio.first_sent            ttfa_ms=778 response_ms=509Copy

Each line carries the event's full name, so grep tool.call finds every tool call.

Logs: frun up --log-format pretty|json, --log-level debug|info|warning|error. Errors include a stable code, whether retrying can help, a suggested fix, and the stack trace (server logs only).

Running below strength is never quiet: if the voice detector didn't load, text-to-speech fell back to the CPU, or the LLM server stopped answering, /health says "status": "degraded" (and, asked with a key, lists each part with the reason), every call's session.start line carries degraded=…, and /metrics has fusion_degraded{component=…}. A server that started in a weaker mode stays visibly so, not just in one line at start-up.

Private by default: logs record text lengths, not what people said. --log-content includes transcripts and replies. API keys are always redacted.

Per turn: time to first audio, end-of-turn wait, transcription delay, STT time, LLM time-to-first-token and tokens/s, TTS time-to-first-audio and real-time factor, playback start, interruptions and how fast generation stopped.

Development

pip install -e ".[dev,talk]"     # or: uv sync --extra dev --extra talk
pytest tests/Copy

Tested on Python 3.11, 3.12 and 3.13.

Package layout: cli/ (frun commands), catalog/ (model catalog, installs, downloads), config.py (settings, profiles, model directory), server.py (FastAPI + WebSocket), contract/ (the interface every model runtime implements), resolver.py (model reference → runtime), registry.py (runtime names and plugins → classes), runtimes/ (one adapter per engine: llama_cpp, ctranslate2, onnx, openai_http), engine/ (orchestrator, scheduler, streaming, barge-in), web/ (the console and the browser client), vad/, telemetry/, testing/ (conformance kit and fakes), audio/ (echo cancellation, duplex).

License

Apache-2.0. Embed fusion-runtime in a commercial product, rebrand it, and ship it closed. What Apache-2.0 asks in return is that you keep the copyright notice and the NOTICE file in what you distribute, and that you don't use the project's name in a way that implies it endorses your product. There is no CLA and no copyright assignment for contributors.

The models pulled by default are permissive too — Whisper MIT, Silero VAD MIT, Qwen2.5 Apache-2.0, Kokoro Apache-2.0 — so the whole default path is clear for commercial use, weights included. A model you point it at yourself carries its own licence, and some open-weight models restrict commercial use or rebranding. Check that one before you ship it.