Local Speech-to-Text for Home Assistant: Whisper over LLM

I had already won the benchmark before I started. That’s how it felt, anyway.

The plan was simple. I run Ollama locally, and gemma4:e4b genuinely lists audio under Capabilities in ollama show — so I can do speech-to-text for Home Assistant Assist entirely in-house, no cloud, no API key. I synthesised a handful of test sentences with edge-tts --voice de-AT-JonasNeural, ran both engines over them, and the result was not close: gemma4 beat faster-whisper base on Austrian German. Where whisper dropped or mangled words, gemma was verbatim. On clean English it was a tie.

Done, I thought. It was not done.

The table that destroyed the benchmark

The moment real microphone material went through — Assist in the browser, about -27 dB RMS, room noise, a human instead of a speech synthesiser — gemma4 fell apart on the same captured WAV files where faster-whisper was verbatim:

real mic clip	gemma4:e4b	faster-whisper base
“close the office shutter and also close shutter number one in the living room”	“schließe den Offen-Schatten am Wohnende des Livengro”	✅ verbatim
“It’s unbelievable, it’s not working”	“Das beliebte Netzwerk”	✅ verbatim

That isn’t “slightly worse”. That’s unusable. And it comes with failure modes a conventional ASR decoder structurally cannot have, because what’s sitting here is a language model with an audio encoder, not a transcription model:

• It answers instead of transcribing. Actual return values: “I’m sorry, but no audio was provided. Please provide the audio you would like me to transcribe.” And once, better still, just: “Transcribe this audio.” Yes. That was the job.
• It hallucinates into random languages. On English speech: जो उसमें ऑफ शटर, в смысле office shot, Ça me lève le réseau.
• It misidentifies the spoken language entirely and then “translates” into whatever it settled on.

The lesson is the actual point of this article, and it generalises well beyond speech recognition:

Benchmarking an ASR backend on synthesised speech is worthless. TTS output is clean, level-normalised and accent-neutral. It hides precisely the weaknesses that matter.

If you test this yourself: capture real utterances (--dump-dir in the bridge script below does exactly that) and measure on those. A fixture that never crossed a room proves nothing about a microphone in a room.

For phone voice notes — close mic, opus, one speaker — gemma4:e4b is still fine. For Assist it isn’t. So the backend is faster-whisper, and the rest of this article is how that gets into Home Assistant.

Home Assistant cannot send audio to your service

My first instinct was a webhook. Assist records, POSTs the file somewhere, I return text. That does not exist. An Assist pipeline will accept an STT engine through exactly two doors:

1. the Wyoming protocol — length-prefixed JSON events plus raw PCM over a plain TCP socket, or
2. a custom stt platform, i.e. a full custom integration.

Wyoming is by far the cheaper door: roughly 150 lines of Python, the engine shows up natively in the STT dropdown, zero edits to any HA config file and no HA restart. The chain:

Mic (Companion app / ESPHome satellite / browser)
  → HA Assist pipeline
    → Wyoming tcp://<host>:10300
      → this script: PCM → WAV → faster-whisper (in-process)

The HA-facing contract is small. You build an AsyncServer plus an AsyncEventHandler with the wyoming Python package and answer four events:

class WhisperEventHandler(AsyncEventHandler):
    async def handle_event(self, event: Event) -> bool:
        if Describe.is_type(event.type):
            await self.write_event(self.info.event())
            return True

        if Transcribe.is_type(event.type):
            transcribe = Transcribe.from_event(event)
            if transcribe.language:
                self._language = transcribe.language
            self._audio = bytearray()
            return True

        if AudioChunk.is_type(event.type):
            chunk = AudioChunk.from_event(event)
            self._rate = chunk.rate
            self._width = chunk.width
            self._channels = chunk.channels
            self._audio.extend(chunk.audio)
            return True

        if AudioStop.is_type(event.type):
            text = ""
            try:
                text = await self._transcribe(bytes(self._audio))
            except Exception:  # noqa: BLE001 - never kill the Assist pipeline
                _LOGGER.exception("transcription failed")
            finally:
                self._audio = bytearray()
            _LOGGER.info("transcript: %r", text)
            # HA opens a fresh connection per utterance.
            await self.write_event(Transcript(text=text).event())
            return False

        return True

Three details in that block were paid for the hard way:

• The return False after the Transcript. Home Assistant opens a fresh connection per utterance. Hold it open and the next one desyncs.
• The except that swallows everything. Let an exception escape and you don’t just lose the transcription — the entire Assist pipeline fails with a generic error. Returning an empty Transcript means Assist merely didn’t hear you, which is a far better failure.
• Read the sample rate off the chunk, not from a constant.

Model size comes down to a single word

English is solved at base. Austrian compound nouns are not — and they fail silently, because a small model emits a plausible-looking word and nothing in the log flags anything. That’s the trap: the pipeline looks perfectly healthy.

11-second fixture, reference sentence containing “Rollo im Büro” and “Terrassenmarkise”:

model	device	latency	output
base	CPU int8	1.7 s	“das Rodo in Büro … Terrasse nach Kise” ✗
small	CPU int8	5.0 s	“das Rollo in Büro … Terrassenakise” ~
large-v3-turbo	CPU int8	21.5 s	“das Rollo im Büro … Terrassenmarkise” ✓

All three identified the language correctly — de @ 0.99–1.00. The failure is vocabulary, not language ID. Anyone debugging language detection here is digging in the wrong hole.

The rule I took from it: size the model for the hardest language in the house, not the easiest — and validate on the compound nouns, not on “turn on the light”.

And large-v3-turbo at 21.5 seconds? Useless for voice. On GPU it’s 1–2 seconds. Except the GPU is spoken for. On the 11 GB RTX 2080 Ti, Ollama loads gemma4:e4b on demand for a different project — a live chatbot — and with a large num_ctx that holds a good 10 GB of the card. Ollama does evict it after a few idle minutes, but parking whisper next to it permanently means the next chatbot request either waits or spills to CPU. Nothing reliably left for whisper.

I could have evicted it. I didn’t, deliberately: before you throw a resident model off the card, find out what depends on it. In this case unloading it would have caused a visible outage on a live site. If you need VRAM, lowering the squatter’s num_ctx is far safer than killing it, and it only costs context length.

So STT runs on CPU with small. A compromise between base’s 1.7 s and large-v3-turbo’s vocabulary. Which means I am sitting on exactly the model that produced “Terrassenakise”. I know.

Forcing --language turns Whisper into a translator

The other recurring trap: pin Whisper to a language and it stops transcribing and starts translating. That was one of the biggest sources of garbage in my logs, and it doesn’t look like a bug — it looks like a bad model.

German and English get spoken at the same microphone here, so the service runs with --language auto. In the script that’s one line:

# None => auto-detect. HA sends the pipeline language on every
# Transcribe event; "auto" on the command line ignores it, which is
# what you want if you speak more than one language at this mic.
lang = None if self.cli_args.language == "auto" else (self._language or None)

What that still does not fix: code-switching inside a single sentence. “Ich geh eine different language an” stays broken. That’s inherent, not a setting.

Also non-optional: vad_filter=True. Home Assistant pads utterances with silence, and without VAD Whisper hallucinates into that silence.

The naming wart I’m owning

The systemd user unit that’s actually running looks like this:

[Unit]
Description=Wyoming STT bridge (Ollama gemma4:e4b) for Home Assistant Assist
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/home/masta/.hermes/bin/wyoming-gemma-stt.py --uri tcp://0.0.0.0:10300 --model small --language auto
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

wyoming-gemma-stt.py, described as “Ollama gemma4:e4b” — running faster-whisper. That isn’t sloppiness, it’s deliberate, and the code says why:

AsrProgram(
    # Name kept as "gemma-ollama" so HA's existing entity
    # stt.gemma_ollama and the pipeline binding keep working.
    name="gemma-ollama",
    description=f"faster-whisper {model} speech-to-text",
    ...
)

Home Assistant derives the entity id and the pipeline binding from the AsrProgram name. Which is why my STT entity is still called stt.gemma_ollama. Change the name when you swap the backend and you have to remove the Wyoming integration, re-add it, and re-pick the engine in every pipeline. Keep it, and replacing the entire ASR backend is invisible to HA — no restart, no clicks.

I kept it. The price is a service whose name lies about its own contents. For a system I maintain alone I’ll take that; on a team I’d eat the migration once and rename.

Two more things that cost time during setup:

• Port 10300 taken twice. A manually backgrounded copy makes the systemd unit crash-loop with OSError: [Errno 98]. pkill -f wyoming-gemma-stt.py first, then systemctl --user enable --now.
• nc -z <your-own-LAN-IP> <port> proves nothing. Loopback bypasses the firewall. With DEFAULT_INPUT_POLICY=DROP in /etc/default/ufw you need an explicit rule no matter what nc says: sudo ufw allow from 192.168.1.0/24 to any port 10300 proto tcp.

The model loads at startup, so after a restart give it about 10 seconds before the service accepts connections.

Second problem: Assist only understands templates

At this point my setup listens, but it still only understands whatever phrase templates the built-in intent engine ships with. “Is anything still open upstairs?” is not a template. On top of that, Assist only matches entities that are explicitly exposed, and matches on their literal name — two entities containing “office shutter” is ambiguous, which is a silent no-op. You fix that with aliases, not by renaming things.

The way out is a different conversation agent. And that’s where you hit a wall:

Home Assistant’s OpenAI integration hard-refuses foreign endpoints. Verbatim: “works only with the official OpenAI API endpoint and does not support OpenAI-API-compatible third-party services, proxies, or alternative…” — core issue #137087, still open as of January 2026. HACS carries an Extended OpenAI Conversation fork that adds base_url.

But the native Ollama integration accepts any URL out of the box. So instead of fighting HA I just spoke Ollama’s dialect: a small shim that impersonates Ollama and forwards everything to the agent framework I already run at home. No HACS, no custom component, no HA restart.

HA probes more than /api/chat. Miss a route and the integration won’t add at all:

route	purpose
GET /	connectivity probe, must return Ollama is running
GET /api/version	version string
GET /api/tags	model list — one fake hermes-agent entry
POST /api/show	per-model metadata / capabilities
POST /api/chat	the turn itself, both stream:true and false

Streaming can be faked. One ndjson chunk carrying the full text with done:false, then one with empty content and done:true — HA accepts it:

await resp.write(json.dumps({
    "model": body.get("model", MODEL_NAME),
    "created_at": now,
    "message": {"role": "assistant", "content": content},
    "done": False,
}).encode() + b"\n")
await resp.write(json.dumps({
    "model": body.get("model", MODEL_NAME),
    "created_at": now,
    "message": {"role": "assistant", "content": ""},
    "done": True,
    "done_reason": "stop",
    "total_duration": elapsed_ns,
    "eval_count": len(content.split()),
}).encode() + b"\n")

And the rule you must never break in a shim like this: never let a backend failure escape as an HTTP error. Return the error text as the assistant message, otherwise Assist just shows a generic failure with no clue in it:

except Exception as exc:  # noqa: BLE001 - always answer HA with something
    _LOGGER.exception("hermes call failed")
    content = f"Hermes unreachable: {exc}"

The shim runs as its own user unit:

[Unit]
Description=Ollama-API shim exposing Hermes as an HA conversation agent
After=network-online.target hermes-gateway.service
Wants=network-online.target

[Service]
Type=simple
EnvironmentFile=%h/.hermes/ha-shim.env
ExecStart=%h/.hermes/bin/ha-hermes-ollama-shim.py --host 0.0.0.0 --port 11435 --hermes-url http://127.0.0.1:8643
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=default.target

Two choices in there are deliberate:

• --hermes-url http://127.0.0.1:8643, not the main profile’s default port. My main profile is the big expensive model with every tool enabled — the last thing I want behind an always-available microphone. What sits behind Assist is a separate, small profile with its own gateway process:

  ExecStart=/home/masta/.hermes/hermes-agent/venv/bin/hermes -p home_assistant gateway run

  That profile runs claude-haiku-4-5. The API server is profile-scoped, so this is a second service, not a config toggle. And Environment=HERMES_PROFILE=... is not enough — the singleton lock keys on get_hermes_home()/gateway.pid, so the second unit crash-loops with “Gateway already running”. You need the -p CLI flag.

• EnvironmentFile=%h/.hermes/ha-shim.env (chmod 600, a single KEY= line). Otherwise the shim reads its API key from the default profile’s .env, and pointing it at a different port gets you 401s. Since _load_key() checks os.getenv("API_SERVER_KEY") first, injecting it via the unit is enough — no script edit.

The agent itself stays bound to 127.0.0.1. Only the shim is exposed; it holds the key so HA never sees it.

And then the good local model lost anyway

Obviously I wanted this local. So: gemma4:12b behind the shim — 92.1% average on the public HA leaderboard, the second-best local option there. The result inverted my expectation exactly:

context	outcome
standalone, ~12-line entity list in the prompt	0.6–0.9 s, correct entity id (cover.rollo_lr_1), best Austrian German of any model tested (“Des schalt i da glei zu”)
behind the agent shim (~31k-token system prompt + full tool schema)	zero tool calls, 30–100 s per reply, answered as a generic chatbot

Asked “Wie viel Strom produziert meine PV-Anlage gerade?”, the verbatim reply was:

“Das kann ich leider nicht sehen. Als KI habe ich keinen Zugriff auf deine Hardware, dein Smart-Home-System…”

— with live ha_* tools sitting unused in its schema. Right there. Labeled. Asked to run a shell command and paste the output, it replied “What is the specific problem or task? …Feel free to just start describing what you need!”

That’s the same mistake as the top of this article, one layer up. A hand-built 12-line context is no more representative of agent use than a TTS file is of a microphone: both remove exactly the pressure that breaks the thing.

Three things I took away:

1. tools in ollama show advertises format support, not competence. A model can emit perfectly valid tool-call syntax and still never decide to call one once the prompt is large.
2. Prompt size is the failure axis, not parameter count. Models in the 12B class degrade into generic assistant chatter rather than failing loudly.
3. Learn the symptom on sight: fluent, friendly, on-topic prose that answers nothing and calls nothing. No amount of prompt wording fixes that.

The leaderboard number isn’t wrong, by the way — it measures something else. It measures HA’s native LLM API with a small tool set. Not a 31k-token harness.

So Haiku sits behind the shim. Not free either: on the same question Haiku took ~3.4 s against Opus’s ~16.8 s, but said “6 shutters” and then listed seven, and lost the nuance Opus caught (that the house has no indoor temperature sensor at all — only the one in the car). Good enough for commands, weaker for inventory and reasoning questions. After a model swap, re-verify accuracy, not just that a reply arrives.

Where I landed

What actually runs today — three user units, all active:

wyoming-gemma-stt.service   → faster-whisper small, CPU int8, --language auto, :10300
ha-hermes-shim.service      → Ollama dialect on :11435 → 127.0.0.1:8643
hermes-gateway-ha.service   → profile "home_assistant", claude-haiku-4-5

Visible inside Home Assistant as:

stt.gemma_ollama            (Wyoming; the name lies, it's whisper)
conversation.hermes         (via the shim)
conversation.home_assistant (built-in)

And now the part I’m not going to dress up: this is not a finished voice assistant.

• My default pipeline is still conversation.home_assistant. The agent pipeline exists alongside it and is deliberately not the preferred one. The agent gets full tool access with no confirmation step — whoever reaches the microphone reaches everything. I don’t want that as the default.
• No wake word. Both pipelines have wake_word_entity: null.
• No spoken reply. Both have tts_engine: null. The answer comes back as text in the chat sheet. The return leg simply isn’t built.
• Push-to-talk, Companion app only — and specifically only the in-app Assist dialog (three-dots menu → Assist → mic inside the chat sheet).
• iOS Shortcuts and the Action button never reach my STT engine. Those routes use Apple’s Dictate text, transcribe on-device, and hand HA a string via “Assist with Provided Input”. The pipeline’s STT engine is never invoked and my service logs nothing. That cost me half an evening, because “it’s not recording” and “the service is dead” look identical from outside. If your Wyoming log stays empty: ask which button you pressed first, before debugging firewall, model or mic permissions.
• ~2.5–3 s of CPU latency for transcription, plus at least ~2 s for an agent reply. Fine for push-to-talk. Far too slow for a wake-word satellite.

What I’d do differently

I’d debug the capture path before the model. --dump-dir writes every utterance to disk; levels, duration and sample rate answer “is the mic even working” in one command. Instead I guessed at the model.

I’d measure on real microphone audio from the start. The entire gemma detour — including the service name that now outlives it — exists only because I believed a benchmark run on synthetic speech — one I had built myself, which should have been the first clue.

And I’d push the big model through the real harness before getting excited about its leaderboard position.

Next up: the TTS return leg, so Assist answers instead of only writing, and an honest attempt at large-v3-turbo on GPU — once I’ve given the chatbot living on that 2080 Ti a smaller num_ctx without killing it in the process.