STT and TTS on a Second GPU: the MSI Laptop Takes Over

At the end of part 1 I promised two things: the TTS return leg, so Assist answers instead of only writing, and an honest attempt at large-v3-turbo on GPU — once I’d given the chatbot living on the RTX 2080 Ti a smaller num_ctx.

That diet never happened.

How far can you cut the context of a model that real people use on a live site before it gets stupid? I still don’t know, because I never dared to find out in production. The fix was already in the house — my MSI laptop, with a second, newer GPU in it.

That’s the unglamorous punchline of this article. I didn’t solve a problem, I relocated it.

The arithmetic everything failed on

The 2080 Ti has 11.2 GB of usable VRAM. Two tenants wanted to live there at once:

• gemma4:e4b — the guest chatbot for the wedding site. A live site, real guests typing into it, requests arriving at my Ollama through an SSH reverse tunnel. It needs 9.6 GB. It’s the only local model that gets Austrian German and the facts right, so “just use a smaller one” wasn’t on the table.
• Whisper large-v3-turbo plus the TTS voice together: 5.2 GB.

9.6 + 5.2 > 11.2. No discussion, no optimization, no trick. The result was that Ollama kept evicting one for the other — and you don’t notice that as an error message, you notice it because one voice request takes twice as long as the one before it.

The only way both models stay loaded permanently is two cards. And I had two cards. I had just never looked at them as two cards.

The second machine: a laptop that used to travel

MSI laptop, NVIDIA GeForce RTX 3070 Laptop GPU, 8 GB (8.59 GB as reported). Ampere, so two generations newer than the 2080 Ti in the server. It runs Windows; the services run inside WSL2 (Ubuntu) with systemd=true, RAM capped at 7 GB, 16 CPU threads.

The state I found that WSL install in is the actual reason this became an article. Present: Python 3.12 and the CUDA driver path from Windows (/dev/dxg, /usr/lib/wsl/lib/libcuda.so). Not present: torch, nvidia-smi, Docker, pip, python3-venv. On top of that a sudo that wants a password, and a WSL user who is not a Windows administrator.

So: everything in userspace with uv. No apt install, no container, no root. That sounds like self-flagellation, but it has one pleasant side effect — whatever runs this way runs traceably out of exactly one directory.

Small aside I tripped over during a network scan: the laptop’s MAC identifies as “Intel Corporate”, not MSI. Laptops advertise the Wi-Fi chipset vendor. If you go looking through your own network for “the MSI”, you won’t find it.

Trap 1: faster-whisper does not pull CUDA in for you

This is the most expensive trap of the whole move, because it is silent. faster-whisper does not install the CUDA libraries with it. If they’re missing, the service doesn’t die — it falls back to cpu/int8 without comment and keeps working like a good boy. The journal gets exactly one line about it:

could not load large-v3-turbo on cuda (Library libcublas.so.12 is not found)

Here’s the difference that one line hides:

Device	time per transcription
cuda/float16	0.47 s
silent fallback to cpu/int8	10.8 s

Fix:

uv pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*

And the lesson I now staple to every reinstall: after each reinstall, grep the journal for on cuda versus on cpu. Not for errors — there are none. When it works, it says:

wyoming-whisper: loaded large-v3-turbo on cuda/float16 in 4.1s

Trap 2: torchaudio pulls in the wrong CUDA

The requirements resolved torchaudio to a CUDA 13 build while torch was on 2.6.0+cu124. The TTS server then died with this:

OSError: libcudart.so.13: cannot open shared object file

The fix is a pin, not debugging:

uv pip install "torchaudio==2.6.0" --torch-backend=cu124

This is the moment you notice that “local and cloud-free” does not mean “without somebody else’s dependencies”. It only means the dependencies sit on my disk instead of in a data center whose terms of service change next year.

Trap 3: the wrong voice, and why it was faster

My favorite trap. espeak-ng is an apt package and I have no root. Without espeak, phonemizer throws:

espeak not installed on your system

HAL synthesis failed as a result — and the server fell back to the Piper voice. So audio came out. The only externally visible symptom was: wrong voice. In the TTS service’s journal, September 17 between 20:00 and 20:05, eight of these:

ERROR hal-tts.server: HAL synthesis failed; trying piper fallback

The clue: Piper renders in about 0.15 s, HAL needs about 0.3 s. So the wrong voice shows up as a suspiciously good benchmark. A system that suddenly got twice as fast is not a win, it’s a hint that something other than what you think is running.

The fix without root: the espeakng-loader wheel ships libespeak-ng.so itself. The systemd unit points PHONEMIZER_ESPEAK_LIBRARY and ESPEAK_DATA_PATH at it, done. No apt, no admin, no calling myself for permission.

Trap 4: the log that lies

The shim on the server (ha-hermes-ollama-shim.py) logs

backend=qwen3-tts/voice-design

as a hardcoded string, entirely regardless of what actually rendered the audio. Believe the log and you end up with a voice you don’t use plus a log line confirming the opposite.

The only trustworthy source is the journal on the TTS box itself: backend=hal-styletts2 versus backend=piper, plus a grep -c 'piper fallback' that returns zero. Last seven days here: 157 renders with backend=hal-styletts2, not a single fallback.

Trap 5: “the laptop is offline” (it isn’t)

The laptop swallows ICMP and refuses every TCP port from outside — Windows Defender Firewall, Public profile. ping: 100% loss. nmap -p-: finds nothing. At the same moment I had an SSH session into WSL open and working fine.

Concluding from a scan that a Windows machine is offline is simply wrong. Test with the real ssh command, never with a ping or a port scan.

Then the second half of the same problem: the Wi-Fi was categorized as Public. A firewall rule scoped to “Private” then never matches — silently. It’s in the list, it’s enabled, it does nothing.

WSL here runs networkingMode=mirrored, which means a WSL service listening on 0.0.0.0 shares the Windows host’s LAN IP. No netsh portproxy needed, just the firewall rule. In an elevated PowerShell:

New-NetFirewallRule -DisplayName "WSL voice services (LAN)" `
  -Direction Inbound -Action Allow -Protocol TCP `
  -LocalPort 10300,11436 -RemoteAddress 192.168.1.0/24 -Profile Any
powercfg /change standby-timeout-ac 0
Set-NetConnectionProfile -InterfaceAlias "Wi-Fi" -NetworkCategory Private

Practical tip: before installing anything heavy, test the port with a throwaway listener:

python3 -m http.server 10300 --bind 0.0.0.0

If that isn’t reachable from outside, it’s not your model’s fault.

And one rule worth tattooing somewhere: never run wsl.exe --terminate remotely. It kills sshd with it — which is exactly the door you’re standing in. Recovery then means walking to the laptop. Restart individual services with systemctl --user restart instead, and nothing else.

What actually runs now

Two systemctl --user services with linger enabled, so they start without an SSH session. Both up continuously since the September 17 reboot, so more than seven days:

service	port	what
wyoming-stt.service	10300	faster-whisper large-v3-turbo, cuda/float16, --language auto, --idle-unload 0
hal-tts.service	11436	HAL voice, StyleTTS2, --preload, 0.84 GiB on cuda

idle_unload=0 on both — permanently resident. That’s the entire point of the move. Nothing evicts anything anymore.

The real ExecStart of the STT unit, including the script name left over from the gemma detour in part 1 (the unit file now carries a comment admitting the name lies):

ExecStart=%h/.hermes/venv-wyoming/bin/python %h/.hermes/bin/wyoming-gemma-stt.py --uri tcp://0.0.0.0:10300 --model large-v3-turbo --language auto --idle-unload 0

On the server, the small shim on port 11435 that the voice pipeline talks to stays where it is. It just points somewhere else now:

ExecStart=%h/.hermes/bin/ha-hermes-ollama-shim.py --host 0.0.0.0 --port 11435 --hermes-url http://127.0.0.1:8643 --stt-host 192.168.1.205 --tts-host 192.168.1.205

Two flags. That is the entire server-side rebuild. The rest was installation work on the laptop.

And the 2080 Ti? It belongs to the chatbot again. Verified at 43 out of 43 layers on the GPU. Ollama still loads and unloads it on demand, which is normal and was never the problem — the problem was the second roommate.

The numbers, including the one that got worse

Full voice turn through the live /voice endpoint, measured to first audio:

	cold	warm
before (everything on the server)	4.97 s	1.58 s
after (STT+TTS on the laptop)	2.29 s	1.81 s

Cold more than halved, because nothing gets evicted and reloaded anymore. Warm got about 0.2 s worse — that’s the LAN hop, and I can’t engineer it away. Latency from server to laptop is 18 ms median over six probes.

I’m writing that down because it’s the honest balance sheet: the move improved the worst case a lot and made the best case slightly worse. A quarter second I’ll trade for never waiting five seconds again.

After the real reboot on September 17, the startup looked like this:

20:21:33  WSL up
20:21:39  wyoming-stt
20:21:53  hal-tts
          first full voice turn: 2.02 s, HAL voice

The first voice turn after any restart takes 6–7 seconds though — lazy load. One-off, and still precisely the moment where you assume it’s broken.

And yes: the TTS return leg from part 1 exists now. The voice turn answers out loud instead of only writing. How that arrives at the microphone end is its own article and deliberately not the topic here.

Honestly: what’s shaky about this setup

Reboot recovery needs a logon. A VBS script in the Windows Startup folder launches WSL hidden — but at logon, not at boot. After that, systemd linger brings both services up. Which means: if Windows grabs an update at 3 a.m. and restarts, the voice assistant is dead until somebody signs in. I’ve accepted that for now. The proper fix would be a scheduled task “AtStartup” running as SYSTEM — that needs admin rights and is untested here. It might start a second WSL instance that collides on the same ports. So I am not claiming it works.

It’s a laptop. Sleep on AC is disabled; on battery it still suspends after two hours, and closing the lid means the voice assistant is gone. This thing used to travel. Now it stays home — as a habit, not a guarantee.

Two machines are two failure points. The voice assistant now depends on a Windows laptop. That isn’t an architecture decision, it’s an availability decision, and it’s worse than before.

Warm latency got slower. 1.58 s → 1.81 s. It’s in the table above, and it’s here too.

WSL is capped at 7 GB of RAM. Thin for bigger models, even though the VRAM would be fine.

A rollback exists. The old units on the server (hal-tts.service and wyoming-gemma-stt.service) are stopped and disabled but still installed. Drop the two flags from the shim, re-enable the old units, done.

And the most work-saving point of all: most readers need none of this. If you have one GPU and no competing live workload on it, just run Whisper on the GPU and go home. This entire setup exists because a wedding chatbot squats on my card and isn’t allowed to leave.

The checklist I built out of those five traps

Only one of the five traps actually crashed. The rest had one thing in common: no red banner, no alert. It just ran wrong, or looked dead while it wasn’t. That’s what this list is for, and I now walk it after every change:

1. Is it actually on the GPU? Grep the journal for on cuda versus on cpu. A transcription taking twenty times longer than it should looks, from inside Home Assistant, like “a bit slow today”.
2. Is the voice I think is rendering actually rendering? Don’t check the shim, it lies by constant. Grep backend= on the TTS box and run grep -c 'piper fallback' as well. Anything but zero is a finding.
3. Did it get unexpectedly fast? Don’t celebrate, investigate. Every time something here got faster, it was a fallback, never an improvement.
4. New port? Check it with python3 -m http.server before installing anything.
5. “Unreachable” is not a statement on Windows. Before debugging the service, establish whether a packet could ever arrive at all — Public versus Private profile — and remember ping proves nothing here.

The list isn’t clever. It’s just the answer to a setup that lied to me more often than it failed.

Where I landed

In part 1 I wrote that you should lower the resident model’s num_ctx rather than shoot it. That was the right instinct and still the wrong road — because I was comparing two options that both assumed there is exactly one card. “Which model has to give way?” was the wrong question from the start.

The answer was a laptop that used to travel and now has a day job. Not elegant, not planned, pretty much the opposite of architecture. But the chatbot has its card, Whisper and the voice have theirs, and nothing evicts anything anymore.

One more thing became clear on the way: the whole move was possible because every building block in this setup is a host and a port. Two flags on the shim and half the voice chain lives on another machine. With any of the cloud services I could have used instead, that question would never even have come up — nothing moves house there, you just cancel.