hyperhive/docs/4-implementation/swarm/stt.md

Speech to text

swarm-stt turns the voice memos people send agents into text, once per memo for the whole swarm. This page covers the core in the swarm-stt crate. The daemon around it, its deploy options and the matrix side follow in later changes.

What a request does

Service::transcribe takes the bytes of an audio file and answers its text:

1. It rejects empty audio, and audio over max_bytes, before any work.
2. It hashes the bytes with SHA-256. The hash names the cache entry.
3. A cache hit answers at once with cached: true.
4. A miss joins the run already in flight for that hash, or starts one. Two requests for one memo that arrive together share one engine run, and only the run's starter sees cached: false.
5. The engine runs one at a time, because whisper uses every core it gets.
6. The text goes into the cache, then back to every waiting request. A failed run caches nothing, so the next request tries again.

The cache

One <sha256>.txt file per memo, in a directory. An entry lives for the TTL counted from its write, and a read of an expired entry removes it. purge removes every expired entry. A write goes to a temporary name first and then moves into place with a rename, so a reader never sees half a transcript.

The engine

The Engine trait takes bytes and returns text. Whisper is the real one: ffmpeg decodes any container to 16 kHz mono WAV, cutting it at max_secs, and whisper-cli from whisper.cpp reads the WAV with the configured model and language detection on. Before anything decodes, sniff checks that the file starts like an audio container (Ogg, WAV, FLAC, MP3, AMR, MP4 or M4A, WebM); the daemon refuses a playlist, an image or a script without starting ffmpeg. Decoding is pipe-only (-protocol_whitelist pipe): the audio comes from rooms with outside people, so a file that names other files or URLs gets no way to open them. The service kills a run that outlasts its timeout (decode and transcribe together) and fails the request, so one hung child can't hold the single engine slot. A scratch directory per run holds the WAV and goes away with the run.

The daemon

swarm-stt connects to the queue as its own OIDC client, with its secret read from the store under its own certificate, as the notification router does. It subscribes to $SWARM.stt.transcribe.* in the queue group swarm-stt and answers each request on its reply subject. The body of a request is the raw audio. The answer is JSON: {"status":"ok","text":…,"cached":…} or {"status":"failed","message":…} (swarm_wire::stt). An agent may publish only on its own $SWARM.stt.transcribe.<agent>, and the service may publish nowhere but the inbox of its own client, so replies leave through NATS reply rights.

The service holds at most MAX_WAITING requests at once, whether running or waiting for the engine's one slot, because each holds its audio in memory, and at most one per asking agent (the last token of the subject), so one agent can't fill every slot. It answers a request past either limit with a failed response saying so. MAX_WAITING times RUN_TIMEOUT_SECS must stay under the 15-minute reply deadline, because a request at the back of the line waits for every run ahead of it; the daemon refuses to start with settings that break that. A failed message tells the sender briefly what went wrong; the tools' own output and the host's error text go to the log only. It removes expired transcripts every hour.

Settings

All are environment variables with the prefix SWARM_STT_; the nix module will set them.

variable	default	meaning
STATE_DIR	required	holds cache/ and scratch/
FFMPEG, WHISPER_CLI, MODEL	required	the two binaries and the ggml model file
CACHE_TTL_SECS	one day	how long a transcript stays
MAX_BYTES	6 MiB	the largest audio file taken; the queue carries 8 MiB
MAX_AUDIO_SECS	300	seconds of audio decoded; ffmpeg drops the rest
RUN_TIMEOUT_SECS	240	the longest one run may take
MAX_WAITING	3	requests held at once
NATS_URL, OIDC_TOKEN_ENDPOINT, OIDC_CLIENT_ID, OIDC_CA_FILE	required, the CA file optional	the queue connection

Two things the daemon doesn't do yet: sweep scratch directories a crash left behind, and set whisper's thread count. The issue 6383 discussion has the choices behind the defaults.

Host setup

The speech-to-text service, in a swarm-stt container on the host that sets deploy.stt.enable. allSwarmServices defaults it on wherever the host has the service's store identity, like the other swarm services. Run it where the inference hardware is: it transcribes with whisper.cpp.

An agent that receives a voice memo sends the audio to the service over the queue, on its own subject, and gets the text back. The service keeps each transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs one whisper run. deploy.stt.settings.modelSize picks the model (small by default, or base), and modelFile takes any other ggml model. The cache time, the limits on file size, audio length and run time, and how many requests it holds at once are settings next to it.

The audio comes from rooms with outside people, so the service runs as its own unprivileged user under a tight systemd sandbox, with a memory ceiling and a CPU quota. Its network isn't restricted: the decoder runs in the same network namespace as the service and can read its credentials, so a decoder exploit could read them and send them anywhere the container can reach. Closing that's for the strict-egress work on service containers, or for running the decode with no network and no credentials. The file's first bytes must look like an audio container before anything decodes it. Its leaf and its queue client work as the notification router's do.