Speech to text

swarm-stt turns the voice memos people send agents into text, once per memo for the whole swarm. This page covers the core in the swarm-stt crate. The daemon around it, its deploy options and the matrix side follow in later changes.

What a request does

Service::transcribe takes the bytes of an audio file and answers its text:

It rejects empty audio, and audio over max_bytes, before any work.
It hashes the bytes with SHA-256. The hash names the cache entry.
A cache hit answers at once with cached: true.
A miss joins the run already in flight for that hash, or starts one. Two requests for one memo that arrive together share one engine run, and only the run's starter sees cached: false.
The engine runs one at a time, because whisper uses every core it gets.
The text goes into the cache, then back to every waiting request. A failed run caches nothing, so the next request tries again.

The cache

One <sha256>.txt file per memo, in a directory. An entry lives for the TTL counted from its write, and a read of an expired entry removes it. purge removes every expired entry. A write goes to a temporary name first and then moves into place with a rename, so a reader never sees half a transcript.

The engine

The Engine trait takes bytes and returns text. Whisper is the real one: ffmpeg decodes any container to 16 kHz mono WAV, cutting it at max_secs, and whisper-cli from whisper.cpp reads the WAV with the configured model and language detection on. Before anything decodes, sniff checks that the file starts like an audio container (Ogg, WAV, FLAC, MP3, AMR, MP4 or M4A, WebM); the daemon refuses a playlist, an image or a script without starting ffmpeg. Decoding is pipe-only (-protocol_whitelist pipe): the audio comes from rooms with outside people, so a file that names other files or URLs gets no way to open them. The service kills a run that outlasts its timeout (decode and transcribe together) and fails the request, so one hung child can't hold the single engine slot. A scratch directory per run holds the WAV and goes away with the run.

The daemon

swarm-stt connects to the queue as its own OIDC client, with its secret read from the store under its own certificate, as the notification router does. It subscribes to $SWARM.stt.transcribe.* in the queue group swarm-stt and answers each request on its reply subject. The body of a request is the raw audio. The answer is JSON: {"status":"ok","text":…,"cached":…} or {"status":"failed","message":…} (swarm_wire::stt). An agent may publish only on its own $SWARM.stt.transcribe.<agent>, and the service may publish nowhere but the inbox of its own client, so replies leave through NATS reply rights.

The service holds at most MAX_WAITING requests at once, whether running or waiting for the engine's one slot, because each holds its audio in memory, and at most one per asking agent (the last token of the subject), so one agent can't fill every slot. It answers a request past either limit with a failed response saying so. MAX_WAITING times RUN_TIMEOUT_SECS must stay under the 15-minute reply deadline, because a request at the back of the line waits for every run ahead of it; the daemon refuses to start with settings that break that. A failed message tells the sender briefly what went wrong; the tools' own output and the host's error text go to the log only. It removes expired transcripts every hour.

Settings

All are environment variables with the prefix SWARM_STT_; the nix module will set them.

variable	default	meaning
`STATE_DIR`	required	holds `cache/` and `scratch/`
`FFMPEG`, `WHISPER_CLI`, `MODEL`	required	the two binaries and the ggml model file
`CACHE_TTL_SECS`	one day	how long a transcript stays
`MAX_BYTES`	6 MiB	the largest audio file taken; the queue carries 8 MiB
`MAX_AUDIO_SECS`	300	seconds of audio decoded; ffmpeg drops the rest
`RUN_TIMEOUT_SECS`	240	the longest one run may take
`MAX_WAITING`	3	requests held at once
`NATS_URL`, `OIDC_TOKEN_ENDPOINT`, `OIDC_CLIENT_ID`, `OIDC_CA_FILE`	required, the CA file optional	the queue connection

Two things the daemon doesn't do yet: sweep scratch directories a crash left behind, and set whisper's thread count. The issue 6383 discussion has the choices behind the defaults.

5.3 KiB Raw Blame History