hyperhive/docs/4-implementation/swarm/stt.md

Speech to text

swarm-stt turns the voice memos people send agents into text, once per memo for the whole swarm. This page covers the core in the swarm-stt crate. The daemon around it, its deploy options and the matrix side follow in later changes.

What a request does

Service::transcribe takes the bytes of an audio file and answers its text:

1. It rejects empty audio, and audio over max_bytes, before any work.
2. It hashes the bytes with SHA-256. The hash names the cache entry.
3. A cache hit answers at once with cached: true.
4. A miss joins the run already in flight for that hash, or starts one. Two requests for one memo that arrive together share one engine run, and only the run's starter sees cached: false.
5. The engine runs one at a time, because whisper uses every core it gets.
6. The text goes into the cache, then back to every waiting request. A failed run caches nothing, so the next request tries again.

The cache

One <sha256>.txt file per memo, in a directory. An entry lives for the TTL counted from its write, and a read of an expired entry removes it. purge removes every expired entry. A write goes to a temporary name first and then moves into place with a rename, so a reader never sees half a transcript.

The engine

The Engine trait takes bytes and returns text. Whisper is the real one: ffmpeg decodes any container to 16 kHz mono WAV, cutting it at max_secs, and whisper-cli from whisper.cpp reads the WAV with the configured model and language detection on. Decoding is pipe-only (-protocol_whitelist pipe): the audio comes from rooms with outside people, so a file that names other files or URLs gets no way to open them. The service kills a run that outlasts its timeout (decode and transcribe together) and fails the request, so one hung child can't hold the single engine slot. A scratch directory per run holds the WAV and goes away with the run.

Settings the daemon will expose

The model (default small), the cache TTL (default one day), the size, length and run-time caps. The daemon change also owns a bound on how many requests may wait for the engine slot, a sweep of scratch directories a crash left behind, and whisper's thread count. The design and the choices behind them are in the discussion on issue 6383.