46 lines
2.3 KiB
Markdown
46 lines
2.3 KiB
Markdown
# Speech to text
|
|
|
|
`swarm-stt` turns the voice memos people send agents into text, once per memo
|
|
for the whole swarm. This page covers the core in the `swarm-stt` crate. The
|
|
daemon around it, its deploy options and the matrix side follow in later
|
|
changes.
|
|
|
|
## What a request does
|
|
|
|
`Service::transcribe` takes the bytes of an audio file and answers its text:
|
|
|
|
1. It rejects empty audio, and audio over `max_bytes`, before any work.
|
|
2. It hashes the bytes with SHA-256. The hash names the cache entry.
|
|
3. A cache hit answers at once with `cached: true`.
|
|
4. A miss joins the run already in flight for that hash, or starts one. Two
|
|
requests for one memo that arrive together share one engine run, and only
|
|
the run's starter sees `cached: false`.
|
|
5. The engine runs one at a time, because whisper uses every core it gets.
|
|
6. The text goes into the cache, then back to every waiting request. A failed
|
|
run caches nothing, so the next request tries again.
|
|
|
|
## The cache
|
|
|
|
One `<sha256>.txt` file per memo, in a directory. An entry lives for the TTL
|
|
counted from its write, and a read of an expired entry removes it. `purge`
|
|
removes every expired entry. A write goes to a temporary name first and then moves
|
|
into place with a rename, so a reader never sees half a transcript.
|
|
|
|
## The engine
|
|
|
|
The `Engine` trait takes bytes and returns text. `Whisper` is the real one:
|
|
`ffmpeg` decodes any container to 16 kHz mono WAV, cutting it at `max_secs`,
|
|
and `whisper-cli` from whisper.cpp reads the WAV with the configured model and
|
|
language detection on. Decoding is pipe-only (`-protocol_whitelist pipe`): the audio comes from
|
|
rooms with outside people, so a file that names other files or URLs gets no
|
|
way to open them. The service kills a run that outlasts its timeout (decode and transcribe
|
|
together) and fails the request, so one hung child can't hold the single
|
|
engine slot. A scratch directory per run holds the WAV and goes away with the run.
|
|
|
|
## Settings the daemon will expose
|
|
|
|
The model (default `small`), the cache TTL (default one day), the size,
|
|
length and run-time caps. The daemon change also owns a bound on how many
|
|
requests may wait for the engine slot, a sweep of scratch directories a crash
|
|
left behind, and whisper's thread count. The design and the choices behind them are in the discussion on
|
|
issue 6383.
|