83 lines
5.3 KiB
Markdown
83 lines
5.3 KiB
Markdown
# Speech to text
|
|
|
|
`swarm-stt` turns the voice memos people send agents into text, once per memo
|
|
for the whole swarm. This page covers the core in the `swarm-stt` crate. The
|
|
daemon around it, its deploy options and the matrix side follow in later
|
|
changes.
|
|
|
|
## What a request does
|
|
|
|
`Service::transcribe` takes the bytes of an audio file and answers its text:
|
|
|
|
1. It rejects empty audio, and audio over `max_bytes`, before any work.
|
|
2. It hashes the bytes with SHA-256. The hash names the cache entry.
|
|
3. A cache hit answers at once with `cached: true`.
|
|
4. A miss joins the run already in flight for that hash, or starts one. Two
|
|
requests for one memo that arrive together share one engine run, and only
|
|
the run's starter sees `cached: false`.
|
|
5. The engine runs one at a time, because whisper uses every core it gets.
|
|
6. The text goes into the cache, then back to every waiting request. A failed
|
|
run caches nothing, so the next request tries again.
|
|
|
|
## The cache
|
|
|
|
One `<sha256>.txt` file per memo, in a directory. An entry lives for the TTL
|
|
counted from its write, and a read of an expired entry removes it. `purge`
|
|
removes every expired entry. A write goes to a temporary name first and then moves
|
|
into place with a rename, so a reader never sees half a transcript.
|
|
|
|
## The engine
|
|
|
|
The `Engine` trait takes bytes and returns text. `Whisper` is the real one:
|
|
`ffmpeg` decodes any container to 16 kHz mono WAV, cutting it at `max_secs`,
|
|
and `whisper-cli` from whisper.cpp reads the WAV with the configured model and
|
|
language detection on. Before anything decodes, `sniff` checks that the file starts like an audio
|
|
container (Ogg, WAV, FLAC, MP3, AMR, MP4 or M4A, WebM); the daemon refuses a playlist, an image or a
|
|
script without starting `ffmpeg`. Decoding is pipe-only (`-protocol_whitelist pipe`): the audio comes from
|
|
rooms with outside people, so a file that names other files or URLs gets no
|
|
way to open them. The service kills a run that outlasts its timeout (decode and transcribe
|
|
together) and fails the request, so one hung child can't hold the single
|
|
engine slot. A scratch directory per run holds the WAV and goes away with the run.
|
|
|
|
## The daemon
|
|
|
|
`swarm-stt` connects to the queue as its own OIDC client, with its secret read
|
|
from the store under its own certificate, as the notification router does. It
|
|
subscribes to `$SWARM.stt.transcribe.*` in the queue group `swarm-stt` and
|
|
answers each request on its reply subject. The body of a request is the raw
|
|
audio. The answer is JSON: `{"status":"ok","text":…,"cached":…}` or
|
|
`{"status":"failed","message":…}` (`swarm_wire::stt`). An agent may publish
|
|
only on its own `$SWARM.stt.transcribe.<agent>`, and the service may publish
|
|
nowhere but the inbox of its own client, so replies leave through NATS reply
|
|
rights.
|
|
|
|
The service holds at most `MAX_WAITING` requests at once, whether running or
|
|
waiting for the engine's one slot, because each holds its audio in memory, and
|
|
at most one per asking agent (the last token of the subject), so one agent
|
|
can't fill every slot. It answers a request past either limit with a `failed`
|
|
response saying so. `MAX_WAITING` times `RUN_TIMEOUT_SECS` must stay under the
|
|
15-minute reply deadline, because a request at the back of the line waits for
|
|
every run ahead of it; the daemon refuses to start with settings that break
|
|
that. A `failed` message tells the sender briefly what went wrong; the tools'
|
|
own output and the host's error text go to the log only. It
|
|
removes expired transcripts every hour.
|
|
|
|
## Settings
|
|
|
|
All are environment variables with the prefix `SWARM_STT_`; the nix module will
|
|
set them.
|
|
|
|
| variable | default | meaning |
|
|
| ------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- |
|
|
| `STATE_DIR` | required | holds `cache/` and `scratch/` |
|
|
| `FFMPEG`, `WHISPER_CLI`, `MODEL` | required | the two binaries and the ggml model file |
|
|
| `CACHE_TTL_SECS` | one day | how long a transcript stays |
|
|
| `MAX_BYTES` | 6 MiB | the largest audio file taken; the queue carries 8 MiB |
|
|
| `MAX_AUDIO_SECS` | 300 | seconds of audio decoded; ffmpeg drops the rest |
|
|
| `RUN_TIMEOUT_SECS` | 240 | the longest one run may take |
|
|
| `MAX_WAITING` | 3 | requests held at once |
|
|
| `NATS_URL`, `OIDC_TOKEN_ENDPOINT`, `OIDC_CLIENT_ID`, `OIDC_CA_FILE` | required, the CA file optional | the queue connection |
|
|
|
|
Two things the daemon doesn't do yet: sweep scratch directories a crash left
|
|
behind, and set whisper's thread count. The issue 6383 discussion has the
|
|
choices behind the defaults.
|