814edd1d67
Each service section lost its container, credential and metrics detail to a Host setup section on its own implementation page, unchanged apart from links. The four agentAuth option rows that sat above the page title moved into the telemetry page's metrics option table.
108 lines
6.7 KiB
Markdown
108 lines
6.7 KiB
Markdown
# Speech to text
|
|
|
|
`swarm-stt` turns the voice memos people send agents into text, once per memo
|
|
for the whole swarm. This page covers the core in the `swarm-stt` crate. The
|
|
daemon around it, its deploy options and the matrix side follow in later
|
|
changes.
|
|
|
|
## What a request does
|
|
|
|
`Service::transcribe` takes the bytes of an audio file and answers its text:
|
|
|
|
1. It rejects empty audio, and audio over `max_bytes`, before any work.
|
|
2. It hashes the bytes with SHA-256. The hash names the cache entry.
|
|
3. A cache hit answers at once with `cached: true`.
|
|
4. A miss joins the run already in flight for that hash, or starts one. Two
|
|
requests for one memo that arrive together share one engine run, and only
|
|
the run's starter sees `cached: false`.
|
|
5. The engine runs one at a time, because whisper uses every core it gets.
|
|
6. The text goes into the cache, then back to every waiting request. A failed
|
|
run caches nothing, so the next request tries again.
|
|
|
|
## The cache
|
|
|
|
One `<sha256>.txt` file per memo, in a directory. An entry lives for the TTL
|
|
counted from its write, and a read of an expired entry removes it. `purge`
|
|
removes every expired entry. A write goes to a temporary name first and then moves
|
|
into place with a rename, so a reader never sees half a transcript.
|
|
|
|
## The engine
|
|
|
|
The `Engine` trait takes bytes and returns text. `Whisper` is the real one:
|
|
`ffmpeg` decodes any container to 16 kHz mono WAV, cutting it at `max_secs`,
|
|
and `whisper-cli` from whisper.cpp reads the WAV with the configured model and
|
|
language detection on. Before anything decodes, `sniff` checks that the file starts like an audio
|
|
container (Ogg, WAV, FLAC, MP3, AMR, MP4 or M4A, WebM); the daemon refuses a playlist, an image or a
|
|
script without starting `ffmpeg`. Decoding is pipe-only (`-protocol_whitelist pipe`): the audio comes from
|
|
rooms with outside people, so a file that names other files or URLs gets no
|
|
way to open them. The service kills a run that outlasts its timeout (decode and transcribe
|
|
together) and fails the request, so one hung child can't hold the single
|
|
engine slot. A scratch directory per run holds the WAV and goes away with the run.
|
|
|
|
## The daemon
|
|
|
|
`swarm-stt` connects to the queue as its own OIDC client, with its secret read
|
|
from the store under its own certificate, as the notification router does. It
|
|
subscribes to `$SWARM.stt.transcribe.*` in the queue group `swarm-stt` and
|
|
answers each request on its reply subject. The body of a request is the raw
|
|
audio. The answer is JSON: `{"status":"ok","text":…,"cached":…}` or
|
|
`{"status":"failed","message":…}` (`swarm_wire::stt`). An agent may publish
|
|
only on its own `$SWARM.stt.transcribe.<agent>`, and the service may publish
|
|
nowhere but the inbox of its own client, so replies leave through NATS reply
|
|
rights.
|
|
|
|
The service holds at most `MAX_WAITING` requests at once, whether running or
|
|
waiting for the engine's one slot, because each holds its audio in memory, and
|
|
at most one per asking agent (the last token of the subject), so one agent
|
|
can't fill every slot. It answers a request past either limit with a `failed`
|
|
response saying so. `MAX_WAITING` times `RUN_TIMEOUT_SECS` must stay under the
|
|
15-minute reply deadline, because a request at the back of the line waits for
|
|
every run ahead of it; the daemon refuses to start with settings that break
|
|
that. A `failed` message tells the sender briefly what went wrong; the tools'
|
|
own output and the host's error text go to the log only. It
|
|
removes expired transcripts every hour.
|
|
|
|
## Settings
|
|
|
|
All are environment variables with the prefix `SWARM_STT_`; the nix module will
|
|
set them.
|
|
|
|
| variable | default | meaning |
|
|
| ------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- |
|
|
| `STATE_DIR` | required | holds `cache/` and `scratch/` |
|
|
| `FFMPEG`, `WHISPER_CLI`, `MODEL` | required | the two binaries and the ggml model file |
|
|
| `CACHE_TTL_SECS` | one day | how long a transcript stays |
|
|
| `MAX_BYTES` | 6 MiB | the largest audio file taken; the queue carries 8 MiB |
|
|
| `MAX_AUDIO_SECS` | 300 | seconds of audio decoded; ffmpeg drops the rest |
|
|
| `RUN_TIMEOUT_SECS` | 240 | the longest one run may take |
|
|
| `MAX_WAITING` | 3 | requests held at once |
|
|
| `NATS_URL`, `OIDC_TOKEN_ENDPOINT`, `OIDC_CLIENT_ID`, `OIDC_CA_FILE` | required, the CA file optional | the queue connection |
|
|
|
|
Two things the daemon doesn't do yet: sweep scratch directories a crash left
|
|
behind, and set whisper's thread count. The issue 6383 discussion has the
|
|
choices behind the defaults.
|
|
|
|
## Host setup
|
|
|
|
The speech-to-text service, in a `swarm-stt` container on the host that sets
|
|
`deploy.stt.enable`. `allSwarmServices` defaults it on wherever the host has the
|
|
service's store identity, like the other swarm services. Run it where the inference
|
|
hardware is: it transcribes with whisper.cpp.
|
|
|
|
An agent that receives a voice memo sends the audio to the service over the
|
|
queue, on its own subject, and gets the text back. The service keeps each
|
|
transcript for a day, keyed by a hash of the audio, so the same memo sent to several agents costs
|
|
one whisper run. `deploy.stt.settings.modelSize` picks the
|
|
model (`small` by default, or `base`), and `modelFile` takes any other ggml
|
|
model. The cache time, the limits on file size, audio length and run time, and
|
|
how many requests it holds at once are settings next to it.
|
|
|
|
The audio comes from rooms with outside people, so the service runs as its own
|
|
unprivileged user under a tight systemd sandbox, with a memory ceiling and
|
|
a CPU quota. Its network isn't restricted: the decoder runs in the same
|
|
network namespace as the service and can read its credentials, so a decoder
|
|
exploit could read them and send them anywhere the container can reach. Closing
|
|
that's for the strict-egress work on service containers, or for running the
|
|
decode with no network and no credentials. The file's first bytes must look like an audio container
|
|
before anything decodes it. Its leaf and its queue client work as the notification router's
|
|
do.
|