# Speech to text

`swarm-stt` turns the voice memos people send agents into text, once per memo
for the whole swarm. This page covers the core in the `swarm-stt` crate. The
daemon around it, its deploy options and the matrix side follow in later
changes.

## What a request does

`Service::transcribe` takes the bytes of an audio file and answers its text:

1. It rejects empty audio, and audio over `max_bytes`, before any work.
2. It hashes the bytes with SHA-256. The hash names the cache entry.
3. A cache hit answers at once with `cached: true`.
4. A miss joins the run already in flight for that hash, or starts one. Two
   requests for one memo that arrive together share one engine run, and only
   the run's starter sees `cached: false`.
5. The engine runs one at a time, because whisper uses every core it gets.
6. The text goes into the cache, then back to every waiting request. A failed
   run caches nothing, so the next request tries again.

## The cache

One `<sha256>.txt` file per memo, in a directory. An entry lives for the TTL
counted from its write, and a read of an expired entry removes it. `purge`
removes every expired entry. A write goes to a temporary name first and then moves
into place with a rename, so a reader never sees half a transcript.

## The engine

The `Engine` trait takes bytes and returns text. `Whisper` is the real one:
`ffmpeg` decodes any container to 16 kHz mono WAV, cutting it at `max_secs`,
and `whisper-cli` from whisper.cpp reads the WAV with the configured model and
language detection on. Before anything decodes, `sniff` checks that the file starts like an audio
container (Ogg, WAV, FLAC, MP3, AMR, MP4 or M4A, WebM); the daemon refuses a playlist, an image or a
script without starting `ffmpeg`. Decoding is pipe-only (`-protocol_whitelist pipe`): the audio comes from
rooms with outside people, so a file that names other files or URLs gets no
way to open them. The service kills a run that outlasts its timeout (decode and transcribe
together) and fails the request, so one hung child can't hold the single
engine slot. A scratch directory per run holds the WAV and goes away with the run.

## The daemon

`swarm-stt` connects to the queue as its own OIDC client, with its secret read
from the store under its own certificate, as the notification router does. It
subscribes to `$SWARM.stt.transcribe.*` in the queue group `swarm-stt` and
answers each request on its reply subject. The body of a request is the raw
audio. The answer is JSON: `{"status":"ok","text":…,"cached":…}` or
`{"status":"failed","message":…}` (`swarm_wire::stt`). An agent may publish
only on its own `$SWARM.stt.transcribe.<agent>`, and the service may publish
nowhere but the inbox of its own client, so replies leave through NATS reply
rights.

The service holds at most `MAX_WAITING` requests at once, whether running or
waiting for the engine's one slot, because each holds its audio in memory. It
answers a request past that with a `failed` response saying it's busy. It
removes expired transcripts every hour.

## Settings

All are environment variables with the prefix `SWARM_STT_`; the nix module will
set them.

| variable                                                            | default                        | meaning                                               |
| ------------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------- |
| `STATE_DIR`                                                         | required                       | holds `cache/` and `scratch/`                         |
| `FFMPEG`, `WHISPER_CLI`, `MODEL`                                    | required                       | the two binaries and the ggml model file              |
| `CACHE_TTL_SECS`                                                    | one day                        | how long a transcript stays                           |
| `MAX_BYTES`                                                         | 6 MiB                          | the largest audio file taken; the queue carries 8 MiB |
| `MAX_AUDIO_SECS`                                                    | 300                            | seconds of audio decoded; ffmpeg drops the rest       |
| `RUN_TIMEOUT_SECS`                                                  | 300                            | the longest one run may take                          |
| `MAX_WAITING`                                                       | 8                              | requests held at once                                 |
| `NATS_URL`, `OIDC_TOKEN_ENDPOINT`, `OIDC_CLIENT_ID`, `OIDC_CA_FILE` | required, the CA file optional | the queue connection                                  |

Two things the daemon doesn't do yet: sweep scratch directories a crash left
behind, and set whisper's thread count. The issue 6383 discussion has the
choices behind the defaults.