swarm-archive: grant only once an archive is up; keep logins across a gap in placements
The archive host creates a ready marker in the store and swarm-controller writes no credentials before it exists, so a swarm without an archive, and a mixed-version swarm, leave every hive backup a logged no-op. A hive with no declaration no longer counts as placing nothing: its copies and its agents' user lines stay. The restic password is pinned by tests across a revoke, a re-placement and a move.
This commit is contained in:
atlas
parent
306aae0378
commit
91f36f39d2
2 changed files with 241 additions and 52 deletions
|
|
@ -43,6 +43,9 @@ Two properties protect the backups from a compromised hive:
|
|||
other. A hive holds the logins of the agents it places and no others. When an
|
||||
agent moves, the controller gives the new hive a new password and deletes the
|
||||
old hive's copy, so the old hive can't reach the repository any more.
|
||||
Only that login rotates. The controller creates an agent's restic password
|
||||
once, when it first places the agent, and a move, a destroy or a re-placement never
|
||||
changes it, because it's the key of the repository.
|
||||
- **Append-only.** A client can add snapshots and nothing else: no delete, no
|
||||
overwrite. The archive host removes old snapshots.
|
||||
|
||||
|
|
@ -98,23 +101,27 @@ users file refresh and the last prune.
|
|||
agent, and the lines above it say why. The alert waits 30 minutes, so an
|
||||
agent placed a few minutes ago doesn't trip it while its login is on its way.
|
||||
The next hourly run retries.
|
||||
- **`swarm-controller has granted this hive no archive credentials`.** The
|
||||
hive has agents and the store holds no login for any of them. Either
|
||||
`swarm-controller` doesn't have the archive yet (update it), or its archive
|
||||
pass fails: look for `archive:` lines in its journal.
|
||||
- **`swarm-controller has granted this hive no archive credentials`.** Not a
|
||||
failure: the hive logs it and succeeds. It means the store holds no login for
|
||||
any agent here, because no archive is running yet, `swarm-controller` is
|
||||
older than the archive, or its archive pass fails (look for `archive:` lines
|
||||
in its journal). If an archive is running and this persists, check the
|
||||
controller.
|
||||
- **The `archive prune failed` alert.** The daily prune couldn't apply the
|
||||
retention to one repository, for example because a backup held its lock for
|
||||
longer than ten minutes. The next prune retries.
|
||||
- **A swarm without an archive.** If no host sets `deploy.archive.enable`, every
|
||||
hive's run fails. Set `deploy.hive-controller.archive.enable = false` on the
|
||||
hives until one does.
|
||||
- **A swarm without an archive.** Nothing fails and nothing alerts. The archive
|
||||
host announces itself in the secret store when it starts, and
|
||||
`swarm-controller` writes no logins before that, so every hive's run finds
|
||||
nothing to back up and says so in its journal.
|
||||
|
||||
## Mixed versions
|
||||
|
||||
A hive that doesn't have the backup unit yet doesn't back up, and nothing else
|
||||
changes. A `swarm-controller` without the archive pass writes no logins, so
|
||||
hives with the unit fail with the message above until it updates. The archive
|
||||
host admits only the users the controller wrote, so until then it admits nobody.
|
||||
hives with the unit log that they have no credentials and succeed until it
|
||||
updates. The archive host admits only the users the controller wrote, so until
|
||||
then it admits nobody.
|
||||
|
||||
## What lives where
|
||||
|
||||
|
|
|
|||
|
|
@ -60,6 +60,7 @@ the rest of the cell says how.
|
|||
| `swarm/hives/<hive>/archive/<agent>` (fields `rest-password`, `restic-password`) | `swarm-controller`, for every agent a hive places, every 5 minutes: a new random REST password when the agent is new on that hive, the agent's restic password from the leaf below; the copy on a hive that no longer places the agent is deleted | the agent's hive: `hive-archive-backup`, under the hive's own store identity | ✅ a move to another hive writes a new REST password, which locks the old hive out | ✅ read at every backup run |
|
||||
| `swarm/services/archive/users/<agent>` (field `htpasswd`) | `swarm-controller`, with the REST password above: its `{SHA}` hash, never the password; deleted when no hive places the agent | `swarm-archive-users` on the archive's host, which renders the users file into `/run` | ✅ rewritten with the password | ✅ re-read every 2 minutes; the server notices the new file without a restart |
|
||||
| `swarm/services/archive/restic/<agent>` (field `password`) | `swarm-controller`, once per agent, on its first placement; never replaced and never deleted, because it opens every backup of the agent | `swarm-archive-prune` on the archive's host, through `RESTIC_PASSWORD_COMMAND` | ❌ a new password would not open the existing repository | ✅ read at every prune |
|
||||
| `swarm/services/archive/ready` | `swarm-archive-users` on the archive's host, once, when none is stored; `swarm-controller` writes no archive credentials before it exists | `swarm-controller`, every 5 minutes | ❌ never rewritten | ✅ read every 5 minutes |
|
||||
| `swarm/services/ci-metrics/forge/token` | `swarm-controller`, at start and every 5 minutes | `swarm-ci-metrics`, `swarm-controller` | ✅ when missing or no longer matching the forge | ✅ read on every pass |
|
||||
| _(not in the store)_ a hive's mTLS leaf | the store's PKI, or you by hand | the hive itself, from disk | must be stated | must be stated |
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue