diff --git a/docs/2-handbook/external-accounts/ssh.md b/docs/2-handbook/external-accounts/ssh.md index 7c15fa710..805a1db7c 100644 --- a/docs/2-handbook/external-accounts/ssh.md +++ b/docs/2-handbook/external-accounts/ssh.md @@ -60,9 +60,11 @@ whole file, at that path. To work inside the agent's container instead, see key changed and you didn't change the machine, don't clear it: something else answers at that address. -This doesn't open the network. If you set the hive's `egress.forward` to -`reject`, the hive refuses the agent's direct connection to the machine's port -until a separate allowance exists; the default `log` lets it through. See +The SSH switch and the CA don't open the network. The default `egress.forward` +of `log` lets the agent's direct connection to the machine's port through. If +you set it to `reject`, the hive refuses the connection unless you gave the +agent a named connection to that host and port (see +[Named connections](#named-connections)). See [the network page](../../4-implementation/networking/network.md). ## 2. Make a machine trust the CA @@ -111,8 +113,16 @@ host and the port. For example `k8s-lab-vm`, `root`, `muede-pc2`, `2222`. The agent's ssh config then holds a `Host k8s-lab-vm` block with that host, user and port, and the connection uses the same certificate as any other. The block follows a change within ten seconds, and it's empty while the agent's SSH -switch is off, so one switch ends both the certificate and the names. A name -doesn't open the network: see the note under section 1. The routes and the +switch is off, so one switch ends both the certificate and the names. + +A named connection also lets the agent through the hive's egress rule. Under +`egress.forward = "reject"` the hive accepts TCP from that agent to the host +and port of each of its names, and from no other agent. The hive looks the +host up itself, every ten minutes, and accepts every IPv4 address it answers +with, except one inside its own agent bridge subnet. A name must therefore +resolve on the hive host the way it does for you. If the lookup fails or +hangs, the hive keeps the last answer. A new name, a removed one or a +flipped switch reaches the hive's rule in about a minute. The routes and the rules a connection must meet are on [the SSH CA implementation page](../../4-implementation/swarm/ssh-ca.md#named-connections). diff --git a/docs/4-implementation/networking/network.md b/docs/4-implementation/networking/network.md index a23bf938d..23dba437b 100644 --- a/docs/4-implementation/networking/network.md +++ b/docs/4-implementation/networking/network.md @@ -737,11 +737,21 @@ also refuses the flow. leaves alone too (see _Intercepted web traffic_). They leave no line, so a `hive-egress-would-drop` line is exactly a flow that `reject` would refuse, and a service container keeps every port and protocol under `reject`. +- **Named SSH connections.** Right after the `direct_mac` rule the chain + accepts TCP from an agent's MAC to an address and port in the `agent_allow` + set, in both modes, so such a flow leaves no count and no log line either. + The set holds, for each agent whose `ssh` switch is on, the addresses its + connections' hosts resolve to and their ports (see + [the SSH CA page](../swarm/ssh-ca.md#egress)). `hive-c0re` renders them into + `services.hyperhive.deploy.hive-controller.egress.connectionsFile` + (`/var/lib/hive-squid/ssh-connections`), and `hive-egress-pin` fills the set + from the file like `direct_mac`, when the file changes and on its timer. The + set is empty while the file is missing or empty. - **What `reject` refuses.** Everything from the bridge to the uplink that the - chain didn't exempt, whether a flow is new or not: TCP on any port but the - 80 and 443 that squid takes, UDP (DNS to an outside resolver, QUIC and NTP - included), ICMP and IPv6. A TCP flow gets a reset, so the connect fails at - once instead of stalling. Everything else gets ICMP administratively + chain didn't exempt or allow, whether a flow is new or not: TCP on any port + but the 80 and 443 that squid takes, UDP (DNS to an outside resolver, QUIC + and NTP included), ICMP and IPv6. A TCP flow gets a reset, so the connect + fails at once instead of stalling. Everything else gets ICMP administratively prohibited. The log lines of this mode carry the prefix `hive-egress-reject ` (and `hive-egress-reject-unmetered `); the rate limit and the counter work as in `log` mode. Still open to every agent are the host's own addresses: the @@ -761,6 +771,11 @@ also refuses the flow. agent in the gap. - An unreadable `unrestricted` file lists no agent, so `unrestricted` agents lose every non-web flow. + - A new named connection, or a flipped `ssh` switch, reaches the set about a + minute after the write: the hive reads the swarm every 30 seconds and the + file's change runs a pass at once. A host that moves to an address the hive + hasn't looked up yet gets refusals for up to ten minutes. An agent whose + connection names a host that never resolved has no allowance for it. - An agent in per-agent `log` mode keeps its web traffic, which squid only observes, and loses every other port and protocol. - **Which traffic.** IPv4 and IPv6 alike (the table is `inet`). The chain never diff --git a/docs/4-implementation/swarm/ssh-ca.md b/docs/4-implementation/swarm/ssh-ca.md index d9ac95ace..1498236f5 100644 --- a/docs/4-implementation/swarm/ssh-ca.md +++ b/docs/4-implementation/swarm/ssh-ca.md @@ -147,7 +147,8 @@ stores a connection and by the agent before it renders one: its own proxy, so no direct connection there can exist. The swarm doesn't check the hive's own bridge subnet, because each hive sets its -own and an agent can move between hives. +own and an agent can move between hives. Each hive drops an address of its own +subnet when it resolves a host (see [Egress](#egress)). `hive-agent`'s `ssh_connections.rs` renders the list each time the harness reads the record (every ten seconds) into `/run/hive-agent-ssh-connections/config`: @@ -159,11 +160,35 @@ agent user, because ssh refuses an included file that another user owns. A controller older than the field drops the list when it next writes the record through `PUT /permissions`. -## Not covered +## Egress -Egress. The default `egress.forward = "log"` lets a direct connection from an -agent to a machine's SSH port through, and `reject` blocks it; there is no -per-agent allowance for it yet. +The hive's forward chain accepts a named connection of an agent that has `ssh` +on, whatever `egress.forward` is (`log` or `reject`). Each hive does this for +the agents it has: + +1. `hive-c0re`'s `workers::egress_connections` reads the permissions the hive + last read from the swarm and, for each agent with `ssh` on, resolves the + `host` of each connection with the host's resolver. An IPv4 literal passes + through. It asks the hosts of a pass together, each for at most five seconds, + so a hung resolver delays the file by that long and no more. It keeps an + answer for ten minutes and keeps the last answer when a lookup fails or + times out (one warning, until the name resolves again). It drops loopback, + link-local, multicast and IPv6 answers, and any address inside the hive's + bridge subnet (`HIVE_NETWORK_SUBNET`) with one warning per host, because a + flow to the bridge never reaches the forward chain. It writes + ` ` lines to `egress.connectionsFile`, beside squid's + users file. +2. `hive-priv egress-pin` maps each agent to the MAC of its port and fills the + `agent_allow` set of the `inet hive_egress` table, in the same transaction + as `direct_mac`. `hive-egress-connections.path` runs a pass when the file + changes. +3. The forward chain accepts `ether saddr . ip daddr . tcp dport @agent_allow` + right after the `direct_mac` rule, so a matching flow is neither counted + nor logged. + +The allowance is for TCP and IPv4 only: the agent bridge has no IPv6 +addressing. An agent whose port has no binding yet has no allowance until it +gets one. See [the network page](../networking/network.md). ## Revocation diff --git a/nix/host-modules/hive-egress.nix b/nix/host-modules/hive-egress.nix index 9c850c84f..bcda6af19 100644 --- a/nix/host-modules/hive-egress.nix +++ b/nix/host-modules/hive-egress.nix @@ -179,7 +179,9 @@ let # so it allows nothing those refuse; a `reject` ends the packet. The MACs # in `direct_mac` (the unrestricted agents and the service # containers, the traffic the intercept chain leaves alone) come first in - # both modes, so a `log` line names what `reject` would refuse. Traffic to + # both modes, so a `log` line names what `reject` would refuse. The TCP + # connections in `agent_allow` (an agent's `ssh_connections`, resolved by + # hive-c0re) are accepted next, with the same effect. Traffic to # the host's own addresses (squid, dnsmasq, the gateway) is delivered # locally and never reaches the forward hook. forwardChain = '' diff --git a/swarm-wire/src/agent_permissions.rs b/swarm-wire/src/agent_permissions.rs index 16acd4dbe..79541cbdd 100644 --- a/swarm-wire/src/agent_permissions.rs +++ b/swarm-wire/src/agent_permissions.rs @@ -98,12 +98,14 @@ pub struct AgentPermissions { #[serde(default, skip_serializing_if = "Option::is_none")] pub max_repos: Option, /// Whether the agent may have its SSH key signed by the swarm's SSH CA. Absent - /// is off. Only the controller reads it: it attaches the SSH signing policy - /// to the agent's store login role while this is `true`. + /// is off. The controller attaches the SSH signing policy to the agent's store + /// login role while this is `true`, and a hive lets the agent's + /// `ssh_connections` through its forward chain while it is. #[serde(default, skip_serializing_if = "Option::is_none")] pub ssh: Option, /// The named SSH connections the operator declared for the agent. The - /// agent renders them into its ssh config while `ssh` is on. Each passes + /// agent renders them into its ssh config and its hive allows them on its + /// forward chain, while `ssh` is on. Each passes /// [`crate::ssh_connection::SshConnection::checked`] before it is stored, /// and names are unique. A controller built before this field decodes the /// record without it and stores it back without it.