egress: the forward chain refuses by default where the intercept is on
egress.forward defaults to reject while egress.intercept.enable is on (a hive controller host turns it on) and to log otherwise, because reject without the intercept would refuse all web traffic and an assertion already forbids it. log stays an explicit choice. The hive-squid VM test runs its log-mode half on the base node, so the base pins forward = log at mkDefault priority and the reject specialisation overrides it. A module-eval case covers the default on a controller host, on a host without the intercept, and an explicit log.
This commit is contained in:
parent
d88a0cc526
commit
b8851743e8
5 changed files with 86 additions and 35 deletions
|
|
@ -61,10 +61,10 @@ key changed and you didn't change the machine, don't clear it: something else
|
||||||
answers at that address.
|
answers at that address.
|
||||||
|
|
||||||
The SSH switch and the CA don't open the network. The default `egress.forward`
|
The SSH switch and the CA don't open the network. The default `egress.forward`
|
||||||
of `log` lets the agent's direct connection to the machine's port through. If
|
is `reject`, so the hive refuses the agent's direct connection to the machine's
|
||||||
you set it to `reject`, the hive refuses the connection unless you gave the
|
port unless you gave the agent a named connection to that host and port (see
|
||||||
agent a named connection to that host and port (see
|
[Named connections](#named-connections)). If you set `egress.forward` to `log`,
|
||||||
[Named connections](#named-connections)). See
|
the connection goes through. See
|
||||||
[the network page](../../4-implementation/networking/network.md).
|
[the network page](../../4-implementation/networking/network.md).
|
||||||
|
|
||||||
## 2. Make a machine trust the CA
|
## 2. Make a machine trust the CA
|
||||||
|
|
|
||||||
|
|
@ -233,9 +233,10 @@ when it speaks HTTP on port 80 or TLS on port 443 and its agent is `restricted`
|
||||||
or in `log` mode: the host redirects that traffic into squid's intercept ports
|
or in `log` mode: the host redirects that traffic into squid's intercept ports
|
||||||
(see _Intercepted web traffic_ below). Everything else goes straight out through
|
(see _Intercepted web traffic_ below). Everything else goes straight out through
|
||||||
the NAT, as before: the `unrestricted` agents, the service containers and every
|
the NAT, as before: the `unrestricted` agents, the service containers and every
|
||||||
other port and protocol. By default nothing blocks that. The host logs the direct
|
other port and protocol. A hive controller host refuses the direct connections
|
||||||
connections of the `restricted` and `log` agents, and `egress.forward =
|
of the `restricted` and `log` agents (`egress.forward = "reject"`, the default
|
||||||
"reject"` refuses them (see _Agent bridge egress log_ below).
|
while the intercept is on); `egress.forward = "log"` only logs them (see _Agent
|
||||||
|
bridge egress log_ below).
|
||||||
|
|
||||||
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
|
Squid filters on the destination hostname: a `CONNECT` target or a plain-HTTP
|
||||||
host. It never decrypts TLS. The one TLS key it loads is a throwaway that the
|
host. It never decrypts TLS. The one TLS key it loads is a throwaway that the
|
||||||
|
|
@ -727,7 +728,8 @@ loopback — the hive-c0re admin socket is a UDS, not TCP.
|
||||||
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
`services.hyperhive.deploy.hive-controller.egress.forward` sets what the `forward` chain of
|
||||||
the `inet hive_egress` table does with traffic the host routes from the agent
|
the `inet hive_egress` table does with traffic the host routes from the agent
|
||||||
bridge to anywhere but the bridge. It takes
|
bridge to anywhere but the bridge. It takes
|
||||||
`accept`, `log` (the default) or `reject`. In `log` mode the chain blocks
|
`accept`, `log` or `reject`; the default is `reject` where the intercept is on
|
||||||
|
(see _The default_ below) and `log` elsewhere. In `log` mode the chain blocks
|
||||||
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
nothing. It logs each new flow and counts every packet. In `reject` mode it
|
||||||
also refuses the flow.
|
also refuses the flow.
|
||||||
|
|
||||||
|
|
@ -756,12 +758,15 @@ also refuses the flow.
|
||||||
(and `hive-egress-reject-unmetered `); the rate limit and the counter work as
|
(and `hive-egress-reject-unmetered `); the rate limit and the counter work as
|
||||||
in `log` mode. Still open to every agent are the host's own addresses: the
|
in `log` mode. Still open to every agent are the host's own addresses: the
|
||||||
proxy ports, DNS and DHCP from the bridge address, and the gateway.
|
proxy ports, DNS and DHCP from the bridge address, and the gateway.
|
||||||
- **Switching on.** Leave `log` running for about a week, then read the
|
- **The default.** `reject` while `egress.intercept.enable` is on, which
|
||||||
`hive-egress-would-drop` lines. Each is a flow that `reject` would refuse; a
|
`hive-squid.nix` turns on for a hive controller host, and `log` on a host
|
||||||
flow that belongs to a legitimate use needs a fix first (a service container
|
without the intercept, because `reject` there would refuse all web traffic
|
||||||
missing from `direct_mac`, an agent that should be `unrestricted`). Then set
|
too. The soak ran with `log` first: the `hive-egress-would-drop` lines are
|
||||||
`services.hyperhive.deploy.hive-controller.egress.forward = "reject"`. Making it the default
|
each a flow that `reject` would refuse, and a flow that belongs to a
|
||||||
is a one-line change of the option's default.
|
legitimate use needed a fix before the switch (a service container missing
|
||||||
|
from `direct_mac`, an agent that should be `unrestricted`). To go back to
|
||||||
|
logging, set
|
||||||
|
`services.hyperhive.deploy.hive-controller.egress.forward = "log"`.
|
||||||
- **What `reject` can break.**
|
- **What `reject` can break.**
|
||||||
- A service container with no pin yet isn't in `direct_mac`, so the
|
- A service container with no pin yet isn't in `direct_mac`, so the
|
||||||
chain refuses it until `hive-egress-pin` binds its port. The bridge table
|
chain refuses it until `hive-egress-pin` binds its port. The bridge table
|
||||||
|
|
|
||||||
|
|
@ -344,10 +344,16 @@ in
|
||||||
"log"
|
"log"
|
||||||
"reject"
|
"reject"
|
||||||
];
|
];
|
||||||
default = "log";
|
default = if egressCfg.intercept.enable then "reject" else "log";
|
||||||
|
defaultText = lib.literalMD ''
|
||||||
|
`"reject"` while `egress.intercept.enable` is on, `"log"` otherwise
|
||||||
|
'';
|
||||||
description = ''
|
description = ''
|
||||||
What the `inet hive_egress` table does with traffic the host forwards
|
What the `inet hive_egress` table does with traffic the host forwards
|
||||||
from the agent bridge to anywhere but the bridge. In `log` and
|
from the agent bridge to anywhere but the bridge. The default is `reject`
|
||||||
|
on a host with the intercept on, which a hive controller host has, and
|
||||||
|
`log` on a host without it, because `reject` without the intercept would
|
||||||
|
refuse every agent's web traffic. In `log` and
|
||||||
`reject` the traffic of the MACs in the `direct_mac` set (the unrestricted
|
`reject` the traffic of the MACs in the `direct_mac` set (the unrestricted
|
||||||
agents and the service containers) is accepted and never logged, and so
|
agents and the service containers) is accepted and never logged, and so
|
||||||
is TCP from an agent to an address and port in the `agent_allow` set
|
is TCP from an agent to an address and port in the `agent_allow` set
|
||||||
|
|
|
||||||
|
|
@ -114,6 +114,14 @@ let
|
||||||
network.enable = true;
|
network.enable = true;
|
||||||
deploy.hive-controller.egress.forward = "reject";
|
deploy.hive-controller.egress.forward = "reject";
|
||||||
};
|
};
|
||||||
|
# The default of `egress.forward`: the controller host has the intercept on,
|
||||||
|
# the host with the network and no controller does not, and a controller host
|
||||||
|
# can still ask for `log`.
|
||||||
|
egressForwardDefault = hiveWithController { };
|
||||||
|
egressForwardDefaultNoIntercept = hive { network.enable = true; };
|
||||||
|
egressForwardLogged = hiveWithController {
|
||||||
|
deploy.hive-controller.egress.forward = "log";
|
||||||
|
};
|
||||||
|
|
||||||
# Every hive defaults the agents' queue address to the queue's name, so the
|
# Every hive defaults the agents' queue address to the queue's name, so the
|
||||||
# only hive without one is a hive told to have none. Also the host still
|
# only hive without one is a hive told to have none. Also the host still
|
||||||
|
|
@ -1628,15 +1636,15 @@ let
|
||||||
&& !(centralToggleOff.systemd.services ? hive-egress);
|
&& !(centralToggleOff.systemd.services ? hive-egress);
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The forward chain is in the ordinary host's table and logs the bridge's
|
# The forward chain of a controller host that asks for `log` is in its
|
||||||
# forwarded flows without a verdict that blocks. The ruleset is read as
|
# table and logs the bridge's forwarded flows without a verdict that
|
||||||
# text; the last rule of the chain is the accept that keeps the log from
|
# blocks. The ruleset is read as text; the last rule of the chain is the
|
||||||
# changing any outcome.
|
# accept that keeps the log from changing any outcome.
|
||||||
name = "the default egress table logs the bridge's forwarded flows and blocks none";
|
name = "the log-mode egress table logs the bridge's forwarded flows and blocks none";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
text = lib.head (
|
text = lib.head (
|
||||||
lib.splitString "table bridge hive_egress {" bare.services.hyperhive.network.egress.ruleset
|
lib.splitString "table bridge hive_egress {" egressForwardLogged.services.hyperhive.network.egress.ruleset
|
||||||
);
|
);
|
||||||
chain = lib.last (lib.splitString "chain forward" text);
|
chain = lib.last (lib.splitString "chain forward" text);
|
||||||
rules = lib.filter (l: l != "") (map lib.trim (lib.splitString "\n" chain));
|
rules = lib.filter (l: l != "") (map lib.trim (lib.splitString "\n" chain));
|
||||||
|
|
@ -1700,7 +1708,7 @@ let
|
||||||
file = "/var/lib/hive-squid/ssh-connections";
|
file = "/var/lib/hive-squid/ssh-connections";
|
||||||
envOf = m: name: m.systemd.services.${name}.environment;
|
envOf = m: name: m.systemd.services.${name}.environment;
|
||||||
in
|
in
|
||||||
inOrder bare
|
inOrder egressForwardLogged
|
||||||
&& inOrder egressForwardReject
|
&& inOrder egressForwardReject
|
||||||
&& lib.hasInfix set (inetOf egressForwardAccept)
|
&& lib.hasInfix set (inetOf egressForwardAccept)
|
||||||
&& !(lib.hasInfix "@agent_allow accept" (rulesetOf egressForwardAccept))
|
&& !(lib.hasInfix "@agent_allow accept" (rulesetOf egressForwardAccept))
|
||||||
|
|
@ -1866,9 +1874,9 @@ let
|
||||||
# `reject` logs and refuses in one rule, as a TCP rule (reset) and a
|
# `reject` logs and refuses in one rule, as a TCP rule (reset) and a
|
||||||
# non-TCP rule (ICMP admin-prohibited) for each of the four log/verdict
|
# non-TCP rule (ICMP admin-prohibited) for each of the four log/verdict
|
||||||
# positions, after the exempt MACs. The ruleset is read as
|
# positions, after the exempt MACs. The ruleset is read as
|
||||||
# text. The `accept` count (the exempt MACs only) and the
|
# text. The `accept` count (the exempt MACs only) and the controller
|
||||||
# `bare` host, whose chain has no refusal, are the controls that the
|
# host that asks for `log`, whose chain has no refusal, are the controls
|
||||||
# lookups find what is there.
|
# that the lookups find what is there.
|
||||||
name = "egress.forward reject refuses the bridge's forwarded flows after the exempt MACs";
|
name = "egress.forward reject refuses the bridge's forwarded flows after the exempt MACs";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
@ -1881,7 +1889,7 @@ let
|
||||||
tcp = lib.filter (l: lib.hasInfix " meta l4proto tcp " l && !(lib.hasInfix "@agent_allow" l)) rules;
|
tcp = lib.filter (l: lib.hasInfix " meta l4proto tcp " l && !(lib.hasInfix "@agent_allow" l)) rules;
|
||||||
other = lib.filter (lib.hasInfix " meta l4proto != tcp") rules;
|
other = lib.filter (lib.hasInfix " meta l4proto != tcp") rules;
|
||||||
logged = lib.filter (lib.hasInfix "log prefix") rules;
|
logged = lib.filter (lib.hasInfix "log prefix") rules;
|
||||||
bareText = bare.services.hyperhive.network.egress.ruleset;
|
loggedText = egressForwardLogged.services.hyperhive.network.egress.ruleset;
|
||||||
in
|
in
|
||||||
!(lib.any (
|
!(lib.any (
|
||||||
a: !a.assertion && lib.hasInfix "deploy.hive-controller.egress" a.message
|
a: !a.assertion && lib.hasInfix "deploy.hive-controller.egress" a.message
|
||||||
|
|
@ -1905,14 +1913,14 @@ let
|
||||||
&&
|
&&
|
||||||
lib.last rules
|
lib.last rules
|
||||||
== ''iifname "hive-br0" oifname != "hive-br0" meta l4proto != tcp reject with icmpx type admin-prohibited''
|
== ''iifname "hive-br0" oifname != "hive-br0" meta l4proto != tcp reject with icmpx type admin-prohibited''
|
||||||
&& !(lib.hasInfix "reject with" bareText)
|
&& !(lib.hasInfix "reject with" loggedText)
|
||||||
&& lib.hasInfix "would-drop" bareText;
|
&& lib.hasInfix "would-drop" loggedText;
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# `reject` without the intercept is refused by an assertion that names
|
# `reject` without the intercept is refused by an assertion that names
|
||||||
# the option and both ways out; the controller host with `reject` (the
|
# the option and both ways out; the controller host with `reject` (the
|
||||||
# intercept on) and `bare` (`log`) are the controls that the lookup
|
# intercept on) and the host without the intercept on its default
|
||||||
# finds nothing for them.
|
# (`log`) are the controls that the lookup finds nothing for them.
|
||||||
name = "egress.forward reject without the intercept is refused";
|
name = "egress.forward reject without the intercept is refused";
|
||||||
ok =
|
ok =
|
||||||
let
|
let
|
||||||
|
|
@ -1928,7 +1936,36 @@ let
|
||||||
&& lib.hasInfix "intercept.enable = true" (lib.head (refused egressRejectNoIntercept)).message
|
&& lib.hasInfix "intercept.enable = true" (lib.head (refused egressRejectNoIntercept)).message
|
||||||
&& lib.hasInfix "keep forward = \"log\"" (lib.head (refused egressRejectNoIntercept)).message
|
&& lib.hasInfix "keep forward = \"log\"" (lib.head (refused egressRejectNoIntercept)).message
|
||||||
&& refused egressForwardReject == [ ]
|
&& refused egressForwardReject == [ ]
|
||||||
&& refused bare == [ ];
|
&& refused egressForwardDefaultNoIntercept == [ ];
|
||||||
|
}
|
||||||
|
{
|
||||||
|
# The default of `egress.forward` follows the intercept: `reject` on the
|
||||||
|
# controller host, whose ruleset then refuses and carries no would-drop
|
||||||
|
# line, and `log` on the host without the intercept, which a `reject`
|
||||||
|
# default would fail to evaluate. A controller host can still ask for
|
||||||
|
# `log`, and its ruleset then only logs.
|
||||||
|
name = "egress.forward defaults to reject with the intercept on and to log without it";
|
||||||
|
ok =
|
||||||
|
let
|
||||||
|
forwardOf = m: m.services.hyperhive.deploy.hive-controller.egress.forward;
|
||||||
|
interceptOf = m: m.services.hyperhive.deploy.hive-controller.egress.intercept.enable;
|
||||||
|
rulesetOf = m: m.services.hyperhive.network.egress.ruleset;
|
||||||
|
unmet =
|
||||||
|
m:
|
||||||
|
lib.filter (
|
||||||
|
a: !a.assertion && lib.hasInfix "hive-controller.egress.forward" a.message
|
||||||
|
) m.assertions;
|
||||||
|
in
|
||||||
|
forwardOf egressForwardDefault == "reject"
|
||||||
|
&& interceptOf egressForwardDefault
|
||||||
|
&& lib.hasInfix "hive-egress-reject" (rulesetOf egressForwardDefault)
|
||||||
|
&& !(lib.hasInfix "hive-egress-would-drop" (rulesetOf egressForwardDefault))
|
||||||
|
&& forwardOf egressForwardDefaultNoIntercept == "log"
|
||||||
|
&& !(interceptOf egressForwardDefaultNoIntercept)
|
||||||
|
&& unmet egressForwardDefaultNoIntercept == [ ]
|
||||||
|
&& forwardOf egressForwardLogged == "log"
|
||||||
|
&& lib.hasInfix "hive-egress-would-drop" (rulesetOf egressForwardLogged)
|
||||||
|
&& !(lib.hasInfix "hive-egress-reject" (rulesetOf egressForwardLogged));
|
||||||
}
|
}
|
||||||
{
|
{
|
||||||
# The redirect of the agents' web traffic: IPv4 TCP 80 and 443 from the
|
# The redirect of the agents' web traffic: IPv4 TCP 80 and 443 from the
|
||||||
|
|
|
||||||
|
|
@ -228,9 +228,12 @@ pkgs.testers.runNixOSTest {
|
||||||
pkgs.mkpasswd
|
pkgs.mkpasswd
|
||||||
pkgs.nftables
|
pkgs.nftables
|
||||||
];
|
];
|
||||||
# The reject-mode half of the test switches to this at its end. The
|
# The log-mode half of the test runs on this node, and the reject-mode
|
||||||
# forward mode is fixed when the node evaluates; the switch reloads
|
# half switches to the specialisation at its end. The forward mode is
|
||||||
# `hive-egress` with the other ruleset.
|
# fixed when the node evaluates; the switch reloads `hive-egress` with
|
||||||
|
# the other ruleset. The default is `reject` while the intercept is on,
|
||||||
|
# so the base pins `log`, at a priority the specialisation overrides.
|
||||||
|
services.hyperhive.deploy.hive-controller.egress.forward = lib.mkDefault "log";
|
||||||
specialisation.reject.configuration.services.hyperhive.deploy.hive-controller.egress.forward =
|
specialisation.reject.configuration.services.hyperhive.deploy.hive-controller.egress.forward =
|
||||||
"reject";
|
"reject";
|
||||||
services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary = "other.t.local";
|
services.hyperhive.deploy.hive-controller.squid.selfcheck.unrestrictedCanary = "other.t.local";
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue