diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 744d02c..b9b8e3e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -62,7 +62,8 @@ jobs: run: | apt-get update apt-get install -y --no-install-recommends iproute2 nftables openssh-server \ - openssh-client curl python3 python3-venv util-linux procps ca-certificates + openssh-client curl python3 python3-venv util-linux procps ca-certificates \ + iptables python3 -m venv /opt/sshuttle /opt/sshuttle/bin/pip install sshuttle==2.0.0 - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 diff --git a/debian/control b/debian/control index 1f387da..4e05851 100644 --- a/debian/control +++ b/debian/control @@ -34,6 +34,7 @@ Depends: python3-gi, sshuttle (>= 1.3.1), Recommends: + iptables, systemd-resolved, Description: NetworkManager VPN plugin that tunnels with sshuttle nm-sshuttle runs sshuttle as a NetworkManager VPN connection. The tunnel is diff --git a/docs/design.md b/docs/design.md index 8a433f3..0fc2f70 100644 --- a/docs/design.md +++ b/docs/design.md @@ -842,7 +842,7 @@ keys: | `dns` | `split` | `none`, `split` or `all`. | | `dns-servers` | `10.1.0.53` | Required for `split` and `all`. IPv4 only. | | `dns-domains` | `corp.example,lab.example` | Routing domains for `split`. | -| `method` | `nft` | `nft` (default) or `nat`. `tproxy` comes later. | +| `method` | `nft` | `nft` (default), `nat`, or `tproxy`, which also carries UDP ([tproxy](later/tproxy.md)). | | `gateway` | `203.0.113.7` | Override the first hop, IPv4 or IPv6. Needed with `ProxyCommand`. | | `probe` | `10.1.0.10:22` | Optional reachability check through the tunnel, `ADDRESS:PORT` or `[IPV6]:PORT`. The service must send first, as an ssh server does (§4.4, "Health"). | | `fail-closed` | `yes` | Install the guard table (default `yes`). | @@ -1260,7 +1260,9 @@ Events and what the supervisor does: state is not evidence (see "Activated is not proof"): - the tunnel unit is active (an unreadable state is no failure); - with `method=nft`, sshuttle's nft table exists and a listener holds its - port; + port; with `method=tproxy`, its mark chain in iptables' `mangle` table + does, for each tunnelled family, and the plugin's rule and route are + there (put back if not); - the guard table exists (re-install it if something flushed the ruleset); - optionally, the `probe` TCP connect through the tunnel, where two failures in a row cause a restart. This covers sshuttle's "process @@ -1469,7 +1471,10 @@ The IPv6 sets are always there, empty for an IPv4-only profile ([IPv6](later/ipv6.md)). The exclude sets hold every address of the first hop and the profile's `exclude`. sshuttle's NAT redirect runs earlier, at priority `dstnat`, so packets it redirected carry conntrack status `dnat` -when they reach the guard. +when they reach the guard. With `method=tproxy` there is no NAT; two rules +ahead of the `dnat` ones accept what sshuttle's mangle chain marked +(`ip daddr @subnets4 meta mark 0x4e530000 accept`, and the same for IPv6; +[tproxy](later/tproxy.md)). **[lab T4]** In the lab, while the tunnel was down, connections failed in about 10 ms instead of leaking to the default route. In the lab's container, @@ -1680,11 +1685,18 @@ below used sshuttle 2.0.0 with the nft method in an Ubuntu 24.04 container | T12 | The IPv6 guard sets: redirected IPv6 passes; tunnel down, IPv6 TCP is refused in milliseconds although a route would deliver it, and the excluded jump host stays reachable over IPv6 || PASS | | T13 | A first hop with only an IPv6 address inside a tunnelled IPv6 subnet: ssh's own connection is not captured || PASS | | T14 | IPv6 off on the host: the plugin's check sees it, sshuttle without `--disable-ipv6` stops, and with it and the IPv6 networks left out it carries IPv4 | PASS | +| T15 | `method=tproxy` behind the plugin's guard and policy routing: DNS pinned to the tunnel link with `IP_UNICAST_IF` is captured and answered via the jump host | PASS | +| T16 | `method=tproxy`: a UDP echo and TCP go through the tunnel, and nothing to the internal network leaves on the uplink | PASS | +| T17 | `method=tproxy`, tunnel down: UDP is refused locally, TCP in milliseconds, nothing leaves on the uplink | PASS | +| T18 | `method=tproxy`: SIGKILL leaves the mangle chains, which neither answer nor leak; the plugin's sweep removes them, and the rule and route stay | PASS | +| T19 | `method=tproxy`: a UDP flow silent for 40 s answers again | PASS | T11 to T13 need IPv6 in the kernel, which the container this table was first written in lacked; they passed in CI's lab job, as did all the others. T1 to T10 and T14 passed in both, with the guard of §4.6, IPv6 sets -included. +included. T15 to T19 use the plugin's own guard, policy routing and sweep, +and give the host a route to the internal network through the jump host, +so that a leak would arrive. The dummy link is replaced by a veth whose peer sits in an empty namespace, because the lab kernel has no `dummy` module. For these tests the two @@ -1817,8 +1829,9 @@ Later, also propose a hook for an externally provided firewall helper - **The park candidate** (M2-26): NM stayed "activating" 4 s into a parked gap with no flip, and a kill in a parked gap left no VPN DNS entry, unlike M2-09. One run is not enough to turn it on by default. - The first CI run logged one SELinux denial that the second did not; its - record was not kept. + Some CI runs log one SELinux denial that others do not: Fedora's + `agetty` reading the console issue file NetworkManager's dispatcher + writes. It is not the plugin's, and M2-15 reports it as INFO. **Third VM run** (CI, 2026-10-11; 482 checks, no failure, no SELinux denial) added M2-29: with a ProxyJump host named inside `~corp.test`, the reconnect after a kill found the name unresolvable and came back @@ -1854,4 +1867,8 @@ Later, also propose a hook for an externally provided firewall helper design in [`later/`](later/README.md). **IPv6 is in:** IPv6 subnets and exclusions, the IPv6 guard sets, an IPv6 first hop and `gateway`, and the IPv6 part of the network key, with unit tests, lab tests T11 to - T14 and VM scenarios V6-01 and V6-02. + T14 and VM scenarios V6-01 and V6-02. **So is `method=tproxy`:** UDP + through the tunnel, the guard's mark rules, policy routing the plugin + owns, the sweep of sshuttle's mangle chains and a per-method health + check, with unit tests, lab tests T15 to T19 and VM scenarios TP-01 to + TP-03. diff --git a/docs/later/README.md b/docs/later/README.md index 7166bbb..e4f37f1 100644 --- a/docs/later/README.md +++ b/docs/later/README.md @@ -2,7 +2,7 @@ The [design](../design.md)'s roadmap (§7) ends with work left out of v1. Each item has a design here. They build on the architecture of §4 and -change nothing in it unless they say so. Implemented: IPv6 and the GNOME editor. +change nothing in it unless they say so. Implemented: IPv6, the GNOME editor and tproxy. | Design | What it adds | Size | |---|---|---| diff --git a/docs/later/gnome-editor.md b/docs/later/gnome-editor.md index 7bd7f26..0a1d996 100644 --- a/docs/later/gnome-editor.md +++ b/docs/later/gnome-editor.md @@ -125,7 +125,7 @@ One column of rows, in the order a user fills them in: | DNS domains | entry; sensitive for "split" | `dns-domains` | | Advanced (expander): first hop | entry, empty means "from ssh -G" | `gateway` | | Advanced: probe | entry, `ADDRESS:PORT` | `probe` | -| Advanced: method | drop-down: nft, nat, and tproxy once it exists | `method` | +| Advanced: method | drop-down: nft, nat, tproxy | `method` | | Advanced: block traffic while down | switch, on by default | `fail-closed` | | Local user | label | `local-user` | @@ -274,6 +274,8 @@ NetworkManager or the plugin. the plugin; DNS servers stay IPv4. The IPv6 tab has the same "Use this connection only for resources on its network" switch, so the note names both tabs. -- **Not done yet:** `method=tproxy` and the keys of several tunnels. Each is - a change to `nm-sshuttle-validate.c` and to the cases when the plugin gets - it; the shared cases fail on whichever side is behind. +- **`method=tproxy`** is in the drop-down and accepted, as in the plugin + ([tproxy](tproxy.md)). +- **Not done yet:** the keys of several tunnels. That is a change to + `nm-sshuttle-validate.c` and to the cases when the plugin gets them; the + shared cases fail on whichever side is behind. diff --git a/docs/later/tproxy.md b/docs/later/tproxy.md index cd2373d..dbad6c3 100644 --- a/docs/later/tproxy.md +++ b/docs/later/tproxy.md @@ -1,7 +1,9 @@ # UDP through `method=tproxy` -Status: design, not implemented. One of the "Later" items of the +Status: implemented. One of the "Later" items of the [design](../design.md) (§7); the index is in [README.md](README.md). +"What was built" at the end lists the results and where the code differs +from this design. v1 tunnels TCP only. sshuttle's `nft` and `nat` methods redirect TCP, plus UDP to port 53 of the `--ns-hosts` servers; any other UDP to a tunnelled @@ -232,12 +234,42 @@ About 120 lines of Python plus tests. `nmcli general reload`; no SELinux denial (`iptables` runs from the helper as `iptables_t`). +## What was built + +The design as above, with these results and details: + +- **DNS works (lab T15):** a query pinned to the tunnel link with + `IP_UNICAST_IF` is rerouted to `lo` by the mark and answered through + the jump host, so `dns=split` and `dns=all` stay allowed with `tproxy`. +- **Slot 0 only for now:** mark `0x4e530000`, table 20051, priority 100. + The functions take the slot, for the multiple tunnels design. +- **The routing is installed whether or not the profile fails closed** + (`fail-closed=no` drops only the guard), and only for the families the + tunnel carries. It is removed for both families. +- **The sweep** runs `iptables -w -t mangle -S` and `ip6tables`, deletes + the `OUTPUT` and `PREROUTING` jumps to stale chains, then flushes and + deletes the chains. Listing does not create a `mangle` table. +- **Health:** a family without a live `sshuttle-m-PORT` chain is a drop, + as a missing nft table is for `nft`; a missing rule or route is put + back. `nft flush ruleset` removes iptables-nft's chains and the guard, + not the rule or route. With iptables-legacy the chains are not in nft + and survive the flush, so only the guard comes back and the tunnel + stays up. Fedora's sshuttle package pulls in iptables-legacy, and in + the test VM `iptables` then uses it (VM TP-02). +- **Packaging:** `Recommends: iptables` (deb) and `iptables-nft` (RPM). + `nm-sshuttle check` fails a `tproxy` profile without them. +- **Tests:** lab T15 to T19 with the plugin's own guard, routing and + sweep, a route to the internal network through the jump host and a + counter on the uplink, so that a leak would show. T18 shows that stale + chains black-hole UDP rather than leak it. VM scenarios TP-01 (UDP, + TCP and split DNS, a gap), TP-02 (killed sshuttle, `nft flush + ruleset`) and TP-03 (`nmcli general reload` and a roam leave the rule + and route alone) pass on Fedora 44. + ## Open questions -- DNS through `tproxy` with `IP_UNICAST_IF` (T15). The answer decides - whether `dns=split` works with this method at all. - Whether `rp_filter` or a firewalld setting drops packets that come back in on `lo` with an uplink source address. The loopback device is normally exempt from reverse path filtering (inferred). - Whether NetworkManager leaves the rule and the route on `lo` alone on - every target release (the VM test above). + every target release. It does on Fedora 44 (VM TP-03). diff --git a/docs/user-guide.md b/docs/user-guide.md index 967922e..c30a755 100644 --- a/docs/user-guide.md +++ b/docs/user-guide.md @@ -106,7 +106,7 @@ The options of `add`: | `--dns-domains` | The domains `split` sends there, subdomains included. `~corp.example` only routes lookups. A plain `corp.example` also completes short names, so `git` becomes `git.corp.example`. Quote a leading `~` in the shell. | | `--gateway` | The address ssh connects to first, when the plugin cannot work it out (`ProxyCommand`). | | `--probe` | `ADDRESS:PORT` (or `[IPV6]:PORT`) of a server on the far side that speaks first, such as an ssh server. The plugin connects to it every 10 s and restarts the tunnel after two failures in a row. Without it, a tunnel that is up but stuck is noticed only when sshuttle exits. | -| `--method` | `nft` (the default) or `nat`. | +| `--method` | `nft` (the default) or `nat`. `tproxy` also sends UDP through the tunnel, such as QUIC or a DNS server you query directly; it needs `iptables` installed. | | `--fail-closed` | `yes` (the default) blocks traffic to the tunnelled networks while the tunnel is down. `no` lets it go out on the local network. | | `--user` | Whose ssh configuration and agent to use. The default is you. | @@ -210,7 +210,8 @@ Common causes: - One tunnel at a time. - TCP and DNS go through the tunnel; other UDP traffic, such as QUIC, - does not. + does so only with `--method tproxy`. A UDP flow that is silent for 30 s + starts afresh. `ping` never goes through: it is blocked. - DNS servers on the far side must have IPv4 addresses. If a split domain answers with IPv6 addresses, list those networks in `--subnets` too, or connections to them go around the tunnel. diff --git a/gnome/nm-sshuttle-editor.c b/gnome/nm-sshuttle-editor.c index e090303..cd55c7a 100644 --- a/gnome/nm-sshuttle-editor.c +++ b/gnome/nm-sshuttle-editor.c @@ -16,7 +16,7 @@ #define RESOURCE "/io/github/dennisklein/nm-sshuttle/nm-sshuttle-editor.ui" static const char *const DNS_MODES[] = {"none", "split", "all"}; -static const char *const METHODS[] = {"nft", "nat"}; +static const char *const METHODS[] = {"nft", "nat", "tproxy"}; /* The rows of the form, in the order of NmssEditor.rows. */ enum { diff --git a/gnome/nm-sshuttle-editor.ui b/gnome/nm-sshuttle-editor.ui index 4f85eba..19fc3e0 100644 --- a/gnome/nm-sshuttle-editor.ui +++ b/gnome/nm-sshuttle-editor.ui @@ -404,6 +404,7 @@ nft nat + tproxy @@ -419,7 +420,7 @@ - How traffic is captured. nft is the default + How traffic is captured. nft is the default; tproxy also carries UDP 0 True 64 diff --git a/gnome/nm-sshuttle-validate.c b/gnome/nm-sshuttle-validate.c index 8de3e24..dc11bb4 100644 --- a/gnome/nm-sshuttle-validate.c +++ b/gnome/nm-sshuttle-validate.c @@ -283,8 +283,8 @@ nmss_check_dns_domains(const char *dns, const char *value, GError **error) gboolean nmss_check_method(const char *value, GError **error) { - if (value && !g_strv_contains((const char *const[]) {"nft", "nat", NULL}, value)) - return fail(error, NMSS_FIELD_METHOD, _("Method: expected nft or nat, not “%s”"), value); + if (value && !g_strv_contains((const char *const[]) {"nft", "nat", "tproxy", NULL}, value)) + return fail(error, NMSS_FIELD_METHOD, _("Method: expected nft, nat or tproxy, not “%s”"), value); return TRUE; } diff --git a/gnome/tests/test-editor.c b/gnome/tests/test-editor.c index 676aad0..d5ec977 100644 --- a/gnome/tests/test-editor.c +++ b/gnome/tests/test-editor.c @@ -66,7 +66,7 @@ cli_connection(void) nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "remote", "corp"); nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "local-user", g_get_user_name()); nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "subnets", "10.0.0.0/8"); - nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "method", "tproxy"); + nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "method", "socks"); nm_setting_vpn_add_data_item(NM_SETTING_VPN(s_vpn), "from-a-newer-cli", "x"); nm_connection_add_setting(connection, s_vpn); return connection; @@ -234,7 +234,7 @@ test_existing_profile(Fixture *f, gconstpointer user_data) g_assert_false(nm_vpn_editor_update_connection(f->editor, f->connection, &error)); g_assert_error(error, NMSS_VALIDATE_ERROR, NMSS_FIELD_METHOD); g_clear_error(&error); - g_assert_cmpstr(nm_setting_vpn_get_data_item(s_vpn, "method"), ==, "tproxy"); + g_assert_cmpstr(nm_setting_vpn_get_data_item(s_vpn, "method"), ==, "socks"); gtk_drop_down_set_selected(GTK_DROP_DOWN(method), 1); /* nat */ g_assert_true(nm_vpn_editor_update_connection(f->editor, f->connection, &error)); diff --git a/lab/README.md b/lab/README.md index 169ccb6..280a9c6 100644 --- a/lab/README.md +++ b/lab/README.md @@ -22,12 +22,13 @@ namespaces, veth links, nft tables and a local user (`nmss-lab`), and it changes the address of its own veth link. ```console -# apt install iproute2 nftables openssh-server openssh-client curl python3 +# apt install iproute2 nftables iptables openssh-server openssh-client curl python3 # pip install sshuttle # or the distro package # SSHUTTLE=$(command -v sshuttle) ./lab/run.sh ``` -It takes about two minutes, most of it in the roaming test (T10). It exits +It takes about three minutes, most of it in the roaming test (T10) and +the silent UDP flow (T19). It exits non-zero if any check fails. ## What it checks @@ -48,8 +49,16 @@ non-zero if any check fails. | T12 | The guard's IPv6 sets let redirected traffic through. With the tunnel down they refuse IPv6 TCP at once, although the host's route would deliver it (shown without the guard), and the excluded jump host stays reachable over IPv6. | | T13 | A first hop reached by its IPv6 address inside a tunnelled IPv6 subnet: with it excluded, ssh's own connection is not captured. | | T14 | With IPv6 off on the host the plugin's check sees it, sshuttle stops without `--disable-ipv6`, and with it (and the IPv6 networks left out, as the plugin does) the tunnel carries IPv4. | +| T15 | `method=tproxy` behind the plugin's guard and policy routing: DNS pinned to the tunnel link with `IP_UNICAST_IF` is captured and answered through the jump host. | +| T16 | `method=tproxy`: a UDP echo and TCP go through the tunnel, and nothing to the internal network leaves on the uplink. | +| T17 | `method=tproxy`, tunnel down: UDP is refused locally and TCP at once, and nothing leaves on the uplink. | +| T18 | `method=tproxy`: SIGKILL leaves sshuttle's mangle chains, which neither answer nor leak; the plugin's sweep removes them and leaves its rule and route. | +| T19 | `method=tproxy`: a UDP flow silent for 40 s answers again. | -T11 to T13 print SKIP on a kernel without IPv6. +T11 to T13 print SKIP on a kernel without IPv6, and T15 to T19 without +`iptables` and `ip6tables`. T15 to T19 use the plugin's code from this +checkout, give the host a route to 10.99.0.0/24 through the jump host and +count what leaves on the uplink. The per-tunnel dummy link is replaced by a veth whose peer sits in an empty namespace, so the lab also runs on kernels built without the `dummy` module. diff --git a/lab/run.sh b/lab/run.sh index 0ee1481..42a36ff 100755 --- a/lab/run.sh +++ b/lab/run.sh @@ -13,12 +13,14 @@ # Where the kernel has IPv6, the same links also carry 2001:db8:51::/64 and # 2001:db8:99::/64 (HTTP on ::10). The host has a route to 2001:db8:99::/64 # through the jump host, so that a leak would actually arrive (T11-T13). +# T15-T19 (method=tproxy) add the same route for 10.99.0.0/24, and a UDP echo +# server on 10.99.0.10:7777. # # Run as root on a disposable VM or container. It creates network # namespaces, veth links, nft tables and a local user, and it changes the # address of its own veth link. Needs: iproute2 (ip, ss), nft, sshd, ssh, # ssh-agent, setpriv, curl, python3 and sshuttle (set SSHUTTLE=/path if it -# is not on PATH). The kill tests match processes whose first argument is +# is not on PATH); iptables and ip6tables for T15-T19. The kill tests match processes whose first argument is # the sshuttle path, so do not run other sshuttle instances meanwhile. set -uo pipefail @@ -43,8 +45,12 @@ LO_IPV6_OFF=$(cat /proc/sys/net/ipv6/conf/lo/disable_ipv6 2>/dev/null || echo 1) [ -x "$SSHUTTLE" ] || { echo "sshuttle not found; set SSHUTTLE=" >&2; exit 2; } WORK=$(mktemp -d /tmp/nmss-lab.XXXXXX) +# sshuttle tells systemd it is ready once its rules are in place. Nothing +# listens here, so with -v that shows in its log as "Error notifying systemd". +export NOTIFY_SOCKET=$WORK/notify chmod 755 "$WORK" -cp "$LAB_SRC/ssh-as-user" "$LAB_SRC/dns_query.py" "$LAB_SRC/dns_server.py" "$WORK/" +cp "$LAB_SRC/ssh-as-user" "$LAB_SRC/dns_query.py" "$LAB_SRC/dns_server.py" \ + "$LAB_SRC/udp_echo.py" "$WORK/" chmod 755 "$WORK"/* AGENT_SOCK=$WORK/agent/agent.sock @@ -55,6 +61,8 @@ pass() { echo "PASS $*"; PASS=$((PASS + 1)); } fail() { echo "FAIL $*"; FAIL=$((FAIL + 1)); } skip() { echo "SKIP $*"; SKIP=$((SKIP + 1)); } check() { local name=$1; shift; if "$@"; then pass "$name"; else fail "$name"; fi; } +# Run Python against the plugin's own code in this checkout. +plugin() { PYTHONPATH=$REPO python3 -c "$1"; } # PIDs of sshuttle clients and firewall helpers started from $SSHUTTLE # ("python3 $SSHUTTLE ..."). Exact match on argv[1], so the shell that runs @@ -72,6 +80,11 @@ teardown() { nft delete table inet "$t" done nft delete table inet "$GUARD_TABLE" 2>/dev/null + nft delete table inet nmss-lab-count 2>/dev/null + plugin 'from nm_sshuttle import guard, tunnel +tunnel.sweep(); guard.remove_policy_routing()' 2>/dev/null + ip route del 10.99.0.0/24 via $JUMP_IP 2>/dev/null + pkill -f "$WORK/udp_echo.py" 2>/dev/null [ -f "$WORK/sshd.pid" ] && kill "$(cat "$WORK/sshd.pid")" 2>/dev/null pkill -f "$WORK/dns_server.py" 2>/dev/null pkill -f "http.server --bind" 2>/dev/null @@ -227,10 +240,11 @@ start_tunnel6() { echo $! } -wait_connected() { +wait_connected() { # until the tunnel is up and sshuttle's rules are in place local i for i in $(seq 1 30); do - grep -q "Connected to server" "$1" && return 0 + grep -q "Connected to server" "$1" && grep -q "Error notifying systemd" "$1" && + return 0 sleep 0.5 done return 1 @@ -245,6 +259,15 @@ wait_exit() { # wait_exit PID SECONDS return 1 } +wait_gone() { # wait_gone SECONDS: until no client or firewall helper is left + local i + for i in $(seq 1 "$1"); do + [ -z "$(sshuttle_pids)" ] && return 0 + sleep 1 + done + return 1 +} + redirect_port() { # redirect_port LOG -> TCP port sshuttle listened on sed -n "s/.*TCP redirector listening on ('127.0.0.1', \([0-9]*\)).*/\1/p" "$1" } @@ -462,8 +485,7 @@ fi echo "== T14 IPv6 off on the host" [ "$HAVE_IPV6" = yes ] && sysctl -qw net.ipv6.conf.lo.disable_ipv6=1 plugin_sees_ipv6() { - PYTHONPATH=$REPO python3 -c \ - 'import sys; from nm_sshuttle.tunnel import ipv6_available; sys.exit(not ipv6_available())' + plugin 'import sys; from nm_sshuttle.tunnel import ipv6_available; sys.exit(not ipv6_available())' } plugin_finds_ipv6_off() { ! plugin_sees_ipv6; } check "T14a the plugin's check finds IPv6 off" plugin_finds_ipv6_off @@ -478,6 +500,107 @@ kill -TERM "$PID" wait_exit "$PID" 10 [ "$HAVE_IPV6" = yes ] && sysctl -qw net.ipv6.conf.lo.disable_ipv6="$LO_IPV6_OFF" +# ------------------------------------------------- method=tproxy (T15-T19) +# The plugin's own guard (with the mark rules), policy routing and sweep, from +# this checkout (docs/later/tproxy.md). The host gets a route to the internal +# network through the jump host, so that a leak would actually arrive, and a +# counter on the uplink shows whether anything to it left the host. +TMARK=0x4e530000 + +start_tproxy() { # start_tproxy LOG -> prints PID + "$SSHUTTLE" -v --disable-ipv6 --method tproxy --tmark $TMARK \ + -e "$WORK/ssh-as-user $LAB_USER $AGENT_SOCK" -r corp -x $JUMP_IP \ + --ns-hosts 10.99.0.53 --to-ns 10.99.0.53 10.99.0.0/24 > "$1" 2>&1 & + echo $! +} + +udp_echo() { # udp_echo TEXT [PAUSE] -> the client's output + python3 "$WORK/udp_echo.py" send 10.99.0.10 7777 "$@" +} + +mangle_chains() { iptables -w -t mangle -S | grep -c '^-N sshuttle-'; } + +uplink_leaks() { # packets to the internal network that left on the uplink + nft list chain inet nmss-lab-count postrouting | grep -o 'packets [0-9]*' | awk '{print $2}' +} + +if command -v iptables > /dev/null && command -v ip6tables > /dev/null; then + ip netns exec internal python3 "$WORK/udp_echo.py" serve 10.99.0.10 7777 \ + > "$WORK/udp.log" 2>&1 & + ip route add 10.99.0.0/24 via $JUMP_IP + nft -f - < "$WORK/dns.log" + + echo "== T15 DNS through method=tproxy, pinned to the tunnel link" + PID=$(start_tproxy "$WORK/t15.log") + check "T15a the tproxy tunnel comes up behind the plugin's guard" wait_connected "$WORK/t15.log" + check "T15b DNS pinned to the tunnel link (IP_UNICAST_IF) is answered" \ + test "$(python3 "$WORK/dns_query.py" 10.99.0.53 git.corp.test $TUNNEL_LINK)" = \ + "ANSWER 10.99.0.10" + check "T15c ... by the internal server, asked by the jump host" \ + grep -q "query from 10.99.0.1: git.corp.test" "$WORK/dns.log" + + echo "== T16 UDP and TCP through the tunnel" + check "T16a UDP echo answers, from the jump host" \ + test "$(udp_echo hello)" = "ECHO hello from 10.99.0.1" + check "T16b TCP to the internal host" test "$(http_code http://10.99.0.10:8080/)" = 200 + check "T16c nothing to the internal network left on the uplink" test "$(uplink_leaks)" = 0 + + echo "== T17 tunnel down: UDP and TCP fail locally" + kill -TERM "$PID" + wait_exit "$PID" 10 + wait_gone 10 # the firewall helper removes the chains after the client exits + check "T17a sshuttle removed its mangle chains" test "$(mangle_chains)" = 0 + check "T17b UDP refused locally" test "$(udp_echo down)" = "ERROR Operation not permitted" + START=$(date +%s%N) + CODE=$(http_code http://10.99.0.10:8080/ 4) + ELAPSED_MS=$(( ($(date +%s%N) - START) / 1000000 )) + check "T17c TCP refused at once (${ELAPSED_MS} ms)" test "$CODE" = 000 -a "$ELAPSED_MS" -lt 1000 + check "T17d nothing left on the uplink" test "$(uplink_leaks)" = 0 + + echo "== T18 SIGKILL leaves the mangle chains; the plugin's sweep removes them" + PID=$(start_tproxy "$WORK/t18.log") + wait_connected "$WORK/t18.log" + sshuttle_pids | xargs -r kill -KILL + sleep 1 + check "T18a stale mangle chains left behind" test "$(mangle_chains)" -gt 0 + check "T18b UDP gets no answer through them" test "$(udp_echo stale)" != "ECHO stale from 10.99.0.1" + check "T18c ... and does not leak" test "$(uplink_leaks)" = 0 + plugin 'from nm_sshuttle import tunnel; tunnel.sweep()' + check "T18d the sweep removes them" test "$(mangle_chains)" = 0 + check "T18e the plugin's rule and route stay" \ + plugin 'import sys; from nm_sshuttle import guard +sys.exit(not guard.policy_routing_present([4]))' + + echo "== T19 a UDP flow silent for 40 s" + PID=$(start_tproxy "$WORK/t19.log") + wait_connected "$WORK/t19.log" + OUT=$(udp_echo quiet 40) + check "T19 the flow answers before and after the pause" \ + test "$OUT" = "ECHO quiet from 10.99.0.1 +ECHO quiet-again from 10.99.0.1" + kill -TERM "$PID" + wait_exit "$PID" 10 + check "T19b the plugin removes the rule and route by their values" \ + plugin 'import sys; from nm_sshuttle import guard +sys.exit(not guard.remove_policy_routing())' + nft delete table inet $GUARD_TABLE + nft delete table inet nmss-lab-count + ip route del 10.99.0.0/24 via $JUMP_IP +else + skip "T15-T19 need iptables and ip6tables" +fi + echo echo "== $PASS passed, $FAIL failed, $SKIP skipped" [ "$FAIL" = 0 ] diff --git a/lab/udp_echo.py b/lab/udp_echo.py new file mode 100644 index 0000000..1895762 --- /dev/null +++ b/lab/udp_echo.py @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: MIT +"""UDP echo for the lab: the server answers every datagram with its text and +the address it came from, so tests can see whether it went through the jump +host. + +usage: udp_echo.py serve ADDRESS PORT + udp_echo.py send ADDRESS PORT TEXT [PAUSE] + +send prints "ECHO from ", "TIMEOUT" or "ERROR ". With +PAUSE it sends twice from the same socket, PAUSE seconds apart, and prints +both answers. +""" +import socket +import sys +import time + + +def serve(addr, port): + s = socket.socket(socket.AF_INET6 if ":" in addr else socket.AF_INET, socket.SOCK_DGRAM) + s.bind((addr, port)) + while True: + data, peer = s.recvfrom(2048) + print(f"datagram from {peer[0]}: {data.decode(errors='replace')}", flush=True) + s.sendto(data + f" from {peer[0]}".encode(), peer) + + +def ask(s, addr, port, text): + try: + s.sendto(text.encode(), (addr, port)) + data, _ = s.recvfrom(2048) + return "ECHO " + data.decode(errors="replace") + except socket.timeout: + return "TIMEOUT" + except OSError as e: + return f"ERROR {e.strerror or e}" + + +def main(): + if sys.argv[1] == "serve": + serve(sys.argv[2], int(sys.argv[3])) + return + addr, port, text = sys.argv[2], int(sys.argv[3]), sys.argv[4] + s = socket.socket(socket.AF_INET6 if ":" in addr else socket.AF_INET, socket.SOCK_DGRAM) + s.settimeout(3) + print(ask(s, addr, port, text)) + if len(sys.argv) > 5: + time.sleep(float(sys.argv[5])) + print(ask(s, addr, port, text + "-again")) + + +if __name__ == "__main__": + main() diff --git a/nm_sshuttle/cli.py b/nm_sshuttle/cli.py index 7ac5247..f9b3809 100644 --- a/nm_sshuttle/cli.py +++ b/nm_sshuttle/cli.py @@ -1,10 +1,11 @@ # SPDX-License-Identifier: MIT """nm-sshuttle: the commands the units and the plugin run. - cleanup stop the tunnel unit, remove nmss0, the guard and stale - sshuttle tables (ExecStopPost of nm-sshuttle.service, and - the plugin's startup cleanup) - sweep remove stale sshuttle tables (the tunnel unit's + cleanup stop the tunnel unit, remove nmss0, the guard, the tproxy + policy routing and stale sshuttle tables and chains + (ExecStopPost of nm-sshuttle.service, and the plugin's + startup cleanup) + sweep remove stale sshuttle tables and chains (the tunnel unit's ExecStartPre and ExecStopPost) exec-tunnel ExecStart of the tunnel unit ssh-as-user sshuttle's ssh command @@ -58,6 +59,7 @@ def cleanup(): if guard.guard_present(): log.info("removing a leftover guard table %s", GUARD_TABLE) ok &= guard.remove_guard() + ok &= guard.remove_policy_routing() tunnel.sweep() try: os.remove(TUNNEL_SPEC) @@ -217,7 +219,8 @@ def ssh_g_as_self(_user, target): return p.stdout -def check(args, nmcli=nmcli_run, ssh_g=ssh_g_as_self, run=subprocess.run): +def check(args, nmcli=nmcli_run, ssh_g=ssh_g_as_self, run=subprocess.run, + iptables_available=None): """Check a profile as the user it belongs to (design §4.2, §4.3). Returns 0 when everything passes. The test login may prompt for a host key @@ -232,6 +235,9 @@ def check(args, nmcli=nmcli_run, ssh_g=ssh_g_as_self, run=subprocess.run): "fix with nmcli or remove and add again)") return 1 print("ok profile fields") + if prof.method == "tproxy" and not (iptables_available or tunnel.iptables_available)(): + print("FAIL method=tproxy needs iptables and ip6tables; install iptables") + return 1 if settings["ipv6"]["auto-route-ext-gw"] != "no": # Profiles added before IPv6 support lack it; NM then adds a host route # towards an IPv6 first hop (docs/later/ipv6.md). @@ -298,7 +304,7 @@ def main(argv=None): for opt in ("exclude", "dns-servers", "dns-domains", "gateway", "probe"): a.add_argument("--" + opt) a.add_argument("--dns", choices=["none", "split", "all"]) - a.add_argument("--method", choices=["nft", "nat"]) + a.add_argument("--method", choices=["nft", "nat", "tproxy"]) a.add_argument("--fail-closed", choices=["yes", "no"]) c = sub.add_parser("check", help="check a profile as its user") c.add_argument("name") diff --git a/nm_sshuttle/const.py b/nm_sshuttle/const.py index a0a72e6..1d3e429 100644 --- a/nm_sshuttle/const.py +++ b/nm_sshuttle/const.py @@ -33,6 +33,12 @@ LINK_ADDRESS = "192.0.0.8" # RFC 7600 IPv4 dummy address (design §4.1) NBNS_SENTINEL = "192.0.0.10" # the reconnect sentinel (design §4.4) GUARD_TABLE = "nm-sshuttle-guard" +# method=tproxy (docs/later/tproxy.md): the mark sshuttle sets ("NS" in the high +# half, nobody else's), the routing table that sends marked packets to lo, and +# the rule's priority, ahead of Tailscale's and wg-quick's rules. +TPROXY_MARK = 0x4e530000 +TPROXY_TABLE = 0x4e53 +TPROXY_PRIORITY = 100 # Overridable for development and tests; packages use the defaults. LIBEXEC = os.environ.get("NM_SSHUTTLE_LIBEXEC", "/usr/libexec/nm-sshuttle") diff --git a/nm_sshuttle/guard.py b/nm_sshuttle/guard.py index 2390468..8c0aada 100644 --- a/nm_sshuttle/guard.py +++ b/nm_sshuttle/guard.py @@ -10,7 +10,7 @@ from .profile import collapse -from .const import GUARD_TABLE, LINK +from .const import GUARD_TABLE, LINK, TPROXY_MARK, TPROXY_PRIORITY, TPROXY_TABLE log = logging.getLogger("nm-sshuttle") @@ -63,13 +63,17 @@ def remove_link(name=LINK): return True -def guard_ruleset(networks, exclude): +def guard_ruleset(networks, exclude, mark=None): """The guard table (design §4.6, docs/later/ipv6.md) as an nft script. sshuttle's redirect runs at priority dstnat, so its packets carry conntrack status dnat when they reach this output hook; everything else to the tunnelled networks is refused at once instead of leaking. Each family has - its own pair of sets; the IPv6 ones are empty for an IPv4-only profile.""" + its own pair of sets; the IPv6 ones are empty for an IPv4-only profile. + + With method=tproxy there is no NAT: sshuttle's mangle chain marks what it + takes (priority mangle, before this hook), so a packet with that mark is + let through (docs/later/tproxy.md).""" def set_def(name, items, version): items = collapse(items, version) body = f"type ipv{version}_addr; flags interval;" @@ -87,6 +91,8 @@ def set_def(name, items, version): " type filter hook output priority filter; policy accept;", " ip daddr @exclude4 accept", " ip6 daddr @exclude6 accept", + *([f" ip daddr @subnets4 meta mark {mark:#010x} accept", + f" ip6 daddr @subnets6 meta mark {mark:#010x} accept"] if mark else []), " ip daddr @subnets4 ct status dnat accept", " ip6 daddr @subnets6 ct status dnat accept", " ip daddr @subnets4 meta l4proto tcp reject with tcp reset", @@ -99,10 +105,10 @@ def set_def(name, items, version): ]) -def install_guard(networks, exclude): +def install_guard(networks, exclude, mark=None): """Install (or replace) the guard table in one nft transaction.""" script = f"table inet {GUARD_TABLE}\ndelete table inet {GUARD_TABLE}\n" - script += guard_ruleset(networks, exclude) + script += guard_ruleset(networks, exclude, mark) ok, out = run(["nft", "-f", "-"], input=script) if not ok: raise RuntimeError(f"cannot install the guard table: {out}") @@ -120,3 +126,71 @@ def remove_guard(): log.warning("cannot remove the guard table: %s", out) return False return True + + +# ------------------------------------------------------- policy routing +# method=tproxy: packets sshuttle marked must be routed to lo, where TPROXY +# hands them to its listener. sshuttle leaves that to the user; the plugin +# owns it and removes it by its exact values, never flushing a table or rules +# it does not own (docs/later/tproxy.md). +def tproxy_mark(slot=0): + return TPROXY_MARK + slot + + +def _ip(family): + return ["ip", "-6"] if family == 6 else ["ip", "-4"] + + +def _rule(slot): + return ["priority", str(TPROXY_PRIORITY + slot), + "fwmark", f"{tproxy_mark(slot):#x}/0xffffffff", "lookup", str(TPROXY_TABLE + slot)] + + +def _local_route(slot): + return ["local", "default", "dev", "lo", "table", str(TPROXY_TABLE + slot)] + + +def _routing_parts(family, slot): + """(rule there, route there) for one family.""" + ok, rules = run(_ip(family) + ["rule", "list"]) + rule = ok and any( + line.startswith(f"{TPROXY_PRIORITY + slot}:") + and f"fwmark {tproxy_mark(slot):#x}" in line + and f"lookup {TPROXY_TABLE + slot}" in line + for line in rules.splitlines()) + ok, routes = run(_ip(family) + ["route", "show", "table", str(TPROXY_TABLE + slot)]) + route = ok and any(line.startswith("local default dev lo") for line in routes.splitlines()) + return rule, route + + +def policy_routing_present(families, slot=0): + """Whether the rule and the route are there for every one of families.""" + return all(all(_routing_parts(f, slot)) for f in families) + + +def install_policy_routing(families, slot=0): + """Install the rule and the route for each family, replacing any copy.""" + for family in families: + for _ in range(10): # "ip rule add" twice makes two rules + if not run(_ip(family) + ["rule", "del", *_rule(slot)])[0]: + break + ok, out = run(_ip(family) + ["rule", "add", *_rule(slot)]) + if ok: + ok, out = run(_ip(family) + ["route", "replace", *_local_route(slot)]) + if not ok: + raise RuntimeError(f"cannot install the policy routing for IPv{family}: {out}") + + +def remove_policy_routing(slot=0): + """Remove the rule and the route for both families; True when none is left.""" + clean = True + for family in (4, 6): + for _ in range(10): + if not run(_ip(family) + ["rule", "del", *_rule(slot)])[0]: + break + if _routing_parts(family, slot)[1]: + run(_ip(family) + ["route", "del", *_local_route(slot)]) + if any(_routing_parts(family, slot)): + log.warning("cannot remove the policy routing for IPv%d", family) + clean = False + return clean diff --git a/nm_sshuttle/profile.py b/nm_sshuttle/profile.py index 952b413..f1a7400 100644 --- a/nm_sshuttle/profile.py +++ b/nm_sshuttle/profile.py @@ -11,6 +11,8 @@ import re from dataclasses import dataclass, field +from .const import TPROXY_MARK + REMOTE_RE = re.compile(r"[A-Za-z0-9._@:%\[\]-]+") DOMAIN_RE = re.compile(r"~?(\.|[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)*\.?)") USER_RE = re.compile(r"[A-Za-z_][A-Za-z0-9_.-]*") # also keeps it one word for sshuttle -e @@ -44,7 +46,8 @@ def tunnel_spec(self, first_hop, ipv6=False): of the first hop's addresses; ipv6 says whether sshuttle tunnels IPv6.""" return {"remote": self.remote, "user": self.user, "subnets": self.subnets, "exclude": self.exclude, "dns": self.dns, "dns_servers": self.dns_servers, - "method": self.method, "first_hop": list(first_hop), "ipv6": ipv6} + "method": self.method, "first_hop": list(first_hop), "ipv6": ipv6, + **({"tmark": TPROXY_MARK} if self.method == "tproxy" else {})} def guarded_networks(self): """Networks the guard makes fail closed: the subnets plus the DNS servers.""" @@ -181,8 +184,8 @@ def parse(settings, getpwnam=pwd.getpwnam): servers, domains = [], [] method = data.get("method", "nft") - if method not in ("nft", "nat"): - raise ProfileError(f"method: expected nft or nat, got {method!r}") + if method not in ("nft", "nat", "tproxy"): + raise ProfileError(f"method: expected nft, nat or tproxy, got {method!r}") gateway = data.get("gateway") or None if gateway: diff --git a/nm_sshuttle/service.py b/nm_sshuttle/service.py index bd8d026..99eb896 100644 --- a/nm_sshuttle/service.py +++ b/nm_sshuttle/service.py @@ -498,17 +498,29 @@ def create_link(self): def remove_link(self): guard.remove_link() - def install_guard(self, networks, exclude): - guard.install_guard(networks, exclude) + def install_guard(self, networks, exclude, mark=None): + guard.install_guard(networks, exclude, mark) def remove_guard(self): guard.remove_guard() + def install_policy_routing(self, families): + guard.install_policy_routing(families) + + def remove_policy_routing(self): + guard.remove_policy_routing() + + def policy_routing_present(self, families): + return guard.policy_routing_present(families) + + def iptables_available(self): + return tunnel.iptables_available() + def sweep(self): tunnel.sweep() - def nft_health(self): - return tunnel.nft_health() + def nft_health(self, method="nft"): + return tunnel.nft_health(method) def ipv6_available(self): return tunnel.ipv6_available() diff --git a/nm_sshuttle/supervisor.py b/nm_sshuttle/supervisor.py index 60c9392..9933120 100644 --- a/nm_sshuttle/supervisor.py +++ b/nm_sshuttle/supervisor.py @@ -31,7 +31,8 @@ from .const import (AC_ACTIVATED, AC_ACTIVATING, AC_DEACTIVATED, AC_DEACTIVATING, AC_NAMES, AC_UNKNOWN, CONN_PORTAL, DEVICE_UNMANAGED, FAIL_CONNECT, FAIL_LOGIN, LINK, LINK_ADDRESS, NBNS_SENTINEL, - ST_INIT, ST_STARTED, ST_STARTING, ST_STOPPED, ST_STOPPING, STATE_NAMES) + ST_INIT, ST_STARTED, ST_STARTING, ST_STOPPED, ST_STOPPING, STATE_NAMES, + TPROXY_MARK) from .tunnel import classify_failure log = logging.getLogger("nm-sshuttle") @@ -115,6 +116,7 @@ def _reset_activation(self): self.stale_device = None self.last_config = None self.guard_installed = False + self.routing_installed = False # method=tproxy (docs/later/tproxy.md) self.activated_once = False self.link_managed = None # result of the unmanaged check (design §4.1) self.stopped_sent = False @@ -263,10 +265,25 @@ def _ip4_config(self, sentinel=False): return cfg def _install_guard(self): + mark = TPROXY_MARK if self.profile.method == "tproxy" else None self.fx.install_guard(self.profile.guarded_networks(), - self.hops + list(self.profile.exclude)) + self.hops + list(self.profile.exclude), mark) self.guard_installed = True + def _install_routing(self): + """method=tproxy: the rule and route that take marked packets to lo. They + stay for the whole activation, like the guard, fail-closed or not.""" + self.fx.install_policy_routing(sorted(self._tunnelled_families())) + self.routing_installed = True + + def _remove_guard(self): + if self.guard_installed: + self.fx.remove_guard() + self.guard_installed = False + if self.routing_installed: + self.fx.remove_policy_routing() + self.routing_installed = False + def _set_hops(self, addrs): """The first hop's addresses; Config's gateway is the first IPv4 one, or the first IPv6 one when it has none. NM needs only one.""" @@ -407,6 +424,9 @@ def _begin_connect(self, settings, sender): return log.warning("IPv6 is off on this host; the IPv6 networks are guarded but not " "tunnelled") + if p.method == "tproxy" and not self.fx.iptables_available(): + self.give_up(FAIL_CONNECT, "method=tproxy needs iptables and ip6tables") + return if self.nm_version and self.nm_version in self._invisible_versions(): log.info("NetworkManager %s never completed an nbns reconnect here; reconnects " "will not show in NetworkManager (invisible reconnect)", self.nm_version) @@ -460,6 +480,8 @@ def _have_first_hop(self, addrs, stale_device): self.ifindex = self.fx.create_link() if self.profile.fail_closed: self._install_guard() + if self.profile.method == "tproxy": + self._install_routing() self.fx.write_tunnel_spec(self.profile.tunnel_spec(self.hops, self.ipv6)) except (OSError, RuntimeError) as e: self.give_up(FAIL_CONNECT, str(e)) @@ -993,7 +1015,8 @@ def _health(self, hid): if self.phase != "up" or hid != self.health_id: return self._timer("health", self.HEALTH_INTERVAL, self._health, hid) - snap = self.fx.nft_health() + method = self.profile.method + snap = self.fx.nft_health(method) if snap is not None: if self.profile.fail_closed and not snap["guard"]: log.warning("the guard table is gone (was the ruleset flushed?); installing it " @@ -1004,10 +1027,19 @@ def _health(self, hid): self.give_up(FAIL_CONNECT, f"cannot restore the guard: {e}") return missing = self._tunnelled_families() - snap["sshuttle"] - if self.profile.method == "nft" and missing: - self._drop("health: no sshuttle table with a listener on its port for " + + if method in ("nft", "tproxy") and missing: + what = "mark chain" if method == "tproxy" else "table" + self._drop(f"health: no sshuttle {what} with a listener on its port for " + " and ".join(f"IPv{f}" for f in sorted(missing))) return + if method == "tproxy" and not self.fx.policy_routing_present( + sorted(self._tunnelled_families())): + log.warning("the policy routing for method=tproxy is gone; installing it again") + try: + self._install_routing() + except (OSError, RuntimeError) as e: + self.give_up(FAIL_CONNECT, f"cannot restore the policy routing: {e}") + return self.fx.tunnel_active(self._callback(self._health_unit, hid)) if self.profile.probe: host, port = self.profile.probe_target() @@ -1237,9 +1269,7 @@ def _remove_link_and_guard(self): self._cancel("link", "link-cap") self.link_removed = True self.fx.remove_link() - if self.guard_installed: - self.fx.remove_guard() - self.guard_installed = False + self._remove_guard() self._maybe_finish() def _maybe_finish(self): @@ -1343,9 +1373,7 @@ def _teardown_and_exit(self, why): def _exit_after_stop(self): if self.phase != "exiting": return - if self.guard_installed: - self.fx.remove_guard() - self.guard_installed = False + self._remove_guard() self._quit_now() @guarded diff --git a/nm_sshuttle/tunnel.py b/nm_sshuttle/tunnel.py index 81f4136..246442d 100644 --- a/nm_sshuttle/tunnel.py +++ b/nm_sshuttle/tunnel.py @@ -30,6 +30,9 @@ SSHUTTLE_TABLE_RE = re.compile(r"^sshuttle-ipv([46])-(\d+)$") # What may go into the generated ssh config unquoted. SSH_NAME_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$") +# method=tproxy: sshuttle's chains in the mangle table (docs/later/tproxy.md). +SSHUTTLE_CHAIN_RE = re.compile(r"^sshuttle-([mtd])-(\d+)$") +IPTABLES = {4: "iptables", 6: "ip6tables"} def bridge_command(user, ssh_config=None): @@ -59,6 +62,8 @@ def wanted(nets): argv = [sshuttle, "-v", "--method", spec.get("method", "nft")] if not ipv6: argv.append("--disable-ipv6") + if spec.get("tmark"): + argv += ["--tmark", f"{spec['tmark']:#x}"] argv += ["-r", spec["remote"]] for net in wanted(first_hops(spec) + list(spec.get("exclude", []))): argv += ["-x", net] @@ -339,10 +344,56 @@ def stale_tables(nft_list_tables, ports): return sshuttle_tables(nft_list_tables, ports, live=False) -def nft_health(): - """What the health check reads from nft (design §4.4, "Health"): is the - guard table there, and for which families (4, 6) does a sshuttle table - have a listener on its port? None when nft cannot be asked.""" +def mangle_chains(iptables_s): + """sshuttle's chains in 'iptables -t mangle -S' output: {name: port}.""" + chains = {} + for line in iptables_s.splitlines(): + parts = line.split() + if len(parts) == 2 and parts[0] == "-N": + m = SSHUTTLE_CHAIN_RE.match(parts[1]) + if m: + chains[parts[1]] = int(m.group(2)) + return chains + + +def stale_chain_commands(iptables_s, ports): + """The iptables arguments that remove the sshuttle chains whose port has no + listener: the jumps from OUTPUT and PREROUTING first, then flush every + chain, then delete them (the t-chain jumps to the d-chain).""" + stale = [c for c, port in mangle_chains(iptables_s).items() if port not in ports] + cmds = [] + for line in iptables_s.splitlines(): + parts = line.split() + if (len(parts) >= 4 and parts[0] == "-A" and parts[1] in ("OUTPUT", "PREROUTING") + and parts[-2] == "-j" and parts[-1] in stale): + cmds.append(["-D"] + parts[1:]) + cmds += [["-F", c] for c in stale] + cmds += [["-X", c] for c in stale] + return cmds + + +def iptables_available(): + """Whether both iptables and ip6tables exist, as sshuttle's tproxy method + needs (its is_supported).""" + return all(shutil.which(cmd, path=SEARCH_PATH) for cmd in IPTABLES.values()) + + +def _mangle(family): + """'iptables -t mangle -S' for one family, or None.""" + try: + p = subprocess.run([IPTABLES[family], "-w", "-t", "mangle", "-S"], capture_output=True, + text=True, timeout=10) + except (OSError, subprocess.TimeoutExpired): + return None + return p.stdout if p.returncode == 0 else None + + +def nft_health(method="nft"): + """What the health check reads from the firewall (design §4.4, "Health"): + is the guard table there, and for which families (4, 6) does sshuttle have + its rules with a listener on their port? Those are a table in nft, or for + method=tproxy a mark chain in iptables' mangle table. None when nft cannot + be asked.""" try: p = subprocess.run(["nft", "list", "tables"], capture_output=True, text=True, timeout=10) @@ -350,23 +401,49 @@ def nft_health(): return None if p.returncode: return None - live = sshuttle_tables(p.stdout, listening_ports(), live=True) - return {"guard": GUARD_TABLE in inet_tables(p.stdout), - "sshuttle": {int(SSHUTTLE_TABLE_RE.match(t).group(1)) for t in live}} + ports = listening_ports() + if method == "tproxy": + families = set() + for version in IPTABLES: + chains = mangle_chains(_mangle(version) or "") + if any(c.startswith("sshuttle-m-") and port in ports for c, port in chains.items()): + families.add(version) + else: + live = sshuttle_tables(p.stdout, ports, live=True) + families = {int(SSHUTTLE_TABLE_RE.match(t).group(1)) for t in live} + return {"guard": GUARD_TABLE in inet_tables(p.stdout), "sshuttle": families} def sweep(): - """Delete sshuttle tables left by a killed tunnel; they only black-hole traffic.""" + """Delete sshuttle tables and chains left by a killed tunnel; they only + black-hole traffic.""" try: out = subprocess.run(["nft", "list", "tables"], capture_output=True, text=True, timeout=10).stdout except (OSError, subprocess.TimeoutExpired) as e: log.warning("sweep: cannot list nft tables: %s", e) - return - for table in stale_tables(out, listening_ports()): + out = "" + ports = listening_ports() + for table in stale_tables(out, ports): p = subprocess.run(["nft", "delete", "table", "inet", table], capture_output=True, text=True, timeout=10) if p.returncode == 0: log.info("sweep: deleted stale table %s", table) else: log.warning("sweep: cannot delete %s: %s", table, p.stderr.strip()) + # method=tproxy leaves iptables chains; through iptables, so that this also + # works with iptables-legacy (docs/later/tproxy.md). + for version, cmd in IPTABLES.items(): + if not shutil.which(cmd, path=SEARCH_PATH): + continue + listing = _mangle(version) + if listing is None: + continue + for args in stale_chain_commands(listing, ports): + p = subprocess.run([cmd, "-w", "-t", "mangle", *args], capture_output=True, + text=True, timeout=10) + if p.returncode: + log.warning("sweep: %s -t mangle %s: %s", cmd, " ".join(args), + p.stderr.strip()) + elif args[0] == "-X": + log.info("sweep: deleted stale chain %s (IPv%d)", args[1], version) diff --git a/packaging/rpm/nm-sshuttle.spec b/packaging/rpm/nm-sshuttle.spec index 5698649..46db475 100644 --- a/packaging/rpm/nm-sshuttle.spec +++ b/packaging/rpm/nm-sshuttle.spec @@ -26,6 +26,8 @@ Requires: util-linux Requires: openssh-clients Requires: systemd-resolved Requires: dbus-common +# method=tproxy runs sshuttle's iptables method (iptables and ip6tables) +Recommends: iptables-nft %description nm-sshuttle runs sshuttle as a NetworkManager VPN connection. The tunnel is diff --git a/test-vm/README.md b/test-vm/README.md index 9577677..15d9cca 100644 --- a/test-vm/README.md +++ b/test-vm/README.md @@ -45,6 +45,9 @@ block is lifted. Each scenario has an id `M2-NN`; its checks are `M2-NNx`. | M2-29 | a ProxyJump host named inside `~corp.test`: resolved from the local network at connect, pinned to that address on the reconnect | | V6-01 | an IPv6 subnet through the tunnel, arriving from the jump host; refused at once in a gap ([IPv6](../docs/later/ipv6.md)) | | V6-02 | the uplink's IPv6 addresses change: a second address in the prefix is no network change, a new prefix restarts a tunnel whose first hop is IPv6, and not an IPv4 one | +| TP-01 | `method=tproxy` ([tproxy](../docs/later/tproxy.md)): sshuttle's mark chain and the plugin's rule and route; UDP echo, TCP and split DNS through the tunnel; UDP and TCP refused at once in a gap; nothing left after the disconnect | +| TP-02 | `method=tproxy`: sshuttle killed while up, then `nft flush ruleset`: the guard comes back, the missing mark chain causes a reconnect, the rule and route stay | +| TP-03 | `method=tproxy`: `nmcli general reload` and a roam to a second uplink leave the rule and route alone | `PASS` and `FAIL` are asserted. `INFO` prints something the design marks as unobserved (for example firewalld's zone binding after a link loss, or what @@ -142,6 +145,6 @@ display (`test-vm/vm.sh viewer`), and create the profile as in The editor needs a real GNOME session; the protocol for it, which also covers building the packages, is [gnome-editor-protocol.md](gnome-editor-protocol.md). It runs on a laptop, or in this VM with `vm.sh up --gnome` and the VNC display. -The VM's preparation installs the packages that build the editor (`gcc`, -`gtk4-devel`, `libnm-devel`, `json-glib-devel`), and `vm.sh run` installs it +The VM's preparation installs what builds the editor (`gcc` and the +`pkgconfig(...)` names the RPM spec lists), and `vm.sh run` installs it with the plugin. diff --git a/test-vm/lib.sh b/test-vm/lib.sh index 5cb9208..586fca3 100644 --- a/test-vm/lib.sh +++ b/test-vm/lib.sh @@ -19,7 +19,6 @@ sc() { systemctl --no-pager "$@"; } jc() { journalctl --no-pager "$@"; } bc() { busctl --system --no-pager "$@"; } rv() { resolvectl --no-pager "$@"; } -jsync() { journalctl --sync > /dev/null 2>&1; } BUS=org.freedesktop.NetworkManager.sshuttle PLUGIN_UNIT=nm-sshuttle.service @@ -49,6 +48,10 @@ WEB_IP6=2001:db8:99::10 HAVE_IPV6=no [ -e /proc/net/if_inet6 ] && HAVE_IPV6=yes CON6=nmtest-corp6 +CONTP=nmtest-corptp # method=tproxy (docs/later/tproxy.md) +UDP_PORT=7777 # UDP echo on $WEB_IP +TPROXY_MARK=0x4e530000 +TPROXY_TABLE=20051 LANB6_CON=nmtest-lanb6 UPLINK="" BLOCK_TABLE=nmtest-block @@ -139,10 +142,10 @@ diag_state() { echo "guard: $(guard_present && echo present || echo absent); tunnel: $(sc is-active "$TUNNEL_UNIT")" \ "; plugin: $(sc is-active "$PLUGIN_UNIT") pid $(plugin_pid)" echo "--- plugin journal tail" - jc -u "$PLUGIN_UNIT" -n 30 -o short-precise 2>&1 + tail -n 3000 "$JOURNAL_LOG" | grep -E "$PLUGIN_LINE" | tail -n 30 } > "$tmp" 2>&1 DETAIL_LINES=200 detail "$1-state" "state when it failed" < "$tmp" - sed 's/^/ | /' "$tmp" | head -n 40 + sed 's/^/ | /' "$tmp" | head -n 70 rm -f "$tmp" } @@ -252,12 +255,15 @@ EOF systemd-run --quiet --collect --unit=nmtest-sshd \ -p NetworkNamespacePath=/run/netns/jump /usr/sbin/sshd -D -e -f "$SSHD_DIR/sshd_config" - systemd-run --quiet --collect --unit=nmtest-dns \ + systemd-run --quiet --collect --unit=nmtest-dns -p SyslogIdentifier=nmtest-dns \ -p NetworkNamespacePath=/run/netns/internal /usr/bin/python3 "$REPO/lab/dns_server.py" $DNS_IP systemd-run --quiet --collect --unit=nmtest-web \ -p NetworkNamespacePath=/run/netns/internal \ /usr/bin/python3 -m http.server --bind $WEB_IP 8080 --directory "$VARDIR/www" - [ "$HAVE_IPV6" = yes ] && systemd-run --quiet --collect --unit=nmtest-web6 \ + systemd-run --quiet --collect --unit=nmtest-udp \ + -p NetworkNamespacePath=/run/netns/internal \ + /usr/bin/python3 "$REPO/lab/udp_echo.py" serve $WEB_IP $UDP_PORT + [ "$HAVE_IPV6" = yes ] && systemd-run --quiet --collect --unit=nmtest-web6 -p SyslogIdentifier=nmtest-web6 \ -p NetworkNamespacePath=/run/netns/internal \ /usr/bin/python3 -m http.server --bind $WEB_IP6 8080 --directory "$VARDIR/www" for i in $(seq 1 50); do @@ -477,28 +483,43 @@ plugin_owns_bus() { # plugin_owns_bus [PID] : alive, and owns the bus name [ -n "$pid" ] && [ "$pid" != 0 ] && kill -0 "$pid" 2> /dev/null && [ "$(bus_owner_pid)" = "$pid" ] } -# log_mark : later plog calls show the plugin unit's journal after this point +# The checks read the journal through one journalctl -f that runs for the +# whole run: fresh reads, by unit, after a cursor or since a time, at times +# missed the plugin's newest entries, which a later read showed. +JOURNAL_LOG=$VARDIR/journal.log +JOURNAL_PID="" +PLUGIN_LINE=' nm-sshuttle-service\[[0-9]+\]: ' +journal_follow() { + journalctl --no-pager --follow --lines=0 -o short-precise > "$JOURNAL_LOG" 2> /dev/null & + JOURNAL_PID=$! +} +journal_stop() { [ -z "$JOURNAL_PID" ] || kill "$JOURNAL_PID" 2> /dev/null; JOURNAL_PID=""; } +journal_since_mark() { tail -c "+$(( ${LOGMARK:-0} + 1 ))" "$JOURNAL_LOG" 2> /dev/null; } +# log_mark : later plog calls show the plugin's journal after this point log_mark() { - jsync - LOGMARK=$(jc -u "$PLUGIN_UNIT" -n 1 -o cat --show-cursor 2> /dev/null | sed -n 's/^-- cursor: //p') + LOGMARK=$(stat -c %s "$JOURNAL_LOG" 2> /dev/null || echo 0) MARK_EPOCH=$(date +%s) } -plog() { # the plugin unit's journal since log_mark (or since the run began) - jsync - if [ -n "$LOGMARK" ]; then - jc -u "$PLUGIN_UNIT" --after-cursor="$LOGMARK" -o short-precise 2>&1 - else - jc -u "$PLUGIN_UNIT" --since "@${MARK_EPOCH:-$START_EPOCH}" -o short-precise 2>&1 - fi -} +plog() { journal_since_mark | grep -E "$PLUGIN_LINE"; } # the plugin's journal since log_mark (or since the run began) plog_has() { plog | grep -q -F -- "$1"; } plog_has_not() { ! plog | grep -q -F -- "$1"; } +# journald tells followers about new entries in batches, so a line shows up in +# JOURNAL_LOG a moment after it was logged. journal_catch_up : until the file +# holds what journald had before this call, which a marker of our own shows. +journal_catch_up() { + local m="nmtest-sync-$RANDOM$RANDOM" + echo "$m" | systemd-cat -t nmtest-sync + wait_for 5 grep -q -F -- "$m" "$JOURNAL_LOG" +} plog_show() { # plog_show PATTERN : the matching lines (a check's evidence); fails without any + journal_catch_up + wait_for 2 plog_has "$1" plog | grep -F -- "$1" | tail -n 8 | grep . } # Python errors and the supervisor's own error paths, since log_mark plog_errors() { local errs + journal_catch_up errs=$(plog | grep -E 'unexpected error|Traceback|error while handling|D-Bus handler error') [ -z "$errs" ] || { echo "$errs" | head -n 10; return 1; } } @@ -559,7 +580,9 @@ uplink_resolved_ok() { # the uplink keeps its servers and resolved's default ro [ "${dr##* }" = yes ] || { echo "$UPLINK default-route is not yes"; rc=1; } return "$rc" } -dnslog_has() { jsync; jc -u nmtest-dns --since "@$START_EPOCH" -o cat | grep -q -F -- "$1"; } +# servers_log NAME : what the test server NAME logged since log_mark (or since the run began) +servers_log() { journal_since_mark | grep -F " $1["; } +dnslog_has() { servers_log nmtest-dns | grep -q -F -- "$1"; } split_lookup_ok() { # a fresh split name resolves, and the query reached the internal server via the jump host local n="u$RANDOM$RANDOM.corp.test" out out=$(rv query --cache=no --legend=no "$n" 2>&1) @@ -576,7 +599,8 @@ outside_lookup_ok() { # a fresh outside name is not sent to the tunnel's DNS se } http_ok() { [ "$(curl -g -s -m 5 -o /dev/null -w '%{http_code}' --noproxy '*' "$1")" = 200 ]; } # web6_from ADDRESS : the IPv6 web server logged a request from ADDRESS since log_mark -web6_from() { jsync; jc -u nmtest-web6 --since "@$MARK_EPOCH" -o cat | grep "^$1 "; } +web6_lines() { servers_log nmtest-web6 | sed 's/^[^]]*\]: //' | grep "^$1 "; } +web6_from() { wait_for 3 web6_lines "$1"; web6_lines "$1"; } # info_or_check ID TEXT CMD... : like check, but only INFO with what CMD saw # when INFO_ONLY lists ID's last letter (an answer, not an assertion) @@ -613,6 +637,10 @@ guard_present() { nft list table inet "$GUARD_TABLE" > /dev/null 2>&1; } guard_gone() { ! guard_present; } sshuttle_tables() { nft list tables 2> /dev/null | awk '$3 ~ /^sshuttle-ipv[46]-[0-9]+$/ {print $3}'; } tunnel_state() { sc is-active "$TUNNEL_UNIT" 2> /dev/null; } +# method=tproxy: sshuttle's chains in iptables' mangle table, the plugin's rule and route +mangle_chains() { iptables -w -t mangle -S 2> /dev/null | awk '$1 == "-N" && $2 ~ /^sshuttle-[mtd]-[0-9]+$/ {print $2}'; } +tproxy_rule() { ip rule list 2> /dev/null | grep "fwmark $TPROXY_MARK lookup $TPROXY_TABLE"; } +tproxy_route() { ip route show table $TPROXY_TABLE 2> /dev/null | grep '^local default dev lo'; } tunnel_active() { [ "$(tunnel_state)" = active ]; } tunnel_gone() { case $(tunnel_state) in inactive|failed|unknown) return 0 ;; *) return 1 ;; esac; } all_clean() { # no link, no guard, no tunnel unit, no stale sshuttle table @@ -721,6 +749,8 @@ leftovers() { guard_present && l="$l guard" tunnel_gone || l="$l tunnel-unit($(tunnel_state))" [ -z "$(sshuttle_tables)" ] || l="$l sshuttle-table" + [ -z "$(mangle_chains)" ] || l="$l sshuttle-mangle-chain" + [ -z "$(tproxy_rule)$(tproxy_route)" ] || l="$l tproxy-routing" echo "${l# }" } clear_leaked_dns() { # a leaked VPN DNS entry would spoil later DNS checks @@ -735,7 +765,7 @@ clear_leaked_dns() { # a leaked VPN DNS entry would spoil later DNS checks reset_state() { local c left unblock_jump - for c in "$CON" "$CON_ZONE" "$CON6"; do + for c in "$CON" "$CON_ZONE" "$CON6" "$CONTP"; do [ -n "$(con_state "$c")" ] && nmcli connection down "$c" > /dev/null 2>&1 done wait_for 30 con_gone @@ -764,13 +794,15 @@ avc_since() { # avc_since EPOCH OFFSET : SELinux denial records since then cleanup() { set +e note "cleaning up" + journal_stop unblock_jump nmcli connection down "$CON" > /dev/null 2>&1 nmcli connection down "$CON_ZONE" > /dev/null 2>&1 nmcli connection down "$CON6" > /dev/null 2>&1 - nmcli connection delete "$CON" "$CON_ZONE" "$CON6" > /dev/null 2>&1 + nmcli connection down "$CONTP" > /dev/null 2>&1 + nmcli connection delete "$CON" "$CON_ZONE" "$CON6" "$CONTP" > /dev/null 2>&1 sc stop "$PLUGIN_UNIT" "$TUNNEL_UNIT" > /dev/null 2>&1 - sc stop nmtest-sshd nmtest-dns nmtest-web nmtest-web6 nmtest-dns-local > /dev/null 2>&1 + sc stop nmtest-sshd nmtest-dns nmtest-web nmtest-web6 nmtest-dns-local nmtest-udp > /dev/null 2>&1 rv revert h-j > /dev/null 2>&1 if [ -f "$VARDIR/ssh-config-written" ]; then rm -f "$HOME_U/.ssh/config" "$VARDIR/ssh-config-written" diff --git a/test-vm/run.sh b/test-vm/run.sh index 617df4b..ef56d32 100755 --- a/test-vm/run.sh +++ b/test-vm/run.sh @@ -29,6 +29,9 @@ # M2-29 a ProxyJump host named inside the tunnelled DNS domain # V6-01 IPv6 subnet through the tunnel, fail closed in a gap # V6-02 roams that change only the uplink's IPv6 addresses +# TP-01 method=tproxy: UDP and TCP through the tunnel, fail closed in a gap +# TP-02 method=tproxy: kill sshuttle and nft flush ruleset while up +# TP-03 method=tproxy: NM leaves the policy routing alone; teardown removes it # # Result ids are M2-NN (scenario) and M2-NNx (a check in it). PASS and FAIL are # asserted; INFO prints something the design marks unobserved. A FAIL prints @@ -55,7 +58,7 @@ while [ $# -gt 0 ]; do --keep) KEEP=1; shift ;; --suspend) SUSPEND=1; shift ;; cleanup) CMD=cleanup; shift ;; - -h|--help) sed -n '3,35p' "$0"; exit 0 ;; + -h|--help) sed -n '3,38p' "$0"; exit 0 ;; *) echo "unknown argument: $1" >&2; exit 2 ;; esac done @@ -199,7 +202,7 @@ bounce_signals_ok() { signal_sequence "$1" | grep -q 'StateChanged:4 StateChange guard_and_refused() { guard_present && refused_fast; } stays_down() { sleep 3; echo "state '$(con_state)'"; con_gone; } zone_con_gone() { [ -z "$(con_state "$CON_ZONE")" ]; } -sigterm_logged() { plog_has "SIGTERM during gap" && plog_has "giving up"; } +sigterm_logged() { plog_show "SIGTERM during gap" && plog_show "giving up"; } new_instance_running() { local p; p=$(plugin_pid); echo "MainPID $p, before $1"; [ -n "$p" ] && [ "$p" != 0 ] && [ "$p" != "$1" ]; } new_instance_started() { plog | grep -o 'starting (pid [0-9]*' | grep -v "pid $1\$" | grep -q .; } idle_exit_done() { plugin_unit_down && ! bus_owned; } @@ -310,6 +313,7 @@ s_reconnect_nofw() { gap_len_ok() { local l + journal_catch_up l=$(plog | grep -o 'reconnected after [0-9.]*' | tail -n 1 | awk '{print $3}') echo "the plugin measured ${l:-no reconnect} s" awk -v l="${l:-0}" 'BEGIN {exit !(l >= 10)}' @@ -617,8 +621,15 @@ s_stoppost() { s_selinux() { SCENARIO=M2-15 note "M2-15 SELinux denials over the whole run" - local n out + local n out other out=$(avc_since "$START_EPOCH" "$START_OFFSET") + # Not the plugin's: Fedora's getty reading the issue file NetworkManager's + # dispatcher writes for the console, seen in some runs + other=$(grep -E 'comm="agetty".*tcontext=[^ ]*:NetworkManager_dispatcher_console_var_run_t:' <<< "$out") + if [ -n "$other" ]; then + result M2-15i INFO "ignored, not the plugin's: $(grep -c . <<< "$other") denial(s) of agetty reading NetworkManager's console issue file" + out=$(grep -v -F -x -f <(echo "$other") <<< "$out") + fi n=$(grep -c . <<< "$out") if [ -z "$out" ]; then result M2-15 PASS "no SELinux denial records during the run" @@ -635,6 +646,7 @@ s_selinux() { # --------------------------------------------------- M2 completion scenarios reconnected_quickly() { # the last reconnect, as the plugin measured it, took at most $1 s local l + journal_catch_up l=$(plog | grep -o 'reconnected after [0-9.]*' | tail -n 1 | awk '{print $3}') echo "the plugin measured ${l:-no reconnect} s" [ -n "$l" ] && awk -v l="$l" -v m="$1" 'BEGIN {exit !(l <= m)}' @@ -1001,7 +1013,7 @@ jump_name_teardown() { fi return 0 } -pin_count() { jsync; jc -u "$TUNNEL_UNIT" --since "@$START_EPOCH" -o cat | grep -c -F "the first hop's name does not resolve"; } +pin_count() { journal_catch_up; grep -c -F "the first hop's name does not resolve" "$JOURNAL_LOG"; } pinned_jump() { grep -A2 "^Host $JUMP_NAME\$" /run/nm-sshuttle-ssh/ssh_config | grep -q "HostName $JUMP_IP"; } s_jump_name() { @@ -1116,6 +1128,120 @@ s_ipv6_roam() { end_checks V6-02 } +# ------------------------------------------------- tproxy (docs/later/tproxy.md) +udp_echo() { python3 "$REPO/lab/udp_echo.py" send $WEB_IP $UDP_PORT "$@"; } +udp_ok() { local out; out=$(udp_echo "$1"); echo "$out"; [ "$out" = "ECHO $1 from 10.99.0.1" ]; } +udp_refused() { local out; out=$(udp_echo gap); echo "$out"; [ "$out" = "ERROR Operation not permitted" ]; } +contp_is() { [ "$(con_state "$CONTP")" = "$1" ]; } +contp_gone() { [ -z "$(con_state "$CONTP")" ]; } +has_mark_chain() { mangle_chains; mangle_chains | grep -q '^sshuttle-m-'; } +routing_present() { tproxy_rule && tproxy_route; } +routing_gone() { [ -z "$(tproxy_rule)$(tproxy_route)" ]; } +tproxy_clean() { + echo "chains: '$(mangle_chains | xargs)'; rule: '$(tproxy_rule)'; route: '$(tproxy_route)'" + [ -z "$(mangle_chains)" ] && routing_gone +} +have_iptables() { command -v iptables > /dev/null && command -v ip6tables > /dev/null; } +# Fedora's sshuttle pulls in iptables-legacy, and which backend iptables uses +# then depends on the install order +iptables_legacy() { iptables -V 2> /dev/null | grep -q legacy; } +# tp_recovered ID IFINDEX : activated again, same link, UDP through the tunnel +tp_recovered() { + if WAIT_POLL=0.5 wait_for 150 contp_is activated; then + result "$1" PASS "activated again ($((WAITED_MS / 1000)) s)" + else + result "$1" FAIL "not activated again within 150 s (state '$(con_state "$CONTP")')" + plog | tail -n 30 | detail "$1" "plugin journal" + return 1 + fi + sleep 1 + check "$1-i" "same ifindex ($2) after the reconnect" test "$(link_ifindex)" = "$2" + check "$1-c" "sshuttle's mark chain is back" has_mark_chain + check "$1-u" "UDP flows through the tunnel again" udp_ok again +} + +s_tproxy() { + begin TP-01 "method=tproxy: UDP and TCP through the tunnel, fail closed in a gap" + have_iptables || { result TP-01 SKIP "iptables and ip6tables are not installed"; return; } + create_profile "$CONTP" "method = tproxy" + connect_ok TP-01 "activation of a method=tproxy profile" "$CONTP" || return + local ifi + ifi=$(link_ifindex) + result TP-01i INFO "$(iptables -V 2>&1)" + check TP-01a "sshuttle has a mark chain in iptables' mangle table" has_mark_chain + check TP-01b "the plugin's rule and local route are in place" routing_present + check TP-01c "UDP echo to $WEB_IP:$UDP_PORT answers, from the jump host" udp_ok hello + check TP-01d "TCP, by address and by a split DNS name" traffic_ok + check TP-01e "a fresh split name resolves through the tunnel (resolved, IP_UNICAST_IF)" split_lookup_ok + block_jump + drop_tunnel + wait_for 15 contp_is activating || { result TP-01f FAIL "no gap: state '$(con_state "$CONTP")'"; gap_end; return; } + check TP-01f "UDP to the subnet is refused at once during the gap (fail closed)" udp_refused + check TP-01g "TCP likewise" refused_fast + check TP-01h "the rule and route stay during the gap" routing_present + gap_end + tp_recovered TP-01r "$ifi" + nmcli connection down "$CONTP" > /dev/null 2>&1 + wait_for 30 contp_gone + check TP-01z "after the disconnect: no mark chain, rule or route left" wait_for 10 tproxy_clean + end_checks TP-01 +} + +s_tproxy_health() { + begin TP-02 "method=tproxy: kill sshuttle and nft flush ruleset while up" + have_iptables || { result TP-02 SKIP "iptables and ip6tables are not installed"; return; } + create_profile "$CONTP" "method = tproxy" + connect_ok TP-02 "activation of a method=tproxy profile" "$CONTP" || return + local ifi t0 v="" + ifi=$(link_ifindex) + drop_tunnel + wait_for 15 plog_has "tunnel dropped" + check TP-02a "plugin log: the tunnel unit stopped, then a reconnect" plog_show "tunnel dropped (the tunnel unit stopped)" + tp_recovered TP-02b "$ifi" || { end_checks TP-02; return; } + log_mark + t0=$(now_ms) + nft flush ruleset + if WAIT_POLL=0.2 wait_for 15 guard_present; then v=$(( $(now_ms) - t0 )); fi + check_le TP-02c "the guard table is back after the flush" "$v" 11000 ms + if iptables_legacy; then # its chains are not in nft, so the flush leaves them + check TP-02d "iptables-legacy: the flush left sshuttle's mark chain, so no reconnect" \ + has_mark_chain + else + wait_for 15 plog_has "no sshuttle mark chain with a listener" + check TP-02d "plugin log: no sshuttle mark chain with a listener, so a reconnect" \ + plog_show "no sshuttle mark chain with a listener" + fi + check TP-02e "the flush left the rule and route alone" routing_present + tp_recovered TP-02f "$ifi" + if fw_running; then + firewall-cmd --reload > /dev/null 2>&1 + result TP-02z INFO "firewalld reloaded after the flush: $(fw_running && echo running)" + fi + end_checks TP-02 +} + +s_tproxy_nm() { + begin TP-03 "method=tproxy: NM leaves the policy routing alone" + have_iptables || { result TP-03 SKIP "iptables and ip6tables are not installed"; return; } + create_profile "$CONTP" "method = tproxy" + connect_ok TP-03 "activation of a method=tproxy profile" "$CONTP" || return + local ifi + ifi=$(link_ifindex) + nmcli general reload > /dev/null 2>&1 + sleep 3 + check TP-03a "after 'nmcli general reload' the rule and route are still there" routing_present + check TP-03b "... and UDP flows" udp_ok reload + if lanb_setup && nmcli --wait 30 connection up "$LANB_CON" >> "$REPORT/raw/roam.txt" 2>&1; then + check TP-03c "plugin log: the network changed" wait_for 10 plog_has "the network changed" + tp_recovered TP-03d "$ifi" + check TP-03e "after the roam the rule and route are still there" routing_present + else + result TP-03c SKIP "cannot activate the second uplink" + fi + lanb_teardown + end_checks TP-03 +} + s_suspend() { begin M2-28 "suspend and resume" if [ "$SUSPEND" != 1 ]; then @@ -1192,6 +1318,8 @@ collect() { preflight cleanup > /dev/null 2>&1 # leftovers of an earlier run mkdir -p "$REPORT/raw" "$VARDIR" +journal_follow +[ "$KEEP" = 1 ] && trap journal_stop EXIT agent_setup topology create_profile "$CON" @@ -1226,6 +1354,9 @@ want M2-28 && s_suspend want M2-29 && s_jump_name want V6-01 && s_ipv6 want V6-02 && s_ipv6_roam +want TP-01 && s_tproxy +want TP-02 && s_tproxy_health +want TP-03 && s_tproxy_nm s_selinux reset_state collect diff --git a/test-vm/vm.sh b/test-vm/vm.sh index 6ddfa3c..87e4257 100755 --- a/test-vm/vm.sh +++ b/test-vm/vm.sh @@ -79,8 +79,9 @@ prepare() { fi say "installing meson, ninja and firewalld in the VM" # gcc and the -devel packages let meson build the editor for GNOME Settings - vm "sudo dnf -y install meson ninja-build firewalld nftables iproute python3-gobject-base sshuttle && - sudo dnf -y install gcc gtk4-devel libnm-devel json-glib-devel && + vm "sudo dnf -y install meson ninja-build firewalld nftables iproute python3-gobject-base sshuttle \ + iptables-nft && + sudo dnf -y install gcc 'pkgconfig(gtk4)' 'pkgconfig(libnm)' 'pkgconfig(json-glib-1.0)' && sudo systemctl enable --now firewalld && sudo loginctl enable-linger $VM_USER && sudo touch /var/lib/nmtest-prepared" < /dev/null || die "preparing the VM failed" diff --git a/tests/fakes.py b/tests/fakes.py index d5c40a7..220be9f 100644 --- a/tests/fakes.py +++ b/tests/fakes.py @@ -58,6 +58,10 @@ def __init__(self): self.probe_cbs = [] self.lock_cbs = [] self.ipv6 = True # IPv6 is on on the host + self.iptables = True # iptables and ip6tables exist + self.guard_mark = None # the guard's tproxy mark, as last installed + self.routing = None # families with tproxy policy routing, or None + self.health_methods = [] # recording helpers def _rec(self, *event): @@ -123,9 +127,16 @@ def session_locked(self, user, done): self.lock_cbs.append(done) # health - def nft_health(self): + def nft_health(self, method="nft"): + self.health_methods.append(method) return None if self.health is None else dict(self.health) + def policy_routing_present(self, families): + return self.routing is not None and set(families) <= self.routing + + def iptables_available(self): + return self.iptables + def ipv6_available(self): return self.ipv6 @@ -172,8 +183,17 @@ def remove_link(self): self._rec("remove_link") self.link = None - def install_guard(self, networks, exclude): + def install_guard(self, networks, exclude, mark=None): self._rec("install_guard", list(networks), list(exclude)) + self.guard_mark = mark + + def install_policy_routing(self, families): + self._rec("install_routing", list(families)) + self.routing = set(families) + + def remove_policy_routing(self): + self._rec("remove_routing") + self.routing = None def remove_guard(self): self._rec("remove_guard") diff --git a/tests/profile-cases.json b/tests/profile-cases.json index 554f2ac..960e03f 100644 --- a/tests/profile-cases.json +++ b/tests/profile-cases.json @@ -97,7 +97,8 @@ {"name": "method nat", "data": {"method": "nat"}}, {"name": "method nft", "data": {"method": "nft"}}, - {"name": "method tproxy before it exists", "data": {"method": "tproxy"}, "error": "method"}, + {"name": "method tproxy", "data": {"method": "tproxy"}}, + {"name": "method that does not exist", "data": {"method": "socks"}, "error": "method"}, {"name": "method that is capitalised", "data": {"method": "NFT"}, "error": "method"}, {"name": "method that is empty", "data": {"method": ""}, "error": "method"}, diff --git a/tests/test_service_dbus.py b/tests/test_service_dbus.py index cb41aa5..feb31d5 100644 --- a/tests/test_service_dbus.py +++ b/tests/test_service_dbus.py @@ -339,10 +339,11 @@ def remove_link(name="nmss0"): monkeypatch.setattr(guard, "create_link", create_link) monkeypatch.setattr(guard, "remove_link", remove_link) monkeypatch.setattr(guard, "link_ifindex", lambda name="nmss0": host["link"]) - monkeypatch.setattr(guard, "install_guard", lambda n, e: host["ops"].append("guard")) + monkeypatch.setattr(guard, "install_guard", lambda n, e, mark=None: host["ops"].append("guard")) monkeypatch.setattr(guard, "remove_guard", lambda: host["ops"].append("unguard")) monkeypatch.setattr(tunnel, "sweep", lambda: None) - monkeypatch.setattr(tunnel, "nft_health", lambda: {"guard": True, "sshuttle": {4, 6}}) + monkeypatch.setattr(tunnel, "nft_health", + lambda method="nft": {"guard": True, "sshuttle": {4, 6}}) monkeypatch.setattr(tunnel, "ipv6_available", lambda: True) monkeypatch.setattr(service.pwd, "getpwnam", fake_getpwnam) monkeypatch.setattr(service.Service, "run_cleanup", lambda self, done: done()) diff --git a/tests/test_tproxy.py b/tests/test_tproxy.py new file mode 100644 index 0000000..4ea6861 --- /dev/null +++ b/tests/test_tproxy.py @@ -0,0 +1,192 @@ +# SPDX-License-Identifier: MIT +"""method=tproxy (docs/later/tproxy.md): sshuttle's mark, the guard's mark +rules, the policy routing, the sweep of mangle chains and the supervisor.""" + +import argparse +from types import SimpleNamespace + +from fakes import NM, bring_up, fake_getpwnam, make, settings +from test_cli import Nmcli, conn, ssh_g_ok +from nm_sshuttle import cli, guard, tunnel +from nm_sshuttle.const import FAIL_CONNECT, ST_STOPPED, TPROXY_MARK +from nm_sshuttle.profile import parse + +SPEC = {"remote": "corp", "user": "alice", "subnets": ["10.0.0.0/8"], "exclude": [], + "dns": "none", "dns_servers": [], "method": "tproxy", "first_hop": ["203.0.113.7"], + "tmark": TPROXY_MARK} + + +# ------------------------------------------------------------- profile, argv +def test_only_a_tproxy_spec_carries_the_mark(): + nft = parse(settings(), fake_getpwnam).tunnel_spec(["203.0.113.7"]) + assert "tmark" not in nft + spec = parse(settings(method="tproxy"), fake_getpwnam).tunnel_spec(["203.0.113.7"]) + assert spec["method"] == "tproxy" and spec["tmark"] == 0x4e530000 + + +def test_sshuttle_argv_passes_the_mark(): + argv = tunnel.sshuttle_argv(SPEC, "sshuttle") + assert argv[argv.index("--method") + 1] == "tproxy" + assert argv[argv.index("--tmark") + 1] == "0x4e530000" + assert "--tmark" not in tunnel.sshuttle_argv(dict(SPEC, method="nft", tmark=None), "s") + + +# ------------------------------------------------------------------- guard +def test_the_guard_accepts_the_mark_ahead_of_the_dnat_rule(): + rules = guard.guard_ruleset(["10.0.0.0/8", "2001:db8::/48"], [], TPROXY_MARK).splitlines() + mark4 = rules.index(" ip daddr @subnets4 meta mark 0x4e530000 accept") + mark6 = rules.index(" ip6 daddr @subnets6 meta mark 0x4e530000 accept") + assert max(mark4, mark6) < rules.index(" ip daddr @subnets4 ct status dnat accept") + assert "meta mark" not in guard.guard_ruleset(["10.0.0.0/8"], []) + + +def fake_ip(monkeypatch, rules="", routes=""): + """Record ip's argv; 'rule del' fails once nothing is left to delete.""" + calls = [] + state = {"rules": 1} + + def run(argv, input=None, timeout=10): + calls.append(argv) + if argv[2:4] == ["rule", "del"]: + state["rules"] -= 1 + return state["rules"] >= 0, "" + if argv[2:4] == ["rule", "list"]: + return True, rules + if argv[2:4] == ["route", "show"]: + return True, routes + return True, "" + monkeypatch.setattr(guard, "run", run) + return calls + + +def test_policy_routing_goes_by_exact_values(monkeypatch): + calls = fake_ip(monkeypatch) + guard.install_policy_routing([4, 6]) + rule = ["priority", "100", "fwmark", "0x4e530000/0xffffffff", "lookup", "20051"] + route = ["local", "default", "dev", "lo", "table", "20051"] + assert ["ip", "-4", "rule", "add", *rule] in calls + assert ["ip", "-6", "route", "replace", *route] in calls + calls.clear() + assert guard.remove_policy_routing() + assert ["ip", "-4", "rule", "del", *rule] in calls + assert not any("flush" in argv for argv in calls) + + +def test_policy_routing_is_read_back_per_family(monkeypatch): + fake_ip(monkeypatch, rules="0:\tfrom all lookup local\n" + "100:\tfrom all fwmark 0x4e530000 lookup 20051\n", + routes="local default dev lo scope host\n") + assert guard.policy_routing_present([4, 6]) + fake_ip(monkeypatch, rules="100:\tfrom all fwmark 0x1 lookup 100\n", + routes="local default dev lo scope host\n") + assert not guard.policy_routing_present([4]) + + +# ------------------------------------------------------------------- sweep +MANGLE = """-P PREROUTING ACCEPT +-P OUTPUT ACCEPT +-N sshuttle-d-12300 +-N sshuttle-m-12300 +-N sshuttle-t-12300 +-N sshuttle-m-12301 +-N other-chain +-A PREROUTING -j sshuttle-t-12300 +-A OUTPUT -j sshuttle-m-12301 +-A OUTPUT -j sshuttle-m-12300 +-A sshuttle-m-12300 -d 10.0.0.0/8 -p udp -m udp -j MARK --set-xmark 0x4e530000/0xffffffff +""" + + +def test_mangle_chains_are_parsed(): + assert tunnel.mangle_chains(MANGLE) == {"sshuttle-d-12300": 12300, + "sshuttle-m-12300": 12300, + "sshuttle-t-12300": 12300, + "sshuttle-m-12301": 12301} + + +def test_the_sweep_removes_the_chains_of_a_port_without_listener(): + cmds = tunnel.stale_chain_commands(MANGLE, ports={12301}) + assert cmds == [["-D", "PREROUTING", "-j", "sshuttle-t-12300"], + ["-D", "OUTPUT", "-j", "sshuttle-m-12300"], + ["-F", "sshuttle-d-12300"], ["-F", "sshuttle-m-12300"], + ["-F", "sshuttle-t-12300"], + ["-X", "sshuttle-d-12300"], ["-X", "sshuttle-m-12300"], + ["-X", "sshuttle-t-12300"]] + assert tunnel.stale_chain_commands(MANGLE, ports={12300, 12301}) == [] + + +def test_health_reads_the_mark_chain_for_tproxy(monkeypatch): + monkeypatch.setattr(tunnel.subprocess, "run", lambda argv, **kw: SimpleNamespace( + returncode=0, stdout=MANGLE if argv[0].endswith("tables") else + "table inet nm-sshuttle-guard\ntable inet sshuttle-ipv4-12300\n")) + monkeypatch.setattr(tunnel, "listening_ports", lambda: {12300}) + assert tunnel.nft_health("tproxy") == {"guard": True, "sshuttle": {4, 6}} + monkeypatch.setattr(tunnel, "listening_ports", lambda: {12399}) + assert tunnel.nft_health("tproxy")["sshuttle"] == set() + + +# -------------------------------------------------------------- supervisor +def test_connect_installs_the_marked_guard_then_the_routing(): + fx, sup = make() + bring_up(fx, sup, method="tproxy") + names = fx.names() + assert names.index("install_guard") < names.index("install_routing") < \ + names.index("start_tunnel") + assert fx.guard_mark == TPROXY_MARK and fx.routing == {4} + + +def test_the_routing_is_installed_without_the_guard_too(): + fx, sup = make() + bring_up(fx, sup, method="tproxy", **{"fail-closed": "no"}) + assert "install_guard" not in fx.names() and fx.routing == {4} + + +def test_the_routing_goes_with_the_guard(): + fx, sup = make() + bring_up(fx, sup, method="tproxy") + sup.disconnect() + fx.advance(0.1) + assert fx.names()[-2:] == ["remove_guard", "remove_routing"] and fx.routing is None + fx2, sup2 = make() + bring_up(fx2, sup2) + sup2.disconnect() + fx2.advance(0.1) + assert "remove_routing" not in fx2.names() + + +def test_connect_fails_without_iptables(): + fx, sup = make() + fx.iptables = False + sup.connect(settings(method="tproxy"), NM) + assert ("failure", FAIL_CONNECT) in fx.events and sup.state == ST_STOPPED + assert "install_guard" not in fx.names() and "start_tunnel" not in fx.names() + fx2, sup2 = make() + fx2.iptables = False + bring_up(fx2, sup2) # nft needs no iptables + assert sup2.phase == "up" + + +def test_health_drops_without_the_mark_chain_and_restores_the_routing(): + fx, sup = make() + bring_up(fx, sup, method="tproxy") + fx.routing = None + fx.advance(10) + assert fx.health_methods[-1] == "tproxy" + assert fx.names()[-1] == "install_routing" and fx.routing == {4} and sup.phase == "up" + fx.health = {"guard": True, "sshuttle": set()} + fx.advance(10) + assert sup.phase == "gap" + + +# --------------------------------------------------------------------- cli +def test_check_wants_iptables_for_tproxy(capsys, monkeypatch): + monkeypatch.setattr(cli.pwd, "getpwnam", lambda n: SimpleNamespace(pw_uid=1000)) + monkeypatch.setattr(cli.pwd, "getpwuid", lambda u: SimpleNamespace(pw_name="dennis")) + data = "remote = corp, local-user = dennis, subnets = 10.0.0.0/8, method = tproxy" + nm = Nmcli({"u1": conn(**{"vpn.data": data})}) + ok = SimpleNamespace(returncode=0) + assert cli.check(argparse.Namespace(name="corp"), nm, ssh_g_ok, lambda a: ok, + lambda: False) == 1 + assert "FAIL method=tproxy needs iptables" in capsys.readouterr().out + assert cli.check(argparse.Namespace(name="corp"), nm, ssh_g_ok, lambda a: ok, + lambda: True) == 0