diff --git a/README.md b/README.md index 52571ff..7c36a35 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ Track what is on your network, what it exposes to the internet, and when that ch `nmap` and Fing answer "what is on my network *right now*". Neither remembers. netdiff records every scan, diffs it against the last one, and tells you what actually changed - a device that appeared at 3am, a printer that quietly opened port 8080, a laptop that moved to a new DHCP lease. -Then `netdiff audit` asks the question those tools do not: **which of these is reachable from outside your house, and why does that matter?** +Then `netdiff audit` asks the question those tools do not: **which of these is reachable from outside your house, and why does that matter?** And `netdiff here` inverts it for wifi you did not choose: **what does this network do to me?** ```console $ netdiff scan 192.168.1.0/24 @@ -151,6 +151,39 @@ One trust boundary is worth naming: SSDP replies are unauthenticated UDP, so any That check holds for every hop, not just the first. A device description can name an absolute `controlURL` that discards the URL we vetted, and any response can redirect, so the control URL is re-checked against the same subnet and redirects are refused outright. The same reasoning covers what gets *printed*: a `verify` line is a command you are told to run, so every value from the network that reaches one - the control URL, a forward's internal client - is validated where it enters, not escaped where it is rendered. +## `netdiff here`: what the network does to *you* + +Every command above asks *what is on this network*. Sitting in a cafe that is the wrong question, and an impolite one - those are not your devices, and the [Scope](#scope) section rules out scanning them. So `netdiff here` inverts it: **not what is on this network, but what does this network do to me.** + +```console +$ netdiff here +here: 10.24.8.0/22 - 1 high, 3 medium, 2 info +gateway 10.24.8.1, resolver(s) 10.24.8.1 + + high this network is reading your encrypted traffic to example.com + medium this network intercepts web traffic - example.com answered 302 + medium this network's resolver invents answers for names that do not exist + medium 2 service(s) on this machine are bound to the network, not loopback + info 31 other device(s) on this network are reachable from here +``` + +Nothing it does is about anyone else here. It asks whether your TLS survives the trip, whether the resolver you were handed tells the truth, whether the ARP table adds up, and which of *your own* ports answer from where you are sitting. + +| Rule | Severity | Fires when | +| --- | --- | --- | +| `here-tls-intercepted` | high | A verified TLS handshake to a public site fails certificate validation - something is terminating your connections and opening new ones onward | +| `here-captive-portal` | medium | A site that never redirects answered with a redirect, so something else answered for it | +| `here-dns-invented` | medium | The resolver returned an address for a name under `.invalid`, which is reserved so that it can never exist | +| `here-dns-redirected` | medium | The resolver answered a public name with a private address. No content network does this | +| `here-arp-claims` | medium | One MAC answers for three or more addresses. Dull explanations exist and the lesson names them | +| `here-own-ports-exposed` | medium | Services on this machine answered on its network address, so they are bound to the network rather than only to loopback | +| `here-client-isolation-off` | info | Explicitly **not** a problem. Other devices are reachable - which cuts both ways | +| `here-client-isolation-on` | info | The good result, printed as such: only the gateway answered | + +**It records nothing.** A cafe network written into `~/.netdiff/history.db` would put strangers' MAC addresses in your `inventory` and make the next scan at home report a wall of `appeared`/`vanished`. `here` is a snapshot of somewhere you are passing through, not a network you track. + +**Two things it deliberately does not claim.** It does not compare the local resolver's answers against a public one: content networks legitimately return different addresses to different resolvers, so a difference is not evidence. The two DNS rules above need no comparison to be certain. And it cannot see an interceptor whose certificate authority your machine already trusts - a corporate laptop with a policy CA installed produces a clean handshake, and nothing observable distinguishes that from an honest network. + ## Why no dependencies, and why no root Most LAN scanners either shell out to `nmap` or send raw ARP frames with `scapy`, and raw frames need root. netdiff does neither. @@ -177,6 +210,8 @@ netdiff scan 192.168.1.0/24 --no-ports # discovery only, no TCP connections netdiff scan 192.168.1.0/24 --no-mdns # skip asking devices what they are netdiff audit 192.168.1.0/24 # what this network exposes, and why it matters netdiff audit --ports top100 3000 5432 # a set, plus whatever else you run +netdiff here # what this network does to you - for wifi you did not choose +netdiff here -v # each finding expanded into its lesson netdiff inventory # every device ever seen, first and last sighting netdiff history # diff the two most recent scans netdiff glossary # every word the output uses, one line each @@ -220,6 +255,8 @@ History lives in `~/.netdiff/history.db` (override with `--db`). It is a plain S - **`--ports top100` is 100 ports, not 65535.** It is nmap's frequency ranking, which is a good answer to "what is worth a timeout" and a bad answer to "what is definitely closed". A service on an unusual port is invisible to both the default set and this one. - **No UPnP gateway means no UPnP findings, not a clean bill of health.** A router with UPnP disabled is a good result, and it is also the common case now. Port forwards you configured by hand do not appear in the UPnP table at all - check your router's admin page for those. - **The audit sees the LAN's exposure, not the internet's view of it.** It reads the forwarding table the router admits to. The only way to know what is actually reachable is to test from outside, which is why every exposure finding hands you that command. +- **`netdiff here` cannot see a trusted interceptor.** If a certificate authority has been installed on this machine - which is how corporate traffic inspection works, with consent - the handshake verifies and there is nothing to report. Nothing observable distinguishes that from an honest network, so it is stated here rather than papered over with a guess. +- **A quiet network looks like an isolated one.** `here-client-isolation-on` says only the gateway answered ARP, which is the observation. Client isolation produces it, and so does being the only guest in the building. The finding is worded as what was seen for that reason. - **The bundled vendor table is small.** It covers common home-network hardware. For full coverage, download the IEEE registry and point `NETDIFF_OUI` at the CSV: ```bash curl -o oui.csv https://standards-oui.ieee.org/oui/oui.csv @@ -230,6 +267,8 @@ History lives in `~/.netdiff/history.db` (override with `--db`). It is a plain S Only scan networks you are responsible for. netdiff is deliberately read-only. In full, what it sends: empty UDP datagrams to provoke ARP, one ICMP echo per device, TCP handshakes, `HEAD /` to HTTP ports, a TLS ClientHello, an SMB negotiate offering one dialect, an SSH version string, a DNS query for `example.com`, the standard DNS-SD question over multicast, and one UPnP request asking the router to list its own port forwards. It never writes to a host, never authenticates, and never changes router configuration. Even so, scanning equipment you do not own is your problem, not the tool's. +`netdiff here` exists because of that last sentence. On a network you did not choose, it adds one DNS query for a name under `.invalid`, one for `example.com`, one HTTP `GET` and one verified TLS handshake to `example.com`, and TCP handshakes to your own address. It sends nothing at all to any other device on the network beyond the same empty datagrams that provoke ARP - every question it asks is about the infrastructure you were handed, or about this machine. + ## Development ```bash @@ -242,6 +281,8 @@ The tests never touch the network. ARP parsing runs against captured `arp -an` a `test_mdns.py` builds its packets with its own helpers rather than with the encoder in `mdns.py`, because a decoder tested only against its own encoder agrees with itself however wrong both are. Half of that file is malformed input - a name pointing at itself, a record claiming to be longer than the packet carrying it - because anything able to send a UDP datagram can send those. +`test_here.py` never touches a network either, which matters more there than anywhere: the interesting cases are all hostile networks - a resolver inventing answers, a box terminating your TLS, a MAC answering for addresses it does not own - and none of them can be arranged on a desk. So the DNS parsing runs against packets the test assembles itself, and every rule is a pure function over what `observe()` gathered. Half of that file asserts something is *not* reported, for a sharper reason than elsewhere: `here` hands its reader a verdict on whether to trust the place they are sitting, and a confident accusation against a network that was fine is worse than a missed finding. + `test_diff.py` covers change detection. `test_audit.py` covers the rules, and roughly half of it asserts that something is *not* reported - an open port, an HTTP 200, a missing security header, a connection error. Those are the important half: the failure mode for a tool like this is not missing a finding, it is inventing one. Every audit rule is a pure function - evidence in, a `Finding` or `None` out - and nothing in `audit.py` opens a socket. That is what makes the security logic testable at all. `Finding.evidence` has no default value, so a finding cannot be constructed without the observation that proves it. diff --git a/netdiff/audit.py b/netdiff/audit.py index 872b7b6..c3abe6d 100644 --- a/netdiff/audit.py +++ b/netdiff/audit.py @@ -23,6 +23,7 @@ from __future__ import annotations +import ipaddress import re from dataclasses import dataclass from datetime import date @@ -393,6 +394,234 @@ def headline(finding) -> str: ), "verify": "curl -sI http://{device}:{port}/ | grep -i server", }, + # --- `netdiff here`: what the network does to you, not what is on it ----- + # + # These fire against infrastructure you were handed rather than devices you + # own, so the bar for each is the same and it is high: an observation that + # has no innocent explanation. "Different from what I expected" is not one. + "here-tls-intercepted": { + "severity": "high", + "title": "this network is reading your encrypted traffic to {host}", + "why": ( + "We opened an ordinary HTTPS connection to {host} - a public site with " + "a certificate from a public authority - and the certificate that came " + "back does not verify. A network that is merely passing traffic along " + "cannot cause that: the certificate is signed by an authority your " + "machine already trusts, and nothing in between has the key needed to " + "produce another one. The likely reading is that something here is " + "terminating your encrypted connections, reading them, and opening new " + "ones onward - so everything you send over this network should be " + "treated as read by a stranger, including passwords and anything " + "already logged in on this machine.\n\n" + "Two things on this computer produce the same failure, and both are " + "worth ruling out first because neither is about the network: a clock " + "set wrongly enough to put today outside the certificate's dates, and " + "a trust store that is empty or unreadable. The evidence line quotes " + "the reason given - anything mentioning expiry points at the clock." + ), + "fix": ( + "Stop using this network for anything that matters, now rather than " + "after you have finished. Do not click through the browser warning - it " + "is the only thing standing between you and the interception, and this " + "is the case it was built for. If you must use the network, use a VPN, " + "which encrypts everything before it touches the local network at all. " + "On a corporate network this may be a deliberate policy with a " + "certificate installed on your machine, in which case you would not be " + "seeing this finding - so it is not that. Check this machine's clock " + "first, though: it is free, and it explains an expiry failure without " + "any of the above being true." + ), + "verify": ( + "openssl s_client -connect {host}:443 /dev/null" + " | openssl x509 -noout -issuer\n" + "The issuer is whoever signed it. On a clean connection that is a\n" + "public certificate authority; here it is whatever is intercepting." + ), + }, + "here-captive-portal": { + "severity": "medium", + "title": "this network intercepts web traffic - {host} answered {status}", + "why": ( + "We asked {host} for its front page and something else replied. That " + "site does not redirect anyone, so the answer did not come from it. " + "This is the ordinary machinery of a captive portal - the hotel or " + "cafe page that wants a room number - and by itself it is unremarkable. " + "What is worth knowing is that the machinery is capable of rewriting " + "any unencrypted request, that it does not always switch off after you " + "have logged in, and that anything on this network sending data over " + "plain HTTP is passing it through whoever runs that box." + ), + "fix": ( + "Log in through the portal if that is what this is, then run `netdiff " + "here` again - a portal that is still intercepting afterwards is doing " + "something other than asking you to agree to terms. Either way, treat " + "unencrypted traffic on this network as public, and prefer a VPN for " + "anything you would not post." + ), + "verify": ( + "curl -sI http://{host}/\n" + "A clean network answers 200 from the site itself. A portal answers\n" + "a redirect to itself, and the Location header names where." + ), + # ponytail: this rests on example.com never issuing a redirect, which is + # IANA's decision and not a property of the protocol. It has held for + # decades. The day it stops, this fires on every network at once - which + # is loud enough that nobody will be quietly misled. + }, + "here-dns-invented": { + "severity": "medium", + "title": "this network's resolver invents answers for names that do not exist", + "why": ( + "We asked {resolver} for a name under `.invalid`, which is reserved so " + "that it can never be registered by anyone, and it returned an address. " + "There is no lookup that could have produced that answer - it was made " + "up. Usually this is an ISP or a hotspot pointing typos at a search or " + "advertising page, which means every mistyped name you enter is logged " + "somewhere with your address beside it. It also breaks the signal " + "software relies on to know a name is wrong, so a program that should " + "fail fast instead connects to whoever is answering." + ), + "fix": ( + "Set a resolver of your own rather than accepting the one this network " + "handed you - 1.1.1.1 or 9.9.9.9 both refuse to do this. On a network " + "you control, look for a 'DNS assist', 'search redirect' or 'web guide' " + "setting at your ISP and turn it off. Note that a network can also " + "intercept port 53 outright, so confirm the change took effect rather " + "than assuming it." + ), + "verify": ( + "dig @{resolver} {name} +short\n" + "Any address at all is a fabricated answer. The correct reply is\n" + "NXDOMAIN, which prints nothing." + ), + }, + "here-dns-redirected": { + "severity": "medium", + "title": "this network's resolver points {name} at an address inside the network", + "why": ( + "{resolver} answered a query for {name} with {address}, which is a " + "private address - one that only exists inside this network. The real " + "site is not there. Whatever is at that address is intercepting " + "connections meant for somewhere else, which is how a captive portal " + "works and also how credentials are collected. This one is not " + "ambiguous: no legitimate content network answers a public name with a " + "private address, so there is no innocent explanation to weigh." + ), + "fix": ( + "If you have not yet passed this network's login page, this is probably " + "that and it will stop once you have. If you have, the network is " + "steering your traffic somewhere of its choosing: stop using it for " + "anything that matters, and use a VPN if you must stay on it." + ), + "verify": ( + "dig @{resolver} {name} +short\n" + "Compare against a resolver this network did not choose:\n" + "dig @1.1.1.1 {name} +short" + ), + }, + "here-arp-claims": { + "severity": "medium", + "title": "one device answers for {count} addresses on this network", + "why": ( + "The MAC address {mac} holds {count} different addresses in this " + "machine's ARP table. ARP has no authentication - any device can answer " + "for any address - so one device claiming many is the signature of " + "something answering for addresses it does not own, which is how " + "traffic between two other machines gets routed through a third. " + "There are dull explanations too, and on most networks it is one of " + "them: a router with several addresses, a bridge, a hypervisor hosting " + "virtual machines. Which it is depends on where you are - on your own " + "network you can probably name the device; on a cafe network you " + "cannot, and that is the point." + ), + "fix": ( + "On a network you control, identify the device and satisfy yourself it " + "is a router or a hypervisor. On one you do not, assume traffic here is " + "observable: use a VPN, and check that sites you care about show a " + "valid certificate rather than clicking through a warning." + ), + "verify": "arp -an | grep -i {mac}\nEach line is an address it claims.", + }, + "here-client-isolation-off": { + "severity": "info", + "title": "{count} other device(s) on this network are reachable from here", + "why": ( + "Not a problem by itself, and the normal case for a home network - it " + "is what lets your laptop print. Worth knowing on a network you do not " + "own, because it works in both directions: those devices are reachable " + "from your machine, and your machine is reachable from them. Public " + "wifi with client isolation switched on would show none of them. This " + "network has it off, so any of the findings about your own open ports " + "in this report are reachable by whoever else is here." + ), + "fix": ( + "Nothing, on your own network. On someone else's, make sure your " + "machine's firewall is on and file sharing is off before you settle in " + "- most operating systems have a 'public network' setting that does " + "both, and it is worth checking it actually got applied when you joined." + ), + "verify": ( + "arp -an\n" + "Every line with a hardware address beside it is a device that\n" + "answered from this network." + ), + }, + "here-client-isolation-on": { + "severity": "info", + "title": "nothing but the gateway answered - this network may isolate its clients", + "why": ( + "Only the router appeared. On public wifi that usually means client " + "isolation is switched on, which is the setting you want: devices can " + "reach the internet and not each other, so nobody here can reach your " + "machine. Stated as the observation rather than the conclusion, because " + "a quiet network looks the same as an isolated one - devices that have " + "not been spoken to may simply not be in the table yet, and a network " + "where you are genuinely the only guest looks identical." + ), + "fix": ( + "Nothing. This is the good result. It does not make the network " + "trustworthy - isolation stops other guests reaching you, and does " + "nothing about whoever runs the network itself." + ), + "verify": ( + "arp -an\n" + "Only the gateway has a hardware address beside it, or the table is\n" + "empty." + ), + }, + "here-own-ports-exposed": { + "severity": "medium", + "title": "{count} service(s) on this machine are bound to the network, not loopback", + "why": ( + "These are your ports, not somebody else's. A service bound only to " + "127.0.0.1 cannot be reached from the network at all; these answered on " + "{address}, this machine's address on this network, so they are offered " + "to it. On your own network that is usually deliberate - it is how file " + "sharing and remote login work. On a network you do not own it is a " + "door left open in a room you do not know, and if this report also says " + "client isolation is off then other people here can knock on it.\n\n" + "What this does not prove is that a connection from another machine " + "would succeed. The check runs from this computer to its own address, " + "and an operating system may answer that without the packet reaching " + "the network - so a host firewall that would refuse the same connection " + "from elsewhere is never consulted. The `verify` command runs it from " + "somewhere that settles it." + ), + "fix": ( + "Turn on your firewall and switch off sharing before joining a network " + "you do not control - macOS has this under Network then Firewall, " + "Windows asks whether a network is public and means it, and Linux has " + "ufw. Where a service only needs to be reachable from this machine, " + "bind it to 127.0.0.1 rather than to every interface." + ), + "verify": ( + "nc -vz {address} {port} # from another device on this network\n" + "Run it from somewhere else, not from this machine. An operating\n" + "system may answer a connection to its own address without the packet\n" + "ever reaching the network, and a firewall that would have refused it\n" + "from elsewhere never gets consulted." + ), + }, "open-ports-noted": { "severity": "info", "title": "{count} open port(s) observed, and not reported as problems", @@ -628,6 +857,192 @@ def rule_dns_recursion(ip: str, evidence: str): return finding("dns-recursion-open", ip, evidence) +# --- `netdiff here` --------------------------------------------------------- +# +# Same contract as every rule above: plain data in, a Finding or None out, no +# sockets. `here.observe()` does the talking. These read its dict. + + +def rule_tls_intercepted(host: str, reason): + """A public certificate that does not verify. Nothing benign does this.""" + if not reason: + # '' is a clean handshake, None is a network we could not reach at all. + # Neither is interception, and only one of them is even about TLS. + return None + return finding( + "here-tls-intercepted", + "network", + f"the certificate for {host} failed verification: {reason}", + host=host, + ) + + +def rule_captive_portal(host: str, response): + """Somebody other than the host answered for it.""" + if response is None: + return None + status, location = response + if not (300 <= status < 400 or status == 511): + return None + where = f", pointing at {location}" if location else "" + return finding( + "here-captive-portal", + "network", + f"GET http://{host}/ answered HTTP {status}{where}", + host=host, + status=status, + ) + + +def rule_dns_invented(resolver: str, name: str, answer): + """An address for a name that is reserved so it can never exist.""" + if answer is None: + return None + rcode, addresses = answer + if rcode != 0 or not addresses: + return None + return finding( + "here-dns-invented", + resolver, + f"{resolver} answered {name} with {', '.join(addresses)}; " + f"the only correct reply is NXDOMAIN", + resolver=resolver, + name=name, + ) + + +def rule_dns_redirected(resolver: str, name: str, answer, private): + """A public name answered with an address that only exists in here. + + `private` is passed in rather than imported so this stays a pure function + over its arguments - the caller owns the one line of address arithmetic. + """ + if answer is None: + return None + _rcode, addresses = answer + inside = [address for address in addresses if private(address)] + if not inside: + return None + return finding( + "here-dns-redirected", + resolver, + f"{resolver} answered {name} with {', '.join(inside)}", + resolver=resolver, + name=name, + address=", ".join(inside), + ) + + +def rule_arp_claims(neighbours, threshold: int): + """One MAC holding several addresses, which ARP allows and nothing prevents.""" + by_mac: dict = {} + for ip, mac in neighbours.items(): + by_mac.setdefault(mac, []).append(ip) + findings = [] + for mac, addresses in sorted(by_mac.items()): + if len(addresses) < threshold: + continue + listed = ", ".join(sorted(addresses, key=ipaddress.ip_address)) + findings.append( + finding( + "here-arp-claims", + mac, + f"{mac} answers for {listed}", + mac=mac, + count=len(addresses), + ) + ) + return findings + + +def rule_client_isolation(others, gateway: str): + """Whether anything but the router answered, reported either way. + + The good result is worth printing. A report that only speaks up about + problems leaves you unable to tell "this network isolates its clients" from + "the check did not run", and on a network you are deciding whether to trust + those are opposite answers. + + `gateway` is passed as '' unless the gateway itself answered ARP. Without + that, an empty table produced "only the gateway answered" - evidence for + something nobody observed - and there are two ordinary ways to get there: a + machine where neither `arp -an` nor `ip neigh` parsed, and a VPN, where the + default route is a tunnel peer that was never on this segment at all. Both + would have been handed the reader as "Nothing. This is the good result." + """ + if not gateway: + # A sweep that did not even find the router proves nothing about + # isolation. Say nothing rather than call the network safe. + return None + if others: + # Sorted numerically, not as text: .102 before .23 reads as a bug, and + # this list is meant to be checked line-by-line against `arp -an`. + listed = ", ".join(sorted(others, key=ipaddress.ip_address)) + return finding( + "here-client-isolation-off", + "network", + f"{len(others)} device(s) answered ARP besides the gateway: {listed}", + count=len(others), + ) + return finding( + "here-client-isolation-on", + "network", + f"only the gateway {gateway} answered ARP on this segment", + ) + + +def rule_own_ports_exposed(address: str, ports): + """Ports of ours bound to the network rather than to loopback.""" + if not ports: + return None + return finding( + "here-own-ports-exposed", + "network", + f"{address} accepted connections on {', '.join(str(p) for p in ports)}", + address=address, + count=len(ports), + # One real port rather than a `PORT` placeholder: the verify line is + # meant to be pasted, and a command you have to edit first is one people + # do not run. + port=ports[0], + ) + + +def here_findings(observed, private) -> list[Finding]: + """Every `here` rule against one `here.observe()` dict. + + Keys are read directly rather than with defaults: `observe()` returns all of + them every time, including the names it asked about and the threshold it + used, so a missing one is a bug and should say so rather than quietly + rendering a finding about the empty string. + + `private` is the address test, passed in for the same reason the rules take + it - this module decides what things mean and owns no arithmetic about what + an address is. + """ + neighbours = observed["neighbours"] or {} + gateway, us = observed["gateway"], observed["us"] + host, invalid = observed["host"], observed["invalid_name"] + others = {ip for ip in neighbours if ip not in (gateway, us)} + + findings = [ + rule_tls_intercepted(host, observed["tls"]), + rule_captive_portal(host, observed["http"]), + # The gateway is only a gateway for this purpose if it answered here. + rule_client_isolation(others, gateway if gateway in neighbours else ""), + rule_own_ports_exposed(us, observed["own_ports"]), + ] + for resolver, answer in observed["nxdomain"].items(): + findings.append(rule_dns_invented(resolver, invalid, answer)) + for resolver, answer in observed["public_name"].items(): + findings.append(rule_dns_redirected(resolver, host, answer, private)) + findings.extend(rule_arp_claims(neighbours, observed["arp_threshold"])) + + return sorted( + (f for f in findings if f), key=lambda f: (SEVERITY_ORDER[f.severity], f.device) + ) + + def audit( devices, gateway=None, banners=None, probes=None, today=None ) -> list[Finding]: diff --git a/netdiff/cli.py b/netdiff/cli.py index 88088a8..cc4dbd9 100644 --- a/netdiff/cli.py +++ b/netdiff/cli.py @@ -1,4 +1,4 @@ -"""Command line interface: scan, audit, glossary, inventory, history.""" +"""Command line interface: scan, audit, here, glossary, inventory, history.""" from __future__ import annotations @@ -13,7 +13,7 @@ from pathlib import Path from . import audit as audit_rules -from . import glossary, mdns, oui, probe, report, store, upnp +from . import glossary, here, mdns, oui, probe, report, store, upnp from .diff import diff, summarise from .scan import ( DEFAULT_PORTS, @@ -57,11 +57,18 @@ def resolve_subnet(args) -> None: Announcing it is not decoration: a scan whose target was inferred has to show its target, or the report is about a network the reader never chose. + + On stderr, because every command that calls this also has a `--json` mode + and a notice on stdout lands inside the document - `netdiff scan --json` + with no subnet emitted something no parser would accept. """ if args.subnet: return args.subnet = local_subnet() - print(f"no subnet given - scanning {args.subnet}, the network this machine is on") + print( + f"no subnet given - scanning {args.subnet}, the network this machine is on", + file=sys.stderr, + ) def cmd_scan(args) -> int: @@ -311,6 +318,49 @@ def cmd_glossary(args) -> int: return 0 +def cmd_here(args) -> int: + """What this network does to you, for a network you did not choose.""" + resolve_subnet(args) + observed = here.observe(args.subnet, ports=resolve_ports(args.ports)) + findings = audit_rules.here_findings(observed, here.is_private) + + if args.json: + print( + json.dumps( + { + "subnet": args.subnet, + "gateway": observed["gateway"], + "resolvers": list(observed["resolvers"]), + "summary": audit_rules.summarise(findings), + "findings": [f.__dict__ for f in findings], + }, + indent=2, + ) + ) + return 0 + + print(f"here: {args.subnet} - {audit_rules.summarise(findings)}") + print( + f"gateway {observed['gateway'] or 'none'}, " + f"resolver(s) {', '.join(observed['resolvers']) or 'none'}\n" + ) + severity = "" + for finding in findings: + if not args.verbose: + print_headline(finding) + continue + if finding.severity != severity: + severity = finding.severity + print(severity.upper()) + print_lesson(finding) + + if not args.verbose: + print() + print("-v adds the evidence each line rests on, why it matters, what to do") + print("about it, and a command you can run yourself to confirm it.") + return 0 + + def cmd_inventory(args) -> int: conn = store.connect(args.db) rows = store.inventory(conn) @@ -442,6 +492,39 @@ def build_parser() -> argparse.ArgumentParser: gloss.add_argument("term", nargs="?", help="a term to explain, e.g. upnp") gloss.set_defaults(func=cmd_glossary) + her = sub.add_parser( + "here", + help="what this network does to you, for one you did not choose", + description=( + "The inverse of `audit`. Not what is on this network - those are not " + "your devices - but what it does to you: whether it reads your " + "encrypted traffic, whether its DNS tells the truth, whether its ARP " + "adds up, and which of your own ports answer from here. Records " + "nothing: a network you are passing through does not belong in your " + "history." + ), + ) + her.add_argument( + "subnet", + nargs="?", + help="CIDR of the network you are on - defaults to working it out", + ) + her.add_argument( + "--ports", + nargs="*", + default=list(DEFAULT_PORTS), + metavar="PORT|SET", + help=f"which of your own ports to check ({', '.join(sorted(PORT_SETS))})", + ) + her.add_argument( + "-v", + "--verbose", + action="store_true", + help="expand every finding into its evidence, why, fix and verify", + ) + her.add_argument("--json", action="store_true") + her.set_defaults(func=cmd_here) + inv = sub.add_parser("inventory", help="every device ever seen") inv.add_argument("--json", action="store_true") inv.set_defaults(func=cmd_inventory) diff --git a/netdiff/here.py b/netdiff/here.py new file mode 100644 index 0000000..46f52fb --- /dev/null +++ b/netdiff/here.py @@ -0,0 +1,352 @@ +"""What this network does to you, rather than what is on it. + +Every other command in netdiff asks *what is here*. On a cafe or hotel network +that is the wrong question and an impolite one: those are not your devices, and +the README's scope section rules out scanning them. So this inverts it. Not +*what is on this network* but *what does this network do to me* - does it read +my TLS, does it lie about DNS, does its ARP add up, and which of my own ports +answer from where I am sitting. Every question is about you or about the +infrastructure you were handed; none is about the stranger at the next table. +The one thing that touches other people's devices is the empty datagram that +provokes ARP, which `scan` already sends and which asks nothing of them. + +The same split as everywhere else: this module opens the sockets and parses the +bytes, `audit.py` decides what any of it means. `observe()` returns a plain dict +and the rules are pure functions over it, so every one of them is tested without +a network - which matters more here than anywhere, because the interesting cases +are hostile networks that are difficult to arrange on purpose. + +Read-only, same as the rest. What this sends, in full: one DNS query for a name +under `.invalid`, one for `example.com`, one HTTP GET to `example.com`, one TLS +handshake to `example.com` that is completed and abandoned, the same empty UDP +datagrams `scan` uses to provoke ARP, and TCP handshakes to your own address. +""" + +from __future__ import annotations + +import ipaddress +import re +import socket +import ssl +import struct +import subprocess +import time +from concurrent.futures import ThreadPoolExecutor + +from .mdns import decode_name +from .probe import DNS_ID, dns_query +from .scan import ( + ARP_SETTLE_SECONDS, + MAX_HOSTS, + local_address, + nudge, + read_arp_table, + scan_ports, +) + +# RFC 2606 reserves `.invalid` so that it can never be registered by anyone. A +# resolver that returns an address for a name underneath it is not resolving, +# it is inventing - there is no lookup that could have produced an answer. +NXDOMAIN_PROBE = "netdiff-does-not-exist.invalid" +NXDOMAIN = 3 + +# RFC 2606 reserves this one too, for documentation and examples, which is why +# the rest of the tool already uses it. It has a stable public address and no +# reason to redirect anybody, so anything other than a plain answer is the +# network talking rather than the host. +PUBLIC_PROBE = "example.com" + +# A MAC holding this many addresses on one segment is worth mentioning. Two is +# ordinary - a router with a second address, a bridged VM, a host with an alias. +# Three starts to look like something answering ARP for addresses it does not +# own, which is what reading someone else's traffic requires. +SUSPICIOUS_CLAIMS = 3 + +# Long enough for a captive portal or an interfering middlebox to answer, short +# enough that a network with no internet at all does not hang the command. Every +# probe runs concurrently, so this is the cost of the slowest one, not the sum. +PROBE_TIMEOUT = 5.0 + + +# ---------------------------------------------------------------- the network's own settings + + +# `netstat -rn -f inet`: "default 192.168.1.1 UGScg en0" +# `ip route`: "default via 192.168.1.1 dev eth0 ..." +_DEFAULT_ROUTE = re.compile(r"^default\s+(?:via\s+)?(\d+\.\d+\.\d+\.\d+)") +# `/etc/resolv.conf` and `scutil --dns` disagree on everything except this. +_NAMESERVER = re.compile(r"nameserver(?:\[\d+\])?\s*:?\s+(\d+\.\d+\.\d+\.\d+)") + + +def default_gateway(runner=subprocess.run) -> str: + """The router this network told us to send everything through, or ''. + + Same `runner` seam as `scan.local_subnet`, for the same reason: the parsing + is the part that can be wrong, and it is only testable if the command is + something a test can hand over. + """ + for cmd in (["netstat", "-rn", "-f", "inet"], ["ip", "route"]): + try: + proc = runner(cmd, capture_output=True, text=True, timeout=10) + except (OSError, subprocess.SubprocessError): + continue + if proc.returncode != 0 or not proc.stdout: + continue + for line in proc.stdout.splitlines(): + match = _DEFAULT_ROUTE.match(line.strip()) + if match: + return match.group(1) + return "" + + +def system_resolvers(runner=subprocess.run, path="/etc/resolv.conf") -> tuple: + """Every resolver this machine is currently configured to ask. + + Which one you use is handed to you by the network over DHCP unless you have + overridden it, so on someone else's wifi this is *their* resolver by + default - which is the whole reason it is worth asking questions of. + + macOS keeps its real configuration in `scutil --dns` and leaves + `/etc/resolv.conf` as a note saying so, so both are read and the results + merged. Order is preserved and duplicates dropped. + """ + found = [] + try: + with open(path, encoding="utf-8", errors="replace") as handle: + text = handle.read() + except OSError: + text = "" + for line in text.splitlines(): + if line.strip().startswith("nameserver"): + match = _NAMESERVER.search(line) + if match: + found.append(match.group(1)) + try: + proc = runner(["scutil", "--dns"], capture_output=True, text=True, timeout=10) + if proc.returncode == 0 and proc.stdout: + found.extend(_NAMESERVER.findall(proc.stdout)) + except (OSError, subprocess.SubprocessError): + pass + return tuple(dict.fromkeys(found)) + + +# ---------------------------------------------------------------- DNS + + +def parse_dns_answers(data: bytes): + """(rcode, addresses) from a reply to our query, or None if it is not one. + + Pure over bytes, like the parsers in `probe.py` and for the same reason: a + resolver that lies is the case worth testing and it is not one you can + arrange on a desk. Names in the answer section may be compressed, so + `mdns.decode_name` walks them - it already refuses pointer loops. + """ + if len(data) < 12: + return None + ident, flags, questions, answers, _, _ = struct.unpack("!HHHHHH", data[:12]) + if ident != DNS_ID or not flags & 0x8000: + return None + offset = 12 + for _ in range(questions): + _, offset = decode_name(data, offset) + offset += 4 + addresses = [] + for _ in range(answers): + if offset >= len(data): + break + _, offset = decode_name(data, offset) + if offset + 10 > len(data): + break + rtype, _rclass, _ttl, rdlen = struct.unpack("!HHIH", data[offset : offset + 10]) + offset += 10 + if offset + rdlen > len(data): + # The record claims to be longer than the packet carrying it, so the + # slice would come back short and a truncated address still parses. + break + if rtype == 1 and rdlen == 4: + addresses.append(".".join(str(b) for b in data[offset : offset + rdlen])) + offset += rdlen + return flags & 0x000F, tuple(addresses) + + +def dns_answers(resolver: str, name: str, timeout: float = PROBE_TIMEOUT): + """Ask one resolver for one name. None if it did not answer at all.""" + try: + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock: + sock.settimeout(timeout) + sock.connect((resolver, 53)) + sock.send(dns_query(name)) + return parse_dns_answers(sock.recv(2048)) + except OSError: + return None + + +def is_private(address: str) -> bool: + """Would an answer of this address mean the network kept you inside it? + + `0.0.0.0` and `127.0.0.1` are the two answers that look private and are not + a redirection: both are how a filtering resolver says *no*, and every + school, guest and pi-hole network uses one of them. Steering you somewhere + and refusing to answer are opposite acts, and calling a block "the network + is collecting your credentials" would be the worst false positive here. + + `is_private` already covers loopback and link-local, so those need no clause + of their own - only the two exclusions do. + """ + try: + parsed = ipaddress.ip_address(address) + except ValueError: + return False + if parsed.is_unspecified or parsed.is_loopback: + return False + return parsed.is_private + + +# ---------------------------------------------------------------- HTTP and TLS + + +_STATUS = re.compile(r"^HTTP/\d\.\d\s+(\d{3})") + + +def parse_http_head(head: str): + """(status, location) from the head of a response, or None if it is not one.""" + lines = head.splitlines() + if not lines: + return None + match = _STATUS.match(lines[0].strip()) + if not match: + return None + location = "" + for line in lines[1:]: + if line.lower().startswith("location:"): + location = line.split(":", 1)[1].strip() + break + return int(match.group(1)), location + + +def http_get(host: str, port: int = 80, timeout: float = PROBE_TIMEOUT): + """Ask a known host for its front page, following nothing. + + Redirects are refused by construction rather than by configuration - there + is no code here that could follow one - which is the same property + `upnp._NoRedirect` buys the other HTTP path in this project. The point is + the redirect itself: it is the answer, not something in the way of one. + """ + request = ( + f"GET / HTTP/1.1\r\nHost: {host}\r\n" + "User-Agent: netdiff\r\nAccept: */*\r\nConnection: close\r\n\r\n" + ) + try: + with socket.create_connection((host, port), timeout) as sock: + sock.settimeout(timeout) + sock.sendall(request.encode("ascii")) + return parse_http_head(sock.recv(2048).decode("utf-8", "replace")) + except OSError: + return None + + +def tls_verified(host: str, port: int = 443, timeout: float = PROBE_TIMEOUT): + """'' if the certificate verified, the reason if it did not, None if unreachable. + + Verification is left **on**, which is the exact opposite of + `probe.tls_certificate` and for the opposite reason. There, a failed check is + the ordinary case and refusing the handshake would mean learning nothing. + Here it is the finding: this is a public host with a certificate from a + public authority, and the only thing between you and it is the network. + + Only a verification failure counts. A connection that never completed means + no internet, and a TLS error that is not about the certificate means a + middlebox or a bad link - neither is evidence of interception, and saying so + anyway would be the confident-sentence-about-nothing failure this project + exists as a reaction to. + + A machine with no trust store is the same failure wearing a disguise, and it + is the one that would have shipped. `unable to get local issuer certificate` + is byte-for-byte what an interception CA produces *and* what a python.org + build whose `Install Certificates.command` was never run produces on every + network it will ever join. An empty store is a fact about this computer, so + the question is unanswerable here rather than answered wrongly. + """ + context = ssl.create_default_context() + if not context.cert_store_stats()["x509_ca"]: + return None + try: + with socket.create_connection((host, port), timeout) as raw: + with context.wrap_socket(raw, server_hostname=host): + return "" + except ssl.SSLCertVerificationError as exc: + return exc.verify_message or str(exc.reason) + except (ssl.SSLError, OSError): + return None + + +# ---------------------------------------------------------------- collection + + +def observe( + subnet: str, ports=(), runner=subprocess.run, timeout=PROBE_TIMEOUT +) -> dict: + """Everything `netdiff here` looks at, gathered into plain data. + + Interpretation happens in `audit.here_findings`, which opens no socket. The + probes run concurrently because each one is a wait rather than work, and on + a network with no internet at all every one of them costs its full timeout - + serially that is a minute of dead air for a command you run while standing + up. + """ + network = ipaddress.ip_network(subnet, strict=False) + if network.num_addresses > MAX_HOSTS: + # Same guard as `scan.discover`, and it has to be repeated rather than + # inherited because this reaches `nudge` by a different route. + raise ValueError( + f"{subnet} holds {network.num_addresses} addresses; " + f"netdiff scans one broadcast segment, up to {MAX_HOSTS}" + ) + us = local_address() + gateway = default_gateway(runner) + resolvers = system_resolvers(runner) or ((gateway,) if gateway else ()) + + # The two addresses in a subnet that are not devices. macOS caches the + # broadcast address as ff:ff:ff:ff:ff:ff after any broadcast traffic, and + # counting that as a neighbour turns "this network isolates its clients" + # into "1 device is reachable from here" - a flipped verdict, not a stray row. + edges = {str(network.network_address), str(network.broadcast_address)} + + def sweep(): + nudge([str(host) for host in network.hosts()]) + time.sleep(ARP_SETTLE_SECONDS) + table = read_arp_table() + return { + ip: mac + for ip, mac in table.items() + if ip not in edges and ipaddress.ip_address(ip) in network + } + + jobs = { + "neighbours": sweep, + "own_ports": lambda: scan_ports(us, ports), + "tls": lambda: tls_verified(PUBLIC_PROBE, timeout=timeout), + "http": lambda: http_get(PUBLIC_PROBE, timeout=timeout), + "nxdomain": lambda: { + r: dns_answers(r, NXDOMAIN_PROBE, timeout=timeout) for r in resolvers + }, + "public_name": lambda: { + r: dns_answers(r, PUBLIC_PROBE, timeout=timeout) for r in resolvers + }, + } + with ThreadPoolExecutor(max_workers=len(jobs)) as pool: + values = dict(zip(jobs, pool.map(lambda job: job(), jobs.values()))) + + # What was asked travels with the answers. The rules render the name and the + # threshold into their findings, and a report that says "a name under + # .invalid resolved" without naming which one is not evidence of anything. + return dict( + values, + us=us, + gateway=gateway, + resolvers=resolvers, + subnet=subnet, + host=PUBLIC_PROBE, + invalid_name=NXDOMAIN_PROBE, + arp_threshold=SUSPICIOUS_CLAIMS, + ) diff --git a/netdiff/probe.py b/netdiff/probe.py index d7b809e..6233fe7 100644 --- a/netdiff/probe.py +++ b/netdiff/probe.py @@ -219,13 +219,14 @@ def smb_dialect(ip: str, port: int = 445, timeout: float = 3.0) -> str: DNS_PROBE_NAME = "example.com" # Fixed rather than random: it only has to match our own reply on a socket we # connected ourselves. Randomising it would be cargo-culted cache-poisoning -# defence for a query whose answer is thrown away. -_DNS_ID = 0x1D1F +# defence for a query whose answer is thrown away. Public because `here.py` +# builds its queries with `dns_query` and has to recognise the replies. +DNS_ID = 0x1D1F def dns_query(name: str = DNS_PROBE_NAME) -> bytes: """One A query with the recursion-desired bit set.""" - header = struct.pack("!HHHHHH", _DNS_ID, 0x0100, 1, 0, 0, 0) + header = struct.pack("!HHHHHH", DNS_ID, 0x0100, 1, 0, 0, 0) return header + encode_name(name) + struct.pack("!HH", 1, 1) @@ -234,7 +235,7 @@ def parse_dns_reply(data: bytes) -> str: if len(data) < 12: return "" ident, flags, _, answers, _, _ = struct.unpack("!HHHHHH", data[:12]) - if ident != _DNS_ID or not flags & 0x8000: + if ident != DNS_ID or not flags & 0x8000: return "" if flags & 0x000F or not flags & 0x0080 or not answers: # rcode must be 0, the recursion-available bit must be set, and there diff --git a/tests/test_audit.py b/tests/test_audit.py index c701799..3b2001c 100644 --- a/tests/test_audit.py +++ b/tests/test_audit.py @@ -421,7 +421,15 @@ def everything(): def test_every_rule_id_used_by_a_rule_exists_in_the_teaching_table(): - assert {f.rule for f in everything()} == set(RULES) + """Both directions: no rule fires without teaching text, none is orphaned. + + `RULES` is one table for two commands. The `here-` half belongs to `netdiff + here`, which `audit()` never fires, and `test_here.py` holds the same + assertion for those. + """ + assert {f.rule for f in everything()} == { + rule for rule in RULES if not rule.startswith("here-") + } def test_dangling_verify_pings_a_bare_address_not_an_ip_colon_port(): diff --git a/tests/test_here.py b/tests/test_here.py new file mode 100644 index 0000000..912ee8b --- /dev/null +++ b/tests/test_here.py @@ -0,0 +1,714 @@ +"""What this network does to you - the checks behind `netdiff here`. + +Nothing here touches a real network. The interesting cases are all hostile +networks - a resolver that invents answers, a box terminating your TLS, a MAC +answering for addresses it does not own - and those are exactly the ones you +cannot arrange on a desk, which is why the parsers are pure over bytes and the +rules are pure over a dict. + +Half of this file asserts that something is **not** reported. `here` runs on +networks belonging to other people and hands its reader a verdict about whether +to trust the place they are sitting; the failure mode that matters is not a +missed finding, it is a confident accusation against a network that was fine. + +The DNS packets are assembled by local helpers rather than by anything in +`netdiff`, for the same reason `test_mdns.py` does it: a decoder tested against +its own encoder agrees with itself however wrong both are. +""" + +import contextlib +import http.server +import socket +import ssl +import struct +import threading + +import pytest + +from netdiff import audit, cli, here +from netdiff.probe import DNS_ID + +# --- DNS packet builders ----------------------------------------------------- + + +def name(text): + """A DNS name: each label length-prefixed, terminated by a zero byte.""" + out = b"" + for label in text.rstrip(".").split("."): + out += bytes([len(label)]) + label.encode() + return out + b"\x00" + + +def reply(question, addresses=(), rcode=0, ident=DNS_ID, recursion=True): + """A reply to our own A query, with an answer per address. + + The recursion-desired bit is echoed back from the query, as a real server + does - so a captured reply and one of these have the same flags word. + """ + flags = 0x8000 | 0x0100 | (0x0080 if recursion else 0) | rcode + header = struct.pack("!HHHHHH", ident, flags, 1, len(addresses), 0, 0) + body = name(question) + struct.pack("!HH", 1, 1) + for address in addresses: + body += ( + b"\xc0\x0c" # a pointer back to the question's name, as real servers send + + struct.pack("!HHIH", 1, 1, 60, 4) + + bytes(int(o) for o in address.split(".")) + ) + return header + body + + +def test_the_wire_format_is_what_we_think_it_is(): + """Pins the builders against bytes, so everything below rests on those. + + Without this the rest of the file only proves the helpers and the parser + agree with each other, which they would even if both were wrong. + """ + data = reply("a.invalid", ["10.0.0.1"]) + assert data[:2] == b"\x1d\x1f", "our query id, echoed back" + assert data[2:4] == b"\x81\x80", "QR and recursion-available set, rcode 0" + assert data[12:] == ( + b"\x01a\x07invalid\x00" # the question's name + b"\x00\x01\x00\x01" # type A, class IN + b"\xc0\x0c" # answer name: pointer to offset 12 + b"\x00\x01\x00\x01\x00\x00\x00\x3c\x00\x04" # A, IN, ttl 60, 4 bytes + b"\x0a\x00\x00\x01" # 10.0.0.1 + ) + + +# --- parsing a reply --------------------------------------------------------- + + +def test_an_answer_yields_its_rcode_and_addresses(): + assert here.parse_dns_answers(reply("example.com", ["93.184.216.34"])) == ( + 0, + ("93.184.216.34",), + ) + + +def test_nxdomain_yields_the_code_and_nothing_else(): + assert here.parse_dns_answers(reply("a.invalid", rcode=3)) == (3, ()) + + +def test_a_reply_to_somebody_elses_query_is_not_ours(): + """A stray datagram on the socket must not be read as our answer.""" + assert ( + here.parse_dns_answers(reply("example.com", ["1.2.3.4"], ident=0x4242)) is None + ) + + +def test_a_question_is_not_an_answer(): + """The QR bit is the only thing separating them.""" + question = struct.pack("!HHHHHH", DNS_ID, 0x0100, 1, 0, 0, 0) + name("example.com") + assert here.parse_dns_answers(question) is None + + +def test_a_record_longer_than_its_packet_is_dropped_not_truncated(): + data = bytearray(reply("example.com", ["93.184.216.34"])) + data[-6:-4] = struct.pack("!H", 64) # rdlen now claims 64 bytes, 4 remain + assert here.parse_dns_answers(bytes(data)) == (0, ()) + + +def test_a_reply_carrying_no_answer_section_is_still_read(): + assert here.parse_dns_answers(reply("example.com")) == (0, ()) + + +@pytest.mark.parametrize("data", [b"", b"\x1d\x1f", b"\xff" * 11]) +def test_bytes_that_are_not_a_reply_produce_nothing(data): + assert here.parse_dns_answers(data) is None + + +# --- what the network told us about itself ----------------------------------- + +NETSTAT = """Routing tables + +Internet: +Destination Gateway Flags Netif Expire +default 192.168.1.1 UGScg en0 +127 127.0.0.1 UCS lo0 +""" + +IP_ROUTE = """default via 10.0.0.1 dev eth0 proto dhcp src 10.0.0.55 metric 100 +10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.55 +""" + +SCUTIL = """DNS configuration + +resolver #1 + search domain[0] : lan + nameserver[0] : 192.168.1.1 + nameserver[1] : 8.8.8.8 + flags : Request A records +""" + + +def runner_for(output, ok=("netstat", "ip", "scutil")): + """Stub `subprocess.run`: the listed commands succeed with `output`.""" + + class Result: + def __init__(self, returncode, stdout): + self.returncode, self.stdout = returncode, stdout + + def run(cmd, **kwargs): + return Result(0, output) if cmd[0] in ok else Result(1, "") + + return run + + +def test_the_gateway_is_read_from_bsd_and_linux_alike(): + assert here.default_gateway(runner_for(NETSTAT)) == "192.168.1.1" + assert here.default_gateway(runner_for(IP_ROUTE)) == "10.0.0.1" + + +def test_no_default_route_is_no_gateway_rather_than_a_guess(): + assert here.default_gateway(runner_for("Destination Gateway Flags")) == "" + assert here.default_gateway(runner_for("", ok=())) == "" + + +def test_resolvers_come_from_resolv_conf_and_scutil_together(tmp_path): + """macOS keeps the real configuration in scutil and leaves a note in the file.""" + conf = tmp_path / "resolv.conf" + conf.write_text("# macOS Notice\nnameserver 1.1.1.1\n") + found = here.system_resolvers(runner_for(SCUTIL), path=str(conf)) + assert found == ("1.1.1.1", "192.168.1.1", "8.8.8.8") + + +def test_a_resolver_named_twice_is_one_resolver(tmp_path): + conf = tmp_path / "resolv.conf" + conf.write_text("nameserver 192.168.1.1\n") + assert here.system_resolvers(runner_for(SCUTIL), path=str(conf)) == ( + "192.168.1.1", + "8.8.8.8", + ) + + +def test_no_resolv_conf_and_no_scutil_is_empty_not_an_error(tmp_path): + assert ( + here.system_resolvers(runner_for("", ok=()), path=str(tmp_path / "nope")) == () + ) + + +# --- HTTP -------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "head, expect", + [ + ("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n", (200, "")), + ( + "HTTP/1.1 302 Found\r\nLocation: http://10.0.0.1/login\r\n\r\n", + (302, "http://10.0.0.1/login"), + ), + ("HTTP/1.0 511 Network Authentication Required\r\n\r\n", (511, "")), + # Header names are case-insensitive and middleboxes are careless with them. + ("HTTP/1.1 307 x\r\nLOCATION: /portal\r\n\r\n", (307, "/portal")), + ], +) +def test_a_status_line_and_its_location_are_read(head, expect): + assert here.parse_http_head(head) == expect + + +@pytest.mark.parametrize("head", ["", "not http at all\r\n\r\n", "\r\n"]) +def test_something_that_is_not_a_response_produces_nothing(head): + assert here.parse_http_head(head) is None + + +class _Redirector(http.server.BaseHTTPRequestHandler): + def do_GET(self): + self.send_response(302) + self.send_header("Location", "http://10.0.0.1/login") + self.end_headers() + + def log_message(self, *args): + pass + + +@contextlib.contextmanager +def _serving(handler): + server = http.server.HTTPServer(("127.0.0.1", 0), handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + try: + yield server.server_address[1] + finally: + server.shutdown() + + +def test_a_redirect_is_read_off_a_real_socket_and_not_followed(): + """End to end over loopback: the request we build, the reply we parse. + + Not following it is the whole point - the redirect is the answer, not + something in the way of one. + """ + with _serving(_Redirector) as port: + assert here.http_get("127.0.0.1", port=port) == (302, "http://10.0.0.1/login") + + +def test_a_host_that_does_not_answer_is_not_a_portal(): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + closed = sock.getsockname()[1] + assert here.http_get("127.0.0.1", port=closed, timeout=1.0) is None + + +# --- TLS --------------------------------------------------------------------- +# +# Only the mapping from exception to verdict is pinned here. Standing up a real +# TLS server would mean committing a private key to a public repository, and the +# thing that can actually be wrong is this ordering: SSLCertVerificationError is +# a subclass of SSLError, which is a subclass of OSError, so catching them in +# the wrong order silently turns "intercepted" into "unreachable". + + +def _raising(exc): + def create_connection(*args, **kwargs): + raise exc + + return create_connection + + +def test_a_certificate_that_does_not_verify_is_reported(monkeypatch): + error = ssl.SSLCertVerificationError("bad") + error.verify_message = "self signed certificate in certificate chain" + monkeypatch.setattr(here.socket, "create_connection", _raising(error)) + assert ( + here.tls_verified("example.com") + == "self signed certificate in certificate chain" + ) + + +def test_a_network_we_cannot_reach_is_not_an_interception(monkeypatch): + """No internet is not a man in the middle, and must not be printed as one.""" + monkeypatch.setattr(here.socket, "create_connection", _raising(OSError("no route"))) + assert here.tls_verified("example.com") is None + + +def test_a_tls_error_that_is_not_about_the_certificate_says_nothing(monkeypatch): + monkeypatch.setattr(here.socket, "create_connection", _raising(ssl.SSLError("eof"))) + assert here.tls_verified("example.com") is None + + +def test_a_machine_with_no_trust_store_accuses_nobody(monkeypatch): + """The false positive that would have shipped, and the loudest one possible. + + A python.org build whose `Install Certificates.command` was never run has an + empty store, and every handshake it makes fails with `unable to get local + issuer certificate` - byte for byte what an interception CA produces. Left + alone, this reports `high: this network is reading your encrypted traffic` + on every network that machine will ever join. + """ + monkeypatch.setattr( + here.ssl, + "create_default_context", + lambda: ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT), + ) + monkeypatch.setattr( + here.socket, + "create_connection", + _raising(AssertionError("must not reach the network")), + ) + assert here.tls_verified("example.com") is None + + +# --- the rules --------------------------------------------------------------- +# +# Pure functions over what `observe()` gathered. Each one has its negative case +# beside it, because on a network that is fine every one of these must be silent. + + +def test_a_failed_verification_is_an_interception(): + found = audit.rule_tls_intercepted("example.com", "self signed certificate") + assert found.severity == "high" + assert "self signed certificate" in found.evidence + + +@pytest.mark.parametrize("reason", ["", None]) +def test_a_clean_or_unreachable_handshake_is_not(reason): + assert audit.rule_tls_intercepted("example.com", reason) is None + + +@pytest.mark.parametrize("status", [302, 307, 511]) +def test_anything_but_an_answer_from_the_host_is_a_portal(status): + found = audit.rule_captive_portal("example.com", (status, "http://10.0.0.1/")) + assert found and "10.0.0.1" in found.evidence + + +@pytest.mark.parametrize("response", [(200, ""), (404, ""), (500, ""), None]) +def test_a_site_answering_for_itself_is_not_a_portal(response): + assert audit.rule_captive_portal("example.com", response) is None + + +def test_an_address_for_a_reserved_name_is_invention(): + found = audit.rule_dns_invented("10.0.0.1", "a.invalid", (0, ("10.0.0.53",))) + assert found and "10.0.0.53" in found.evidence + + +@pytest.mark.parametrize("answer", [(3, ()), (0, ()), (2, ()), None]) +def test_a_resolver_that_admits_the_name_does_not_exist_is_correct(answer): + """NXDOMAIN, an empty answer, SERVFAIL and silence are all honest replies.""" + assert audit.rule_dns_invented("10.0.0.1", "a.invalid", answer) is None + + +def test_a_public_name_answered_with_a_private_address_is_a_redirect(): + found = audit.rule_dns_redirected( + "10.0.0.1", "example.com", (0, ("192.168.1.1",)), here.is_private + ) + assert found and "192.168.1.1" in found.evidence + + +@pytest.mark.parametrize( + "answer", + [(0, ("93.184.216.34",)), (0, ("23.192.228.80", "23.215.0.136")), (3, ()), None], +) +def test_a_public_address_or_no_answer_is_not_a_redirect(answer): + """Two different public addresses for one name is a CDN, not a hijack.""" + assert ( + audit.rule_dns_redirected("1.1.1.1", "example.com", answer, here.is_private) + is None + ) + + +@pytest.mark.parametrize("blocked", ["0.0.0.0", "127.0.0.1"]) +def test_a_resolver_refusing_to_answer_is_not_a_resolver_steering_you(blocked): + """Both look private to `ipaddress` and neither is a redirection. + + They are how a filtering resolver says no - school wifi, guest wifi, a + pi-hole. Refusing to answer and steering you somewhere are opposite acts, + and this rule tells its reader the network may be collecting credentials. + """ + assert not here.is_private(blocked) + assert ( + audit.rule_dns_redirected( + "10.0.0.1", "example.com", (0, (blocked,)), here.is_private + ) + is None + ) + + +def test_one_mac_answering_for_many_addresses_is_reported(): + neighbours = { + "10.0.0.1": "aa:bb:cc:00:00:01", + "10.0.0.2": "aa:bb:cc:00:00:01", + "10.0.0.3": "aa:bb:cc:00:00:01", + "10.0.0.9": "aa:bb:cc:00:00:02", + } + found = audit.rule_arp_claims(neighbours, threshold=3) + assert len(found) == 1 + assert found[0].device == "aa:bb:cc:00:00:01" + assert "10.0.0.1, 10.0.0.2, 10.0.0.3" in found[0].evidence + + +def test_a_router_with_a_second_address_is_not_reported(): + """Two is ordinary - an alias, a bridge, a router holding a second address.""" + neighbours = {"10.0.0.1": "aa:bb:cc:00:00:01", "10.0.0.2": "aa:bb:cc:00:00:01"} + assert audit.rule_arp_claims(neighbours, threshold=3) == [] + + +def test_other_devices_answering_means_isolation_is_off(): + found = audit.rule_client_isolation({"10.0.0.7", "10.0.0.8"}, "10.0.0.1") + assert found.rule == "here-client-isolation-off" + assert found.severity == "info", "reachable neighbours are normal, not a problem" + + +def test_only_the_gateway_answering_is_reported_as_the_good_result(): + """The good news has to be printed, or it cannot be told from a check that + did not run - and on a network you are deciding whether to trust, those are + opposite answers.""" + found = audit.rule_client_isolation(set(), "10.0.0.1") + assert found.rule == "here-client-isolation-on" + + +def test_isolation_says_nothing_when_we_have_no_gateway(): + """No default route means the sweep proves nothing either way.""" + assert audit.rule_client_isolation(set(), "") is None + + +@pytest.mark.parametrize( + "neighbours", + [ + {}, # neither `arp -an` nor `ip neigh` parsed + {"10.0.0.7": "aa:bb:cc:00:00:07"}, # a VPN: the route points off-segment + ], +) +def test_a_gateway_that_did_not_answer_cannot_be_evidence_that_it_did(neighbours): + """`only the gateway answered` must not be printed about an empty table. + + Two ordinary ways to get there: a machine where the ARP command did not + parse, and a VPN, where the default route is a tunnel peer that was never on + this segment. Both used to hand the reader "Nothing. This is the good + result." about a check that had not run. + """ + findings = audit.here_findings( + observation(neighbours=neighbours, gateway="10.8.0.1"), here.is_private + ) + assert not [f for f in findings if f.rule.startswith("here-client-isolation")] + + +def test_our_own_open_ports_are_reported_with_the_address_they_answered_on(): + found = audit.rule_own_ports_exposed("10.0.0.55", (22, 445)) + assert found and "10.0.0.55" in found.evidence and "22, 445" in found.evidence + + +def test_a_machine_with_nothing_listening_is_not_a_finding(): + assert audit.rule_own_ports_exposed("10.0.0.55", ()) is None + + +# --- assembling them --------------------------------------------------------- + + +def observation(**overrides): + """A clean network: nothing here should produce a single finding.""" + return dict( + { + "subnet": "10.0.0.0/24", + "us": "10.0.0.55", + "gateway": "10.0.0.1", + "resolvers": ("10.0.0.1",), + "neighbours": {"10.0.0.1": "aa:bb:cc:00:00:01"}, + "own_ports": (), + "tls": "", + "http": (200, ""), + "nxdomain": {"10.0.0.1": (3, ())}, + "public_name": {"10.0.0.1": (0, ("93.184.216.34",))}, + "host": "example.com", + "invalid_name": "a.invalid", + "arp_threshold": 3, + }, + **overrides, + ) + + +HERE_RULES = {rule for rule in audit.RULES if rule.startswith("here-")} + + +def test_a_network_that_behaves_produces_only_the_isolation_note(): + findings = audit.here_findings(observation(), here.is_private) + assert [f.rule for f in findings] == ["here-client-isolation-on"] + + +def test_every_here_rule_in_the_teaching_table_can_actually_fire(): + """The other half of `test_audit.py`'s assertion, for this command's rules. + + A rule id with teaching text and no code path is a lesson nobody can ever + be shown, and nothing else in the suite would notice. + """ + fired = {f.rule for f in audit.here_findings(observation(), here.is_private)} + fired |= {f.rule for f in audit.here_findings(hostile(), here.is_private)} + assert fired == HERE_RULES + + +def hostile(**overrides): + """One network doing every single thing this command looks for.""" + return observation( + tls="self signed certificate in certificate chain", + http=(302, "http://10.0.0.1/login"), + nxdomain={"10.0.0.1": (0, ("10.0.0.53",))}, + public_name={"10.0.0.1": (0, ("10.0.0.53",))}, + neighbours={ + "10.0.0.1": "aa:bb:cc:00:00:01", + "10.0.0.2": "aa:bb:cc:00:00:01", + "10.0.0.3": "aa:bb:cc:00:00:01", + }, + own_ports=(22,), + **overrides, + ) + + +def test_a_hostile_network_produces_all_of_it_worst_first(): + findings = audit.here_findings( + observation( + tls="self signed certificate in certificate chain", + http=(302, "http://10.0.0.1/login"), + nxdomain={"10.0.0.1": (0, ("10.0.0.53",))}, + public_name={"10.0.0.1": (0, ("10.0.0.53",))}, + neighbours={ + "10.0.0.1": "aa:bb:cc:00:00:01", + "10.0.0.2": "aa:bb:cc:00:00:01", + "10.0.0.3": "aa:bb:cc:00:00:01", + }, + own_ports=(22,), + ), + here.is_private, + ) + assert findings[0].rule == "here-tls-intercepted" + assert [f.severity for f in findings] == sorted( + (f.severity for f in findings), key=audit.SEVERITY_ORDER.get + ) + assert {f.rule for f in findings} >= { + "here-tls-intercepted", + "here-captive-portal", + "here-dns-invented", + "here-dns-redirected", + "here-arp-claims", + "here-own-ports-exposed", + } + + +def test_every_resolver_is_asked_and_each_one_answers_for_itself(): + findings = audit.here_findings( + observation( + resolvers=("10.0.0.1", "10.0.0.2"), + nxdomain={"10.0.0.1": (0, ("10.0.0.53",)), "10.0.0.2": (3, ())}, + ), + here.is_private, + ) + invented = [f for f in findings if f.rule == "here-dns-invented"] + assert [f.device for f in invented] == ["10.0.0.1"], "only the one that lied" + + +# --- gathering --------------------------------------------------------------- + + +def test_observe_returns_exactly_the_keys_the_rules_read(monkeypatch): + """The seam between the two halves, and nothing else was testing it. + + `here_findings` reads its keys with no defaults, on purpose. But the only + dict it was ever run against was this file's hand-written fixture, which + duplicates the key set - so renaming a key in `observe` left every test + green and the command raising KeyError on its first real run. This drives + the real `observe` with stubbed collectors and feeds the result straight in. + + It also pins the `zip(jobs, pool.map(...))` pairing: each stub returns a + distinguishable marker, so a result landing under the wrong key shows up. + """ + monkeypatch.setattr(here, "local_address", lambda: "10.0.0.55") + monkeypatch.setattr(here, "default_gateway", lambda runner: "10.0.0.1") + monkeypatch.setattr(here, "system_resolvers", lambda runner: ("10.0.0.1",)) + monkeypatch.setattr(here, "nudge", lambda hosts: None) + monkeypatch.setattr(here.time, "sleep", lambda seconds: None) + monkeypatch.setattr( + here, + "read_arp_table", + lambda: {"10.0.0.1": "aa:bb:cc:00:00:01", "10.0.0.255": "ff:ff:ff:ff:ff:ff"}, + ) + monkeypatch.setattr(here, "scan_ports", lambda ip, ports: (22,)) + monkeypatch.setattr(here, "tls_verified", lambda host, timeout: "") + monkeypatch.setattr(here, "http_get", lambda host, timeout: (200, "")) + monkeypatch.setattr( + here, + "dns_answers", + lambda resolver, name, timeout: ( + (3, ()) if name.endswith(".invalid") else (0, ("93.184.216.34",)) + ), + ) + + observed = here.observe("10.0.0.0/24", ports=(22,)) + + assert observed["us"] == "10.0.0.55" + assert observed["own_ports"] == (22,) + assert observed["tls"] == "" and observed["http"] == (200, "") + assert observed["nxdomain"] == {"10.0.0.1": (3, ())} + assert observed["public_name"] == {"10.0.0.1": (0, ("93.184.216.34",))} + assert observed["host"] == here.PUBLIC_PROBE + assert observed["invalid_name"] == here.NXDOMAIN_PROBE + assert observed["arp_threshold"] == here.SUSPICIOUS_CLAIMS + assert observed["neighbours"] == {"10.0.0.1": "aa:bb:cc:00:00:01"}, ( + "the broadcast address is not a device, and counting it as one would " + "turn an isolated network into one with a neighbour" + ) + # The real contract: this dict is what the rules consume. + assert [f.rule for f in audit.here_findings(observed, here.is_private)] == [ + "here-own-ports-exposed", + "here-client-isolation-on", + ] + + +def test_a_subnet_too_big_to_be_one_segment_is_refused(): + """The same guard `scan.discover` has - `here 10.0.0.0/8` would materialise + 16.7M strings before a packet moved. `cli.main` turns this into exit 2.""" + with pytest.raises(ValueError, match="one broadcast segment"): + here.observe("10.0.0.0/8") + + +# --- the command ------------------------------------------------------------- + + +@pytest.fixture +def here_output(monkeypatch, capsys): + """Run `netdiff here` against a canned observation, return its stdout.""" + + def run(*flags, **overrides): + monkeypatch.setattr(here, "observe", lambda *a, **k: observation(**overrides)) + assert cli.main(["here", "10.0.0.0/24", *flags]) == 0 + return capsys.readouterr().out + + return run + + +def test_the_default_view_is_one_line_per_finding(here_output): + out = here_output(own_ports=(22, 445)) + assert " medium " in out and " info " in out + assert " evidence " not in out, "the lesson is something you ask for" + assert " fix " not in out + + +def test_verbose_carries_every_field(here_output): + out = here_output("-v", own_ports=(22, 445)) + for field in ("evidence", "why", "fix", "verify"): + assert field in out + + +def test_both_views_report_the_same_findings(here_output): + """Brevity may drop detail. It may never drop a finding.""" + assert here_output(own_ports=(22,)).count("here-") == here_output( + "-v", own_ports=(22,) + ).count("here-") + brief = here_output(own_ports=(22,)) + assert "1 medium, 1 info" in brief + + +def test_the_gateway_and_resolvers_are_named(here_output): + """A verdict about a network has to say which network it is about.""" + out = here_output() + assert "10.0.0.1" in out and "10.0.0.0/24" in out + + +def test_here_records_nothing(monkeypatch, capsys, tmp_path): + """A network you are passing through does not belong in your history. + + Recording one would put strangers' MAC addresses in `netdiff inventory` and + make the next scan at home report a wall of appeared/vanished. + """ + db = tmp_path / "history.db" + monkeypatch.setattr(here, "observe", lambda *a, **k: observation()) + assert cli.main(["--db", str(db), "here", "10.0.0.0/24"]) == 0 + capsys.readouterr() + assert not db.exists(), "here opened the history database" + + +def test_json_carries_every_field_and_what_was_asked(monkeypatch, capsys): + import json + + monkeypatch.setattr(here, "observe", lambda *a, **k: observation(own_ports=(22,))) + assert cli.main(["here", "10.0.0.0/24", "--json"]) == 0 + payload = json.loads(capsys.readouterr().out) + assert payload["gateway"] == "10.0.0.1" + assert payload["resolvers"] == ["10.0.0.1"] + assert {f["rule"] for f in payload["findings"]} == { + "here-own-ports-exposed", + "here-client-isolation-on", + } + + +def test_an_inferred_subnet_is_announced_without_landing_in_the_json( + monkeypatch, capsys +): + """The notice is a diagnostic, not data, so it goes to stderr. + + It has to be said - a report whose target was guessed must show its target - + but on stdout it lands inside the document, and `--json` with no subnet + emitted something no parser would accept. Shared by scan and audit too. + """ + import json + + monkeypatch.setattr(here, "observe", lambda *a, **k: observation()) + monkeypatch.setattr(cli, "local_subnet", lambda: "10.0.0.0/24") + assert cli.main(["here", "--json"]) == 0 + captured = capsys.readouterr() + assert json.loads(captured.out)["subnet"] == "10.0.0.0/24" + assert "no subnet given" in captured.err + + +def test_every_here_rule_can_be_explained_without_scanning(capsys): + """`--explain` reads the same table these findings are built from.""" + for rule in [r for r in audit.RULES if r.startswith("here-")]: + assert cli.main(["audit", "--explain", rule]) == 0 + assert "{" not in capsys.readouterr().out, f"{rule} leaked a placeholder" diff --git a/tests/test_scan_and_store.py b/tests/test_scan_and_store.py index 02e0a8f..29876ef 100644 --- a/tests/test_scan_and_store.py +++ b/tests/test_scan_and_store.py @@ -175,7 +175,9 @@ def test_an_omitted_subnet_is_filled_in_and_announced(monkeypatch, capsys): args = type("Args", (), {"subnet": None})() cli.resolve_subnet(args) assert args.subnet == "10.1.2.0/24" - assert "10.1.2.0/24" in capsys.readouterr().out, "a scan must show its target" + captured = capsys.readouterr() + assert "10.1.2.0/24" in captured.err, "a scan must show its target" + assert captured.out == "", "and must not put the notice inside --json output" def test_an_explicit_subnet_is_left_alone(monkeypatch, capsys):