diff --git a/CHANGELOG.md b/CHANGELOG.md index b82663e..978c670 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,12 +4,31 @@ ### Added +- Release hardening documentation with an install/run guide and release + checklist covering source cleanliness, audits, desktop validation, + PyInstaller smoke, security boundaries, and release artifacts. +- Performance Pipeline foundation: bounded packet intake, WebSocket Event + Aggregator, Batch Persistence / Storage Backpressure, Flow-aware Worker + Pool, Live Ring Buffer, and CI-safe benchmark/soak validation. +- Service Attribution / Destination Intelligence with metadata-only + confidence scoring and conservative handling for encrypted/CDN traffic. +- Incident Correlation Engine with redacted incident timelines, severity, + confidence, investigation guidance, and read-only Markdown summary export. - Protocol Intelligence Expansion for TCP, DNS, HTTP, and visible TLS metadata. - Saved Display Filters with built-in recipes and safe field suggestions. - Protocol Statistics and safe Packet Search in the Inspect workspace. - Expanded Expert Info and redacted protocol/inspection summary reports. - Offline PCAP parity for protocol intelligence summaries. +### Changed + +- Replaced remaining naive UTC timestamp call sites with timezone-aware UTC + timestamps. +- Refreshed frontend development lockfile dependencies with compatible + patch/minor updates and cleaned desktop Electron audit findings. +- Reduced expected-error test log noise while preserving production logging and + failure-path coverage. + ## v0.2.0 - Agent & Fleet Monitoring Release - 2026-06-05 ### Added diff --git a/README.md b/README.md index 6da04d4..fc2df55 100644 --- a/README.md +++ b/README.md @@ -263,6 +263,8 @@ and surfaced in Ops Snapshot metrics. - [Deep Packet Inspection](docs/DEEP_PACKET_INSPECTION.md) - [Agent Operational QA Checklist](docs/AGENT_QA_CHECKLIST.md) - [Deployment Overview](docs/DEPLOYMENT_OVERVIEW.md) +- [Install And Run Guide](docs/INSTALL.md) +- [Release Checklist](docs/RELEASE_CHECKLIST.md) - [Release QA Checklist](docs/RELEASE_QA_CHECKLIST.md) - [v0.2.0 Release Notes](docs/RELEASE_NOTES_v0.2.0.md) - [Remote Sensor](docs/REMOTE_SENSOR.md) @@ -307,7 +309,7 @@ Prerequisites: Install everything on Windows: ```powershell -cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro" +cd .\Netbotpro powershell -ExecutionPolicy Bypass -File .\scripts\dev\setup.ps1 ``` @@ -336,7 +338,7 @@ Copy `.env.example` to `.env` for local experiments. The default secure path is Start the local web stack: ```powershell -cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro" +cd .\Netbotpro powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-local.ps1 ``` @@ -355,14 +357,14 @@ http://127.0.0.1:5173 Start the Electron desktop shell after dependencies are installed: ```powershell -cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro\desktop\electron" +cd .\Netbotpro\desktop\electron npm run dev ``` Start a remote sensor on a server you own or administer: ```powershell -cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro" +cd .\Netbotpro powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-sensor.ps1 -BindHost 0.0.0.0 -Port 8765 -AllowedOrigins "http://YOUR_DASHBOARD_HOST:5173" ``` @@ -524,6 +526,8 @@ GitHub Actions builds CI on push and pull request. Desktop release artifacts are - [Agent mode](docs/AGENT_MODE.md) - [Agent QA checklist](docs/AGENT_QA_CHECKLIST.md) - [Deployment overview](docs/DEPLOYMENT_OVERVIEW.md) +- [Install and run guide](docs/INSTALL.md) +- [Release checklist](docs/RELEASE_CHECKLIST.md) - [Release QA checklist](docs/RELEASE_QA_CHECKLIST.md) - [Desktop shell](docs/DESKTOP_SHELL.md) - [Web migration notes](docs/WEB_MIGRATION.md) diff --git a/docs/INSTALL.md b/docs/INSTALL.md new file mode 100644 index 0000000..547eb03 --- /dev/null +++ b/docs/INSTALL.md @@ -0,0 +1,142 @@ +# NetBotPro Install And Run Guide + +NetBotPro is a local-first defensive network monitoring and analysis tool. +Only run live capture, Remote Sensor mode, or Agent telemetry on systems and +networks you own or have explicit permission to administer. + +## Requirements + +- Python 3.13 is recommended for source and CI parity. +- Node.js 20 or newer is recommended. Node 20.17+ avoids newer Electron + tooling engine warnings. +- Npcap on Windows for live packet capture. +- Administrator privileges on Windows, or root/capture capabilities on Linux, + when live packet capture is required. +- Git and PowerShell 7 or Windows PowerShell for the provided scripts. + +Offline PCAP analysis, report review, docs, and most UI development paths do +not require packet-capture privileges. + +## Supported OS Notes + +- Windows is the strongest validated desktop target. +- Linux supports source runs, backend services, and staged desktop packaging. +- macOS runs backend/frontend CI paths, but desktop release validation is not + currently the strongest release target. + +Live capture behavior depends on OS drivers, interface visibility, privileges, +and what traffic is observable from the selected host/interface. + +## Windows Notes + +Install Npcap before using live capture. Reopen the terminal after installing +Npcap, and run the desktop app or backend terminal as Administrator when live +capture or firewall actions are needed. + +Antivirus or endpoint security products can flag packet capture tools, +PyInstaller bundles, or unsigned desktop artifacts. Review alerts carefully and +allow only builds you created from trusted source. + +## Backend From Source + +```powershell +git clone https://github.com/XploitGost/Netbotpro.git +cd Netbotpro +python -m venv .venv +.\.venv\Scripts\python.exe -m pip install --upgrade pip +.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt +.\.venv\Scripts\python.exe -m backend.app.desktop_entry +``` + +For a direct API server run: + +```powershell +$env:NETBOT_LOCAL_TOKEN = "use-a-long-random-token" +.\.venv\Scripts\python.exe -m uvicorn backend.app.main:app --host 127.0.0.1 --port 8765 +``` + +The backend status endpoint is available at: + +```text +http://127.0.0.1:8765/api/status +``` + +## Frontend From Source + +```powershell +cd frontend +npm ci +npm run dev +``` + +Open: + +```text +http://127.0.0.1:5173 +``` + +If the backend runs on a non-default address, pass the API and websocket URLs +through the app query string: + +```text +http://127.0.0.1:5173/?api=http://127.0.0.1:8765/api&ws=ws://127.0.0.1:8765/ws +``` + +## Desktop App + +```powershell +cd desktop\electron +npm ci +npm run dev +``` + +The desktop shell creates or forwards a local token, starts the backend, and +loads the built frontend through the hardened Electron preload bridge. The +renderer does not receive Node.js access. + +## Packaged Backend + +Build the backend runtime with the existing PyInstaller flow: + +```powershell +python -m PyInstaller packaging\pyinstaller\netbotpro_backend.spec --clean --noconfirm +python scripts\release\stage_backend_runtime.py +python scripts\qa\packaged_backend_smoke.py +``` + +The staged runtime should include required backend assets such as +`service_fingerprints.json` and should pass the packaged backend smoke test +without exposing tokens in logs or status payloads. + +## Offline PCAP Mode + +Offline PCAP analysis is local and does not require live capture privileges. +Use the UI upload flow or the backend `/api/analyze-pcap` endpoint with `.pcap` +or `.pcapng` files. Upload size and file type are restricted by the backend. + +Offline output includes redacted packet summaries, protocol intelligence, +flow/conversation summaries, timelines, and risk distributions. It does not +decrypt TLS or collect credentials. + +## Common Troubleshooting + +| Symptom | Likely Cause | Fix | +| --- | --- | --- | +| No capture interfaces | Npcap/libpcap missing or not visible | Install Npcap/libpcap, reopen the terminal, and run the app with capture privileges. | +| Permission denied for capture | Terminal or desktop app lacks privileges | Start PowerShell or the desktop app as Administrator, or grant capture capabilities on Linux. | +| Frontend cannot reach backend | Backend is stopped, wrong port, or blocked by firewall | Confirm `http://127.0.0.1:8765/api/status`, then restart the backend. | +| Protected API returns `401` | Missing or wrong local token | Use the launcher-generated token file or set `NETBOT_LOCAL_TOKEN`. | +| Websocket fails with `403` | Token or allowed origin mismatch | Confirm `NETBOT_ALLOWED_ORIGINS` and websocket token subprotocol configuration. | +| Electron download fails | Network, proxy, cache, or registry issue | Re-run `npm ci` in `desktop\electron` on a network that can reach Electron releases, or configure an approved Electron cache/mirror. | +| Port already in use | Another backend or dev server is running | Stop the old process or set a different `NETBOT_PORT` / Vite port. | +| Python dependency install fails | Old pip, incompatible Python, or wheel download issue | Upgrade pip and use Python 3.13 where possible. | +| Node/npm install fails | Old Node/npm, cache issue, or corporate proxy | Use Node 20.17+ and retry `npm ci` after clearing the npm cache if needed. | +| Antivirus flags the build | Packet capture and unsigned bundles can look unusual | Verify the source, checksums, and build pipeline before allowing the artifact. | + +## Safe Use Reminder + +NetBotPro is for defensive monitoring, troubleshooting, education, and +authorized security analysis. It intentionally does not add command/control, +remote shell, credential collection, TLS decryption, MITM, browser history +scraping, cookie/session inspection, or Agent raw packet/payload/PCAP +forwarding. diff --git a/docs/RELEASE_CHECKLIST.md b/docs/RELEASE_CHECKLIST.md new file mode 100644 index 0000000..2083b4e --- /dev/null +++ b/docs/RELEASE_CHECKLIST.md @@ -0,0 +1,106 @@ +# Release Checklist + +Use this checklist before tagging or publishing a release candidate. + +## Source State + +- [ ] Work from the intended `main` commit or release branch. +- [ ] Confirm `git status --short --branch` is clean. +- [ ] Confirm versions, release notes, and tag naming are consistent. +- [ ] Confirm no large temporary files are staged. +- [ ] Confirm `.runtime/`, benchmark output, `node_modules/`, `dist/`, + `build/`, caches, local env files, and generated ZIPs are ignored. +- [ ] Confirm `.gitattributes` handles Python/Markdown/JSON/JavaScript as LF + and PowerShell/batch scripts as CRLF. +- [ ] Confirm docs do not contain machine-specific local paths. + +## Backend + +- [ ] `python -m pip install -r requirements-dev.txt` +- [ ] `python -m pip check` +- [ ] `python -m unittest discover -s tests -v` +- [ ] Review warnings and expected-error tests. +- [ ] Confirm no naive UTC timestamp deprecation warnings remain. +- [ ] Confirm expected-error logs do not make CI unreadable. + +## Frontend + +- [ ] `cd frontend` +- [ ] `npm ci` +- [ ] `npm run test:ui` +- [ ] `npm run build` +- [ ] `npm audit --omit=dev` +- [ ] `npm audit` +- [ ] If full dev audit requires a risky major upgrade, document the + dev-only scope and follow-up. + +## Desktop + +- [ ] `cd desktop\electron` +- [ ] `npm ci` +- [ ] `npm audit --omit=dev` +- [ ] `npm audit` +- [ ] Validate Electron entrypoints with `node --check main.cjs` and + `node --check preload.cjs`. +- [ ] Run `scripts\qa\electron_smoke.ps1`. +- [ ] Confirm startup logging works and does not expose secrets. +- [ ] Confirm packaged desktop dependencies install in a network environment + where Electron downloads are reachable. + +## PyInstaller And Packaged Backend + +- [ ] `python -m PyInstaller packaging\pyinstaller\netbotpro_backend.spec --clean --noconfirm` +- [ ] `python scripts\release\stage_backend_runtime.py` +- [ ] Confirm `service_fingerprints.json` is included. +- [ ] Confirm required backend data/config files are included. +- [ ] `python scripts\qa\packaged_backend_smoke.py` +- [ ] Confirm health and monitoring smoke checks pass. +- [ ] Confirm bundled logs/status payloads do not expose tokens. + +## Security Boundaries + +- [ ] No command/control. +- [ ] No remote shell. +- [ ] No file collection. +- [ ] No raw packet forwarding from Agent. +- [ ] No raw payload forwarding from Agent. +- [ ] No PCAP forwarding from Agent. +- [ ] No TLS decryption or MITM. +- [ ] No credential collection. +- [ ] No IPS or automatic response actions. +- [ ] No AI autonomous actions. +- [ ] No browser history scraping, cookie/session inspection, browser + extension injection, or keylogging. +- [ ] Agent/Fleet telemetry-only boundary is unchanged. +- [ ] Remote Sensor boundary is unchanged. +- [ ] Metrics, reports, exports, and incident summaries are redacted. + +## Performance + +- [ ] Run the CI-safe benchmark smoke: + `python benchmarks/soak_test_pipeline.py --duration-sec 10 --events-per-sec 200 --flows 20 --ci-safe --output .runtime/benchmarks/release-hardening-smoke` +- [ ] Confirm bounded queues and buffers remain visible in Ops Snapshot. +- [ ] Confirm packet queue, event aggregator, worker pool, persistence, live + ring buffer, service attribution, and incident sections are present. +- [ ] Treat benchmark smoke as structural validation, not a production capacity + claim. + +## Release Artifacts + +- [ ] Source ZIP. +- [ ] Packaged backend runtime. +- [ ] Desktop package where applicable. +- [ ] SHA256 checksums. +- [ ] Changelog and release notes. +- [ ] Release workflow status. +- [ ] Artifact names include the intended version. + +## Known Limitations To Confirm + +- [ ] Live capture requires authorization, drivers, and privileges. +- [ ] Service Attribution confidence is limited by ECH, DoH, VPNs, NAT, CDN + fronting, and weak metadata. +- [ ] Incident Engine is currently bounded/in-memory unless a later persistence + step is added. +- [ ] Server Mode is not a full production multi-node deployment yet. +- [ ] AI Analyst is not implemented in this release. diff --git a/docs/REMOTE_SENSOR.md b/docs/REMOTE_SENSOR.md index afc6536..25ddc73 100644 --- a/docs/REMOTE_SENSOR.md +++ b/docs/REMOTE_SENSOR.md @@ -20,7 +20,7 @@ Use this mode only for systems, servers, and networks where you have explicit le ## Start the sensor on the server ```powershell -cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro" +cd .\Netbotpro powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-sensor.ps1 -BindHost 0.0.0.0 -Port 8765 -AllowedOrigins "http://YOUR_DASHBOARD_HOST:5173" -CaptureMode metadata ``` diff --git a/tests/test_release_readiness.py b/tests/test_release_readiness.py index 14ffe73..2819432 100644 --- a/tests/test_release_readiness.py +++ b/tests/test_release_readiness.py @@ -58,6 +58,8 @@ def test_release_docs_exist_and_readme_links_them(self): readme = self._read("README.md") for relative_path in [ "docs/DEPLOYMENT_OVERVIEW.md", + "docs/INSTALL.md", + "docs/RELEASE_CHECKLIST.md", "docs/RELEASE_QA_CHECKLIST.md", "docs/RELEASE_NOTES_v0.2.0.md", ]: @@ -66,6 +68,44 @@ def test_release_docs_exist_and_readme_links_them(self): self.assertIn(f"v{TARGET_VERSION}", readme) + def test_release_hardening_docs_and_source_cleanliness_rules(self): + install = self._read("docs/INSTALL.md").lower() + checklist = self._read("docs/RELEASE_CHECKLIST.md").lower() + gitignore = self._read(".gitignore") + gitattributes = self._read(".gitattributes") + docs_text = "\n".join( + [ + self._read("README.md"), + self._read("docs/INSTALL.md"), + self._read("docs/RELEASE_CHECKLIST.md"), + self._read("docs/REMOTE_SENSOR.md"), + ] + ) + + self.assertIn("npcap", install) + self.assertIn("packaged backend", install) + self.assertIn("electron download", install) + self.assertIn("safe use", install) + self.assertIn("source state", checklist) + self.assertIn("no command/control", checklist) + self.assertIn("naive utc timestamp", checklist) + self.assertIn("service_fingerprints.json", checklist) + self.assertNotRegex(docs_text, r"C:\\Users\\") + + for ignored in [ + ".runtime/", + "node_modules/", + "frontend/dist/", + "desktop/electron/dist/", + "__pycache__/", + ".venv/", + "*.zip", + ]: + self.assertIn(ignored, gitignore) + + self.assertIn("*.py text eol=lf", gitattributes) + self.assertIn("*.ps1 text eol=crlf", gitattributes) + def test_flow_analysis_docs_and_architecture_are_linked(self): readme = self._read("README.md") architecture = self._read("docs/ARCHITECTURE.md")