Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,31 @@

### Added

- Release hardening documentation with an install/run guide and release
checklist covering source cleanliness, audits, desktop validation,
PyInstaller smoke, security boundaries, and release artifacts.
- Performance Pipeline foundation: bounded packet intake, WebSocket Event
Aggregator, Batch Persistence / Storage Backpressure, Flow-aware Worker
Pool, Live Ring Buffer, and CI-safe benchmark/soak validation.
- Service Attribution / Destination Intelligence with metadata-only
confidence scoring and conservative handling for encrypted/CDN traffic.
- Incident Correlation Engine with redacted incident timelines, severity,
confidence, investigation guidance, and read-only Markdown summary export.
- Protocol Intelligence Expansion for TCP, DNS, HTTP, and visible TLS metadata.
- Saved Display Filters with built-in recipes and safe field suggestions.
- Protocol Statistics and safe Packet Search in the Inspect workspace.
- Expanded Expert Info and redacted protocol/inspection summary reports.
- Offline PCAP parity for protocol intelligence summaries.

### Changed

- Replaced remaining naive UTC timestamp call sites with timezone-aware UTC
timestamps.
- Refreshed frontend development lockfile dependencies with compatible
patch/minor updates and cleaned desktop Electron audit findings.
- Reduced expected-error test log noise while preserving production logging and
failure-path coverage.

## v0.2.0 - Agent & Fleet Monitoring Release - 2026-06-05

### Added
Expand Down
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -263,6 +263,8 @@ and surfaced in Ops Snapshot metrics.
- [Deep Packet Inspection](docs/DEEP_PACKET_INSPECTION.md)
- [Agent Operational QA Checklist](docs/AGENT_QA_CHECKLIST.md)
- [Deployment Overview](docs/DEPLOYMENT_OVERVIEW.md)
- [Install And Run Guide](docs/INSTALL.md)
- [Release Checklist](docs/RELEASE_CHECKLIST.md)
- [Release QA Checklist](docs/RELEASE_QA_CHECKLIST.md)
- [v0.2.0 Release Notes](docs/RELEASE_NOTES_v0.2.0.md)
- [Remote Sensor](docs/REMOTE_SENSOR.md)
Expand Down Expand Up @@ -307,7 +309,7 @@ Prerequisites:
Install everything on Windows:

```powershell
cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro"
cd .\Netbotpro
powershell -ExecutionPolicy Bypass -File .\scripts\dev\setup.ps1
```

Expand Down Expand Up @@ -336,7 +338,7 @@ Copy `.env.example` to `.env` for local experiments. The default secure path is
Start the local web stack:

```powershell
cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro"
cd .\Netbotpro
powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-local.ps1
```

Expand All @@ -355,14 +357,14 @@ http://127.0.0.1:5173
Start the Electron desktop shell after dependencies are installed:

```powershell
cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro\desktop\electron"
cd .\Netbotpro\desktop\electron
npm run dev
```

Start a remote sensor on a server you own or administer:

```powershell
cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro"
cd .\Netbotpro
powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-sensor.ps1 -BindHost 0.0.0.0 -Port 8765 -AllowedOrigins "http://YOUR_DASHBOARD_HOST:5173"
```

Expand Down Expand Up @@ -524,6 +526,8 @@ GitHub Actions builds CI on push and pull request. Desktop release artifacts are
- [Agent mode](docs/AGENT_MODE.md)
- [Agent QA checklist](docs/AGENT_QA_CHECKLIST.md)
- [Deployment overview](docs/DEPLOYMENT_OVERVIEW.md)
- [Install and run guide](docs/INSTALL.md)
- [Release checklist](docs/RELEASE_CHECKLIST.md)
- [Release QA checklist](docs/RELEASE_QA_CHECKLIST.md)
- [Desktop shell](docs/DESKTOP_SHELL.md)
- [Web migration notes](docs/WEB_MIGRATION.md)
Expand Down
142 changes: 142 additions & 0 deletions docs/INSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
# NetBotPro Install And Run Guide

NetBotPro is a local-first defensive network monitoring and analysis tool.
Only run live capture, Remote Sensor mode, or Agent telemetry on systems and
networks you own or have explicit permission to administer.

## Requirements

- Python 3.13 is recommended for source and CI parity.
- Node.js 20 or newer is recommended. Node 20.17+ avoids newer Electron
tooling engine warnings.
- Npcap on Windows for live packet capture.
- Administrator privileges on Windows, or root/capture capabilities on Linux,
when live packet capture is required.
- Git and PowerShell 7 or Windows PowerShell for the provided scripts.

Offline PCAP analysis, report review, docs, and most UI development paths do
not require packet-capture privileges.

## Supported OS Notes

- Windows is the strongest validated desktop target.
- Linux supports source runs, backend services, and staged desktop packaging.
- macOS runs backend/frontend CI paths, but desktop release validation is not
currently the strongest release target.

Live capture behavior depends on OS drivers, interface visibility, privileges,
and what traffic is observable from the selected host/interface.

## Windows Notes

Install Npcap before using live capture. Reopen the terminal after installing
Npcap, and run the desktop app or backend terminal as Administrator when live
capture or firewall actions are needed.

Antivirus or endpoint security products can flag packet capture tools,
PyInstaller bundles, or unsigned desktop artifacts. Review alerts carefully and
allow only builds you created from trusted source.

## Backend From Source

```powershell
git clone https://github.com/XploitGost/Netbotpro.git
cd Netbotpro
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install --upgrade pip
.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
.\.venv\Scripts\python.exe -m backend.app.desktop_entry
```

For a direct API server run:

```powershell
$env:NETBOT_LOCAL_TOKEN = "use-a-long-random-token"
.\.venv\Scripts\python.exe -m uvicorn backend.app.main:app --host 127.0.0.1 --port 8765
```

The backend status endpoint is available at:

```text
http://127.0.0.1:8765/api/status
```

## Frontend From Source

```powershell
cd frontend
npm ci
npm run dev
```

Open:

```text
http://127.0.0.1:5173
```

If the backend runs on a non-default address, pass the API and websocket URLs
through the app query string:

```text
http://127.0.0.1:5173/?api=http://127.0.0.1:8765/api&ws=ws://127.0.0.1:8765/ws
```

## Desktop App

```powershell
cd desktop\electron
npm ci
npm run dev
```

The desktop shell creates or forwards a local token, starts the backend, and
loads the built frontend through the hardened Electron preload bridge. The
renderer does not receive Node.js access.

## Packaged Backend

Build the backend runtime with the existing PyInstaller flow:

```powershell
python -m PyInstaller packaging\pyinstaller\netbotpro_backend.spec --clean --noconfirm
python scripts\release\stage_backend_runtime.py
python scripts\qa\packaged_backend_smoke.py
```

The staged runtime should include required backend assets such as
`service_fingerprints.json` and should pass the packaged backend smoke test
without exposing tokens in logs or status payloads.

## Offline PCAP Mode

Offline PCAP analysis is local and does not require live capture privileges.
Use the UI upload flow or the backend `/api/analyze-pcap` endpoint with `.pcap`
or `.pcapng` files. Upload size and file type are restricted by the backend.

Offline output includes redacted packet summaries, protocol intelligence,
flow/conversation summaries, timelines, and risk distributions. It does not
decrypt TLS or collect credentials.

## Common Troubleshooting

| Symptom | Likely Cause | Fix |
| --- | --- | --- |
| No capture interfaces | Npcap/libpcap missing or not visible | Install Npcap/libpcap, reopen the terminal, and run the app with capture privileges. |
| Permission denied for capture | Terminal or desktop app lacks privileges | Start PowerShell or the desktop app as Administrator, or grant capture capabilities on Linux. |
| Frontend cannot reach backend | Backend is stopped, wrong port, or blocked by firewall | Confirm `http://127.0.0.1:8765/api/status`, then restart the backend. |
| Protected API returns `401` | Missing or wrong local token | Use the launcher-generated token file or set `NETBOT_LOCAL_TOKEN`. |
| Websocket fails with `403` | Token or allowed origin mismatch | Confirm `NETBOT_ALLOWED_ORIGINS` and websocket token subprotocol configuration. |
| Electron download fails | Network, proxy, cache, or registry issue | Re-run `npm ci` in `desktop\electron` on a network that can reach Electron releases, or configure an approved Electron cache/mirror. |
| Port already in use | Another backend or dev server is running | Stop the old process or set a different `NETBOT_PORT` / Vite port. |
| Python dependency install fails | Old pip, incompatible Python, or wheel download issue | Upgrade pip and use Python 3.13 where possible. |
| Node/npm install fails | Old Node/npm, cache issue, or corporate proxy | Use Node 20.17+ and retry `npm ci` after clearing the npm cache if needed. |
| Antivirus flags the build | Packet capture and unsigned bundles can look unusual | Verify the source, checksums, and build pipeline before allowing the artifact. |

## Safe Use Reminder

NetBotPro is for defensive monitoring, troubleshooting, education, and
authorized security analysis. It intentionally does not add command/control,
remote shell, credential collection, TLS decryption, MITM, browser history
scraping, cookie/session inspection, or Agent raw packet/payload/PCAP
forwarding.
106 changes: 106 additions & 0 deletions docs/RELEASE_CHECKLIST.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Release Checklist

Use this checklist before tagging or publishing a release candidate.

## Source State

- [ ] Work from the intended `main` commit or release branch.
- [ ] Confirm `git status --short --branch` is clean.
- [ ] Confirm versions, release notes, and tag naming are consistent.
- [ ] Confirm no large temporary files are staged.
- [ ] Confirm `.runtime/`, benchmark output, `node_modules/`, `dist/`,
`build/`, caches, local env files, and generated ZIPs are ignored.
- [ ] Confirm `.gitattributes` handles Python/Markdown/JSON/JavaScript as LF
and PowerShell/batch scripts as CRLF.
- [ ] Confirm docs do not contain machine-specific local paths.

## Backend

- [ ] `python -m pip install -r requirements-dev.txt`
- [ ] `python -m pip check`
- [ ] `python -m unittest discover -s tests -v`
- [ ] Review warnings and expected-error tests.
- [ ] Confirm no naive UTC timestamp deprecation warnings remain.
- [ ] Confirm expected-error logs do not make CI unreadable.

## Frontend

- [ ] `cd frontend`
- [ ] `npm ci`
- [ ] `npm run test:ui`
- [ ] `npm run build`
- [ ] `npm audit --omit=dev`
- [ ] `npm audit`
- [ ] If full dev audit requires a risky major upgrade, document the
dev-only scope and follow-up.

## Desktop

- [ ] `cd desktop\electron`
- [ ] `npm ci`
- [ ] `npm audit --omit=dev`
- [ ] `npm audit`
- [ ] Validate Electron entrypoints with `node --check main.cjs` and
`node --check preload.cjs`.
- [ ] Run `scripts\qa\electron_smoke.ps1`.
- [ ] Confirm startup logging works and does not expose secrets.
- [ ] Confirm packaged desktop dependencies install in a network environment
where Electron downloads are reachable.

## PyInstaller And Packaged Backend

- [ ] `python -m PyInstaller packaging\pyinstaller\netbotpro_backend.spec --clean --noconfirm`
- [ ] `python scripts\release\stage_backend_runtime.py`
- [ ] Confirm `service_fingerprints.json` is included.
- [ ] Confirm required backend data/config files are included.
- [ ] `python scripts\qa\packaged_backend_smoke.py`
- [ ] Confirm health and monitoring smoke checks pass.
- [ ] Confirm bundled logs/status payloads do not expose tokens.

## Security Boundaries

- [ ] No command/control.
- [ ] No remote shell.
- [ ] No file collection.
- [ ] No raw packet forwarding from Agent.
- [ ] No raw payload forwarding from Agent.
- [ ] No PCAP forwarding from Agent.
- [ ] No TLS decryption or MITM.
- [ ] No credential collection.
- [ ] No IPS or automatic response actions.
- [ ] No AI autonomous actions.
- [ ] No browser history scraping, cookie/session inspection, browser
extension injection, or keylogging.
- [ ] Agent/Fleet telemetry-only boundary is unchanged.
- [ ] Remote Sensor boundary is unchanged.
- [ ] Metrics, reports, exports, and incident summaries are redacted.

## Performance

- [ ] Run the CI-safe benchmark smoke:
`python benchmarks/soak_test_pipeline.py --duration-sec 10 --events-per-sec 200 --flows 20 --ci-safe --output .runtime/benchmarks/release-hardening-smoke`
- [ ] Confirm bounded queues and buffers remain visible in Ops Snapshot.
- [ ] Confirm packet queue, event aggregator, worker pool, persistence, live
ring buffer, service attribution, and incident sections are present.
- [ ] Treat benchmark smoke as structural validation, not a production capacity
claim.

## Release Artifacts

- [ ] Source ZIP.
- [ ] Packaged backend runtime.
- [ ] Desktop package where applicable.
- [ ] SHA256 checksums.
- [ ] Changelog and release notes.
- [ ] Release workflow status.
- [ ] Artifact names include the intended version.

## Known Limitations To Confirm

- [ ] Live capture requires authorization, drivers, and privileges.
- [ ] Service Attribution confidence is limited by ECH, DoH, VPNs, NAT, CDN
fronting, and weak metadata.
- [ ] Incident Engine is currently bounded/in-memory unless a later persistence
step is added.
- [ ] Server Mode is not a full production multi-node deployment yet.
- [ ] AI Analyst is not implemented in this release.
2 changes: 1 addition & 1 deletion docs/REMOTE_SENSOR.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Use this mode only for systems, servers, and networks where you have explicit le
## Start the sensor on the server

```powershell
cd "C:\Users\ASIA SYSTEM\Desktop\netbotpro"
cd .\Netbotpro
powershell -ExecutionPolicy Bypass -File .\scripts\dev\start-sensor.ps1 -BindHost 0.0.0.0 -Port 8765 -AllowedOrigins "http://YOUR_DASHBOARD_HOST:5173" -CaptureMode metadata
```

Expand Down
40 changes: 40 additions & 0 deletions tests/test_release_readiness.py
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ def test_release_docs_exist_and_readme_links_them(self):
readme = self._read("README.md")
for relative_path in [
"docs/DEPLOYMENT_OVERVIEW.md",
"docs/INSTALL.md",
"docs/RELEASE_CHECKLIST.md",
"docs/RELEASE_QA_CHECKLIST.md",
"docs/RELEASE_NOTES_v0.2.0.md",
]:
Expand All @@ -66,6 +68,44 @@ def test_release_docs_exist_and_readme_links_them(self):

self.assertIn(f"v{TARGET_VERSION}", readme)

def test_release_hardening_docs_and_source_cleanliness_rules(self):
install = self._read("docs/INSTALL.md").lower()
checklist = self._read("docs/RELEASE_CHECKLIST.md").lower()
gitignore = self._read(".gitignore")
gitattributes = self._read(".gitattributes")
docs_text = "\n".join(
[
self._read("README.md"),
self._read("docs/INSTALL.md"),
self._read("docs/RELEASE_CHECKLIST.md"),
self._read("docs/REMOTE_SENSOR.md"),
]
)

self.assertIn("npcap", install)
self.assertIn("packaged backend", install)
self.assertIn("electron download", install)
self.assertIn("safe use", install)
self.assertIn("source state", checklist)
self.assertIn("no command/control", checklist)
self.assertIn("naive utc timestamp", checklist)
self.assertIn("service_fingerprints.json", checklist)
self.assertNotRegex(docs_text, r"C:\\Users\\")

for ignored in [
".runtime/",
"node_modules/",
"frontend/dist/",
"desktop/electron/dist/",
"__pycache__/",
".venv/",
"*.zip",
]:
self.assertIn(ignored, gitignore)

self.assertIn("*.py text eol=lf", gitattributes)
self.assertIn("*.ps1 text eol=crlf", gitattributes)

def test_flow_analysis_docs_and_architecture_are_linked(self):
readme = self._read("README.md")
architecture = self._read("docs/ARCHITECTURE.md")
Expand Down