Skip to content

Commit bce90d8

Browse files
hackrangehackrange
committed
External sites: the company's domains from outside, with dynamic scans
A new administrators only module, off until Settings turns it on. Off hides the menu entry and every page and API call under it answers 404. List domains and subdomains with an application and tags, or import the company domains and the hosts in repositories' web addresses. Each site gets its outside view: DNS, network owner, open ports, TLS and the old versions it still accepts, headers, a hygiene grade, a firewall block page check and a screenshot from a headless Chromium. Dynamic scans three ways. Run here with the same profiles as nextgen-dast, from read only baseline checks up through testssl, nuclei, nikto, wapiti, ffuf, sqlmap and dalfox, all pinned in the image by checksum. Start one on a nextgen-dast server and read it back. Or record one from any server by hand or through the API: severity, result address, when it ran and which server. An engine that is missing is shown as skipped. Path findings that only hit a site's catch all page are left out. Each site is mapped to the repositories behind it from web addresses, diagrams, application names and host names. A scan reads routes out of their default branch first and hands them to the engines. A combined risk reads the last scan and the open code findings together. A host marked no scan, or under a domain marked that way, is never scanned; its telemetry is one browser visit. AI triage of a scan and a risk read of a site go through the AI analysis model, are counted on the AI cost page as dast-triage and external-risk, and keep their own token counts and cost. The API is under /api/v1/external with a new external token scope that only an administrator's token can carry. Every id is checked against the site it hangs off, and repositories and findings are read through the asker's scope.
1 parent 0e6bd7c commit bce90d8

29 files changed

Lines changed: 5655 additions & 8 deletions

‎Dockerfile‎

Lines changed: 67 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,10 +28,11 @@ RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-reco
2828
&& rm -rf /var/lib/apt/lists/*
2929

3030
WORKDIR /wheels
31-
COPY requirements.txt requirements-scanner.txt ./
32-
# Two sets, built separately, because they resolve separately.
31+
COPY requirements.txt requirements-scanner.txt requirements-dast.txt ./
32+
# Three sets, built separately, because they resolve separately.
3333
RUN python3 -m pip wheel --wheel-dir /wheels/app -r requirements.txt \
34-
&& python3 -m pip wheel --wheel-dir /wheels/scanner -r requirements-scanner.txt
34+
&& python3 -m pip wheel --wheel-dir /wheels/scanner -r requirements-scanner.txt \
35+
&& python3 -m pip wheel --wheel-dir /wheels/dast -r requirements-dast.txt
3536

3637
# ==============================================================================
3738
# Stage 2, the image that runs
@@ -78,6 +79,17 @@ RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install-reco
7879
libglib2.0-0t64 libpango-1.0-0 libpangoft2-1.0-0 libharfbuzz0b \
7980
libfontconfig1 libcairo2 libgdk-pixbuf-2.0-0 \
8081
fonts-dejavu-core \
82+
# External sites. whois names who owns a site's network. Nikto is
83+
# perl, testssl.sh is bash over openssl and wants hexdump, dig and ps.
84+
whois perl libnet-ssleay-perl libio-socket-ssl-perl libjson-perl \
85+
libxml-writer-perl libtimedate-perl libtext-csv-perl libwww-perl \
86+
libhtml-parser-perl \
87+
bsdextrautils dnsutils procps \
88+
# What headless Chromium loads to draw a page for a screenshot.
89+
# Fonts too: with none, every page with text on it comes out blank.
90+
libnss3 libnspr4 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 \
91+
libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxfixes3 \
92+
libxrandr2 libgbm1 libasound2t64 libxshmfence1 fonts-liberation \
8193
&& rm -rf /var/lib/apt/lists/*
8294

8395
# The clients the other two version control systems need are deliberately not
@@ -122,6 +134,23 @@ ARG GOSEC_SHA256_ARM64=c71244ec8d37488fd479d0d26990968fee03ece48b305d8224ac0c1eb
122134
# gosec reads types through go/packages, which shells out to `go list`. With
123135
# no toolchain it exits zero having analysed nothing, so the compiler comes
124136
# with it or the engine is worse than absent.
137+
# The dynamic scanning engines for External sites, the same set nextgen-dast
138+
# drives. Go binaries checked against their projects' published sums; the
139+
# two script engines are release tarballs checked against the sum of the
140+
# tarball itself.
141+
ARG NUCLEI_VERSION=3.11.1
142+
ARG NUCLEI_SHA256_AMD64=ea63d4ae232808cd7c6bc00d0142428e231fab59dae01042246097d195835ab6
143+
ARG NUCLEI_SHA256_ARM64=8044e3d9768ba0a744b2872c1a87e813006f013da97ca9f50f7661a4203bec07
144+
ARG DALFOX_VERSION=3.2.4
145+
ARG DALFOX_SHA256_AMD64=aaef3f3d26d2de6d4e61ef5295151d9de36e111981618136ac7ec54b578582b8
146+
ARG DALFOX_SHA256_ARM64=5c65627fd82b987eddf1296cf97b7862a401a364a05cf7262e59c9e5b5407d60
147+
ARG FFUF_VERSION=2.3.0
148+
ARG FFUF_SHA256_AMD64=b2a3c725fcb9da175159682f54d6e9149f2905b00d84d155e7efc5d599975ceb
149+
ARG FFUF_SHA256_ARM64=e2d712615d53da797ee929006e41f2df5a7960a5353133db9c150ddaa22ae4e8
150+
ARG TESTSSL_VERSION=3.2.4
151+
ARG TESTSSL_SHA256=98528f8a0ac07f1e226efaa8ead438247df8efcb8fee4e056a937ab82a305490
152+
ARG NIKTO_VERSION=2.6.1
153+
ARG NIKTO_SHA256=d1ca1acb05d81a5a6f374c0afdd76b33afa0089278631a20673c0210a71d992f
125154
ARG GO_VERSION=1.27.2
126155
ARG GO_SHA256_AMD64=ecbadb99091a3f46e31f5f934b068b1864eafa7995211b39eaddf76996045fe5
127156
ARG GO_SHA256_ARM64=94f3e30b8e374bc285e7dadc11e0865726b9bc6e85b841ccceaabc0214c6b7c8
@@ -134,13 +163,15 @@ RUN set -eux; \
134163
gr_sum=$GRYPE_SHA256_AMD64; th_sum=$TRUFFLEHOG_SHA256_AMD64; \
135164
os_sum=$OSV_SHA256_AMD64; tv_sum=$TRIVY_SHA256_AMD64; \
136165
zz_sum=$ZIZMOR_SHA256_AMD64; gs_sum=$GOSEC_SHA256_AMD64; \
137-
go_sum=$GO_SHA256_AMD64 ;; \
166+
go_sum=$GO_SHA256_AMD64; nu_sum=$NUCLEI_SHA256_AMD64; \
167+
df_arch=x86_64; df_sum=$DALFOX_SHA256_AMD64; ff_sum=$FFUF_SHA256_AMD64 ;; \
138168
arm64) gl_arch=arm64; go_arch=arm64; tv_arch=ARM64; zz_arch=aarch64; \
139169
gl_sum=$GITLEAKS_SHA256_ARM64; sy_sum=$SYFT_SHA256_ARM64; \
140170
gr_sum=$GRYPE_SHA256_ARM64; th_sum=$TRUFFLEHOG_SHA256_ARM64; \
141171
os_sum=$OSV_SHA256_ARM64; tv_sum=$TRIVY_SHA256_ARM64; \
142172
zz_sum=$ZIZMOR_SHA256_ARM64; gs_sum=$GOSEC_SHA256_ARM64; \
143-
go_sum=$GO_SHA256_ARM64 ;; \
173+
go_sum=$GO_SHA256_ARM64; nu_sum=$NUCLEI_SHA256_ARM64; \
174+
df_arch=aarch64; df_sum=$DALFOX_SHA256_ARM64; ff_sum=$FFUF_SHA256_ARM64 ;; \
144175
*) echo "unsupported architecture: $arch" >&2; exit 1 ;; \
145176
esac; \
146177
dl=/tmp/tools; mkdir -p "$dl"; \
@@ -155,6 +186,11 @@ RUN set -eux; \
155186
fetch "$gh/zizmorcore/zizmor/releases/download/v${ZIZMOR_VERSION}/zizmor-${zz_arch}-unknown-linux-gnu.tar.gz" zizmor.tgz "$zz_sum"; \
156187
fetch "$gh/securego/gosec/releases/download/v${GOSEC_VERSION}/gosec_${GOSEC_VERSION}_linux_${go_arch}.tar.gz" gosec.tgz "$gs_sum"; \
157188
fetch "https://go.dev/dl/go${GO_VERSION}.linux-${go_arch}.tar.gz" go.tgz "$go_sum"; \
189+
fetch "$gh/projectdiscovery/nuclei/releases/download/v${NUCLEI_VERSION}/nuclei_${NUCLEI_VERSION}_linux_${go_arch}.zip" nuclei.zip "$nu_sum"; \
190+
fetch "$gh/hahwul/dalfox/releases/download/v${DALFOX_VERSION}/dalfox-v${DALFOX_VERSION}-linux-${df_arch}.tar.gz" dalfox.tgz "$df_sum"; \
191+
fetch "$gh/ffuf/ffuf/releases/download/v${FFUF_VERSION}/ffuf_${FFUF_VERSION}_linux_${go_arch}.tar.gz" ffuf.tgz "$ff_sum"; \
192+
fetch "https://codeload.github.com/testssl/testssl.sh/tar.gz/refs/tags/v${TESTSSL_VERSION}" testssl.tgz "$TESTSSL_SHA256"; \
193+
fetch "https://codeload.github.com/sullo/nikto/tar.gz/refs/tags/${NIKTO_VERSION}" nikto.tgz "$NIKTO_SHA256"; \
158194
tar -xzf "$dl/gitleaks.tgz" -C /usr/local/bin gitleaks; \
159195
tar -xzf "$dl/syft.tgz" -C /usr/local/bin syft; \
160196
tar -xzf "$dl/grype.tgz" -C /usr/local/bin grype; \
@@ -165,22 +201,44 @@ RUN set -eux; \
165201
tar -xzf "$dl/gosec.tgz" -C /usr/local/bin gosec; \
166202
tar -xzf "$dl/go.tgz" -C /usr/local; \
167203
ln -sf /usr/local/go/bin/go /usr/local/bin/go; \
204+
python3 -c "import zipfile,sys; zipfile.ZipFile(sys.argv[1]).extract('nuclei', '/usr/local/bin')" "$dl/nuclei.zip"; \
205+
chmod 0755 /usr/local/bin/nuclei; \
206+
tar -xzf "$dl/dalfox.tgz" -C /usr/local/bin --strip-components=1 \
207+
"dalfox-v${DALFOX_VERSION}-linux-${df_arch}/dalfox"; \
208+
tar -xzf "$dl/ffuf.tgz" -C /usr/local/bin ffuf; \
209+
mkdir -p /opt/gcr/engines/testssl /opt/gcr/engines/nikto; \
210+
tar -xzf "$dl/testssl.tgz" -C /opt/gcr/engines/testssl --strip-components=1; \
211+
tar -xzf "$dl/nikto.tgz" -C /opt/gcr/engines/nikto --strip-components=1; \
212+
ln -sf /opt/gcr/engines/testssl/testssl.sh /usr/local/bin/testssl.sh; \
213+
printf '#!/bin/sh\nexec perl /opt/gcr/engines/nikto/program/nikto.pl "$@"\n' > /usr/local/bin/nikto; \
214+
chmod 0755 /usr/local/bin/nikto; \
168215
rm -rf "$dl"; \
169216
gitleaks version; syft version; grype version; trivy --version; zizmor --version; \
170217
gosec --version; go version; \
218+
nuclei -version; ffuf -V; dalfox --version; testssl.sh --version | head -3; \
171219
trufflehog --version || true
172220

173221
# Python packages go in their own venv. Ubuntu marks the system python as
174222
# externally managed, and arguing with that just breaks later apt upgrades.
175223
COPY --from=builder /wheels /wheels
176-
COPY requirements.txt requirements-scanner.txt /tmp/
224+
COPY requirements.txt requirements-scanner.txt requirements-dast.txt /tmp/
177225
RUN python3 -m venv /opt/gcr/venv \
178226
&& /opt/gcr/venv/bin/pip install --no-index --find-links=/wheels/app \
179227
-r /tmp/requirements.txt \
180228
&& python3 -m venv /opt/gcr/scanner-venv \
181229
&& /opt/gcr/scanner-venv/bin/pip install --no-index --find-links=/wheels/scanner \
182230
-r /tmp/requirements-scanner.txt \
183-
&& rm -rf /wheels /tmp/requirements.txt /tmp/requirements-scanner.txt
231+
&& python3 -m venv /opt/gcr/dast-venv \
232+
&& /opt/gcr/dast-venv/bin/pip install --no-index --find-links=/wheels/dast \
233+
-r /tmp/requirements-dast.txt \
234+
&& ln -sf /opt/gcr/dast-venv/bin/wapiti /usr/local/bin/wapiti \
235+
&& ln -sf /opt/gcr/dast-venv/bin/sqlmap /usr/local/bin/sqlmap \
236+
&& PLAYWRIGHT_BROWSERS_PATH=/opt/gcr/browsers \
237+
/opt/gcr/dast-venv/bin/python -m playwright install chromium-headless-shell \
238+
&& chmod -R a+rX /opt/gcr/browsers \
239+
&& wapiti --version && sqlmap --version \
240+
&& rm -rf /wheels /tmp/requirements.txt /tmp/requirements-scanner.txt \
241+
/tmp/requirements-dast.txt
184242

185243
# It runs as a normal user. Nothing here needs root, and this thing clones
186244
# code it did not write.
@@ -212,6 +270,8 @@ ENV GCR_DATA_DIR=/data \
212270
GCR_SCANNER_BIN=/opt/gcr/bin/ghscan \
213271
GCR_SCANNER_VENV=/opt/gcr/scanner-venv \
214272
GCR_PORT=8080 \
273+
PLAYWRIGHT_BROWSERS_PATH=/opt/gcr/browsers \
274+
GCR_NUCLEI_TEMPLATES=/home/gcr/.cache/nuclei-templates \
215275
PATH="/opt/gcr/venv/bin:/opt/gcr/scanner-venv/bin:/opt/gcr/bin:${PATH}"
216276

217277
USER gcr

‎app/core/ai_cost.py‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,17 @@ def _cost(tokens: dict, rate: dict) -> Decimal:
4343
return total
4444

4545

46+
def cost_of(cfg: dict, out: dict):
47+
"""What one answer from ai.complete cost, or None when no prices are set."""
48+
rate = prices(cfg)
49+
if all(v is None for v in rate.values()):
50+
return None
51+
return _cost({"input": out.get("input_tokens") or 0,
52+
"output": out.get("output_tokens") or 0,
53+
"cache_read": out.get("cache_read_tokens") or 0,
54+
"cache_write": out.get("cache_write_tokens") or 0}, rate)
55+
56+
4657
def _blank() -> dict:
4758
return {"calls": 0, "failed": 0, "cut_off": 0,
4859
"input": 0, "output": 0, "cache_read": 0, "cache_write": 0}

‎app/core/api_reference.py‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -80,6 +80,10 @@ def _finding_filters():
8080
("Trends", "How your numbers have moved over time."),
8181
("Report files", "The whole estate report files. Administrators only."),
8282
("Editors", "What the editor plugins use."),
83+
("External sites", "The company's domains and subdomains, their dynamic "
84+
"scans and the repositories behind them. Needs the "
85+
"external scope on an administrator's token, and External "
86+
"sites switched on in Settings."),
8387
]
8488

8589

@@ -382,6 +386,148 @@ def endpoints() -> list:
382386
"only": "secrets,code"},
383387
"example": {"counts": {"credential": 0, "code": 0}, "findings": []},
384388
},
389+
] + _external_endpoints()
390+
391+
392+
def _ext(id_, method, path, summary, description, params=None, body=None,
393+
body_example=None, example=None):
394+
e = {"id": id_, "group": "External sites", "method": method, "path": path,
395+
"scope": "external", "permission": "external.manage",
396+
"summary": summary, "description": description,
397+
"params": params or [], "example": example or {}}
398+
if body is not None:
399+
e["body"] = body
400+
e["body_example"] = body_example
401+
return e
402+
403+
404+
def _f(name, kind, required, description):
405+
return {"name": name, "type": kind, "required": required, "description": description}
406+
407+
408+
_ASSET = {"id": 12, "host": "shop.example.com", "kind": "subdomain",
409+
"parent_domain": "example.com", "application": "Storefront",
410+
"tags": ["ecommerce"], "no_scan": False, "scan_blocked": False,
411+
"telemetry": {"at": "2026-11-03T10:00:00+00:00", "grade": "B",
412+
"category": "live", "status_code": 200, "ip": "203.0.113.10",
413+
"open_ports": [80, 443], "tls_days_left": 61,
414+
"screenshot_url": "https://gcr.example.com/external/12/screenshot.png"},
415+
"last_dast": {"scanned_at": "2026-11-01T02:00:00+00:00", "severity": "high",
416+
"result_url": "https://dast.example.com/assessment/88",
417+
"server": "nextgen-dast", "status": "current"},
418+
"repositories": [{"full_name": "acme/storefront", "score": 95, "linked": True,
419+
"reasons": ["host named in its web addresses"],
420+
"open_findings": {"high": 2}}],
421+
"combined_risk": {"level": "critical",
422+
"reasons": ["last dynamic scan: high",
423+
"open high findings in the code behind it",
424+
"the site and its code both have problems"]}}
425+
_SCAN = {"id": 40, "asset_id": 12, "server": "nextgen-dast", "source": "recorded",
426+
"status": "done", "scanned_at": "2026-11-01T02:00:00+00:00",
427+
"severity": "high", "counts": {"critical": 0, "high": 1, "medium": 3,
428+
"low": 5, "info": 2},
429+
"result_url": "https://dast.example.com/assessment/88"}
430+
_RECORD_BODY = [
431+
_f("server", "string", True, "The DAST server that ran it, by name."),
432+
_f("severity", "string", False, "The worst found: critical, high, medium, low, "
433+
"info or none. Worked out from counts or "
434+
"findings when left out."),
435+
_f("result_url", "string", False, "Where the result can be read."),
436+
_f("scanned_at", "string", False, "When it ran, a date or ISO time. Now when "
437+
"left out. Not in the future."),
438+
_f("counts", "object", False, "Findings per severity."),
439+
_f("findings", "array", False, "Each with title, severity, url, evidence, cwe "
440+
"and remediation. Up to 3000."),
441+
_f("scan_id", "string", False, "The scan's id on the server."),
442+
_f("note", "string", False, "Anything else."),
443+
]
444+
445+
446+
def _external_endpoints() -> list:
447+
asset_id = _p("asset_id", "The site's id.", where="path", kind="integer", example=12)
448+
scan_id = _p("scan_id", "The scan's id.", where="path", kind="integer", example=40)
449+
return [
450+
_ext("ext-list", "GET", "/external/assets", "List sites",
451+
"Every site with its telemetry, last dynamic scan, the repositories "
452+
"behind it and their open findings, and the combined risk.",
453+
[_p("q", "Search the host and application."),
454+
_p("host", "Exactly this host."), _p("application", "Exactly this application."),
455+
_p("tag", "Only sites with this tag."),
456+
_p("severity", "Only sites whose last scan was this bad.",
457+
choices=["critical", "high", "medium", "low", "info", "none"])] + PAGING,
458+
example={"total": 1, "limit": 100, "offset": 0, "assets": [_ASSET]}),
459+
_ext("ext-add", "POST", "/external/assets", "Add a site",
460+
"Adds a domain or subdomain and reads its telemetry. Answers 200 with "
461+
"the existing site when it is already listed.",
462+
body=[_f("host", "string", True, "The host name. A whole address is fine."),
463+
_f("application", "string", False, "The application it serves."),
464+
_f("tags", "array", False, "Application tags."),
465+
_f("no_scan", "boolean", False, "True when it must never be scanned, "
466+
"a SaaS product for example."),
467+
_f("no_scan_reason", "string", False, "Why not."),
468+
_f("notes", "string", False, "Anything else."),
469+
_f("telemetry", "boolean", False, "False to skip the first read.")],
470+
body_example={"host": "shop.example.com", "application": "Storefront",
471+
"tags": ["ecommerce"]}, example=_ASSET),
472+
_ext("ext-get", "GET", "/external/assets/{asset_id}", "One site",
473+
"The same as one row of the list.", [asset_id], example=_ASSET),
474+
_ext("ext-patch", "PATCH", "/external/assets/{asset_id}", "Change a site",
475+
"Change the application, tags, notes or no scan. Fields left out "
476+
"stay as they are.", [asset_id],
477+
body=[_f("application", "string", False, "The application."),
478+
_f("tags", "array", False, "Replaces the tags."),
479+
_f("notes", "string", False, "Notes."),
480+
_f("no_scan", "boolean", False, "Never scan it."),
481+
_f("no_scan_reason", "string", False, "Why not.")],
482+
body_example={"no_scan": True, "no_scan_reason": "Vendor hosted"},
483+
example=_ASSET),
484+
_ext("ext-delete", "DELETE", "/external/assets/{asset_id}", "Remove a site",
485+
"Removes the site and its scan history. 409 while a scan runs.",
486+
[asset_id], example={}),
487+
_ext("ext-telemetry", "POST", "/external/assets/{asset_id}/telemetry",
488+
"Read telemetry again",
489+
"Queues a fresh read: DNS, ports, TLS, headers and a screenshot. "
490+
"Passive for a site marked no scan.", [asset_id],
491+
example={"queued": True}),
492+
_ext("ext-scans", "GET", "/external/assets/{asset_id}/scans", "A site's scans",
493+
"Every dynamic scan of the site, newest first.", [asset_id] + PAGING,
494+
example={"total": 1, "limit": 100, "offset": 0, "scans": [_SCAN]}),
495+
_ext("ext-scan", "GET", "/external/assets/{asset_id}/scans/{scan_id}",
496+
"One scan", "One scan with its findings, which engines ran, and "
497+
"the AI triage and what it cost.", [asset_id, scan_id],
498+
example=dict(_SCAN, findings=[{"title": "Git repository metadata is served",
499+
"severity": "high",
500+
"url": "https://shop.example.com/.git/HEAD"}])),
501+
_ext("ext-record", "POST", "/external/assets/{asset_id}/scans", "Record a scan",
502+
"Records a scan run elsewhere: its severity, result address, when it "
503+
"ran and which server ran it. Becomes the site's last scan when it "
504+
"is the newest.", [asset_id], body=_RECORD_BODY,
505+
body_example={"server": "nextgen-dast", "severity": "high",
506+
"result_url": "https://dast.example.com/assessment/88",
507+
"scanned_at": "2026-11-01T02:00:00Z",
508+
"counts": {"high": 1, "medium": 3}}, example=_SCAN),
509+
_ext("ext-record-host", "POST", "/external/scans", "Record a scan by host",
510+
"The same, by host name, for a server that does not know the ids "
511+
"here. 404 for a host that is not listed, unless create is true.",
512+
body=[_f("host", "string", True, "The host that was scanned."),
513+
_f("create", "boolean", False, "Add the host when it is not listed.")]
514+
+ _RECORD_BODY,
515+
body_example={"host": "shop.example.com", "server": "nextgen-dast",
516+
"severity": "medium", "create": True}, example=_SCAN),
517+
_ext("ext-start", "POST", "/external/assets/{asset_id}/dast", "Start a scan",
518+
"Starts a scan on this server or on a nextgen-dast server. 409 for a "
519+
"site that may not be scanned or already has one running. Standard "
520+
"and up, and any DAST server, send attack payloads and need confirm.",
521+
[asset_id],
522+
body=[_f("profile", "string", False, "baseline, quick, standard, "
523+
"thorough or premium."),
524+
_f("server", "string", False, "builtin, or a nextgen-dast server's name."),
525+
_f("confirm", "boolean", False, "True to say you may scan it."),
526+
_f("ai", "boolean", False, "True to have the model triage the "
527+
"findings when it ends. Counted on "
528+
"the AI cost page.")],
529+
body_example={"profile": "quick", "ai": True},
530+
example=dict(_SCAN, source="builtin", status="queued", severity="")),
385531
]
386532

387533

0 commit comments

Comments
 (0)