Observability

Observability & Distributed Tracing

Every significant work unit in GloomProxy emits OpenTelemetry-style spans that are assembled into traces, persisted, and visualized as a waterfall. Use it to understand execution timing, debug failed scans, or trace attack chains across subsystems.

Trace model

Trace (trace_id = job_id / run_id / result_id)
  └── Root span   (no parent_span_id)
        ├── Child span  (parent_span_id = root)
        │     └── SpanEvents  (progress logs attached to the span)
        └── Child span

Every span carries:

Field	Description
trace_id	Groups all spans for one work unit
span_id	Unique span identifier
parent_span_id	Links to parent span (null for root)
subsystem	`scanner
status	`ok
duration_ms	Wall time in milliseconds
attributes	Key-value metadata (sensitive fields auto-redacted)
span_events	Timestamped progress log entries
error	Error message if status = error

Subsystems traced automatically

Subsystem	What generates spans
scanner	Scan job lifecycle + per-finding child spans
replay	Replay execution start → complete/error
workflow	Workflow run lifecycle + step events
proxy	HTTP request + response intercept events
auth	Session refresh and update events
worker	Job assignment + worker offline events
plugin	Plugin discovery at startup

Sensitive field redaction

Attributes matching known sensitive keys (password, token, authorization, cookie, api_key, credentials, etc.) are replaced with [REDACTED] before persistence. Raw credentials never appear in spans.

Performance

The collector uses a bounded async queue (20,000 spans max) with batch writes (200 spans per flush, every 2 seconds). Intake is non-blocking — spans are dropped on queue full rather than blocking the event loop. Monitor the drop rate at GET /api/observability/collector.

Observability API

Method	Path	Description
GET	/api/observability/traces	List traces (?subsystem=&status=&limit=)
GET	/api/observability/traces/{id}	All spans in a trace
GET	/api/observability/traces/{id}/timeline	Waterfall timeline
GET	/api/observability/spans/{id}/lineage	Ancestor chain up to root
GET	/api/observability/spans/{id}/descendants	All child spans
GET	/api/observability/stats	Aggregate span counts per subsystem
GET	/api/observability/collector	Collector health (queue, drop rate)