Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,12 @@ CIBERWEBSCAN_CACHE_ENABLED=true # -> cache.enabled
CIBERWEBSCAN_API_AUTH_SERVER_SECRET= # -> api.auth.server_secret (HMAC for logs only)
CIBERWEBSCAN_DOWNLOAD_REQUIRE_SAME_USER=true # -> download.require_same_user (token bound to issuing key)

# Reverse proxies (client identification). Empty = X-Forwarded-For always ignored.
# Declare proxy endpoints only; the proxy must overwrite or append (rightmost)
# X-Forwarded-For. The API runs Uvicorn with proxy_headers=False, so
# FORWARDED_ALLOW_IPS has no effect on client identification.
# CIBERWEBSCAN_API_TRUSTED_PROXIES=127.0.0.1 # -> api.trusted_proxies (IPs/CIDRs)

# Analysis / CVE API keys
# NVD_API_KEY / VULNERS_API_KEY are read directly by the CVE clients, or use:
# CIBERWEBSCAN_ANALYSIS_CVE_NVD_API_KEY=...
Expand Down
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@ X-API-Key: <your-api-key>
### Middleware

- **RequestLoggingMiddleware**: Logs method, path, status, duration for every request
- **RateLimitingMiddleware**: In-memory sliding window per IP, returns 429 with `Retry-After` header
- **RateLimitingMiddleware**: In-memory per-client counter in a fixed 60s window, returns 429 with `Retry-After` header. The client identity is the connection peer; `X-Forwarded-For` is honoured only when that peer is declared in `api.trusted_proxies` (empty by default = header always ignored), resolved right-to-left hop by hop (`api/client_ip.py`, shared with auth log attribution)

### Models

Expand Down Expand Up @@ -424,6 +424,7 @@ def test_analyze_route(mock_service_class):
| API routes | `src/ciberwebscan/api/routes/` |
| API models | `src/ciberwebscan/api/models/requests.py`, `responses.py` |
| API auth | `src/ciberwebscan/api/auth.py` |
| Client IP resolver | `src/ciberwebscan/api/client_ip.py` |
| API middleware | `src/ciberwebscan/api/middleware.py` |
| API download helper | `src/ciberwebscan/api/helpers/download_helper.py` |
| Pydantic config | `src/ciberwebscan/config/models.py` |
Expand Down
67 changes: 67 additions & 0 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -922,6 +922,8 @@ each use:
`GET /api/download/{token}`, so toggling it applies to the next download request.
- **API auth `server_secret`** (`api.auth.server_secret`): read per request but used only for log
identifiers; the auto-generated default is fixed for the lifetime of the process.
- **Trusted proxies** (`api.trusted_proxies`): re-read on every request when resolving the client
IP for rate limiting and auth log attribution, so proxies can be declared or removed at runtime.
- **HTTP client settings** (timeouts, retries, rate limits): apply to HTTP clients created after
the change; operations already in flight keep the settings they started with.
- **Scan defaults** (scraping, analysis, attack options): apply to operations started after the
Expand Down Expand Up @@ -1005,6 +1007,9 @@ CIBERWEBSCAN_API_AUTH_SERVER_SECRET=""
CIBERWEBSCAN_API_RATE_LIMIT_ENABLED=true
CIBERWEBSCAN_API_RATE_LIMIT_REQUESTS_PER_MINUTE=60

# Client identification (reverse proxies allowed to set X-Forwarded-For)
CIBERWEBSCAN_API_TRUSTED_PROXIES="127.0.0.1"

# CORS
CIBERWEBSCAN_API_CORS_ORIGINS="http://localhost:3000,https://example.com"
CIBERWEBSCAN_API_CORS_ALLOW_CREDENTIALS=true
Expand All @@ -1027,6 +1032,68 @@ The API includes rate limiting to prevent abuse:
- **Configurable** via `CIBERWEBSCAN_API_RATE_LIMIT_REQUESTS_PER_MINUTE`
- **429 status code** returned when exceeded
- **Retry-After header** included in rate limit responses
- **Per-client counter**: each client identity has its own bucket (see
[Client identification](#client-identification-and-trusted-proxies) below)

### Client identification and trusted proxies

Clients are identified by their **connection peer address**. The
`X-Forwarded-For` header is **ignored by default** (secure default) and is only
honoured when the peer address is declared in `api.trusted_proxies`:

```bash
CIBERWEBSCAN_API_TRUSTED_PROXIES="127.0.0.1" # single proxy
CIBERWEBSCAN_API_TRUSTED_PROXIES="10.0.0.0/8,::1" # CIDR and IPv6 accepted
```

Rules applied (`ciberwebscan.api.client_ip.resolve_client_ip`, shared by the
rate limiter and auth log attribution):

1. Peer missing or not parseable as an IP (e.g. unix socket) → that fallback
identity is used and `X-Forwarded-For` is never consulted.
2. `api.trusted_proxies` empty (default) → the header is ignored.
3. Peer outside the trusted list → the header is ignored, even when the list
is non-empty.
4. Peer is a trusted proxy → the chain is walked **from right to left**; the
first hop not vouching for by a trusted proxy is the client. Every hop is
validated with `ipaddress` (IPv4, IPv6 and IPv4-mapped forms are
normalized); a malformed hop stops the walk and the last trusted hop wins.
Entries are validated as IP addresses or CIDR networks at configuration
time and stored in canonical network form.

Reverse proxy requirements for correct attribution:

- The proxy must **overwrite** `X-Forwarded-For` or **append** (rightmost) the
address it observed. Transparent/pass-through proxies that do not touch the
header must **not** be declared.
- Declare only the proxy's own endpoint addresses (`/32` / `/128`), never a
range that also contains end clients — a client inside a trusted range could
forge a chain that the walk would believe.

Effect of not configuring it behind a reverse proxy: all clients share the
proxy's peer address, so they share a single rate-limit bucket (60 rpm global)
and logs show the proxy IP. Spoofing is not possible either way.

Interaction with Uvicorn's proxy handling: both launch paths of this API
(`ciberwebscan api` and `python -m ciberwebscan.api.app`) start Uvicorn with
`proxy_headers=False`, so Uvicorn never rewrites `request.client` from
forwarded headers and `FORWARDED_ALLOW_IPS` is irrelevant — the resolver
above is the single authority for client identity. Deployments that start
the app under a different ASGI server (or a custom Uvicorn invocation) with
forwarded-header trust enabled — especially `FORWARDED_ALLOW_IPS=*`, which
installs the _leftmost_, client-controlled header value as the peer — void
this policy and are not supported. The `Forwarded` header (RFC 7239) is not
honoured by this API.

Invalid values: `api.trusted_proxies` entries must be valid IP addresses or
CIDR networks. Catch-all networks (`0.0.0.0/0`, `::/0`, and IPv4-mapped
ranges covering the whole mapped IPv4 space), IPv4-mapped networks wider
than `/96`, and IPv6 zone identifiers (`fe80::1%eth0`) are rejected: they
would either trust every peer as a proxy or over-match silently. Any
invalid value at startup is reported loudly and, like any
invalid configuration value, makes the loader fall back to the default
configuration (see [Configuration](CONFIGURATION.md)); `PUT /api/config` and
`ciberwebscan config set` validate without mutating.

### CORS

Expand Down
54 changes: 53 additions & 1 deletion docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Examples that work out of the box:
- `CIBERWEBSCAN_ANALYSIS_CVE_NVD_API_KEY` → `analysis.cve.nvd_api_key`
- `CIBERWEBSCAN_CACHE_MAX_SIZE_MB` → `cache.max_size_mb`
- `CIBERWEBSCAN_API_RATE_LIMIT_REQUESTS_PER_MINUTE` → `api.rate_limit.requests_per_minute`
- `CIBERWEBSCAN_API_TRUSTED_PROXIES` → `api.trusted_proxies` (comma-separated IPs/CIDRs)
- `CIBERWEBSCAN_API_CORS_ORIGINS` → `api.cors_origins`
- `CIBERWEBSCAN_API_CORS_ALLOW_CREDENTIALS` → `api.cors_allow_credentials`
- `CIBERWEBSCAN_API_CORS_ALLOW_METHODS` → `api.cors_allow_methods`
Expand Down Expand Up @@ -556,6 +557,7 @@ Configure the FastAPI server, authentication, and CORS settings.
"enabled": true,
"requests_per_minute": 60
},
"trusted_proxies": [],
"cors_origins": ["http://localhost:3000"],
"cors_allow_credentials": true,
"cors_allow_methods": ["GET", "POST", "PUT", "DELETE", "PATCH"],
Expand All @@ -575,6 +577,7 @@ Configure the FastAPI server, authentication, and CORS settings.
| `api.auth.server_secret` | `""` | Secret for HMAC log obfuscation only (see below) |
| `api.rate_limit.enabled` | `true` | Rate limiting enabled by default |
| `api.rate_limit.requests_per_minute` | `60` | Max requests per minute (1-10000) |
| `api.trusted_proxies` | `[]` | Reverse proxies allowed to set `X-Forwarded-For` |
| `api.cors_origins` | `[]` | Allowed CORS origins (empty = no cross-origin) |
| `api.cors_allow_credentials` | `false` | Allow credentials in cross-origin requests |
| `api.cors_allow_methods` | see below | HTTP methods allowed for CORS |
Expand All @@ -585,6 +588,54 @@ Configure the FastAPI server, authentication, and CORS settings.

**Default `cors_allow_headers`:** `["Authorization", "Content-Type", "X-API-Key"]`

#### `api.trusted_proxies` (client identification)

Clients of the API are identified by their **connection peer address**. The
`X-Forwarded-For` header is ignored unless the peer is declared in
`api.trusted_proxies`, so the default (`[]`) is the safe choice for direct
exposure: no header can forge a client identity, bypass the rate limiter, or
consume the quota attributed to another address.

- **Accepts**: IP addresses (IPv4/IPv6) or CIDR networks, as a list or a
comma-separated string. Entries are validated with `ipaddress` and stored in
canonical network form (`127.0.0.1` → `127.0.0.1/32`); host bits in a CIDR
are masked with a warning (`10.0.0.1/8` → `10.0.0.0/8`); IPv4-mapped IPv6
networks (`::ffff:10.0.0.0/104`) are converted to IPv4 (`10.0.0.0/8`).
- **Invalid values** (e.g. `not-an-ip`, `1.2.3.4/33`, catch-all networks
like `0.0.0.0/0` or `::/0`, IPv4-mapped ranges covering the whole mapped
IPv4 space such as `::ffff:10.0.0.0/96`, and IPv6 zone identifiers like
`fe80::1%eth0`) raise a validation
error. Via `PUT /api/config` or `config set` this means a 400 / non-zero
exit without mutating the configuration; at startup (config file or
environment), the loader's pre-existing policy applies: it logs an error and
runs on the **entire default configuration**.
- **Read per request** — changes apply without restarting the server (see
[Runtime changes vs restart-required
settings](#runtime-changes-vs-restart-required-settings)).
- **Trust model**: declare only the proxy endpoints (e.g. `127.0.0.1`,
`10.0.0.1/32`). A range that also contains end clients lets a client inside
it forge a forwarded chain that the right-to-left walk would believe. Only
proxies that **overwrite** or **append** (rightmost) `X-Forwarded-For` may be
declared; pass-through proxies must not be.
- **Behind a reverse proxy without configuring this**: every request is
attributed to the proxy's own address, so all users share a single
rate-limit bucket and logs name the proxy — no spoofing, but quota sharing.
- **Uvicorn interaction**: both launch paths (`ciberwebscan api` and
`python -m ciberwebscan.api.app`) start Uvicorn with `proxy_headers=False`,
so Uvicorn never rewrites `request.client` from forwarded headers and
`FORWARDED_ALLOW_IPS` is irrelevant — the `api.trusted_proxies` policy is
the single authority. Running the app under a different ASGI server with
forwarded-header trust enabled (especially `FORWARDED_ALLOW_IPS=*`, which
installs the client-controlled leftmost header value as the peer) voids
this policy and is not supported. The `Forwarded` header is ignored.

```yaml
api:
trusted_proxies:
- 127.0.0.1
- 10.0.0.0/8
```

#### `api.auth.server_secret` (log obfuscation only)

`api.auth.server_secret` (default `""`) is used **only** to obfuscate API keys in logs: log identifiers are derived from HMAC-SHA256 of the key under this secret. It takes **no part in authorization** — the authenticated identity (`GET /api/auth/me`) and download-token ownership are derived from a stable key identifier instead, so downloads work regardless of this value.
Expand Down Expand Up @@ -880,7 +931,8 @@ configuration object in place, so every `get_config()` consumer sees them immedi

- **Applied at next use (no restart):** `api.auth.api_keys` (API authentication is checked per
request), `api.auth.server_secret` (log obfuscation only, read per request — the auto-generated
default stays stable until the process restarts) and `download.require_same_user` (checked on
default stays stable until the process restarts), `api.trusted_proxies` (client identity for
rate limiting and auth logs is resolved per request), and `download.require_same_user` (checked on
every `GET /api/download/{token}`), HTTP client settings used when a client is created for a new
operation, and scraping/analysis/attack defaults read per scan.
- **Require a restart:** `api.cors.*`, `api.rate_limit.*`, `download.cleanup_interval_seconds`,
Expand Down
1 change: 1 addition & 0 deletions src/ciberwebscan/api/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,4 +153,5 @@ async def general_exception_handler(
port=app_config.api.port,
reload=True,
log_level=app_config.logging.level.lower(),
proxy_headers=False,
)
8 changes: 3 additions & 5 deletions src/ciberwebscan/api/auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
from fastapi.security import APIKeyHeader
from pydantic import BaseModel

from ciberwebscan.api.client_ip import resolve_client_ip
from ciberwebscan.config.loader import get_config
from ciberwebscan.utils.logging import _resolve_server_secret, mask_key_for_logging

Expand Down Expand Up @@ -147,11 +148,8 @@ async def verify_api_key(


def _get_client_ip(request: Request) -> str:
"""Extract client IP from request, handling proxies."""
forwarded = request.headers.get("X-Forwarded-For", "").split(",")[0].strip()
if forwarded:
return forwarded
return request.client.host if request.client else "unknown"
"""Resolve client IP for log attribution using the shared proxy policy."""
return resolve_client_ip(request)


async def get_current_user(
Expand Down
137 changes: 137 additions & 0 deletions src/ciberwebscan/api/client_ip.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
"""
Shared client IP resolution for the API layer.

Client identity comes from the connection peer (``request.client``).
``X-Forwarded-For`` is only consulted when the peer belongs to the declared
``api.trusted_proxies`` list, in which case the forwarded chain is walked from
right to left until the first address that a trusted proxy did not vouch for.
With no trusted proxies configured (the default) the header is always ignored.
"""

from __future__ import annotations

import ipaddress
import logging
from collections.abc import Callable

from fastapi import Request

from ciberwebscan.config.loader import get_config

logger = logging.getLogger(__name__)

_UNKNOWN_CLIENT = "unknown"

_IPv4Address = ipaddress.IPv4Address
_IPv6Address = ipaddress.IPv6Address
_IPv4Network = ipaddress.IPv4Network
_IPv6Network = ipaddress.IPv6Network

_ClientAddress = _IPv4Address | _IPv6Address
_ClientNetwork = _IPv4Network | _IPv6Network


def _unwrap_mapped(address: _ClientAddress) -> _ClientAddress:
"""Return the IPv4 form of an IPv4-mapped IPv6 address (else itself)."""
if isinstance(address, _IPv6Address) and address.ipv4_mapped is not None:
return address.ipv4_mapped
return address


def _parse_ip(value: str) -> _ClientAddress | None:
"""Parse a single hop, or return None when it is not a valid IP."""
try:
return _unwrap_mapped(ipaddress.ip_address(value.strip()))
except ValueError:
return None


def _trusted_networks() -> list[_ClientNetwork]:
"""Parse ``api.trusted_proxies`` for this request (fail closed)."""
try:
entries = get_config().api.trusted_proxies
except Exception:
# Any unexpected config failure must not turn into a 500 or an
# open header: empty list = X-Forwarded-For stays ignored.
logger.warning(
"Could not load api.trusted_proxies; ignoring X-Forwarded-For",
exc_info=True,
)
return []
networks: list[_ClientNetwork] = []
for entry in entries:
try:
networks.append(ipaddress.ip_network(str(entry), strict=False))
except ValueError:
logger.warning("Ignoring invalid api.trusted_proxies entry: %r", entry)
Comment thread
HC-ONLINE marked this conversation as resolved.
Dismissed
return networks


def _forwarded_for(request: Request) -> str:
"""Join all X-Forwarded-For header lines (Starlette semantics)."""
headers = request.headers
# Starlette's Headers exposes getlist(); mapping-style test doubles
# may only provide get().
getlist: Callable[[str], list[str]] | None = getattr(headers, "getlist", None)
if getlist is not None:
return ", ".join(getlist("x-forwarded-for"))
value = headers.get("X-Forwarded-For", "")
return value if isinstance(value, str) else ""


def resolve_client_ip(request: Request) -> str:
"""
Resolve the client identity for rate limiting and log attribution.

Args:
request: The incoming request.

Returns:
The client IP as a string. The connection peer wins whenever
``X-Forwarded-For`` cannot be trusted: empty ``api.trusted_proxies``
(default), peer outside that list, or peer not parseable as an IP
(in which case the shared ``"unknown"`` bucket is used rather than
echoing a non-IP peer string). Behind trusted proxies the chain is
validated hop by hop from right to left and a malformed entry stops
the walk (last trusted hop wins).
"""
peer = request.client.host if request.client else None
peer_ip = _parse_ip(peer) if isinstance(peer, str) and peer else None
if peer_ip is None:
# No usable peer (unix socket, test client, missing scope): never
# consult a client-controlled header for identity, and never echo
# an arbitrary peer string (it could be client-controlled under a
# misconfigured upstream proxy).
return _UNKNOWN_CLIENT

forwarded = _forwarded_for(request)
if not forwarded.strip():
return str(peer_ip)

trusted = _trusted_networks()
if not trusted:
logger.debug(
"Ignoring X-Forwarded-For: no trusted proxies configured (%s)", peer_ip
)
return str(peer_ip)
if not any(peer_ip in network for network in trusted):
logger.debug(
"Ignoring X-Forwarded-For: peer %s is not a trusted proxy", peer_ip
)
return str(peer_ip)

# Peer is a declared proxy: walk the chain right to left (appended hops
# are written by our proxies; the client controls only the left end).
result = str(peer_ip)
for entry in reversed(forwarded.split(",")):
entry = entry.strip()
if not entry:
break
hop = _parse_ip(entry)
if hop is None:
# Malformed hop: stop before touching client-controlled data.
break
if not any(hop in network for network in trusted):
return str(hop)
result = str(hop)
return result
Loading
Loading