Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,8 @@ CIBERWEBSCAN_CACHE_ENABLED=true # -> cache.enabled
# API auth / downloads
CIBERWEBSCAN_API_AUTH_SERVER_SECRET= # -> api.auth.server_secret (HMAC for logs only)
CIBERWEBSCAN_DOWNLOAD_REQUIRE_SAME_USER=true # -> download.require_same_user (token bound to issuing key)
CIBERWEBSCAN_DOWNLOAD_MAX_TOKENS=100 # -> download.max_tokens (max live tokens in the in-memory registry)
CIBERWEBSCAN_DOWNLOAD_MAX_TOTAL_BYTES=1073741824 # -> download.max_total_bytes (byte budget for cached exports)

# Reverse proxies (client identification). Empty = X-Forwarded-For always ignored.
# Declare proxy endpoints only; the proxy must overwrite or append (rightmost)
Expand Down
13 changes: 11 additions & 2 deletions docs/API.md
Original file line number Diff line number Diff line change
Expand Up @@ -648,6 +648,13 @@ that issued it. The `401` case above (`token belongs to different user`) is cont
share tokens across API keys, because `false` disables that access control entirely. See
[CONFIGURATION.md — Download](CONFIGURATION.md#download).

**When no token is issued:** export-enabled POST responses return `200` with
`download_token: null` (and no `download_url`) while the export itself stays available on
disk. This happens when downloads are disabled (`download.enabled=false`) or when the
registry budget is exhausted (`download.max_tokens` / `download.max_total_bytes` — see
[CONFIGURATION.md — Download](CONFIGURATION.md#download)). A `GET /api/download/{token}`
against a disabled service returns `503`.

**Example:**

```bash
Expand Down Expand Up @@ -1041,9 +1048,11 @@ CIBERWEBSCAN_API_CORS_EXPOSE_HEADERS="X-Custom-Header"

# Downloads
CIBERWEBSCAN_DOWNLOAD_ENABLED=true
CIBERWEBSCAN_DOWNLOAD_TOKEN_EXPIRY_MINUTES=60
CIBERWEBSCAN_DOWNLOAD_MAX_RETRY_ATTEMPTS=3
CIBERWEBSCAN_DOWNLOAD_RETENTION_SECONDS=1800
CIBERWEBSCAN_DOWNLOAD_MAX_RETRIES=3
CIBERWEBSCAN_DOWNLOAD_REQUIRE_SAME_USER=true
CIBERWEBSCAN_DOWNLOAD_MAX_TOKENS=100
CIBERWEBSCAN_DOWNLOAD_MAX_TOTAL_BYTES=1073741824
```

### Rate Limiting
Expand Down
39 changes: 29 additions & 10 deletions docs/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -481,7 +481,9 @@ Configure the download tokens returned by export-enabled API requests (`download
"download": {
"enabled": true,
"retention_seconds": 1800,
"require_same_user": true
"require_same_user": true,
"max_tokens": 100,
"max_total_bytes": 1073741824
}
}
```
Expand All @@ -505,17 +507,34 @@ The value is read from the configuration on every download request, so it applie

The returned-attempt refund is capped: it can never push a token above `download.max_retries`, and it never recreates a deleted token. If you lower `max_retries` at runtime, refunds on tokens created under the previous (higher) value may be refused, leaving their counter unchanged — attempts are never inflated beyond the current configuration.

#### Registry budget: `download.max_tokens` / `download.max_total_bytes`

Each download token keeps a full in-memory copy of its export file, so total memory grows with every live token. Two budget keys bound that:

- **`download.max_tokens`** (default `100`, range 1-10000) — maximum number of live tokens in the registry.
- **`download.max_total_bytes`** (default `1073741824` = 1 GiB, range 1 MiB-1024 GiB) — maximum total bytes held by all live tokens.

Admission is atomic and happens **before** the export file is read: a request whose token would exceed either limit is rejected without loading the file into memory. When a budget is exhausted:

- The export itself still completes and is written to disk normally.
- The response returns `200` with `download_token: null` (no token, no `download_url`).
- Budget is freed when a token is downloaded (deleted), expires (`retention_seconds`), or is removed by the cleanup job — there is no eviction of live tokens.

Raise these values only when the host has the RAM for them: `max_total_bytes` is a hard cap on cached-export memory across the whole process. The rejection is logged as a warning identifying the exhausted limit.

#### Default values (quick reference)

| Key | Default | Description |
| ----------------------------------- | --------: | ------------------------------------------------------- |
| `download.enabled` | `true` | Download service enabled (tokens + `GET /api/download`) |
| `download.retention_seconds` | `1800` | Seconds before a download token expires (60-86400) |
| `download.max_file_size_mb` | `500` | Max export file size accepted for download (MB) |
| `download.max_retries` | `3` | Max download attempts per token (1-10) |
| `download.cleanup_interval_seconds` | `300` | Cleanup scheduler interval (requires restart) |
| `download.require_same_user` | `true` | Bind each download token to the API key that issued it |
| `download.stream_chunk_size` | `1048576` | Streaming chunk size in bytes |
| Key | Default | Description |
| ----------------------------------- | -----------: | ------------------------------------------------------- |
| `download.enabled` | `true` | Download service enabled (tokens + `GET /api/download`) |
| `download.retention_seconds` | `1800` | Seconds before a download token expires (60-86400) |
| `download.max_file_size_mb` | `500` | Max export file size accepted for download (MB) |
| `download.max_retries` | `3` | Max download attempts per token (1-10) |
| `download.cleanup_interval_seconds` | `300` | Cleanup scheduler interval (requires restart) |
| `download.require_same_user` | `true` | Bind each download token to the API key that issued it |
| `download.stream_chunk_size` | `1048576` | Streaming chunk size in bytes |
| `download.max_tokens` | `100` | Max live download tokens held in memory (1-10000) |
| `download.max_total_bytes` | `1073741824` | Byte budget across all live tokens (1 MiB-1024 GiB) |

### Cache

Expand Down
10 changes: 10 additions & 0 deletions src/ciberwebscan/api/helpers/download_helper.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

from fastapi import Request

from ciberwebscan.config.loader import get_config
from ciberwebscan.export import EXPORT_FORMATS
from ciberwebscan.services.base import ServiceResult
from ciberwebscan.services.download_service import DownloadService
Expand Down Expand Up @@ -70,6 +71,15 @@ def enrich_response_with_token(
if result.export_path is None:
return result.data, None, None

# Downloads disabled is an expected state, not a failure: skip the
# service call so the (already gated) export does not emit a warning
# per request. The service remains the authoritative admission gate.
if not get_config().download.enabled:
download_service.logger.debug(
"Download token skipped: downloads are disabled (download.enabled=false)"
)
return result.data, None, None

# Generate token from the exported file with the real export format
# (previously hardcoded to "json", which mislabeled every download).
token_result = download_service.generate_download_token(
Expand Down
4 changes: 4 additions & 0 deletions src/ciberwebscan/config/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,10 @@ class DownloadConfig(BaseModel):
cleanup_interval_seconds: Annotated[int, Field(ge=60, le=3600)] = 300
require_same_user: bool = True
stream_chunk_size: Annotated[int, Field(ge=1024, le=10 * 1024 * 1024)] = 1024 * 1024
max_tokens: Annotated[int, Field(ge=1, le=10000)] = 100
max_total_bytes: Annotated[int, Field(ge=1_048_576, le=1_099_511_627_776)] = (
1_073_741_824
)


# =============================================================================
Expand Down
Loading
Loading