Webhook delivery service

.gitignore

@@ -13,3 +13,9 @@ issue.md
 
 # JavaScript dependencies
 node_modules/
+**/node_modules/
+**/coverage/
+
+# Next.js build output
+.next/
+**/.next/

README.md

@@ -27,7 +27,10 @@ Utility-contracts/
 │   │   ├── src/test.rs             # Test suite
 │   │   └── Cargo.toml
 │   └── price_oracle/               # Price oracle contract
+├── webhook-delivery-service/       # High-performance off-chain Webhook service with retry & SSRF shielding (TS)
 ├── meter-simulator/                # Device simulator (JS)
+├── usage-dashboard/                # Real-time Next.js analytics & Webhook monitor dashboard
+├── docs/                           # Architecture, deployment and operational runbooks
 ├── examples/                       # Usage examples
 ├── scripts/                        # Deployment scripts
 ├── .github/workflows/ci.yml        # CI pipeline
@@ -36,6 +39,14 @@ Utility-contracts/
 └── EMERGENCY_RUNBOOK.md            # Emergency procedures
 ```
 
+### Webhook Delivery Service
+
+An enterprise-grade, high-performance off-chain delivery daemon for real-time Soroban alerts (e.g. `LowBalanceAlert`, device tampers).
+- **Performance**: `< 100ms` P99 ingestion latency target via an asynchronous event-driven memory queue.
+- **Robust Security**: Includes HMAC-SHA256 and Ed25519 signature headers, strict replay protection windowing, and thorough SSRF IP/DNS blacklisting.
+- **Resiliency**: Built-in exponential backoff retry schedules with full randomized jitter to survive downstream subscriber downtimes and network drops.
+- **Operational Guides**: See [WEBHOOK_ARCHITECTURE.md](docs/WEBHOOK_ARCHITECTURE.md), [WEBHOOK_DEPLOYMENT.md](docs/WEBHOOK_DEPLOYMENT.md), and [WEBHOOK_RUNBOOK.md](docs/WEBHOOK_RUNBOOK.md).
+
 ## Architecture
 
 ### Variable Rate Tariffs

docs/WEBHOOK_ARCHITECTURE.md

@@ -0,0 +1,103 @@
+# Webhook Delivery Service Architecture
+
+This document describes the design and architecture of the enterprise-grade **Webhook Delivery Service** built for the Utility-Protocol suite.
+
+## 1. Overview and Problem Statement
+
+Decentralized smart contracts on Soroban cannot make direct out-of-band HTTP requests. To deliver real-time notifications (such as low balance alerts, tamper detection, or stream status updates) to external client servers and subscriber endpoints, an off-chain Webhook Delivery Service is required.
+
+### 1.1 Technical Bounds & SLA Targets
+- **Scope**: System-wide integration affecting all protocol components.
+- **Latency**: Critical ingestion paths must execute in **< 100ms P99**.
+- **Availability**: High availability target of **99.99% uptime**.
+- **Security**: Robust cryptographic signatures, strict replay attack protection, and rigorous Server-Side Request Forgery (SSRF) defense.
+
+---
+
+## 2. Architecture Diagram
+
+```
++--------------------------+
+|  Soroban Smart Contract  |
++------------+-------------+
+             | (Events/Alerts)
+             v
++------------+-------------+
+|    On-chain Poller /     |
+|   Webhook Generator      |
++------------+-------------+
+             |
+             | POST /webhooks
+             v
++------------+-------------------------------------------------------------+
+| Webhook Delivery Service (Node.js/TypeScript)                            |
+|                                                                          |
+|  +-----------------------+      +-------------------------------------+  |
+|  |   API Ingestion       | ---> |        Async In-Memory Queue        |  |
+|  |   (<10ms Response)    |      | (Maintains < 100ms P99 Latency)     |  |
+|  +-----------------------+      +------------------+------------------+  |
+|                                                    |                     |
+|                                                    v                     |
+|                                 +------------------+------------------+  |
+|                                 |       Background Dispatcher         |  |
+|                                 +------------------+------------------+  |
+|                                                    |                     |
+|            +---------------------------------------+------------------+  |
+|            |                                       |                  |  |
+|            v                                       v                  v  |
+|  +-------------------+                   +-------------------+  +-----+  |
+|  | Security Engine:  |                   | Retry Controller: |  | Prom|  |
+|  | SSRF Shield       |                   | Exponential       |  | Mets|  |
+|  | Ed25519/HMAC-SHA  |                   | Backoff & Jitter  |  +-----+  |
+|  +-------------------+                   +-------------------+           |
++------------+---------------------------------------+---------------------+
+             |
+             | Signed POST Request
+             v
++------------+-------------+
+|  Client Webhook Endpoint |
++--------------------------+
+```
+
+---
+
+## 3. Core Capabilities
+
+### 3.1 Asynchronous Delivery Engine (<100ms P99)
+- **Non-blocking Ingestion**: When an alert/webhook payload is received, the ingestion API immediately validates the payload schema, enqueues it to an in-memory queue, and returns `202 Accepted`. This decouples request ingestion from network delivery, guaranteeing extremely low response latency (typically **<10ms**).
+- **Asynchronous Dispatching**: An event-driven background loop processes the queue, executing up to configured maximum concurrent requests to avoid thread starvation or exhausting client connections.
+
+### 3.2 Secure Cryptographic Signatures
+To ensure authenticity and integrity, outgoing requests are cryptographically signed using two methods:
+1. **HMAC-SHA256**: Generates a signature hash using a shared secret.
+2. **Ed25519 Asymmetric Signatures**: Generates a signature of the message bytes using the Webhook Service's private key, which can be verified by the recipient using the service's public key.
+
+#### Custom Headers:
+- `X-Webhook-Timestamp`: POSIX epoch timestamp of the transmission.
+- `X-Webhook-Signature-256`: `t=<timestamp>,v1=<hmac-sha256-signature>`
+- `X-Webhook-Signature-Ed25519`: `t=<timestamp>,v1=<ed25519-signature>`
+
+### 3.3 Replay Attack Protection
+Recipient endpoints verify the signature and check that the difference between the `X-Webhook-Timestamp` and their current local clock is within a tolerance window (e.g., **5 minutes**). This prevents attackers from eavesdropping on and replaying historical messages.
+
+### 3.4 Strict SSRF (Server-Side Request Forgery) Defense
+The security engine validates destination URLs before dispatching requests:
+- **Private/Local IP Filtering**: Rejects URLs resolving to private subnets (`127.0.0.0/8`, `10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`, `169.254.169.254` metadata endpoint, etc.).
+- **Protocol Whitelisting**: Strictly restricts protocols to `http:` and `https:`.
+- **Port Whitelisting**: Rejects requests targeting non-standard administrative/internal ports.
+
+### 3.5 Exponential Backoff with Random Jitter
+Transient network drops, rate limits (HTTP 429), and short-term receiver outages are mitigated by a multi-attempt retry model:
+- **Retry Schedule**: Uses base backoff factor ($t_{backoff} = \min(t_{max}, t_{base} \times 2^{attempt})$).
+- **Full Jitter**: Prevents "thundering herd" issues by introducing randomized delay ($t_{jitter} = \text{random}(0, t_{backoff})$).
+- **Max Retries**: Defaulted to **5 attempts** before a webhook is classified as failed.
+
+---
+
+## 4. Monitoring & Metrics
+
+The service registers Prometheus counters and histograms to measure health indicators:
+- `webhook_delivery_attempts_total`: Total POST attempts, labeled by status code, target host, and attempt count.
+- `webhook_delivery_duration_seconds`: Histogram of endpoint response latency.
+- `webhook_queue_size_current`: Gauge representing current queue occupancy.
+- `webhook_failures_total`: Total dropped or exhausted delivery alerts.

docs/WEBHOOK_DEPLOYMENT.md

@@ -0,0 +1,98 @@
+# Webhook Delivery Service Deployment & Rollout Guide
+
+This document describes the deployment strategy, Blue-Green rollout procedure, and Canary Analysis parameters for the off-chain Webhook Delivery Service.
+
+---
+
+## 1. Rollout Overview
+Our goal is to deploy updates with **zero downtime** and verify performance targets ($<100\text{ms}$ P99 latency) under real-world traffic patterns before fully promoting new builds.
+
+### SLA & Quality Gates
+During rollouts, the deployment pipeline must automatically monitor the following quality gates:
+- **Error Rate**: $<0.01\%$ (HTTP 5xx responses).
+- **Latency**: P99 latency $<100\text{ms}$ on ingestion endpoints (`POST /webhooks`).
+- **CPU / Memory**: No memory leaks or sustained CPU utilization exceeding $80\%$.
+
+---
+
+## 2. Blue-Green Deployment Strategy
+
+We employ a classic Blue-Green deployment model to completely isolate the production environment ("Blue") from the staging environment of the upcoming release ("Green").
+
+```
+                  +------------------+
+                  |  Traffic Router  |
+                  |   (Nginx / ALB)  |
+                  +--------+---------+
+                           |
+             +-------------+-------------+
+             |                           |
+             v                           v
+     +---------------+           +---------------+
+     |  Active Blue  |           |  Idle Green   |
+     |   (v1.0.0)    |           |   (v1.1.0)    |
+     +---------------+           +---------------+
+```
+
+### Step 1: Deploy Green Infrastructure
+1. Provision a separate instance group or container fleet running the new version (Green).
+2. Configure the Green environment to use a dedicated staging endpoint (e.g. `http://green.webhook.internal`).
+3. Set up environment variables to match production, ensuring the Green database/caches are fully isolated or configured for backward-compatible schema changes.
+
+### Step 2: Green Health & Sanity Check
+Run an automated verification suite against the Green service before routing any live traffic:
+```bash
+# 1. Verify health endpoint
+curl -s -f http://green.webhook.internal/health
+
+# 2. Test simple ingestion call
+curl -s -X POST -H "Content-Type: application/json" \
+  -d '{"payload":{"event":"test"}, "url":"https://httpbin.org/post", "secret":"test_secret"}' \
+  http://green.webhook.internal/webhooks
+```
+
+### Step 3: Switch Traffic Router
+Once the Green environment passes all sanity tests:
+1. Update the load balancer (Nginx or ALB) configurations to redirect $100\%$ of incoming traffic from Blue to Green.
+2. Monitor active connections on Blue and allow a **5-minute graceful drain window** to complete any outstanding retry attempts or delivery backlogs.
+3. Shut down or idle the Blue infrastructure.
+
+---
+
+## 3. Canary Analysis Strategy
+
+For high-risk updates, a Canary deployment strategy is used to gradually shift traffic and observe behavior.
+
+```
+Traffic Shift:  [ 90% Production (Blue) ]  ===>  [ 10% Canary (Green) ]
+```
+
+### Step-by-Step Canary Rollout
+1. **Stage 1 (10% Traffic)**: Route $10\%$ of live webhook ingestion traffic to the new Green (Canary) instances.
+2. **Analysis Window (30 Minutes)**:
+   - Query Canary Prometheus metrics: `http://canary.webhook.internal/metrics`.
+   - Verify P99 latency:
+     ```promql
+     histogram_quantile(0.99, sum(rate(webhook_delivery_duration_seconds_bucket[5m])) by (le))
+     ```
+   - Verify success rates:
+     ```promql
+     sum(rate(webhook_delivery_attempts_total{status=~"2.."}[5m])) / sum(rate(webhook_delivery_attempts_total[5m])) * 100
+     ```
+3. **Stage 2 (50% Traffic)**: Increase traffic allocation to $50\%$ if error rates are zero and P99 latency is $<100\text{ms}$. Monitor for an additional 15 minutes.
+4. **Stage 3 (100% Traffic)**: Promote Canary to full production (100% traffic) and deprecate the old version.
+
+---
+
+## 4. Rollback Plan
+
+If any quality gate is breached (e.g., P99 latency spikes above 100ms or 5xx error rates exceed 0.1%), trigger an **instantaneous rollback**:
+
+```bash
+# 1. Immediate traffic reversion via ALB/Nginx configuration change
+nginx -s reload -c /etc/nginx/nginx-blue-stable.conf
+
+# 2. Force drain Green queue (if needed, transfer pending jobs back to Blue)
+# 3. Terminate Green container group
+```
+Using Nginx or ALB configuration symlinks allows traffic redirection to take place in **<1 second**, minimizing blast radius during unexpected issues.

docs/WEBHOOK_RUNBOOK.md

@@ -0,0 +1,81 @@
+# Webhook Delivery Service Runbook & Disaster Recovery
+
+**Service Name**: Webhook Delivery Service
+**Criticality**: Tier-1 (High Availability, 99.99% Target)
+**P99 Latency SLA**: < 100ms Ingestion
+
+---
+
+## 1. Diagnostics & Monitoring Checklists
+
+If an incident or alert is triggered, use these fast-path diagnostic endpoints to isolate the root cause:
+
+### Step 1: Health Status Query
+```bash
+curl -s http://webhook-service.internal/health
+```
+**Expected Output**:
+```json
+{ "status": "UP", "queueSize": 0 }
+```
+
+### Step 2: Queue size and Summary Statistics
+```bash
+curl -s http://webhook-service.internal/stats
+```
+*Check `queueSize` and `successRate`. If `queueSize` is climbing (> 1000) and `successRate` is dropping (< 95%), a downstream receiver or network partition is likely causing delivery failures.*
+
+### Step 3: View Recent Failure Logs
+```bash
+curl -s http://webhook-service.internal/logs | grep -E '"status":"FAILED"|"status":"RETRYING"'
+```
+
+---
+
+## 2. Emergency Escalation Workflows
+
+### Scenario 1: Queue Backlog Accumulation (`queueSize` climbing rapidly)
+- **Symptom**: `webhook_queue_size_current` is sustained above 500.
+- **Root Cause**: Downstream client webhook endpoints are offline, rate-limiting requests (HTTP 429), or experiencing extreme latencies, clogging the background processor thread.
+- **Remedy Actions**:
+  1. Increase horizontal scale (increase replica count of the webhook containers) to expand total delivery concurrency.
+  2. Increase the maximum attempts limit temporarily or lower the HTTP timeout value from 5s to 2s to prune slow connections faster.
+  ```bash
+  # Example to scale replicas in Kubernetes
+  kubectl scale deployment webhook-delivery-service --replicas=10
+  ```
+
+### Scenario 2: High Memory / OOM Crashing
+- **Symptom**: Webhook process crashes with "Out of Memory" or CPU utilization is constantly at 100%.
+- **Root Cause**: Memory leak in the in-memory queue or too many pending retries under extreme ingestion spikes.
+- **Remedy Actions**:
+  1. Terminate container and force restart to release leaked memory buffer.
+  2. Implement rate limiting on ingestion endpoints to protect the memory boundaries.
+  3. Deploy a permanent out-of-process persistent queue (like Redis or RabbitMQ) if traffic spikes are consistently exceeding the memory boundaries.
+
+---
+
+## 3. Webhook Secret Key Rotation Procedure
+
+To maintain high security, shared webhook secrets must be rotated every **180 days**, or immediately upon key compromise:
+
+### Step 1: Generate New Shared Secret
+Generate a secure, cryptographically random key:
+```bash
+node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
+# Output example: b98b816a13d9cfc892809e20a2e39958e2bfb73be5c6138be6e3557e49c7bc29
+```
+
+### Step 2: Implement "Double-Signing" Transition Period
+To prevent breaking integrations during rotation:
+1. Configure the Webhook Service to temporarily sign payloads with **both** the old secret and the new secret.
+2. Provide the new secret key to the subscriber.
+3. Once the subscriber updates their webhook receiver to verify using the new secret, remove the old secret from the active signing list.
+
+---
+
+## 4. SSRF Prevention Audits
+
+To ensure the safety of the off-chain system, the SSRF (Server-Side Request Forgery) engine must be audited after any networking or DNS upgrades:
+1. Verify that the URL parser correctly flags subnets by running integration tests.
+2. Inspect server firewalls, ensuring egress traffic is strictly barred from routing to cloud provider private IP ranges and internal Kubernetes API service accounts.