diff --git a/README.md b/README.md index 2a092ec..fc2eabc 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,11 @@ Provider-initiated, device-completed firmware updates with Ed25519 signature ver Verified via 15 property tests with 100+ randomized cases each, covering pause/resume cycles, rounding direction, and overflow protection. + +### Chaos Engineering in Staging + +Staging resilience exercises are governed by the [Chaos Engineering Testing Blueprint](docs/runbooks/chaos-engineering-staging.md). The blueprint defines approved fault scenarios, security guardrails, P99 and availability SLOs, monitoring requirements, and blue-green/canary rollout steps for chaos-enabled staging deployments. + ### Security Properties - **Nonce sync** prevents replay attacks on IoT heartbeats diff --git a/docs/runbooks/chaos-engineering-staging.md b/docs/runbooks/chaos-engineering-staging.md new file mode 100644 index 0000000..5fc5d26 --- /dev/null +++ b/docs/runbooks/chaos-engineering-staging.md @@ -0,0 +1,161 @@ +# Chaos Engineering Testing Blueprint for Staging + +**Environment:** Staging only +**Status:** Required before production chaos experiments +**Performance SLO:** Critical contract and simulator paths must remain below 100 ms P99 during steady-state and recover below 100 ms P99 after each experiment. +**Availability SLO:** Staging services must maintain 99.99% monthly availability outside approved experiment windows. +**Security gate:** Every experiment plan must pass security review before execution. + +## 1. Goals and guardrails + +Chaos testing validates that the Utility Protocol can tolerate realistic service, network, oracle, simulator, dashboard, and contract-adapter failures without silently violating billing, settlement, or safety invariants. + +The staging program has four goals: + +1. Prove that critical paths recover to the performance SLO after injected faults. +2. Confirm that emergency controls, pauses, velocity limits, oracle fallbacks, and runbooks are actionable. +3. Verify observability by requiring alerts, dashboard panels, and incident notes for every experiment. +4. Exercise blue-green and canary deployment procedures before any production rollout. + +Non-negotiable guardrails: + +- Never run destructive chaos experiments against mainnet or production identities. +- Use staging-only signing keys, meters, dashboards, and notification channels. +- Keep a named experiment commander, communications lead, and rollback approver online for every run. +- Stop immediately if customer-like balances, settlement accounting, or admin authorization checks diverge from expected values. +- Preserve logs, traces, contract invocation IDs, and dashboard screenshots for the post-experiment review. + +## 2. Architecture + +```mermaid +flowchart LR + Commander[Experiment commander] --> Scheduler[Chaos scheduler] + Scheduler --> Injector[Fault injectors] + Injector --> Contracts[Soroban contracts on staging] + Injector --> Simulator[Meter simulator] + Injector --> Oracle[Price/oracle adapters] + Injector --> Dashboard[Usage dashboard] + Contracts --> Metrics[Metrics and traces] + Simulator --> Metrics + Oracle --> Metrics + Dashboard --> Metrics + Metrics --> Alerts[Alert manager] + Alerts --> Commander + Metrics --> Review[Experiment evidence store] +``` + +### Components + +| Component | Responsibility | Required controls | +|---|---|---| +| Experiment commander | Owns go/no-go, abort, and rollback decisions | On-call for entire run | +| Chaos scheduler | Starts and stops approved experiments | Allowlist of experiment IDs and time windows | +| Fault injectors | Inject latency, process restarts, network loss, stale oracle data, simulator clock drift, and dashboard API failures | Staging namespace and identity isolation | +| Metrics and traces | Capture latency, error rate, availability, invariant checks, ledger submission outcomes, and alert health | Retention for at least 30 days | +| Alert manager | Pages staging responders when thresholds breach | Dedicated staging routing keys | +| Evidence store | Stores plans, approvals, screenshots, logs, and review notes | Immutable experiment folder per run | + +## 3. Experiment lifecycle + +1. **Design** — Write the hypothesis, blast radius, services affected, expected alerts, SLO thresholds, abort criteria, and rollback steps. +2. **Security review** — Confirm the experiment cannot reach production keys, production network endpoints, real funds, or unapproved customer data. +3. **Readiness check** — Confirm staging green status, dashboards loading, alert routes tested, and rollback owner online. +4. **Blue-green deploy** — Deploy the chaos-enabled configuration to the green staging slice while blue remains stable. +5. **Canary analysis** — Route 5% of staging simulator traffic to green for 15 minutes and compare P99 latency, error rate, and invariant checks. +6. **Experiment execution** — Run one fault at a time unless an approved scenario explicitly requires compound failures. +7. **Abort or recover** — Stop the experiment if any abort threshold triggers; otherwise allow automated recovery to complete. +8. **Review** — Record outcome, timeline, metrics, alert quality, runbook gaps, and follow-up issues. + +## 4. Initial experiment catalog + +| ID | Scenario | Fault injection | Expected behavior | Abort criteria | +|---|---|---|---|---| +| CE-STG-001 | Meter simulator network partition | Drop outbound simulator traffic to contract adapter for 5 minutes | Backoff retries occur; no duplicate billing events are accepted | Duplicate settled usage, critical P99 > 100 ms for 10 minutes after recovery, or error rate > 5% | +| CE-STG-002 | Stale oracle data | Freeze oracle adapter responses at an old timestamp | Freshness checks reject stale price data and surface alerts | Any stale price accepted for settlement | +| CE-STG-003 | Contract adapter latency | Add 250 ms artificial adapter latency for 10 minutes | Dashboard degrades gracefully; contract calls remain bounded by retry budget | Retry storm, queue depth rising for 5 consecutive minutes, or critical P99 fails recovery SLO | +| CE-STG-004 | Dashboard API failure | Return HTTP 503 from usage API for 3 minutes | UI shows degraded state and alert fires without affecting contract state | Blank dashboard, unhandled client exception, or missing alert | +| CE-STG-005 | Settlement worker restart | Restart staging settlement worker during active simulator traffic | Idempotency prevents double settlement and resumes from last checkpoint | Replayed settlement, missed checkpoint, or manual data repair required | +| CE-STG-006 | Emergency pause drill | Trigger staging-only pause flow for selected meters | Claims and deductions halt; operator can verify and unpause according to runbook | Unauthorized caller succeeds or authorized pause fails | + +## 5. Core validation logic + +Each experiment must publish machine-readable results with this minimum schema: + +```json +{ + "experiment_id": "CE-STG-001", + "started_at": "2026-07-17T00:00:00Z", + "ended_at": "2026-07-17T00:05:00Z", + "hypothesis_passed": true, + "critical_p99_ms": 87, + "availability_percent": 99.995, + "alerts_fired": ["staging-meter-partition"], + "invariant_failures": [], + "rollback_required": false +} +``` + +Automated checks must fail the experiment when any of the following occurs: + +- Critical-path P99 is greater than 100 ms for more than 10 minutes after recovery. +- Staging availability outside the approved experiment window drops below 99.99% monthly-equivalent availability. +- A security control, authorization requirement, nonce check, or replay protection check fails. +- Billing, streaming, settlement, or oracle freshness invariants fail. +- Required alerts or dashboards do not show the injected failure. + +## 6. Monitoring, alerting, and dashboards + +Create or update staging dashboards with these panels before enabling recurring chaos runs: + +- Critical-path P50/P95/P99 latency by service and contract operation. +- Availability and error budget burn for staging services. +- Contract invocation success, failure, and retry counts. +- Meter simulator publish rate, retry count, duplicate event count, and clock drift. +- Oracle freshness age, last successful update, and rejected stale submissions. +- Settlement queue depth, checkpoint age, idempotency replays, and failed settlements. +- Emergency pause state, challenge count, and recovery duration. +- Alert delivery latency and notification success rate. + +Alert thresholds: + +| Alert | Threshold | Severity | +|---|---|---| +| `CriticalPathP99High` | P99 > 100 ms for 5 minutes outside injection window | Page | +| `StagingAvailabilityBurn` | 2% monthly error budget consumed in 1 hour | Page | +| `OracleFreshnessViolation` | Stale oracle data accepted or freshness age beyond configured limit | Page | +| `DuplicateSettlementDetected` | Any duplicate settlement invariant failure | Page | +| `ChaosExperimentMissingAlert` | Expected alert absent 2 minutes after injection | Ticket | +| `RecoveryTimeExceeded` | Recovery exceeds scenario-specific objective | Page | + +## 7. Blue-green and canary rollout + +1. Keep the blue staging slice on the last known-good configuration. +2. Deploy chaos hooks, feature flags, and dashboards to green. +3. Run smoke tests against green using staging keys only. +4. Shift 5% of simulator traffic to green for 15 minutes. +5. Promote to 25%, then 50%, then 100% only if P99 latency, availability, error rate, and invariant checks remain within SLO. +6. Roll back to blue if any page-level alert fires, if an invariant fails, or if canary analysis reports statistically significant regression. + +## 8. Runbook checklist + +Before every experiment: + +- [ ] Approved experiment plan is linked in the tracking issue. +- [ ] Security review is complete. +- [ ] Staging identities and endpoints are verified. +- [ ] Dashboards and alert routes are healthy. +- [ ] Rollback command and owner are documented. +- [ ] Experiment commander has announced start time and blast radius. + +During every experiment: + +- [ ] Announce injection start and stop timestamps. +- [ ] Watch P99 latency, availability, invariant checks, and expected alerts. +- [ ] Abort immediately when an abort criterion triggers. + +After every experiment: + +- [ ] Attach metrics, logs, traces, screenshots, and result JSON to the issue. +- [ ] Record whether the hypothesis passed. +- [ ] Create follow-up issues for missing telemetry, runbook gaps, failed alerts, or service defects. +- [ ] Confirm staging returned to baseline for 30 minutes.