Docs/update readme

docs/cloud-providers.md

@@ -0,0 +1,145 @@
+# Running against cloud providers
+
+CloudOracle supports four resource sources, selected at runtime with the `CLOUDORACLE_PROVIDER` env var: **synthetic** (default, no cloud account required), **aws**, **gcp**, **azure**. The analyzer, report, and dashboard work identically with all four — they only differ in where the resource inventory comes from.
+
+> **Tested status.** The **synthetic** and **AWS** providers have been exercised end-to-end against a live AWS account during development. The **GCP** and **Azure** providers are implemented against their respective SDKs with the same structure and the code compiles + unit-tests pass, **but they have not been run against live GCP / Azure subscriptions** because I don't have credentials for those clouds at the time of writing. Field-mapping tests use struct literals; the SDK call paths themselves are unverified. If you test either, please open an issue with what you find.
+
+## Synthetic (default, no setup)
+
+No credentials, no network calls — the app generates realistic EC2 / RDS / EBS / Lambda records locally. Ideal for demos, CI, and trying the dashboard in seconds.
+
+```bash
+docker compose up --build
+docker compose exec app /app/cloudoracle seed --count 120
+# open http://localhost:8080
+```
+
+Tunables:
+- `SYNTHETIC_COUNT` (default `100`) — how many resources to generate per `seed`.
+- `SYNTHETIC_ACCOUNT` (default `synthetic-account`) — account ID baked into the records.
+
+The synthetic provider is what 99% of demos use. Everything else in the v1 guide — findings, exports, trend tracking, dashboard — works with synthetic data without any cloud credentials.
+
+## AWS (verified)
+
+**1. IAM user with read-only access.** In the AWS Console → IAM → Users → Create user, attach:
+- `ReadOnlyAccess`
+- `AWSBillingReadOnlyAccess`
+
+Grab the access key + secret. For least-privilege in production, the minimum set is:
+
+```
+ec2:DescribeInstances, ec2:DescribeVolumes
+rds:DescribeDBInstances, rds:ListTagsForResource
+lambda:ListFunctions, lambda:ListTags
+ce:GetCostAndUsage
+sts:GetCallerIdentity
+```
+
+**2. Configure a local profile.** In `~/.aws/credentials` (or `%USERPROFILE%\.aws\credentials` on Windows):
+
+```ini
+[cloudoracle]
+aws_access_key_id = AKIA...
+aws_secret_access_key = ...
+region = us-east-2
+```
+
+The profile name `cloudoracle` and region `us-east-2` are the defaults. Override with `AWS_PROFILE=xxx` and `AWS_REGION=eu-west-1` if you use different names.
+
+**3. Run the app on the host** (so it can read `~/.aws/credentials`), pointing at the Postgres container:
+
+```bash
+docker compose up -d postgres              # DB only in Docker
+export CLOUDORACLE_PROVIDER=aws
+go run ./cmd/oracle seed                   # fetches real EC2/RDS/EBS/Lambda, upserts, snapshots
+go run ./cmd/oracle analyze                # runs rules → findings on real data
+go run ./cmd/oracle serve --port 8080      # dashboard + API
+```
+
+The STS `GetCallerIdentity` call at startup validates credentials immediately — if the profile is misconfigured or keys are expired, you get the error right away instead of halfway through a scan.
+
+**Running inside Docker with AWS creds** (if you want `docker compose up app` against AWS), pass the creds as env vars to the `app` service in `docker-compose.yml`:
+
+```yaml
+environment:
+  CLOUDORACLE_PROVIDER: aws
+  AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
+  AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
+  AWS_REGION: us-east-2
+```
+
+The AWS SDK v2 auto-picks these up without needing a profile file. Recommended only for demos — for prod/CI, use IAM roles via instance metadata or IRSA on EKS, not static keys.
+
+**Cost:** `Describe*` / `List*` calls are free. A full `seed` against a typical account is ~5-10 API calls total.
+
+## GCP (untested against a live account)
+
+> Implemented but not verified against a real GCP project.
+
+Expected flow:
+
+1. Enable APIs on your project: Compute Engine, Cloud SQL Admin, Cloud Functions.
+2. Set up Application Default Credentials:
+   - Dev: `gcloud auth application-default login`
+   - Prod: `GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa.json`
+3. Export `GOOGLE_CLOUD_PROJECT=your-project-id`.
+
+Required IAM roles (least privilege):
+
+```
+compute.instances.list, compute.disks.list
+cloudsql.instances.list
+cloudfunctions.functions.list
+```
+
+Then:
+
+```bash
+docker compose up -d postgres
+export CLOUDORACLE_PROVIDER=gcp
+export GOOGLE_CLOUD_PROJECT=your-project-id
+go run ./cmd/oracle seed
+go run ./cmd/oracle serve --port 8080
+```
+
+Since this path hasn't been exercised end-to-end, expect to debug the SDK call mapping on first run.
+
+## Azure (untested against a live account)
+
+> Implemented but not verified against a real Azure subscription.
+
+Expected flow:
+
+1. Export `AZURE_SUBSCRIPTION_ID=<your-subscription-guid>`.
+2. Authenticate via one of:
+   - Dev: `az login`
+   - Service principal: `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_SECRET`
+   - Managed Identity (when the app runs on Azure)
+
+The provider uses `DefaultAzureCredential`, which tries all methods in order.
+
+Required RBAC role: `Reader` on the subscription. Production scope:
+
+```
+Microsoft.Compute/virtualMachines/read
+Microsoft.Compute/disks/read
+Microsoft.Sql/servers/read, Microsoft.Sql/servers/databases/read
+Microsoft.Web/sites/read
+```
+
+Then:
+
+```bash
+docker compose up -d postgres
+export CLOUDORACLE_PROVIDER=azure
+export AZURE_SUBSCRIPTION_ID=00000000-0000-0000-0000-000000000000
+go run ./cmd/oracle seed
+go run ./cmd/oracle serve --port 8080
+```
+
+Same caveat as GCP: no live-account run has been done, so treat first execution as a validation exercise.
+
+---
+
+For env var reference, see [configuration.md](configuration.md).

docs/configuration.md

@@ -0,0 +1,33 @@
+# Configuration
+
+Reference for every environment variable CloudOracle reads. All vars are loaded once at startup by `internal/config.Load()` and injected into the cloud, LLM, and DB layers — no component reaches for `os.Getenv` on its own.
+
+| Variable      | Default       | Description           |
+|--------------|---------------|-----------------------|
+| `CLOUDORACLE_PROVIDER` | `synthetic` | Cloud provider: `aws`, `gcp`, `azure`, or `synthetic` |
+| `AWS_PROFILE` | `cloudoracle` | AWS shared-config profile to use |
+| `AWS_REGION` | `us-east-2` | AWS region to scan |
+| `GOOGLE_CLOUD_PROJECT` | _(unset)_ | GCP project ID (required when provider is `gcp`) |
+| `AZURE_SUBSCRIPTION_ID` | _(unset)_ | Azure subscription ID (required when provider is `azure`) |
+| `SYNTHETIC_COUNT` | `100` | Default number of synthetic resources to generate |
+| `SYNTHETIC_ACCOUNT` | `synthetic-account` | Default account ID for synthetic data |
+| `CLOUD_SERVICE_TIMEOUT` | `30s` | Per-service timeout for each cloud API call (Go duration string) |
+| `DB_HOST`    | `localhost`   | PostgreSQL host       |
+| `DB_PORT`    | `5432`        | PostgreSQL port       |
+| `DB_USER`    | `oracle`      | Database user         |
+| `DB_PASSWORD`| `oracle_dev`  | Database password     |
+| `DB_NAME`    | `cloudoracle` | Database name         |
+| `LLM_PROVIDER`     | _(auto)_ | Force a specific LLM provider: `gemini`, `claude`, or `openai`. If unset, auto-detects based on which API key is present. |
+| `LLM_TIMEOUT`      | `30s` | HTTP timeout for LLM API calls (Go duration string) |
+| `LLM_MAX_RETRIES`  | `3` | Number of retries on transient LLM failures (429, 5xx, network errors). Set to `0` to disable. |
+| `LLM_BASE_DELAY`   | `500ms` | Initial backoff between retries; doubles on each attempt with full jitter |
+| `LLM_MAX_DELAY`    | `30s` | Cap for the per-retry wait (also caps `Retry-After` headers) |
+| `GEMINI_API_KEY`   | _(unset)_ | API key for Google Gemini (`gemini-2.5-flash`)     |
+| `ANTHROPIC_API_KEY`| _(unset)_ | API key for Anthropic Claude (`claude-haiku-4-5`)  |
+| `OPENAI_API_KEY`   | _(unset)_ | API key for OpenAI (`gpt-4o-mini`)                 |
+| `LOG_LEVEL`        | `info`    | Log level: `debug`, `info`, `warn`, or `error`     |
+| `LOG_FORMAT`       | `text`    | Log format: `text` (human-readable) or `json` (structured)  |
+
+---
+
+For the design of the LLM provider layer and the analyzer rule engine, see [architecture.md](architecture.md). For per-cloud setup details (profiles, credentials, IAM scopes), see [cloud-providers.md](cloud-providers.md).

docs/testing.md

@@ -0,0 +1,52 @@
+# Testing
+
+The project has two tiers of tests:
+
+- **Unit tests** (171, no external dependencies): pure-function tests for the analyzer, generator, LLM providers, LLM retries, PDF report, exporters, cloud mapping, real-provider fetchers, and central config validation. Run with `go test ./internal/...`.
+- **Integration tests** (12, require Docker): exercise the real Postgres path via [testcontainers-go](https://golang.testcontainers.org/) — insert/upsert behavior, transaction rollback, snapshot aggregation, and a full end-to-end seed → analyze flow against a containerized Postgres 16. Run with `go test -tags=integration ./internal/db/ ./internal/e2e/`.
+
+Integration tests share a single Postgres container per process and `TRUNCATE … RESTART IDENTITY CASCADE` between cases — fast (sub-millisecond reset on small tables) and hermetic enough for our schema. The helper lives at `internal/db/dbtest/postgres.go` and is gated by the `integration` build tag, so the testcontainers dependency stays out of the unit-test compile path. If Docker isn't running, the helper calls `t.Skip` with a clear message rather than failing — running the binary without Docker just skips the integration cases.
+
+The CI workflow at `.github/workflows/test.yml` runs both tiers on every push and PR. GitHub-hosted Ubuntu runners have Docker preinstalled, so the integration job needs no extra service container.
+
+## Unit test coverage
+
+- **Per-rule tests**: each detection rule (`ec2-idle`, `rds-oversized`, `ebs-orphan`, `lambda-over-provisioned`) has happy-path, negative, and boundary tests.
+- **Boundary testing**: CPU thresholds, age cutoffs, memory limits, and invocation counts are explicitly tested at their exact values to catch off-by-one errors.
+- **Aggregator tests**: `Analyze` is tested for empty input, mixed input, false-positive prevention, and correct savings-descending ordering.
+- **LLM provider tests**: all three providers (Gemini, Claude, OpenAI) are tested against mock HTTP servers using `httptest`, covering success responses, API errors, empty payloads, error fields, and context cancellation.
+- **Provider factory tests**: auto-detection order (Gemini > Claude > OpenAI), explicit selection, missing keys, and unknown providers.
+- **Prompt builder tests**: total calculations, severity breakdowns, service rollups, top-5 limiting, and empty input handling.
+- **PDF generation tests**: file creation, AI summary inclusion/exclusion, empty findings, 100-finding page-break stress test, invalid paths, and all severity color codes.
+- **Export tests**: JSON round-trip, CSV header + row layout, numeric formatting, RFC 4180 escaping of commas/quotes/newlines, and empty-findings handling for both formats.
+- **Generator tests**: correct count, valid services/regions/types, non-negative costs, timestamp ordering, and service distribution.
+- **Config tests**: default values, custom values, timeout parsing (valid and invalid durations), empty-env fallback, and DSN assembly.
+- **Cloud mapping tests**: AWS SDK type → `shared.Resource` conversion with struct literals (no AWS calls, no credentials needed).
+- **Real-provider fetcher tests**: every cloud provider (AWS, GCP, Azure) is exercised end-to-end against fake SDK clients — pagination exhaustion, per-service API errors, graceful degradation when one service fails, and edge cases (nil hardware profile on Azure VMs, nil settings on Cloud SQL, web apps mixed with function apps in the Azure `/sites` collection).
+- **LLM retry tests**: the shared retry transport is verified against `httptest` servers — retries until success, respects `MaxRetries` cap, honors `Retry-After` headers, replays the request body on every attempt, retries transport-level errors (not just non-2xx), bails out on context cancellation, and returns immediately on non-retryable statuses (401, 4xx other than 408/429).
+- **Config validation tests**: every invalid input shape (non-numeric port, out-of-range port, unknown enum value, negative integer, malformed Go duration, zero/negative duration), every cross-field rule (provider=gcp without project, provider=azure without subscription, LLM_PROVIDER set without matching API key), and the multi-error accumulator that lists all problems at once instead of failing on the first.
+
+## Integration test coverage
+
+- **Insert + upsert**: round-trip through a real Postgres, asserting that `ON CONFLICT DO UPDATE` updates the right columns (`monthly_cost`, `usage_metric`, `updated_at`) without overwriting `created_at`.
+- **Transaction rollback**: a failing batch (one row that overflows `NUMERIC(10,2)`) rolls back the whole batch, leaving pre-existing rows untouched.
+- **Snapshot aggregation**: a mixed set of resources across multiple `(account, service)` tuples produces exactly the expected snapshot rows, with correct counts and per-tuple cost totals.
+- **Snapshot windowing**: the `--days` filter on the `trend` command actually filters via SQL — old snapshots are excluded from short windows and included in long ones.
+- **End-to-end seed → analyze**: a deterministic resource set engineered to fire each rule once, inserted via `InsertResources`, read back via `ListResources`, and analyzed — asserts every rule fires exactly once and findings are sorted by potential savings descending.
+- **End-to-end with synthetic data**: 50 random resources generated by `SyntheticProvider`, full round-trip through the DB, analyzer must produce *some* findings (the generator skews toward waste patterns).
+- **Re-seed idempotency**: running insert three times on the same fixed-ID set ends with the same row count — proves the seed flow is safe to re-run on a schedule.
+
+## Running the suite
+
+```bash
+# Unit tests (no Docker required)
+go test ./internal/...
+
+# Integration tests (Docker must be running)
+go test -tags=integration ./internal/db/ ./internal/e2e/
+
+# Both, verbose
+go test -tags=integration -v ./internal/...
+```
+
+All rules are pure functions (`Resource -> *Finding`), which makes them trivially testable without mocks, fixtures, or test databases. The code was designed to be testable from the start — not tested after the fact.