Feature/3.0#23
Merged
Merged
Conversation
Part A of milestone 8.1: expose CloudOracle's cost data over an HTTP API the upcoming Python agent (insights-agent/) can call as a LangGraph tool. Why two endpoints instead of just one: the agent's reasoning improves when it can both compare providers (cost-summary) and drill into a single provider's service mix (cost-by-service). The /api/v1/* prefix keeps the dashboard's existing /api/* routes untouched so the embedded React UI is not coupled to the agent's auth model — only v1 requires X-API-Key, v0 dashboard endpoints stay open as before. Data semantics: there is no Cost Explorer / Billing integration yet, so the v1 endpoints approximate period spend from the cost_snapshots table (average projected monthly rate per (account, service), scaled by days/30). Every response carries data_source="snapshots_approximation" and a human-readable note so downstream clients can surface the disclaimer to the user. Real CUR ingestion is a follow-up. Internals: - Added APIConfig (CLOUDORACLE_API_KEY / _API_PORT / _API_SHUTDOWN_TIMEOUT) - New db.ListSnapshotsInRange (inclusive [start, end]) - authMiddleware (constant-time compare) + requestIDMiddleware - apiData interface — single data dep for both v0 dashboard and v1 handlers, replacing scattered db.* calls so tests can run without Postgres. Coverage on internal/api/ is now 86%. - Graceful shutdown via Server.Run honouring SIGINT/SIGTERM with a configurable timeout; runServe refuses to start without an API key. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…raction First commit of the LangGraph-based FinOps agent that consumes the Go /api/v1 cost endpoints. Lays down the project skeleton, config / logging mirroring the Go side (stderr + text|json switch matching slog), and an LLMProvider ABC so future providers (Claude, OpenAI) are additive without touching the graph code. - pyproject.toml with uv, Python 3.12, pytest+coverage (>=80% gate), ruff (strict select), mypy (strict mode) - config.Settings (pydantic-settings) fails fast on missing required keys - logging.setup mirrors Go slog semantics (LOG_LEVEL/LOG_FORMAT, stderr) - llm.LLMProvider ABC + llm.GeminiProvider (default gemini-2.5-flash to match the Go side and avoid drift) - tests cover config validation, logging wiring, and provider construction (real Google SDK patched out so no network in CI); 100% coverage Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…t endpoints
CloudOracleClient (httpx.AsyncClient) wraps the v1 cost endpoints with
X-API-Key auth, configurable timeout, and per-request X-Request-ID
generated with the same 24-hex shape as the Go server's newRequestID —
so a Python-side log line can be correlated with the Go-side log by
request_id without manual stitching.
build_tools() exposes the client methods as LangChain StructuredTools
with rich descriptions that tell the LLM how to surface the
`data_source: snapshots_approximation` caveat to the end user.
Errors are propagated, not swallowed:
- 4xx/5xx → CloudOracleAPIError with status / Go `code` / request_id
- timeouts and network failures → CloudOracleTransportError
- malformed JSON or non-object body → CloudOracleAPIError
Inputs validated locally before issuing the HTTP call:
- ISO YYYY-MM-DD format and start <= end
- provider in {aws, gcp, azure} (also normalizes casing)
- top in [1, 1000]
Tests use pytest-httpx to mock the Go API; cover happy paths, all error
classes (401, 4xx, 5xx, non-JSON, non-object), timeout, network error,
and every local-validation branch. Coverage on tools/cloudoracle.py: 96%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
graph/basic.py exposes build_graph(llm, tools) and ask(graph, question) returning an AgentResult (answer + ordered tool_calls + raw messages). The system prompt is short on purpose — long static instructions tend to drift from the model's actual behavior; the tool docstrings carry the domain-specific guidance about the snapshot caveat. Cross-cutting fix in tools/cloudoracle.py: the LangChain tool wrappers now translate CloudOracleAPIError / CloudOracleTransportError / ValueError into ToolException, which langgraph's ToolNode catches and surfaces back to the LLM as a tool observation. Letting the original RuntimeError propagate aborts the whole graph run, which is the wrong UX for a transient 5xx or a malformed date — the model should see the error and either retry or explain to the user. Tests use a hand-rolled ScriptedChatModel that implements bind_tools and returns pre-scripted AIMessages. Covers: tool selection happy path, two-tool sequential invocation, no-tool-call (off-scope) path, tool-error recovery, and the multimodal AIMessage.content variant Gemini returns. 51 tests, ~97% coverage. Note: create_react_agent is deprecated in langgraph 1.0 (moved to langchain.agents); sub-hito 8.4 will refactor to a hand-rolled supervisor pattern. Until then we filter the warning in pyproject.toml to keep test output clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Local-only state from the Claude Code harness (session transcripts, agent scratch space, etc.) should not end up in the repo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`uv run python -m insights_agent.main "<question>"` (or the `insights-agent` console script) wires Settings → logging → GeminiProvider → CloudOracleClient → ReAct graph and prints either the natural-language answer or a JSON envelope (`--json`). `--verbose` streams the tool calls the model made to stderr so the operator can see which /api/v1 endpoint was actually consulted. Top-level error handling maps the realistic failure modes to distinct exit codes (130 on Ctrl-C, 2 on missing/invalid config, 1 on runtime errors) so callers in shell pipelines can branch deterministically. Includes the tiny `build_graph` signature widening `list[BaseTool] → Sequence[BaseTool]` so the CLI can pass the result of `build_tools` without an extra conversion at the call site. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… diagram `insights-agent/README.md` was a stub. Replace it with a self-contained setup-in-under-10-minutes guide covering: prerequisites, `uv sync`, the seven env vars (required vs optional, defaults), CLI flags and exit codes, an end-to-end smoke test the operator can run by hand against a local Go server, dev workflow (pytest / ruff / mypy), and a one-page architecture pointer table mapping concerns to source files. Root README gains an "AI Insights Agent" section with a Mermaid arch diagram (User → CLI → LangGraph → Gemini → tools → Go API → Postgres) and a roadmap update marking sub-hitos 8.0 / 8.1 done with the remaining 8.2–8.7 items listed so readers can place this work in the larger plan. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Standardize the whole repo on English for comments, docstrings, and code
example queries so a reader on the project doesn't have to bounce between
languages. The change is text-only — no identifiers, behavior, or tests
move.
Translations:
- All `sub-hito 8.x` references in code → `milestone 8.x`
(pyproject.toml, cost_handlers.go, llm/__init__.py, graph/basic.py,
tools/cloudoracle.py).
- `internal/cloud/*_test.go` test docstrings and inline comments.
- `internal/report/pdf.go` section markers and field comments.
- Sample queries in `insights-agent/README.md` and the matching scripted
AIMessage / `ask(...)` query in `tests/test_graph.py` — the existing
asserts (`"$150"`, `"snapshots"`) still match the new English answer
so the test still passes.
Plus pre-existing working-tree formatting in `README.md` (single-line
badges, table padding in the v2 callout, an extra blockquote blank line,
and a Mermaid example query already updated to English) folded into the
same commit since it was already staged-adjacent and is the same kind
of language/cosmetic cleanup.
Verified after: `uv run pytest` (58/58, 91.83% coverage), `uv run ruff
check .`, `uv run mypy src/`, and `go test ./internal/cloud/... ./internal/api/...
./internal/report/...` all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…endpoint Milestone 8.2 (more tools): expose the rule-based analyzer findings as agent-friendly savings recommendations. Go: new authed GET /api/v1/recommendations handler that runs analyzer.Analyze over the current inventory, with optional provider/severity filters and a top cap. Totals (total_count, total_monthly_savings_usd, by_severity) describe the full filtered set before the cap. Carries data_source: "heuristic_rules" to distinguish heuristic estimates from the snapshot-derived cost endpoints. Python: CloudOracleClient.recommendations() + cloudoracle_recommendations tool with a rich docstring; system prompt updated to surface the heuristic_rules caveat. Validation errors map to ToolException so the ReAct loop can recover. Tests: 8 Go handler tests; extended Python tool tests. Both suites green (internal/api; 65 Python tests, 92% coverage, ruff + mypy clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 8.2 (more tools): answer "is my spend growing?" with a per-day cost time series. Go: new authed GET /api/v1/cost-trends handler over ListTrends(days). Returns the per-day series plus a precomputed first/latest/change summary (absolute_usd, percent_from_first, direction up/down/flat) so the agent phrases the trend without crunching the array. percent_from_first is null when the first day is zero. Optional provider filter recomputes each day's total from that day's per-service breakdown. days clamps to 1..365. Shares the snapshots_approximation data_source with the cost endpoints. Python: CloudOracleClient.cost_trends() + cloudoracle_cost_trends tool with a rich docstring steering trend/over-time questions here (vs cost_summary for a single period). Validation errors map to ToolException. Tests: 9 Go handler tests (delta/direction, provider recompute, days clamp, zero-first nil percent, flat, empty, auth, error); extended Python tool tests. Both suites green (internal/api; 71 Python tests, 93% coverage, ruff + mypy clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Milestone 8.2 complete (more tools): answer "what do I have?" with a resource inventory summary. Go: new authed GET /api/v1/inventory handler over ListResources, aggregating counts and projected monthly cost by provider and by (provider, service). Optional provider filter; top cap applies only to by_service so the totals stay accurate when the list is truncated. Because resources carry AccountID, the "functions" provider disambiguation (gcp vs azure) is exact here. Distinct data_source: live_inventory — costs are summed per-resource projected monthly rates from the latest scan, not billed spend. Python: CloudOracleClient.inventory() + cloudoracle_inventory tool with a docstring steering "what do I have?" / footprint questions here (vs cost_summary for spend over a range). Validation errors map to ToolException. Tests: 7 Go handler tests (aggregation, provider filter, top cap with accurate totals, functions disambiguation, auth, empty, error); extended Python tool tests. Both suites green (internal/api; 77 Python tests, 93% coverage, ruff + mypy clean). The agent now ships 5 tools across 5 authenticated v1 endpoints, closing milestone 8.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ne 8.3) Add a sixth agent tool, finops_knowledge_search, that retrieves from a curated FinOps knowledge base for conceptual / policy / how-to questions the HTTP tools can't answer (rightsizing, commitment discounts, data-source caveats, cost allocation, glossary). Architecture (RAG kept in Python, where LangChain lives; the Go server stays a clean data API): - knowledge/: 5 packaged markdown notes, shipped in the wheel. - rag/corpus.py: load + chunk markdown to Documents (offline-testable). - rag/embeddings.py: EmbeddingsProvider ABC + Gemini impl, mirroring the llm/ provider pattern. - rag/store.py: langchain-postgres PGVector factory + store-agnostic retriever. - rag/ingest.py: ingest_corpus() core + insights-agent-ingest console script. - tools/knowledge.py: build_knowledge_tool(retriever) -> finops_knowledge_search, formatting results with [source: file — title] citations; errors map to ToolException so the ReAct loop can recover. Wiring is optional and gated on DATABASE_URL: with it unset the agent runs with just the five HTTP tools and no Postgres dependency. config.py gains database_url / embeddings_model / knowledge_collection / rag_top_k; main.py adds the knowledge tool only when a pgvector DB is configured; the system prompt steers conceptual questions to it. docker-compose switches Postgres to pgvector/pgvector:pg16 (drop-in). Tested fully offline (no DB, no embeddings API): corpus chunking, and the real retrieval + citation path via InMemoryVectorStore + DeterministicFakeEmbedding. 100 Python tests, 92% coverage, ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tone 8.4)
Replace create_react_agent on the production path with an explicit StateGraph:
START → supervisor → {worker} → supervisor → … → synthesize → END
- supervisor routes by tool call: bound with one routing tool per specialist
plus `finish`, the tool it calls names the next hop. Routing via tool calls
(not with_structured_output) keeps the node driveable by the scripted fake
model the suite already uses.
- three specialist workers, each a hand-rolled ReAct loop (_run_react, the
actual create_react_agent replacement) over a tool subset:
cost_analyst (cost-summary/by-service/trends/inventory),
savings_advisor (recommendations + knowledge),
concept_expert (knowledge). A worker contributes one summarizing message;
its tool churn stays local so the supervisor/synthesizer see a clean
transcript.
- synthesize composes the final answer from the findings, in the user's
language, with data-source caveats and citations.
- a hop cap bounds the supervisor loop so a model that never emits `finish`
still terminates.
main.py now builds the supervisor; graph/basic.py (create_react_agent) is
retained as the simple graph and still owns the shared AgentResult /
_stringify_content helpers the supervisor reuses.
Tests: test_supervisor.py drives it end-to-end with the scripted model —
single-worker route→tool→finish→synthesize, two-specialist routing, off-scope
finish-without-worker, hop cap, plus _run_react and _to_text units. 109 Python
tests, 93% coverage, ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…llback (8.5) Three of milestone 8.5's production guardrails, wrapping every run through guardrails/runner.py:run_guarded (the single entry point the CLI and the upcoming HTTP surface share): - Cost/usage caps: RunLimits (max_hops, max_tool_calls, max_worker_iters) from settings, threaded into the supervisor graph and the worker ReAct loop. When a cap is hit the supervisor stops dispatching and synthesizes from what it has, so a confused or injected loop can't run up unbounded LLM/tool cost. Workers now also surface their tool observations through the graph state. - Layered answer validation (guardrails/validation.py): deterministic grounding first — every monetary figure in the answer must match a number in the tool observations, an unmatched figure is a hard fail; then an optional LLM judge for a second opinion when the answer makes numeric claims that pass the deterministic layer. - Deterministic fallback (guardrails/fallback.py): on a run exception (quota, timeout) or a failed validation, return an honest no-LLM answer rendering the raw tool data (or stating nothing was retrieved) instead of a fabricated narrative or a raw traceback. config gains MAX_HOPS / MAX_TOOL_CALLS / MAX_WORKER_ITERS / ENABLE_ANSWER_VALIDATION / ENABLE_LLM_JUDGE; main wires run_guarded and the --json output now includes fallback_used + the validation verdict. Tests cover figure extraction, grounding (pass/fail/tolerance), the judge layers, fallback rendering, and run_guarded (happy / exception / invalid / disabled). 131 Python tests, 93% coverage, ruff + mypy clean. HTTP surface follows next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the agent over HTTP, sharing one runtime with the CLI:
- runtime.py: GeminiAgentRunner assembles the model + client + tools + graph +
run limits once and exposes ask() through the guardrails. The CLI (main.py)
now uses it too, so the two entry points behave identically; the RAG
knowledge-tool builder moved here from main.
- api/app.py: FastAPI create_app with GET /health and POST /ask
({query} -> {answer, tool_calls, fallback_used, validation}). The stack is
built once in the lifespan; optional X-API-Key auth via AGENT_API_KEY (same
convention as the Go server). create_app(runner=...) injects a fake runner so
the surface is testable without Gemini / a live Go server / Postgres.
- api/serve.py: insights-agent-serve console script (uvicorn).
- config gains AGENT_HOST / AGENT_PORT / AGENT_API_KEY.
Tests drive the surface with FastAPI's TestClient and an injected fake runner:
health, ask happy path + metadata, empty-query 422, fallback passthrough,
auth enforced/open, and that an injected runner isn't closed by the app. 138
Python tests, 91% coverage, ruff + mypy clean.
Milestone 8.5 complete: cost caps + layered validation + deterministic fallback
+ HTTP surface.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…action (8.7) Replace the hard-wired snapshot cost path in the v1 endpoints with a billing.Source interface so a real billing integration can be swapped in by config, starting with AWS Cost Explorer. - internal/billing: CostRecord / Report / Source / SourceError, and CostExplorerSource — a GetCostAndUsage query grouped by SERVICE over the period (CE's exclusive end handled), summed across time buckets and pages, returning real unblended cost with data_source "billing_aws_cost_explorer". The CE client is narrowed to an injectable interface (mocked in tests), the same pattern internal/cloud uses for EC2/RDS. - internal/api: snapshotSource implements billing.Source over the existing cost_snapshots aggregation (preserves data_source "snapshots_approximation" and the snapshot_query_failed code exactly). The cost-summary / cost-by-service handlers now group normalized records and echo the report's dynamic data_source; the snapshot-specific aggregateByProvider/ByService helpers are gone. Server gains a WithBillingSource option (default snapshots). - config: CLOUDORACLE_BILLING_PROVIDER (snapshots | aws_cost_explorer). cmd builds the CE source from AWS_REGION/AWS_PROFILE when selected and falls back to snapshots (loudly) if init fails. Tests: CE source (bucket/page summation, exclusive-end TimePeriod, error wrapping, missing-metric skip) with a fake client; api handlers against an injected non-snapshot source (dynamic data_source, provider filter, error code). Existing snapshot cost tests pass unchanged. The agent's FinOps corpus documents the new real-billing data source. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The internal/diff golden tests (byte-exact Markdown/narrative fixtures) failed on Windows checkouts: core.autocrlf=true with no .gitattributes rewrote the fixtures to CRLF while the renderer emits "\n". The committed fixture content was already correct, so this adds `* text=auto eol=lf` (plus binary markers) to keep LF in the working tree on every platform. All Go tests now pass. Also adds docs/v3-guide.md — the Insights Agent guide (supervisor + RAG + guardrails architecture, the /api/v1 contract and data_source semantics, real billing via AWS Cost Explorer, CLI/HTTP usage) alongside v1/v2-guide — and links it from the README.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.