Event-native metrics layer: business metrics that move when events happen — measured 1.1 s p50 event-to-metric freshness on production defaults. Live entity lookups, typed contracts, dual-language SDKs, and release-gated delivery for people, dashboards, services, and AI agents alike.
BI on a replica answers yesterday's questions. Support, ops, and merch workflows need current orders, metrics, and health signals at the moment of decision — not a stale warehouse snapshot, not a pile of one-off service adapters, and not a cache that quietly serves 30-second-old numbers.
AgentFlow's axis is event → live metric: every metric declares which events move it (a contract-tested lineage graph), and the serving layer keeps reads fresh by invalidating its cache when events arrive — a measured behavior, not a slogan (docs/freshness-benchmark.md). One serving boundary on top of that axis:
- streaming ingestion for operational events (validated, enriched, journaled)
- a semantic layer that exposes entities, metrics, lineage, and query endpoints
- typed, versioned contracts — each metric ships with its source events and a staleness budget
- Python and TypeScript clients that speak the same API surface
Consumers are whoever needs the number now: humans, dashboards, downstream services, and AI agents — agents are one consumer, not the product.
- Measured event-to-metric freshness — an event entering the pipeline is reflected in
GET /v1/metrics/*in 1.06 s p50 / 1.99 s p95 on production defaults (event-driven cache invalidation, no webhook registration), tunable to 238 ms p50; a plain TTL cache on the same pipeline sits at ~15 s. Reproducible viapython scripts/benchmark_freshness.py→ freshness benchmark - Lineage as a contract — all six metrics declare their source events, serving table, and a 2.5 s p95 staleness budget in versioned contracts, exposed through
/v1/catalogand/v1/contractsand pinned by tests against the actual write path - Published release line through
v2.0.0on PyPI (agentflow-runtime,agentflow-client) and npm (@yuliaedomskikh/agentflow-client) via OIDC Trusted Publishers with SLSA provenance on every artifact - Tested and gated — 1,500+ unit tests plus a broad Windows no-Docker suite; CI enforces 13 required status checks (lint, schema, unit, integration, helm, perf, terraform, bandit, safety, npm-audit, trivy, contract, build-smoke) through branch protection
- Dual SDK parity across Python and TypeScript — retries, circuit breakers, batching, pagination, contract pinning, idempotency keys,
as_ofhistorical reads — over sub-second entity lookups (p5038–55 ms, p99167 mson local hardware) - Security in the hot path — tenant isolation on every read surface, parameterized queries,
sqlglotAST validation for NL-to-SQL, fail-closed auth, secret scrubbing, and a Bandit gate for new findings - Production-shaped extras — two CDC paths (hardened Debezium/Kafka Connect + a ClickHouse per-branch fan-out), on-call runbooks, and a narrated demo of the DV2 multi-branch warehouse
Upgrading from v1.0.x? See the v1.1 migration guide before installing.
Prerequisites:
- Python
3.11+ make- Docker Compose (
make demostarts Redis and the ClickHouse serving store)
PowerShell 7+:
git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
. .\scripts\setup.ps1
make demomacOS / Linux:
git clone https://github.com/brownjuly2003-code/agentflow.git
cd agentflow
source ./scripts/setup.sh
make demomake demo starts Redis and ClickHouse, seeds demo data through the full pipeline (validated events land in the ClickHouse serving store), and serves the API on http://localhost:8000. Swagger UI is available at http://localhost:8000/docs.
Try it:
curl http://localhost:8000/v1/entity/order/ORD-20260404-1001
curl -X POST http://localhost:8000/v1/query \
-H "Content-Type: application/json" \
-d '{"question":"Show me top 3 products"}'Local demo runs without API-key enforcement unless you explicitly configure AGENTFLOW_API_KEYS_FILE.
Event sources -> Kafka -> Flink -> Iceberg --------\
-> Semantic layer -> FastAPI -> Agent / SDK
Local demo -> local_pipeline -> ClickHouse ------/
(DuckDB stays the local lake / test store)
Stack:
- Ingestion: Kafka producers, Debezium/Kafka Connect CDC, and a local synthetic pipeline
- Processing: Flink plus validation and enrichment stages
- Storage: Iceberg for production-shaped tables; ClickHouse is the serving store (ADR 0006 — ReplacingMergeTree upserts,
final=1reads), DuckDB the local-dev / test store - Serving: FastAPI, contract registry, lineage, search, and operational endpoints
- Orchestration: Dagster
- IaC: Terraform, Helm, Docker Compose, and a Fly.io demo config
See docs/architecture.md for the detailed design, trade-offs, and deployment topologies.
CDC source capture is standardized on Debezium/Kafka Connect; downstream consumers use the canonical AgentFlow CDC contract defined in ADR 0005.
| Area | Files |
|---|---|
| API core | src/serving/api/ |
| Semantic layer | src/serving/semantic_layer/ |
| Python SDK | sdk/agentflow/ |
| TypeScript SDK | sdk-ts/src/ |
| Agent integrations | integrations/agentflow_integrations/ (LangChain, LlamaIndex, CrewAI, MCP) |
| Flink jobs | src/processing/flink_jobs/ |
| Test suites | tests/ |
| Design decisions | docs/decisions/ (ADRs) |
| Public site | site/ |
| IaC | infrastructure/terraform/, infrastructure/dv2/, helm/, k8s/ |
| DV2.0 warehouse | warehouse/agentflow/dv2/ (hubs / links / satellites + real-dataset loader) |
Core
- Architecture — system context, data flow, failure modes
- API Reference — endpoint-by-endpoint curl / Python / TypeScript examples
- Operational Runbook + On-Call Runbooks — local stack, CDC capture, and production-incident playbooks
- Security Audit — threat model, controls, and evidence
- Glossary — interview-ready explanations of the core technical terms
- Interactive Technical Walkthrough — MkDocs Material guide (Mermaid architecture, SDK, deployment, observability)
Deep dives
- DV2.0 Multi-Branch Extension — Data Vault 2.0 model for mid-market e-com (5 locations / 3 jurisdictions): schema, end-to-end flow, demo evidence
- CDC Deployment Plan — Debezium/Kafka Connect rollout
- Competitive Analysis · Release Readiness · Cost Analysis
- Fly.io Demo Deploy — minimal hosted demo
- Contributing · Changelog
# verified release slice
python -m pytest tests/unit tests/integration tests/sdk -q
# benchmark and regression gate
python scripts/run_benchmark.py
python scripts/check_performance.py --baseline docs/benchmark-baseline.json --current .artifacts/load/results.json --max-regress 20
# benchmark trend: [.github/perf-history.json](.github/perf-history.json) is appended on every main push;
# render the history locally with `make perf-plot` (writes docs/perf/history.html).
# contracts and security
python scripts/generate_contracts.py --check
bandit -r src sdk --ini .bandit --severity-level medium -f json -o .tmp/bandit-current.json
python scripts/bandit_diff.py .bandit-baseline.json .tmp/bandit-current.jsonv2.0.0 is the current release line — PyPI agentflow-runtime /
agentflow-client and npm @yuliaedomskikh/agentflow-client, all
published via OIDC Trusted Publishers with SLSA provenance attestations.
CI on main is green across all 13 required checks.
The v1.1.0 → v2.0.0 arc landed in seven increments on top of a security
audit-closure sprint:
v1.1.0— audit closure: tenant isolation across every read surface, SQL guard centralized onsqlglot, entity allowlist enforcement, fail-closed auth, secret rotation, Helm hardening, OpenAPI drift gate, and the required status checks.v1.2.0— DV2 multi-branch warehouse: 38 Data Vault 2.0 tables (8 hubs / 8 links / 22+ satellites), an Argo Workflowsdv2-refreshtemplate, a dbt project (3 mart models + 12 tests), and per-branch CDC fan-out via ClickHouseMaterializedPostgreSQL.v1.3.0—helm/kafka-connecthardening matched tohelm/agentflow(NetworkPolicy + PDB + securityContext), live Helm validation across both charts, and the narrated DV2 demo (terminal + web-UI + dbt docs).v1.4.0— maintenance: on-call runbooks,SECURITY.md, issue/PR templates, contract/DORA CI hardening, repo hygiene, and a dependency wave (mypy, Terraform AWS provider, TypeScript, GitHub Actions, Vitest). No runtime API changes fromv1.3.0.v1.5.0— security & correctness hardening: argon2id key hashing with an O(1) peppered lookup index (M-C4), an NL→SQL guard bypass fix (typedread_csv/read_parquetscan functions now denied in projection position),sqlglotcontrol-byte and mutation-target repairs, and a strict-mypyexpansion across the orchestration and freshness slices. No public API changes.v1.6.0— the architecture-fixing release: ClickHouse becomes the shipped serving engine (pipeline sink,ReplacingMergeTreerow versions, backend-routed event scan, a dedicated CI E2E lane against a real ClickHouse), PII protection moves from the removed app-level string-parse gate to engine-enforced vault governance (fail-closed column grants, per-jurisdiction officer roles, row policies,SQL SECURITY DEFINERviews — every live adversarial probe green), plus the vendored NL→SQL generation engine (LangGraph, routed through GraceKelly), the DV2 raw vault on PostgreSQL withLISTEN/NOTIFYfreshness, the MinIO-backed PyIceberg catalog, and the OpenSSF Scorecard channel (5.8 → 7.0).v2.0.0— the demo universe re-founded and the scale path shipped: the business legend re-pinned end-to-end to an own-brand kitchen-appliance importer in ₽ (breaking for the retired fashion-retail/USD surfaces), the external real-retailer dataset removed outright (breaking: loader deleted, its at-scale benchmark retired as historical), the control plane externalized to PostgreSQL behind theControlPlaneStoreport (ADR 0010, six slices incl. the Helm scale profile), three operational read surfaces split out of the agent catalog (ADR 0011: Order 360, stuck-orders worklist, exception inbox), and the three-node demo topology (ADR 0012) implemented and deployed live on Hugging Face Spaces — plus the G2 audit closure (spec/seed consistency, journal-scan hardening, live evidence re-captures).
The tagged line and main are in sync as of v2.0.0. See the
changelog for full detail.
This is a reference data-engineering project. The streaming, warehouse, and deployment artifacts (Flink, Iceberg, Helm, Terraform, k8s) are exercised against a local pipeline and a kind cluster in CI rather than a managed cloud. Wiring it to a live production source needs inputs that live outside the repo — CDC source onboarding (runbook ready in docs/operations/cdc-production-onboarding.md), a public benchmark on production-grade hardware, and an external pen-test attestation.
| Admin UI | API docs |
|---|---|
 |
 |
| Landing page | Benchmark run |
|---|---|
 |
 |
Capture notes and publish-time checks are listed in docs/publication-checklist.md.
MIT. See LICENSE.
Built as a data-engineering reference project. Initial release cycle
2026-04-10 → 2026-04-20, with post-audit hardening and the DV2
extension landing through v1.4.0. Architecture decisions are recorded as
ADRs in docs/decisions/.