Skip to content

README.md

Latest commit

 

History

History
92 lines (75 loc) · 4.34 KB

README.md

File metadata and controls

92 lines (75 loc) · 4.34 KB
Raw

Example Fixtures

Synthetic, deterministic business data covering every format the flyquery ingest pipeline supports. Use these to:

  • Run an end-to-end smoke test on a fresh deployment.
  • Demo the NL → SQL pipeline without leaking real customer data.
  • Cover regression tests for the parsers, the smart column-name proposer, the describe stage, and the schema KB.

Every file is generated by ../scripts/generate_examples.py. Re-run that script to regenerate -- output is deterministic (seeded RNG) so the fixtures are stable across machines.

Files

Path Format Rows Notes
csv/customers.csv CSV 120 Customer dimension. tier enum, country_code ISO 3166.
csv/sales_orders.csv CSV 500 Order fact table. References customers.csv by customer_id.
json/products.json JSON 50 Array-of-objects product catalogue.
jsonl/events.jsonl JSONL 1 500 Line-delimited user-event log.
xlsx/sales_orders.xlsx XLSX 500 Clean single-sheet tabular file (same data as the CSV).
xlsx/financials_dashboard.xlsx XLSX - Dashboard-style -- multiple sections (Activos, P&L, Ratios) separated by blank rows + label-only header bands. Exercises the section detector + smart column-name proposer + numeric-pseudo-header detector.
parquet/transactions.parquet Parquet 2 000 Financial transactions, includes a currency_code

How the dashboard XLSX stresses the pipeline

financials_dashboard.xlsx deliberately recreates the failure patterns that drove the May-2026 ingest hardening:

  • Multiple data tables in one sheet -- the section extractor must pick them apart via the empty-row boundaries.
  • A label-only header row (Activos, Cuenta de Pérdidas y Ganancias, Ratios principales) -- the section detector must treat these as section labels, not as the data header.
  • Numeric pseudo-header in the P&L section -- the row that DuckDB would naturally sniff as a header is actually data. The column-name proposer must replace columnNN synthetic names with business labels (revenue, cogs, etc).
  • Mixed Spanish / English -- the describe agent + grounding agent must translate (Activos ↔ Assets, Cuenta de Pérdidas y Ganancias ↔ Profit & Loss).

Smoke-test recipe

# 1. Spin up the stack
docker compose up -d postgres redis minio ollama
uv sync
uv run alembic upgrade head

# 2. Start the server
uv run uvicorn flyquery.main:app --host 127.0.0.1 --port 8520 &

# 3. Create workspace + dataset
WS=$(curl -sS -X POST http://localhost:8520/api/v1/workspaces \
  -H 'X-Tenant-Id: demo' -H 'X-Workspace-Id: examples' \
  -H 'Content-Type: application/json' \
  -d '{"slug":"examples","name":"Examples"}' | jq -r .id)

DS=$(curl -sS -X POST http://localhost:8520/api/v1/datasets \
  -H "X-Tenant-Id: demo" -H "X-Workspace-Id: $WS" \
  -H 'Content-Type: application/json' \
  -d '{"name":"smoke","description":"Synthetic fixtures."}' | jq -r .id)

# 4. Upload every format
for f in examples/csv/*.csv \
         examples/json/*.json \
         examples/jsonl/*.jsonl \
         examples/xlsx/*.xlsx \
         examples/parquet/*.parquet; do
  echo "▶ uploading $f"
  curl -sS -X POST "http://localhost:8520/api/v1/datasets/$DS/files" \
    -H "X-Tenant-Id: demo" -H "X-Workspace-Id: $WS" \
    -F "file=@$f" > /dev/null
done

# 5. Ask a few NL questions
curl -sS http://localhost:8520/api/v1/query \
  -H "X-Tenant-Id: demo" -H "X-Workspace-Id: $WS" \
  -H 'Content-Type: application/json' \
  -d "{\"question\": \"What is the total revenue by country?\", \"dataset_id\": \"$DS\"}"

CI coverage

The integration test tests/integration/test_examples_ingest.py walks every file under examples/ and asserts the upload succeeds + the resulting catalogue contains the expected table count. Add new fixtures here and the test will pick them up automatically.