fleet-forge

Mini robot data platform focused on high‑throughput fleet telemetry ingestion, schema validation, hybrid batch/stream processing, and storage indexing.

What’s here now

• gRPC telemetry contract: proto/telemetry.proto
• Ingest gateway (gRPC server + schema validation + local log publisher)
• Edge simulator (async fleet generator)
• Stream processor, query API, and observer dashboard
• Infra scaffolding: Redpanda (Kafka API), Postgres, local object storage
• Architecture + roadmap docs

Quickstart (local)

Docker Compose stack

Use this path when you want Kafka/Postgres plus the ingest gateway, stream processor, query API, and dashboard started for you.

cd infra
docker-compose up -d

Then generate gRPC code for local clients such as the edge simulator:

cd ..
bash scripts/gen_proto.sh

Run the edge simulator:

uv sync
uv run --project . --directory services/edge_sim \
  python -m app.main --traffic-profile demo-issues --vehicles 50 \
  --batch-size 10 --schema-version 2

Dashboard: http://localhost:8080

The dashboard includes controls for starting/stopping demo traffic against the edge simulator and shows ingest/stream health, latency histograms, storage counts, database metadata, and recent ingest/stream errors. Its traffic panel exposes slider knobs for vehicles, batch size, vehicle skew, invalid point rate, duplicate batch rate, publish delay jitter, and stream error rate.

Manual services

Use this path when you want to run services directly with uv. Start only Kafka/Postgres first:

cd infra
docker-compose up -d redpanda postgres
cd ..
bash scripts/gen_proto.sh
uv sync

Start ingest gateway:

uv run --project . --directory services/ingest_gateway \
  env PUBLISHER_BACKEND=kafka python -m app.main

Start the stream processor in another shell:

uv run --project . --directory services/stream_processor \
  python -m app.main

Start the query API in another shell:

uv run --project . --directory services/query_api \
  uvicorn app.main:app --host 0.0.0.0 --port 8000

Run the edge simulator in another shell:

uv run --project . --directory services/edge_sim \
  python -m app.main --traffic-profile demo-issues --vehicles 50 \
  --batch-size 10 --schema-version 2

Edge simulator shortcut:

bash scripts/edge_sim_demo.sh

Full stack shortcut (Docker Compose stack + demo generator):

bash scripts/full_stack_demo.sh

Stop stack:

bash scripts/stop_stack.sh

With PUBLISHER_BACKEND=kafka, telemetry flows through Redpanda into the stream processor, Postgres, and local object storage. If the ingest gateway is run with the local log publisher instead, accepted batches append to /tmp/fleetforge_ingest.log by default.

Local Storage

Objects are written to the local filesystem (default STORAGE_ROOT=/tmp/fleetforge_storage).

The stream processor writes gzip-compressed JSON objects by default. Metadata is inserted into Postgres with ON CONFLICT DO NOTHING, so replayed duplicate batch chunks are counted as deduped rather than inserted again.

Error Logs

Simulated and runtime processing errors are written as JSON lines:

• Ingest errors: /tmp/fleetforge_errors/ingest.log
• Stream errors: /tmp/fleetforge_errors/stream.log

The Docker Compose stack mounts these paths and the dashboard tails them.

Metrics

• Ingest metrics: http://localhost:8001/metrics
• Stream processor metrics: http://localhost:8002/metrics

CLI Observer

uv run --project . python scripts/observer.py

Dashboard

uv run --project . --directory services/observer_dashboard \
  uvicorn app.main:app --host 0.0.0.0 --port 8080

The Docker Compose stack also starts ingest/stream/query and the dashboard on http://localhost:8080. When run in Compose, dashboard traffic controls target ingest_gateway:50051; when run locally, they target localhost:50051 unless EDGE_SIM_TARGET is set. Use the dashboard slider knobs to adjust the live demo generator without restarting the stack.

Load Test (1,000 vehicles)

uv run --project . python scripts/load_test.py --vehicles 1000 --rate-hz 1 --batch-size 5 --duration-s 60

More dynamic traffic:

uv run --project . python scripts/load_test.py --vehicles 1000 --rate-mode sine --rate-min-hz 0.3 --rate-max-hz 3 --batch-size 10 --batch-size-jitter 5 --duration-s 120

Demo-issue traffic:

uv run --project . python scripts/load_test.py --traffic-profile demo-issues --vehicles 100 --batch-size 10 --duration-s 120

The demo-issues profile enables sine-wave rates, batch-size jitter, per-vehicle skew, invalid points, duplicate delivered-batch replay, publish delay jitter, occasional dropped client sends, and simulated stream processor errors. You can tune those independently with flags such as --vehicle-skew, --invalid-point-rate, --duplicate-batch-rate, --publish-delay-jitter-ms, and --stream-error-rate.

Integration Tests

RUN_INTEGRATION=1 uv run --project . pytest -q

Docs

• docs/ARCHITECTURE.md
• docs/ROADMAP.md
• docs/WORK_SPLIT.md