Streaming ML Features

A local streaming feature and inference pipeline for auditable anomaly scoring. Labeled BTCUSDT replays enter Redpanda, a worker computes six rolling features, applies a packaged logistic model, persists decisions to ClickHouse, and exposes quality and operational metrics in Streamlit.

Engineering focus

• One feature definition shared by online and offline paths.
• Point-in-time correctness through feature_time and known_at guards.
• Chronological 56/14/30 train/validation/evaluation split to prevent leakage.
• Model-versus-rule evaluation using precision, recall, F1, and alert rate.
• Inference timeout fallback that preserves feature ingestion and writes a DLQ record.
• Deterministic model artifact with preprocessing, threshold, split, and metrics.

Architecture

labeled replay -> Redpanda -> feature processor -> feature/prediction topics
                                    |          |
                               ClickHouse   inference DLQ
                                    |
                                Streamlit dashboard

The rolling window stores the latest 20 events per symbol and derives log return, realized volatility, buy/sell imbalance, mean inter-arrival time, cancel ratio, and signed price impact. Contracts live under contracts/; the checked model artifact is artifacts/logistic-anomaly-v1.json.

Quality gates

The model artifact reports evaluation F1 0.883721 versus rule-baseline F1 0.833333 on the same deterministic replay. Accuracy is not the promotion gate. Runtime records parity violations and blocks silent divergence between the online and offline feature implementations. Every 97th event injects an inference timeout; the feature is still published, the decision uses the rule fallback, and the failure is routed to the DLQ.

Run locally

Prerequisite: Docker Desktop with Compose v2. The first build downloads images and dependencies; runtime processing calls no external API.

cp .env.example .env
make check
make train
make up
make status
make verify

Local interfaces:

• Dashboard: http://localhost:48504
• Redpanda Console: http://localhost:48084
• ClickHouse health: http://localhost:48123/ping

Stop with make down; make clean deletes project volumes.

Verification evidence

VERIFICATION.md records deterministic retraining, zero online/offline parity violations, active model and fallback paths, and runtime latency for the newest events under the strict known_at guard.

Trade-offs and limits

• One partition and one symbol make ordering easy to inspect but do not test horizontal scaling.
• Feature state exists in worker memory; durable state and recovery remain an explicit extension before making production-readiness claims.
• The DLQ demonstrates isolation and retry payloads but has no automatic retry consumer.