You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
precompute_engine cannot ingest data from a local file #301
The benchmark pipeline (asap-tools/execution-utilities/benchmark/) currently uses Arroyo's single_file_custom connector to replay local dataset files (ClickBench, H2O) through the sketch pipeline. Since precompute_engine is intended to replace Arroyo + asap-summary-ingest, the benchmark needs an equivalent way to feed file data into it.
precompute_engine currently accepts data only via two HTTP endpoints:
POST /api/v1/write — Prometheus remote write (Snappy + Protobuf)
POST /api/v1/import — VictoriaMetrics remote write (Zstd + Protobuf)
There is no way to point the engine at a local JSON or CSV file and have it ingest the data directly. Running the benchmark against precompute_engine therefore requires an out-of-process HTTP sender that re-encodes file rows into the wire format — adding external dependencies and unnecessary round-trip overhead that the Arroyo file connector did not have.
Activity