Push hugging face storage resolution into vortex-cloud rather than vortex-python by robert3005 · Pull Request #9199 · vortex-data/vortex

robert3005 · 2026-08-05T14:53:34Z

We should treat hf:// urls as any other object store, right now we are keeping all of it in vortex-python which is not quite as versatile as having a full fledged object store.

fixes #5379

codspeed-hq · 2026-08-05T15:17:47Z

Merging this PR will degrade performance by 21.03%

❌ 1 regressed benchmark
✅ 1928 untouched benchmarks
⏩ 51 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`decode_varbin[(1000, 2)]`	61.5 µs	77.9 µs	-21.03%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing claude/vortex-hugging-face-format-1asxf3 (40fbfa5) with develop (60b52b7)²}

51 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on develop (27859f7) during the generation of this report, so 60b52b7 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

…cheme Address review on #9199: * `hf::supports_scheme` is only asked by the registry, so it does not need to be `pub`. Its doc no longer intra-doc-links the OpenDAL counterpart either, since that path does not exist in an hf-only build. * "spelled" -> "passed" in the `TokenArg` doc. * The Python bindings should not carry Rust feature gates: the `opendal` feature of `vortex-python` is gone and the OpenDAL-backed stores (COS, and GooseFS from the rebase) are always compiled in, like `HfStore`. The ImportError placeholders, the feature language in the stubs and docs, and the cfg gates in `lib.rs`/`io.rs` all go with it. Other consumers (JNI) keep the opt-in Cargo feature. The `vortex._lib.CosStore`/`GoosefsStore` nitpick entries mirror the `HfStore` one, since the simplified re-exports resolve annotations to the native module path. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd

…e registry Vortex could not read from the Hugging Face Hub. `hf://` was understood only by `vortex.datasets`, which parsed the URI in Python and rewrote it into an authenticated `HTTPStore`, so `vx.open("hf://...")` failed and Polars, DuckDB and DataFusion could not open Hub URLs at all. A Hub repository is a set of files behind an HTTP endpoint that honours range requests, so serving one needs no cloud SDK: an `object_store::http::HttpStore` rooted at the repository's `resolve` prefix, carrying a bearer token when one is available, is the whole implementation. Reads therefore keep every `ClientOptions` setting a caller passes -- timeouts, retries, proxy configuration, `allow_http` -- unlike the OpenDAL-backed schemes, whose bridge owns its own HTTP client. Unlike every other scheme in the registry, the store is rooted at a (repository, revision) pair rather than at the URL authority, since both occupy URL path segments. It therefore reports the in-repository path itself and lets the registry mount it that deep; getting this wrong would send the repository name to the Hub as part of the file path, so it is covered from both the build-and-cache and the cached-store branch. Listing is not supported. The Hub does not implement WebDAV `PROPFIND`, which is how `object_store`'s HTTP store lists a prefix. Opening a known path -- `head` plus ranged `get`, both plain HTTP -- is what a Vortex scan needs, and callers that must expand a glob can list through the Hub's own API first. The `hf` feature adds no cloud SDK, only `http` and `percent-encoding`, so `vortex-python` enables it unconditionally rather than making it opt-in the way `opendal` is. Towards #5379. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd

…reads `hf://` `vortex.datasets` built Hub URLs and authorization headers by hand because the reader could not resolve `hf://` itself. It can now, so the URL building (`hf_hub_url`), the saved-login lookup (`get_token`), the endpoint-relative path rewriting, and the separate prefix-rooted store for revisions containing `/` all collapse into emitting `hf://` URIs and letting the reader resolve them. Two cases an `hf://` URI cannot express keep a store of their own, both rooted at the encoded `resolve` prefix: * a `token` string, which the reader has no way to see; * `token=False`, which must suppress the credentials the reader would otherwise read from `HF_TOKEN` or the saved login. Routing this through `hf://` would have quietly authenticated a read the caller asked to be anonymous, so it is covered by its own test. `token=True` asks for exactly the saved login the reader already finds, so it joins the default on the `hf://` path. Listing stays here. The Hub serves no listing over the object-store protocol, so expanding a glob or a directory still goes through the Hub API, which is why `_parse_hf_uri` and the `HfApi` call remain. File paths are percent-encoded on their way into the URI, since they become URI segments; Vortex decodes them back into the object key. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd

…x.open` `vortex.datasets` still hand-built a Hugging Face `resolve` URL and an authorization header for the two cases an `hf://` URI cannot express: a token held in a variable, and a read that must stay anonymous despite the environment offering credentials. Expose the store class those cases actually want. `HfStore(repo_id, *, repo_type, revision, token, endpoint)` wraps the `HfConfig` / `make_hf_store` pair the way `CosStore` wraps `CosConfig`. `token` follows `huggingface_hub`'s own convention, so a caller can pass its value straight through: `None`/`True` take whatever the environment offers, `False` forces an anonymous read, and a string is used directly. `revision` is passed literally -- unlike in a URL, the store percent-encodes a revision containing `/` itself. `vx.open` previously accepted only the `pyo3-object_store` classes, which is why `datasets.py` carried a comment that `CosStore` could not be read from. It now takes the same `AnyVortexStore` extraction `read_url`/`write` use, so `HfStore` works there -- and `CosStore` does too, removing that limitation. Two things this surfaced: * `HfConfig` built `ClientOptions::default()`, ignoring `ALLOW_HTTP`. Every other scheme picks it up because `parse_url_opts` reads its configuration from the environment, so a plain-HTTP `HF_ENDPOINT` -- a self-hosted Hub, or a test double -- behaved inconsistently. It is honoured now, which is what lets the new end-to-end read test exercise the real path against a local server. * `HfRepoType` gained `FromStr` so the binding can take `repo_type="dataset"`, accepting the plural the Hub spells in its URLs as well. With the store class in place, `datasets.py` no longer knows the Hub endpoint at all: the `huggingface_hub.constants` import and the last `HTTPStore` use are gone. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd

…cheme Address review on #9199: * `hf::supports_scheme` is only asked by the registry, so it does not need to be `pub`. Its doc no longer intra-doc-links the OpenDAL counterpart either, since that path does not exist in an hf-only build. * "spelled" -> "passed" in the `TokenArg` doc. * The Python bindings should not carry Rust feature gates: the `opendal` feature of `vortex-python` is gone and the OpenDAL-backed stores (COS, and GooseFS from the rebase) are always compiled in, like `HfStore`. The ImportError placeholders, the feature language in the stubs and docs, and the cfg gates in `lib.rs`/`io.rs` all go with it. Other consumers (JNI) keep the opt-in Cargo feature. The `vortex._lib.CosStore`/`GoosefsStore` nitpick entries mirror the `HfStore` one, since the simplified re-exports resolve annotations to the native module path. Signed-off-by: Robert Kruszewski <robert@spiraldb.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Lc5zw7Le2T3pakDEUdKTYd

robert3005 added the changelog/feature A new feature label Aug 5, 2026

robert3005 force-pushed the claude/vortex-hugging-face-format-1asxf3 branch from 4014ea0 to 80f90fc Compare August 5, 2026 19:30

robert3005 requested review from AdamGS, a10y, danking and onursatici August 6, 2026 18:27

a10y reviewed Aug 6, 2026

View reviewed changes

Comment thread vortex-cloud/src/hf/mod.rs Outdated

a10y reviewed Aug 6, 2026

View reviewed changes

Comment thread vortex-python/src/hf_store.rs Outdated

a10y reviewed Aug 6, 2026

View reviewed changes

Comment thread vortex-python/src/lib.rs

a10y approved these changes Aug 6, 2026

View reviewed changes

robert3005 force-pushed the claude/vortex-hugging-face-format-1asxf3 branch from 80f90fc to b4f6493 Compare August 6, 2026 19:16

robert3005 and others added 4 commits August 7, 2026 10:15

robert3005 force-pushed the claude/vortex-hugging-face-format-1asxf3 branch from b4f6493 to 40fbfa5 Compare August 7, 2026 10:15

robert3005 enabled auto-merge (squash) August 7, 2026 10:23

robert3005 merged commit c1ae775 into develop Aug 7, 2026
73 of 74 checks passed

robert3005 deleted the claude/vortex-hugging-face-format-1asxf3 branch August 7, 2026 10:23

robert3005 mentioned this pull request Aug 7, 2026

Read hf:// URLs from DuckDB by mounting stores where the registry says #9265

Merged

Navigation Menu

Uh oh!

Push hugging face storage resolution into vortex-cloud rather than vortex-python - #9199

Push hugging face storage resolution into vortex-cloud rather than vortex-python#9199
robert3005 merged 4 commits into
developfrom
claude/vortex-hugging-face-format-1asxf3

robert3005 commented Aug 5, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Aug 5, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

robert3005 commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will degrade performance by 21.03%

Performance Changes

Footnotes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robert3005 commented Aug 5, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 5, 2026 •

edited

Loading