Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 138 additions & 4 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

68 changes: 62 additions & 6 deletions docs/api/python/store/opendal.rst
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
==================
OpenDAL (COS, OSS)
==================
===========================
OpenDAL (COS, OSS, GooseFS)
===========================

Vortex can read from and write to Tencent Cloud COS and Alibaba Cloud OSS through
`OpenDAL <https://opendal.apache.org/>`_, which provides native service support.
Vortex can read from and write to Tencent Cloud COS, Alibaba Cloud OSS, and Tencent Cloud
GooseFS through `OpenDAL <https://opendal.apache.org/>`_, which provides native service
support.

These stores are available only when Vortex is built with the ``opendal`` feature
(e.g. ``maturin develop --features opendal`` or ``cargo build -p vortex-jni --features opendal``).
Expand All @@ -13,7 +14,7 @@ These stores are available only when Vortex is built with the ``opendal`` featur

* - Scheme
- Service
- Endpoint variable
- Endpoint / master variable
- Credential variables
* - ``cos://``
- Tencent Cloud COS
Expand All @@ -23,6 +24,10 @@ These stores are available only when Vortex is built with the ``opendal`` featur
- Alibaba Cloud OSS
- ``OSS_ENDPOINT``
- ``ALIBABA_CLOUD_ACCESS_KEY_ID``, ``ALIBABA_CLOUD_ACCESS_KEY_SECRET``
* - ``goosefs://``
- Tencent Cloud GooseFS
- ``GOOSEFS_MASTER_ADDR``
- (optional) ``auth_type`` / ``auth_username`` properties

:class:`vortex.store.CosStore`
==============================
Expand All @@ -47,6 +52,31 @@ These stores are available only when Vortex is built with the ``opendal`` featur
:param disable_config_load: When ``True``, disable OpenDAL's automatic config loading
and rely only on the explicit configuration. Defaults to ``False``.

:class:`vortex.store.GoosefsStore`
==================================

.. py:class:: vortex.store.GoosefsStore(master_addr, *, root=None, block_size=None, chunk_size=None, write_type=None, auth_type=None, auth_username=None)

A Tencent Cloud GooseFS object store, backed by OpenDAL. Construct it with explicit
configuration and pass it to
:func:`vortex.io.read_url` / :func:`vortex.io.write` via the ``store=`` argument,
exactly like the built-in S3/Azure/GCS stores.

The class is only available when Vortex is built with the ``opendal`` feature; on
a default build, instantiating it raises :class:`ImportError`.

:param master_addr: GooseFS master address(es). Single master:
``"10.0.0.1:9200"``. HA (comma-separated):
``"10.0.0.1:9200,10.0.0.2:9200,10.0.0.3:9200"``.
:param root: Optional key prefix applied to every operation.
:param block_size: Block size in bytes for new files (default: 64 MiB).
:param chunk_size: Chunk size in bytes for streaming RPCs (default: 1 MiB).
:param write_type: Default write type: ``"must_cache"``, ``"cache_through"``,
``"through"``, or ``"async_through"``.
:param auth_type: Authentication type: ``"nosasl"`` or ``"simple"`` (default:
``"simple"``).
:param auth_username: Authentication username (default: current OS user).

Reading from COS
================

Expand Down Expand Up @@ -91,3 +121,29 @@ configuration comes from the environment variables OpenDAL's OSS builder reads
import vortex as vx

a = vx.io.read_url("oss://my-bucket/path/to/dataset.vortex")

Reading from GooseFS
====================

Pass a ``goosefs://`` URL directly. The master address is taken from the URL authority, or
from the ``GOOSEFS_MASTER_ADDR`` environment variable when the authority is empty:

.. code-block:: python

import vortex as vx

a = vx.io.read_url("goosefs://10.0.0.1:9200/path/to/dataset.vortex")

Or configure explicitly with :class:`~vortex.store.GoosefsStore` and pass it to
:func:`vortex.io.read_url` via ``store=``:

.. code-block:: python

from vortex.io import read_url
from vortex.store import GoosefsStore

store = GoosefsStore(master_addr="10.0.0.1:9200")

# When `store=` is supplied, the path is a key within the store, so the scheme and
# master address are not part of the path passed to read_url.
a = read_url("path/to/dataset.vortex", store=store)
7 changes: 4 additions & 3 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,10 +52,11 @@

nitpicky = True # ensures all :class:, :obj:, etc. links are valid
nitpick_ignore = [
# `vortex.store.CosStore` is re-exported through the private `vortex.store._cos` module,
# and the `ObjectStore` type alias resolves to the private path. The public class is
# fully documented in `opendal.rst`; the private path is intentionally not.
# `vortex.store.CosStore` / `GoosefsStore` are re-exported through private modules,
# and the `ObjectStore` type alias resolves to those private paths. The public
# classes are fully documented in `opendal.rst`; the private paths are intentionally not.
("py:class", "vortex.store._cos.CosStore"),
("py:class", "vortex.store._goosefs.GoosefsStore"),
]

doctest_global_setup = "import pyarrow; import vortex; import vortex as vx; import random; random.seed(a=0)"
Expand Down
9 changes: 8 additions & 1 deletion vortex-cloud/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,15 @@ oss = [
"object_store/cloud",
"opendal/services-oss",
]
# Tencent Cloud GooseFS, the `goosefs://` scheme.
goosefs = [
"dep:opendal",
"dep:object_store_opendal",
"dep:tracing",
"opendal/services-goosefs",
]
# Every OpenDAL-backed service.
opendal = ["cos", "oss"]
opendal = ["cos", "oss", "goosefs"]

[lints]
workspace = true
Loading
Loading