[Vedaant-J]

Stack: #53 → #112 → #50 → #60 → #64; #114 on #112; #115 on #64. Merge bottom to top.

Summary

A /Volumes/<catalog>/<schema>/<volume>/... path is reachable only through the FUSE mount on Databricks compute, so a training job on a laptop, EC2 box or CI cannot publish a model or export scores to the volume a Databricks job reads. This PR makes such a path a file destination of DatabricksConnector from any machine. The sink implements #112's FileStoreSink and reads and writes the volume through the Databricks Files API with the connector's identity, reusing #112's writers. Files up to 4 GiB are plain objects that Spark reads directly; larger ones are stored as 4 GiB parts (the Files API caps one upload at 5 GiB). No SQL warehouse is used.

Usage

root = "/Volumes/main/kumo/artifacts"
trainer.fit(training_table, snapshot=snapshot, export_config=ModelOutputConfig(
    databricks, path=f"{root}/models/churn"))
trainer = Trainer.load_from_connector(databricks, path=f"{root}/models/churn")
trainer.predict(prediction_table, snapshot=snapshot, output_config=PredictionOutputConfig(
    databricks, output_paths={"predictions": f"{root}/scores/"}))

The identity needs USE CATALOG, USE SCHEMA, READ VOLUME (load) and WRITE VOLUME (export).

Back-compat

Situation	Behaviour
Table name on a DatabricksConnector	Unchanged: COPY INTO through the staging volume.
Unity Catalog registry publish and fetch (location=)	Unchanged.
/Volumes/... path on a DatabricksConnector	Was refused; now written through the Files API.
/Volumes/... string root on Databricks compute	Unchanged: a FileConnector over FUSE.
Export into a volume directory that already holds files	Fails before scoring starts: Output directory '/Volumes/.../scores/' already contains data from another export. Choose an empty destination path.
Upload or download failure	FileNotFoundError, PermissionError or OSError with the volume path; no signed URL in the message. A failed write cancels its upload, so nothing is committed.
A file larger than 4 GiB read by Spark	A directory of <name>.chunks/<generation>/ parts; only kumo reads it as one file.

Testing

• kumo-databricks/tests/test_volume.py (10 tests, in-memory Files API in tests/fakes.py): reads across part boundaries; sized writes plain up to a part; failed and refused uploads; a download that ignores Range is refused; directories and missing paths; bundle publish and fetch, Parquet and frame exports through a volume path; an occupied directory refused before scoring.
• Live with the rest of the series applied, on a test workspace with synthetic data, off-cluster and in a cluster job: 3.0 GiB and 4.29 GiB files round-trip with matching CRCs, a read across the 4 GiB part boundary is exact, and a kumo.mlops fit and predict with the artifact root on a volume writes snapshots, models and predictions there.

Change map: see the kumo-pr-change-map comment.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: NVIDIA/kumo-engine
• Branch: vedaant/mlops-dbx-volume
• File: .coderabbit.yaml
• Commit: 9125de4
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Enterprise
• Run ID: 5af94af6-0c90-4332-b92c-9bc8e66cc070

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[Vedaant-J]

Change map

File	Function / scope	What changed	Why
packages/kumo-databricks/src/kumo_databricks/volume.py	new: VolumeHandler, volume_filesystem, is_volume_path	Chunked filesystem over the Files API; 4 GiB parts, ranged reads	Volumes cap one upload at 5 GiB
packages/kumo-databricks/src/kumo_databricks/sink.py	_DatabricksTables, DatabricksSinkBackend: holds_files, filesystem	Table methods split out; FileStoreRouting sends volume paths through FilesystemSink	Exports and bundles on /Volumes/... paths
packages/kumo-databricks/src/kumo_databricks/registry.py	_address	Error names the volume path form	Points users at the supported path
packages/kumo-databricks/README.md	"Artifacts on a volume" section	Usage, file layout, required grants	Setup the code cannot show
packages/kumo-databricks/tests/fakes.py	FakeVolume, FakeMetadata, FakeStreamingResponse	In-memory volume with ranged downloads	Tests run without a workspace
packages/kumo-databricks/tests/test_volume.py	new tests	Parts, failures, permissions, bundles, exports, occupied directory	Covers the volume sink