Skip to content

r2.md

Latest commit

 

History

History
61 lines (38 loc) · 3.97 KB

r2.md

File metadata and controls

61 lines (38 loc) · 3.97 KB
Raw

Cloudflare R2 hosting

R2 is the primary host for the u16 shard sets (DESIGN A14). Egress is free, so miners can Range-read as much as they need. Code: hypertrain.storage (client, publisher, doctor); CLI: hypertrain.datasets publish.

1. Bucket

Dashboard: R2 > Create bucket (e.g. hypertrain-data; location hint near the aggregator). Default storage class is fine; Infrequent Access adds a retrieval fee and suits only old shard sets. Do not add a lifecycle rule that deletes shard keys: they are content-addressed and referenced by signed DatasetSpecs. A rule that aborts incomplete multipart uploads is harmless. Temp keys of doctor live under _doctor/.

2. API token (least privilege)

R2 > Manage API tokens > Create: permission Object Read & Write, scoped to this bucket only. Put the Access Key ID and Secret in an untracked .env (copy .env.example; never commit it) or the process environment. Variables: R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, R2_BUCKET, R2_PUBLIC_BASE, optional R2_ENDPOINT (MinIO or any S3 endpoint). Environment variables override --env-file values.

3. Public read

Bucket > Settings > Custom Domains (recommended; caching and WAF) or enable the r2.dev URL (rate limited, testing only). Set R2_PUBLIC_BASE to that URL, no trailing slash. Check Range support:

curl -s -o /dev/null -D- -H 'Range: bytes=0-15' "$R2_PUBLIC_BASE/manifest.json" | head -3   # HTTP/2 206

CORS (Bucket > Settings > CORS policy), needed only for browser readers:

[{"AllowedOrigins": ["*"], "AllowedMethods": ["GET", "HEAD"],
  "AllowedHeaders": ["Range"], "ExposeHeaders": ["Content-Range", "Content-Length", "ETag"],
  "MaxAgeSeconds": 86400}]

4. Check everything

uv run --frozen python -m hypertrain.storage doctor [--env-file .env] [--cors-origin https://app.example]

It does PUT (If-None-Match), HEAD (size and sha256 metadata), signed Range GET, public Range GET (must be 206, plus CORS if asked) and DELETE on a temp key. Exit 0 = all ok; 2 = missing variables (named); 1 = a check failed.

5. Publish a shard set

uv run --frozen python -m hypertrain.datasets build --mix od-a-v1 --out data/od --cache cache
uv run --frozen python -m hypertrain.datasets verify data/od
uv run --frozen python -m hypertrain.datasets publish data/od --to r2 --env-file .env [--prefix sets/od-a-v1]

Order: shards at shard_uri_template keys (default {shard_sha256}), then build_record.json, then manifest.json last, so a reader never sees a manifest with missing shards. Each PUT sends x-amz-content-sha256 and stores x-amz-meta-sha256; shard keys use If-None-Match: * (immutable). Re-running HEADs each key and skips matching objects, so an interrupted run resumes. At the end --verify-units N (default 8) Range-reads random units through R2_PUBLIC_BASE and checks the unit sha256s; a failure is non-zero. Shards are at most 64 MiB, so single PUT is enough (no multipart).

6. Publish teacher labels

uv run --frozen python -m hypertrain.datasets publish-labels <label_cache_dir> --prefix labels/<teacher>-<mix> --env-file .env

Every file is uploaded under the prefix at its relative path (existing identical ones are skipped), then labels-index.json (path, size, sha256 per file) last. Consumers read the index first and verify each file against it.

7. How miners find it

The signed DatasetSpec carries shard_uri_template, shard_sha256_root and unit_sha256_root. The base URL (R2_PUBLIC_BASE plus the prefix) is the operator-supplied entry in the base list handed to fetch.prefetch(..., bases=[...]), with the HF mirror second. The base is not signed: miners check bytes against the signed unit hashes, so a wrong or hostile base can only fail, not poison.

Not on R2 (local by design)

Checkpoints, theta, miner uploads (the challenge object store, hypertrain.data.store, has its own S3Store), the ledger and the holdout (private until reveal) are not part of the R2 dataset flow.