Skip to content

feat(worker): open the ModelScope ModelArtifact source end to end - #699

Merged
thxCode merged 11 commits into
mainfrom
feat/model-artifact-modelscope
Sep 29, 2026
Merged

thxCode merged 11 commits into
mainfrom
feat/model-artifact-modelscope

Conversation

@thxCode

@thxCode thxCode commented Sep 29, 2026 •

Copy link
Copy Markdown
Collaborator

What type of PR is this?

/kind enhancement
/kind api-change
/area worker
/area testing

What this PR does / why we need it:

Opens the modelScope source of ModelArtifact end to end. The API has carried a reserved
modelScope member since the Hugging Face spec with admission refusing it; this PR implements
what opening it needs and turns the member on, at behavior parity with the Hugging Face source,
with the three ModelScope-specific failure modes handled so none of them can silently deliver
wrong content.

  • Admission accepts the member with the same repository and revision rules as Hugging Face
    (the hub source shape is shared), defaults an unset revision to master, and makes
    allowPatterns / ignorePatterns legal on a ModelScope source.
  • Resolution (controller) pins an immutable commit and a manifest digest:
    • A non-commit revision resolves through commits?Ref=<rev> and is cross-checked against the
      repository's own git endpoint — the HTTP smart-protocol ref advertisement, answered by the
      operator in process (no git binary, no subprocess). A disagreement between the hub's index
      and its git fails as SourceUnavailable.
    • The listing recovers from the API's silent 3000-entry truncation by re-listing per directory
      (Root=<dir>, descending only into subtrees that came back truncated), and refuses a
      directory with 3000 or more direct children rather than ship a partial manifest.
    • Errors classify by the envelope Code (ModelScope reports every access failure as 404) into
      RevisionNotFound / AccessDenied / SourceUnavailable; revalidation is a single HEAD
      and follows the same confirm-then-revoke staircase as Hugging Face.
  • Node delivery (plugin): the materializer picks a ModelScope client by the source's kind
    (read from the artifact, volume attributes unchanged); file URLs 302 to a content-addressed
    CDN that honors byte ranges, every file verifies against its manifest sha256 while
    streaming. NodeModelStoreHub gains an optional modelScopeEndpoint (the one API change); a
    ModelScope artifact on a node whose configuration predates the field is refused with a message
    naming the worker upgrade, never resolved against the wrong hub.
  • Engine delivery: vLLM and SGLang render their measured switches
    (VLLM_USE_MODELSCOPE / SGLANG_USE_MODELSCOPE, shared MODELSCOPE_CACHE, a token as
    MODELSCOPE_API_TOKEN from the artifact's Secret, and MODELSCOPE_DOMAIN as the bare host
    derived from the endpoint Setting). The runner ModelScope SDK floor (≥ 1.39.1, which accepts a
    commit as the revision) is documented, not enforced — measured: the current CUDA vLLM runner
    lines carry 1.37.1, the Ascend vLLM line and SGLang 0.5.18 runners meet the floor, and a user
    can always name a runner image of their own.
  • Free behaviors: everything downstream keys on the manifest digest, so peer sync, progress
    aggregation, prefetch and the placement preference work for a ModelScope artifact with no new
    code; e2e proves the peer-sync leg.
  • Docs & configuration: the artifact page owns the source (reason-table rows, the
    cross-check, the truncation recovery, all-sha256 manifests with cross-hub digests
    incomparable, the engine env table, the runner SDK floor); docs/settings.md gains
    model-artifact-modelscope-endpoint (default https://www.modelscope.cn).

Which issue(s) this PR links to:

None

Special notes for your reviewer:

  • e2e on kind (v1.29.14, three nodes): case-113 (the ModelScope source case, five legs,
    including peer sync between nodes) and case-97's moved admission rows (23 PASS) are green;
    evidence lives in the task directory.
  • Chart case-1 green; make lint, make lint docs, check-docs, check-specs, check-agents-shell
    and the full unit-test set green; generated code verified with zero diff in a clean checkout.
  • Known measured limit, stated in the docs: a ModelScope private repository with a valid token
    but no grant maps to AccessDenied with the shared "does not exist or is not accessible"
    message — distinguishing it needs a second ModelScope account, which the test environment does
    not have.

Does this PR introduce a user-facing change?

ModelArtifact gains a `modelScope` source: weights hosted on ModelScope resolve to an immutable
commit and a verified manifest, deliver to nodes through the same cache chain as Hugging Face
(including peer sync), and render for vLLM and SGLang engines. Engine delivery requires a runner
image whose bundled ModelScope SDK is ≥ 1.39.1 — the documentation lists which measured runners
meet the floor today and how to name your own.

NodeModelStoreHub gains an optional modelScopeEndpoint — empty in a spec an
older worker wrote, admitted like the Hugging Face endpoint — and the
model-artifact-modelscope-endpoint Setting feeds it through the node layer,
so the plugin resolves ModelScope artifacts against the same hub the
controller does. The prefetch warm-up image default follows the mirrored-*
pattern. Generated code from the private gen tree, replayed with zero drift.

Task 4 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
…dates

A full commit is taken as is; a branch or tag resolves through the commits
endpoint and is cross-checked against the repository's git endpoint itself —
the smart protocol's ref advertisement, answered in process (the worker image
ships no git binary and the endpoint gates on the user agent, both measured).
A disagreement refuses as SourceUnavailable. The listing walks repo/files and
recovers from the API's silent truncation at 3000 entries, refusing a
directory it cannot enumerate; every file carries the hub's sha256.
Revalidation is one HEAD of the repo endpoint; the token check reads
users/me; failures classify by the envelope code with the status as fallback.

Task 2, 3 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
The member validates with the shared hub rules, its revision defaults to
master, the union refusal names it, and patterns are legal on it; the
reserved-member refusal and its message are gone.

Task 1 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
…aircase

The Hugging Face resolve/revalidate staircase generalizes to a hub interface;
the ModelScope client plugs into it unchanged. The Secret watch enqueues
ModelScope artifacts and a rejected token warns through the hub's own check.

Task 5 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
…nment

A ModelScope source resolves through the same Node and Engine deliveries
against its own endpoint Setting; an engine download reads the SDK's names —
cache, bare-host domain, the engine's own use-switch, and the token from the
artifact's Secret. The ModelScope names join the artifact-owned environment
admission refuses.

Task 6 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
The download source carries its hub's kind from the artifact and the
environment builds the hub of that kind from the node's configuration; a
ModelScope source on a configuration without the endpoint waits naming the
worker upgrade, never resolved against the Hugging Face endpoint. The mount
rule accepts a resolved artifact of either hub.

Task 7 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
The accepted member, its resolution with the git cross-check and the
truncation walk, its reason table and revalidation HEAD, the all-sha256
manifest note, the hub-source delivery table, the ModelScope engine
environment and the runner SDK floor with the measured runner list, and the
model-artifact-modelscope-endpoint Setting row.

Task 8 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
case-113 resolves against the real hub with the git cross-check, materializes
on a cold node hashing to the hub's own sha256, pulls from a peer on the
second node, and pins the commit through a conforming SDK. case-97's
ModelScope row becomes an admission acceptance and the case table gains 113.

Task 9 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
Opens the reserved modelScope member end to end: branch resolution
cross-checked against git, a listing that recovers from the API's silent
truncation, errors classified by the envelope code, node and engine delivery,
a second hub endpoint through the node's effective configuration, and the
runner SDK floor documented with the measured runner list. All nine tasks
green, including case-113's five legs on a local kind cluster.

Task 9 of model-artifact-modelscope.

Signed-off-by: thxCode <thxcode0824@gmail.com>
@gpustack-code-review

gpustack-code-review Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

Reviewed eb36be7..377aaea only; earlier commits in this PR were reviewed in a previous run.

Comment thread pkg/modelartifact/modelscope.go
Comment thread pkg/modelartifact/modelscope.go Outdated
Comment thread pkg/modelmanager/materialize/materialize.go
@thxCode thxCode added kind/enhancement New feature or request kind/api-change Adds, removes, or changes an API area/worker The worker control plane and its controllers area/testing End-to-end test infrastructure labels Sep 29, 2026
- drop HEAD from the cross-check's fallback list: the commits endpoint
  answers master for a misspelled Ref, and a HEAD fallback would confirm
  exactly that silent rewrite (mutation-verified)
- enforce the tree-entry bound incrementally as the Hugging Face walk does,
  instead of after the whole walk
- an Environment failure for one hub kind no longer stops the remaining
  sources; unknown kinds are refused by name instead of resolved as Hugging
  Face
- gitURL keeps the endpoint's path prefix

Task 9 of model-artifact-modelscope (review round 1).

Signed-off-by: thxCode <thxcode0824@gmail.com>
Task 9 of model-artifact-modelscope (review round 1).

Signed-off-by: thxCode <thxcode0824@gmail.com>
@thxCode

thxCode commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Round-1 disposition of the summary-routed findings (fixes in dcefedd):

  1. report.go L208 — unknown kinds resolved as Hugging Face — Fixed. default now refuses an unrecognized kind (including the empty kind a non-hub source would carry) as InvalidRequest naming it; only HubHuggingFace resolves as Hugging Face. Test: TestEnvironmentRejectsAnUnknownKind.

  2. model_artifact.go L141 — hub construction before the Secret checks — Kept as is. The construction costs two in-memory Setting reads plus a ConfigMap GET only when a CA bundle is configured; the Secret check runs first thing inside reconcileHubSource, and the wasted work happens only on the rare missing-Secret path, which is paced by the unavailable retry. Moving construction below the checks would split hub construction from the staircase that owns it for that marginal path.

  3. node_model_store.go — regenerated output must ship with the field — Already satisfied. The generated deepcopy, protobuf, CRD, openapi and applyconfiguration output ships in the same commit as the field (49c7568), replayed in the private gen tree with zero drift.

  4. value.go L442 — the warm-up image default moves in an otherwise-scoped PR — Intentional, coordinator-directed. The mirrored default makes the image pullable in clusters without Docker Hub access, the same pattern as the other mirrored-* defaults; the Setting remains for clusters with their own registry arrangement. Happy to split it into its own PR if you prefer it isolated.

  5. modelscope.go L378 — gitURL drops the endpoint's path prefix — Fixed. The endpoint's path is kept (https://git.example.com/ms → /ms/qwen/repo.git). Test: TestModelScopeGitURLKeepsTheEndpointPath.

@thxCode
thxCode merged commit 63ca654 into main Sep 29, 2026
9 checks passed
@thxCode
thxCode deleted the feat/model-artifact-modelscope branch September 29, 2026 08:48
thxCode added a commit that referenced this pull request Sep 29, 2026
* feat(worker): add the modelartifact expecteddigest anchor

- spec.expectedDigest asserts the manifest digest a hub source must resolve to;
  admission accepts it on hub sources only, refusing claim and image sources each
  with the reason their identity rules it out, and the whole-spec immutability
  ratchet covers it
- status.resolved.digestSource records where a resolved digest came from: Hub, the
  hub's own listing, or Expected, the spec's anchor
- the stale modelscope member comment, left reserved-shaped by #699, now describes
  the opened source
- generated deepcopy, crd, protobuf, openapi and applyconfiguration artifacts

Task 1 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* feat(worker): assert the expectedDigest at resolution and revalidation

- a resolution whose manifest digest differs from the anchor is refused as
  DigestMismatch with both digests in the message, and DigestMismatch joins the
  revoking reasons
- the anchored revalidation re-lists the tree at the resolved commit and
  re-compares the digest instead of probing one file; the unanchored probe is
  unchanged
- a SourceUnavailable confirmed on the refusal staircase writes the anchor as
  the identity (digestSource Expected, no revision or counts) and the artifact
  never contacts the hub again
- the hub interface grows the manifest-returning ListManifest both clients
  already serve
- mutations prove the comparison and the anchored revalidation tests go red

Task 2 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* feat(worker): block engine delivery of an anchored artifact, peers-only chain

- an anchored artifact under Engine delivery waits with AnchorNeedsNodeDelivery,
  the message naming why: an engine downloads by repository and revision and
  cannot anchor-verify what it fetched, and an Expected identity has no commit
- an Expected identity's node chain is peers only: the manifest comes from a
  peer's published listing bound to the digest (Puller.FetchManifest), and with
  no peer holding the tree the mount fails naming the digest nothing holds
- the hub-identity chain is unchanged, still listing anchor-checked before any
  byte moves

Task 3 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* docs(model-store): document the expectedDigest anchor and its delivery

- artifact.md: the expectedDigest field and its hub-only admission, the
  anchored resolution and revalidation states, digestSource, the confirmed
  SourceUnavailable fallback and what an Expected identity delivers, the
  shipped access model for Expected identities, the AnchorNeedsNodeDelivery
  row, the migration contract, and the downgrade-window note
- node-store.md: the Expected identity's peers-only manifest and the
  SourceUnavailable row's anchored case
- README index: the artifact page's description carries the anchor

Task 4 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* test(e2e): case 114 asserts the expectedDigest anchor end to end

- a match resolves with digestSource Hub; a foreign anchor is refused
  DigestMismatch with both digests in the message
- a twice-unreachable hub writes the anchor as the identity with the hub's
  request log flat, and the Expected identity mounts from the peer with the
  hub log still flat
- Engine delivery holds both anchored artifacts with AnchorNeedsNodeDelivery
  and no Pod
- the peer leg skips on one worker or with E2E_C114_OFFLINE=1; the run restores
  both Settings it flips

Task 5 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* fix(worker): an Expected identity's files carry no hub URL, and the KV pin

- fetchTree built every file's URL by calling the hub, which is nil on the
  peers-only path and would panic before a peer byte was pulled; the URL is
  now empty when there is no hub, and the per-file loop's no-hub branch was
  already the loud no-source failure
- the KV identity pin the spec's acceptance names: the digest's leading digits
  for a hub artifact and for an Expected identity alike, the UID's hash only
  for a claim

Task 5 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* fix(e2e): case 114 passes the anchor at the spec's indentation

The artifact helper appends its last argument under spec:, six spaces landed
the field inside the huggingFace member and strict decoding refused every
anchored artifact the case created.

Task 5 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* fix(e2e): case 114's engine-block roles name an image

The replica render synthesizes the runner image from the instance type's
observed hardware; the cluster's generic type has none, so the render failed
before WeightsReady could carry the anchor block. Name the stock python image
on the role — the block creates no Pod whatever the image is.

Task 5 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* docs(specs): ship the ModelArtifact expected digest

T1-T6 delivered: the anchor's admission, assertion at resolution and
revalidation, the confirmed-outage Expected identity, the peers-only chain,
the Engine block, the documentation, and the e2e case. The plan-gate
adjudications are recorded in the spec's Open Questions and Alternatives.

Task 6 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

* fix(worker): an Expected identity's listing failure keeps its cause and class

fromPeersOnly replaced FetchManifest's error wholesale: a cancellation under
the listing — the waiter window firing — was misclassified as
SourceUnavailable and consumed the digest's backoff, and every other
underlying cause never reached the plugin's log. A cancellation now keeps its
canceled class (no backoff, the shape fetchTree's wait already uses), and any
other failure logs its cause beside the loud no-source. The ledger's
tenant-free message is unchanged.

Task 5 of model-artifact-expected-digest.

Signed-off-by: thxCode <thxcode0824@gmail.com>

---------

Signed-off-by: thxCode <thxcode0824@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/testing End-to-end test infrastructure area/worker The worker control plane and its controllers kind/api-change Adds, removes, or changes an API kind/enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant