fix(llm): pin Hugging Face metadata revisions

[pjb157]

Overview

Prevent model-card registration failures when a worker and frontend resolve different commits of the same mutable Hugging Face revision.

Root cause

A ModelDeploymentCard publishes checksums computed from the metadata files in the snapshot loaded by the worker, but its Hugging Face metadata locations identify only the repository or another mutable revision such as main.

The worker and frontend resolve that location independently and may have different cache states or resolve it at different times. If the mutable revision advances, the frontend can fetch bytes from a different commit. Checksum verification then correctly rejects the metadata, but the otherwise healthy worker cannot register.

This is a time-of-check/time-of-use problem: immutable checksums were paired with a mutable artifact address.

Details

• Detect when a worker model path is an hf-hub snapshot and record its 40-character commit in a new optional source_revision model-card field.
• Keep source_path as the canonical repository identifier and preserve the served model name.
• Resolve checksummed metadata files directly from source_path@source_revision, bypassing mutable cache refs.
• Download only the metadata files referenced by the card, using the existing cancellation-safe isolated hf-hub runtime.
• Preserve the ModelExpress fallback for cards without an immutable revision and preserve local-path and self-hosted metadata behavior.

Why this belongs in Dynamo

Deployment-level mitigations can reduce the chance of a mismatch, but they do not make the model card self-consistent:

• Requiring a commit in every configured model identifier shifts correctness to each operator and can couple an external model name to artifact versioning.
• Pinning worker images or pre-populated caches does not guarantee that a separately running frontend resolves the same Hub commit.
• DYN_SELF_HOST_METADATA is a useful transport when a reachable worker system-status server is available, but it deliberately degrades to shared-storage/Hub resolution when that server is absent and may be disabled. Correctness of the fallback should not depend on deployment topology.
• Embedding metadata contents in NATS would also remove the second Hub lookup, but would turn discovery into a blob-transfer protocol with larger messages, retention/lifecycle questions, and a broader compatibility surface.

The model card already carries the metadata checksums and source location. Adding the immutable revision that identifies those bytes is the smallest protocol-level fix and applies to every deployment without imposing a particular transport.

Compatibility

The revision is intentionally separate from source_path:

• New frontend with old worker: source_revision is absent, so resolution follows the existing legacy path.
• Old frontend with new worker: the optional field is ignored and the unchanged source_path continues to use the legacy path.
• source_revision alone does not change mdcsum, allowing mixed-version rolling upgrades when the checksummed metadata is otherwise identical.
• Canonical model-ID consumers such as served-name handling and performance-model lookup continue to receive the original source path.

Once both sides understand source_revision, metadata is resolved from the exact worker commit.

Where should the reviewer start?

Start with lib/llm/src/local_model.rs, where the worker derives source_revision from the resolved cache snapshot. Then review lib/llm/src/model_card.rs, where the frontend applies that revision while resolving the card metadata.

The exact cache and network behavior is implemented in lib/llm/src/hub/huggingface.rs.

Validation

• cargo fmt --all -- --check
• cargo clippy -p dynamo-llm --no-default-features --lib --tests -- -D warnings
• cargo test -p dynamo-llm --no-default-features --lib -- --test-threads=1 (1,744 passed; 3 ignored)
• uvx pre-commit run --files lib/llm/src/hub.rs lib/llm/src/hub/huggingface.rs lib/llm/src/local_model.rs lib/llm/src/model_card.rs
• git diff --check
• Applied cleanly to current ai-dynamo/main (9f616aed66f1de3601ee8184bba53a61728467fa) with the new targeted tests and clippy passing there.

Regression coverage includes:

• stale main pointing at a different cached snapshot;
• empty-cache network download requesting the exact commit URL;
• optional-field and legacy-mdcsum compatibility;
• mutable or stale source revisions being replaced during exact resolution;
• served-model-name preservation; and
• cancellation of detached hf-hub chunk tasks.

Related Issues

This PR is not linked to an issue:

• Confirmed — no related issue

[Copilot]

Pull request overview

This PR hardens Hugging Face–backed model-card metadata resolution by pinning “hf://” sources to an immutable commit SHA derived from the worker’s resolved hf-hub cache snapshot, preventing frontend/worker registration mismatches when a mutable revision (e.g. main) advances.

Changes:

• Worker-side: record repo@<commit-sha> in the model card when the local model path is an hf-hub snapshot, while keeping the served/display name stable.
• Frontend-side: when resolving model-card metadata for hf://repo@<commit>/*, fetch/validate metadata directly from that commit snapshot (cached or downloaded), avoiding mutable cache refs.
• Add regression tests for stale main refs, revision preservation in URIs, and served-name compatibility.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
lib/llm/src/model_card.rs	Resolve hf:// metadata via pinned repo@commit snapshots when available; add regression tests.
lib/llm/src/local_model.rs	Pin worker-emitted source_path to the snapshot commit SHA without altering served name; add unit tests.
lib/llm/src/hub/huggingface.rs	Add helpers to resolve only pinned metadata files from a commit revision (cache-first, offline-aware) and tests.
lib/llm/src/hub.rs	Add from_pinned_hf_metadata entrypoint and wire it into hub exports.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.