Revisit modular diffusers weight-duplication workaround now that huggingface_hub has a shared blobs cache #14802

huggingface_hub v1.32.0 ships a shared blobs cache: Xet-backed files are now deduplicated locally across repos (best-effort, via symlinks, backward compatible with older versions and opt-out possible). See the release notes and huggingface/huggingface_hub#4498.

Concretely, downloading two repos that share components now only pays the download cost once — e.g. Qwen/Qwen-Image and Qwen/Qwen-Image-Edit share ~29% of their data (16.8G/57.7G), FLUX.1-dev / FLUX.1-schnell ~18% (10.3GB), Kwai-Kolors/Kolors / Kolors-Inpainting ~56% (>20GB).

This is relevant to modular diffusers repositories, which duplicate weights from the original repos. We currently have a workaround so that weights are fetched from the original repositories instead of the modular repo, precisely because cross-repo local dedup was missing. With the shared blobs cache in place, that workaround may no longer be necessary — modular repos would be a particularly striking use case for the new dedup (near-zero incremental download for duplicated weights).

Proposed follow-ups:

• Verify dedup actually kicks in for modular diffusers repos (Xet-enabled, identical blobs) and measure the gain.
• If confirmed, plan removal/simplification of the "download from original repo" workaround, bumping the huggingface_hub floor as needed and keeping a fallback for older versions.

Requested by Sayak Paul - Slack thread - Agent trace