Add support for content addressable gems

[jenshenny]

This issue will track the progress of implementing content addressable gems and skinny binary support. Sub issues will be created that will provide more detail on implementation in the CLI and server.

Background & Motivation

Today, precompiled gems ship as "fat" binaries: a single .gem (e.g. my_gem-1.1.0-arm64-darwin.gem) bundles a native binary for every supported Ruby version, even though a consumer only ever uses one.

To better support gems with native extensions, RubyGems/RubyGems.org teams agreed on a design to support "skinny" binaries instead — one small artifact per Ruby ABI — making installs faster (smaller downloads) and simplifying how maintainers build and ship precompiled gems.

The naming blocker

Gem files are named <gem name>-<version>-<platform>.gem with a uniqueness constraint. Multiple skinny binaries for the same name, version, and platform would collide.

To fix this, we replace the platform string with an 8-character checksum derived from the file contents (content-addressable naming):

my_gem-1.1.0-arm64-darwin.gem    # current
my_gem-1.1.0-bb65ajk8.gem        # content-addressable

An alternative is to encode Ruby ABI + platform tags directly into the filename. That solves the same collision problem, but it keeps growing the name as we add more distinguishing dimensions (ABI, platform, and later system requirements, Ruby engine, etc.). Decoupling that metadata from the name entirely — moving it into compact-index requirements — keeps gem names short and fixed-length, and lets us add new matching dimensions later without another naming-scheme change.

[jenshenny]

CLI changes

1. Building (Design not finalized)

• Detect skinny gems — recognize a platformed gem pinned to a single ABI (both ~> X.Y.Z and rake-compiler's >= X.Y, < X.(Y+1).dev).
• gem build produces <name>-<version>-<sha>.gem for skinny gems. The gem is built in memory, the content address is computed from the bytes, and the file is written in a single pass. Fat/source builds are unchanged.
• --ruby-abi selector lets a maintainer request a skinny build explicitly. If it conflicts with required_ruby_version the client raises; if required_ruby_version is absent the appropriate ~> value is added.
• The version suffix is carried as an optional extra field on the stub line (to_ruby), so full_name can reconstruct <name>-<version>-<sha> offline without the .gem bytes.

2. Installing

Install flows we need to make sure are supported:

• gem install <gem> from a remote source selects and installs the correct skinny binary for the running Ruby.
• bundle install from a remote source resolves and installs the skinny binary.
• bundle install --local resolves a skinny binary from the local cache (lockfile round-trip).
• gem install ./<name>-<version>-<sha>.gem installs a local content-addressed file directly.
• require works after install, and a re-install is idempotent.
• An old client transparently falls back to the fat binary (never requests the skinny gem).
• No matching skinny binary → falls back to fat, then source, without stranding.
• The on-disk install directory is consistent between the remote and --local paths.

Resolution

The platform moves out of the filename and into a platform: requirement on the compact-index line, gated by rubygems:>= so old clients ignore skinny rows:

1.0.0          |checksum:…,ruby:>= 3.1,rubygems:>= 3.3.22            # source
1.0.0-x86_64-linux |checksum:…,ruby:>= 3.1                              # fat
1.0.0-9f3c1a2b |checksum:…,ruby:~> 3.3.0,rubygems:>= 4.1.0.dev,platform:= x86_64-linux   # skinny

• Preference order: skinny → fat → source. When a skinny binary compatible with the running Ruby exists, the client prefers it exclusively; otherwise it falls back to the fat/source variant so a usable binary is never stranded. Because per-minor ~> ranges are disjoint, at most one skinny qualifies — there's no skinny-vs-skinny tie-break.
• Bundler (endpoint_specification.rb) parses the platform: token into a platform_requirement and matches compatibility on the real platform (since the version token holds the SHA). match_platform.rb implements the prefer-skinny-else-fall-back selection.
• Old clients drop skinny rows because their RubyGems version doesn't satisfy the rubygems:>= gate (and the SHA token doesn't match a real platform anyway), so they transparently install the fat binary.

4. Other CLI commands

• gem list -r, gem info -r, gem search -r decode the token to the real platform via the platform: field, never display the hash, and add a Ruby ABI column.
• gem push accepts --platform and --ruby selectors and resolves the matching file by reading each file's internal spec.
• Local files / errors — any output or error message should likewise decode and show the real platform rather than the hash.