Improve local developer experience for parallel git worktrees with a shared, versioned toolchain cache

[simonrozsival]

Summary

Building dotnet/android from multiple Git worktrees on the same machine is unnecessarily expensive and can be unsafe.

Each worktree provisions some tools independently, while other tools are installed into shared but partially unversioned directories. This causes repeated downloads and extraction, duplicated disk usage, version conflicts between branches, and races when multiple worktrees build or clean in parallel.

We should introduce a configurable, versioned cache/store for downloaded and installed build tools while keeping branch-specific mutable outputs isolated to each worktree.

Motivation

A common development workflow is:

1. Keep several branches checked out using git worktree.
2. Run make prepare and builds in multiple worktrees.
3. Switch between branches that reference different SDK or toolchain versions.
4. Occasionally build multiple worktrees concurrently.

Today this can require downloading and installing several gigabytes repeatedly. On one development machine, the Android toolchain occupies roughly 11 GB and its archive cache another 5 GB, excluding per-worktree .NET SDK installations and NuGet/Gradle content.

The current layout also allows one worktree to remove or replace tools being used by another worktree.

Current provisioning inventory

Strictly, the current make prepare target does not download every tool needed by a complete build. LLVM and most Android tooling are first acquired during make all. This issue covers the complete local provisioning lifecycle rather than only the literal prepare target.

Phase	Downloads
make prepare	dotnet-install script and exact .NET SDK
make prepare	10 Android runtime packs and 12 Mono/Emscripten workload manifest packages
make prepare	Microsoft OpenJDK archive and checksum
make prepare	Java.Interop NuGet restore, including its conditional runtime pack
make prepare	Recursive Java.Interop Git submodules
make all	Android command-line tools, Platform Tools, Build Tools, CMake, emulator, system image, Maven repository, docs, sources, NDK, and selected Android platforms
make all	xamarin-android-toolchain-<version>.7z from dotnet/android-native-tools
make all	Bundletool
make all	Gradle distributions and dependencies for R8, Manifest Merger, ProGuard rules, Java source utilities, and tests
Workload setup	dotnet workload install for android, android-deps, and optionally MAUI
Binding builds	Maven artifacts, POM files, and the Microsoft NuGet package map
Tests	Additional emulator system images through sdkmanager and remote test assets

The project named src/binutils is particularly easy to overlook. It downloads an archive containing LLVM 18.1 tools and libraries, including llc, llvm-mc, llvm-objcopy, llvm-strip, and LLD for Windows, macOS, and Linux.

Existing cache behavior

Some package managers already provide suitable global, versioned caches:

Mechanism	Current cache
NuGet	NUGET_PACKAGES / the user NuGet package cache
Gradle	GRADLE_USER_HOME, normally ~/.gradle
Maven bindings	MavenCacheDirectory, already user-global by default
Android archives	AndroidToolchainCacheDirectory, normally ~/android-archives

These should generally remain package-manager-owned caches rather than being reimplemented.

The problematic areas are installed SDKs/toolchains, archive retention, cache versioning, and concurrent access.

Current problems

Per-worktree duplication

The exact .NET SDK is installed into:

bin/<Configuration>/dotnet

Each worktree and configuration can therefore contain another complete copy of the same SDK.

Shared but unversioned installations

The Android toolchain currently defaults to ~/android-toolchain, but several destinations do not include their complete version:

jdk-21
ndk
sdk/platform-tools
sdk/emulator

Branches requiring different versions can replace one another's installations.

Shared state is removed by normal clean operations

Several BeforeTargets="Clean" targets delete globally useful content:

• _CleanOpenJDK deletes the installed JDK and downloaded archive.
• _CleanBinutils deletes the extracted LLVM/native toolchain and archive.
• _CleanBundleTool deletes its downloaded archive.
• _CleanAndroidSdkComponents removes installed SDK/NDK components.

Running make clean in one worktree can therefore force another worktree to reinstall tools or leave it with missing files.

No cross-process coordination

The shared MSBuild downloader has retry and checksum support, but it does not coordinate separate build processes.

Extraction targets generally remove and recreate their destination in place. Two worktrees requesting the same component concurrently can race, corrupt an installation, or delete a directory while the other process is using it.

The local .NET SDK is mutable

The local SDK cannot simply be redirected to one shared directory. Build and workload targets modify:

packs/
sdk-manifests/
metadata/
template-packs/
library-packs/

Different worktrees may contain different locally built Android packs and workload manifests.

Proposed design

Introduce a single environment variable for the prepare/build cache, for example:

MA_PREPARE_CACHE_DIR

When unset, it should default to a directory under the repository so the existing single-worktree experience remains self-contained. Developers using worktrees could set it globally:

export MA_PREPARE_CACHE_DIR="$HOME/.cache/dotnet-android"

A possible layout is:

$MA_PREPARE_CACHE_DIR/
  v1/
    downloads/
      sha256/<hash>/<filename>
    store/
      <host-rid>/
        dotnet/<exact-sdk-version>/
        jdk/<exact-jdk-version>/
        ndk/<exact-package-revision>/
        native-tools/<exact-version>/
        android-sdk/<toolchain-fingerprint>/
    locks/
    tmp/

The v1 component is a cache-layout schema version that can be incremented if extraction or layout semantics change.

Storage rules

Content	Proposed handling
Downloaded archives	Shared and keyed by checksum where available
Base .NET SDK	Shared, immutable, keyed by exact SDK version and host RID
Worktree .NET SDK	Small facade at the existing bin/<Configuration>/dotnet path
OpenJDK	Shared, immutable, keyed by exact version, OS, and architecture
Android NDK	Shared, immutable, keyed by exact package revision, OS, and architecture
LLVM/native tools	Shared archive and extraction keyed by XABinutilsVersion and host
Android SDK	Versioned writable instance keyed by a fingerprint of its baseline package set
Locally built workload packs and manifests	Worktree-local
Derived product outputs	Worktree-local
NuGet, Gradle, and Maven caches	Continue using their existing global caches

.NET SDK facade

To preserve existing paths and behavior, bin/<Configuration>/dotnet should remain worktree-local.

It can be assembled from the shared base SDK by:

• Copying the dotnet muxer and required root files.
• Linking or junctioning immutable host, sdk, and shared directories.
• Seeding writable SDK content where required.
• Keeping packs, sdk-manifests, metadata, template-packs, and library-packs worktree-local.
• Exporting DOTNET_ROOT and disabling multilevel lookup for deterministic resolution.
• Falling back to copying when links or hardlinks are unavailable or cross filesystem boundaries.

The immutable store should not include Configuration; Debug and Release use the same base SDK. Only their worktree-local facades and built packs need configuration separation.

Android SDK considerations

The Android SDK cannot be entirely read-only:

• License acceptance writes beneath the SDK root.
• Tests invoke sdkmanager to install emulator images.
• Additional packages may be installed as part of local testing.

The initial implementation can use a complete writable Android SDK directory keyed by a toolchain fingerprint containing all baseline package identities, versions, hashes, host OS, architecture, and cache schema.

Branches with different toolchain fingerprints would coexist. Branches with the same fingerprint could share the same installation with locking around mutations.

Component-level deduplication or assembled SDK views can be considered later if duplicated extracted SDK content becomes significant.

Atomicity and concurrency

Every shared installation should:

1. Acquire a cross-process lock for its cache key.
2. Check for a valid completion manifest.
3. Download or extract into a unique staging directory.
4. Verify checksums and expected files.
5. Write a .complete.json manifest containing the version, hash, host, and layout schema.
6. Atomically publish the staging directory when possible.
7. Handle cross-volume and Windows file-locking constraints explicitly.
8. Remove or repair incomplete installations on the next run.

The current timestamp-based source.properties and version-file sentinels are not sufficient to distinguish versions or guarantee that extraction completed successfully.

Clean and pruning semantics

Normal build cleanup must not evict globally shared content.

make clean should remove only worktree-local outputs and views. Shared cache deletion should require an explicit operation such as:

make clean-prepare-cache

A future pruning command could remove versions that have not been used recently, but automatic eviction is not required for the first implementation.

Compatibility

Existing explicit overrides must remain supported:

AndroidToolchainCacheDirectory
AndroidToolchainDirectory
AndroidSdkDirectory
AndroidNdkDirectory
JavaSdkDirectory
NUGET_PACKAGES
MavenCacheDirectory
GRADLE_USER_HOME

The existing Android NDK restriction on paths containing spaces must remain enforced.

CI currently caches ~/android-archives and uses explicit Android toolchain paths. Migration should avoid unexpectedly invalidating the existing multi-gigabyte cache.

Suggested implementation phases

Each phase should be independently shippable and leave unmigrated components using their existing paths.

Phase 1: Add opt-in cache-root plumbing

• Read MA_PREPARE_CACHE_DIR consistently from Make, shell/PowerShell bootstrap scripts, and MSBuild.
• Define derived download, store, lock, and temporary directories plus a cache-layout schema version.
• Preserve current component paths when the variable is unset during the initial rollout.
• Preserve precedence for explicit component properties: explicit property, then the new cache root, then the legacy default.
• Add a diagnostic command such as make prepare-cache-info that prints every resolved path and whether each component has migrated to the new cache.

Phase 2: Make shared downloads and installations safe

• Add cross-process locking to the shared downloader.
• Download and extract through unique staging paths and publish atomically.
• Add versioned .complete.json manifests containing the expected version, checksum, host, and layout schema.
• Make interrupted or corrupt installations self-healing.
• Stop normal Clean targets from deleting downloaded archives or shared installations.
• Add an explicit cache-cleaning command such as make clean-prepare-cache.

This safety work should land before redirecting additional mutable or unversioned destinations into one shared directory.

Phase 3: Pilot with LLVM/native tools

Migrate src/binutils first:

• Store the dotnet/android-native-tools archive beneath the new download root.
• Store its extracted LLVM/native toolchain beneath a versioned host-specific store path.
• Keep the final files copied into worktree-local product outputs.
• Leave all consumers unchanged because they continue to consume the existing local output paths.

This component is a useful pilot because its archive and extraction are already versioned and its shared state does not need to be mutated after installation.

Phase 4: Migrate other low-risk archives

Move archive caching one component at a time while leaving installation and consumer paths unchanged:

• Bundletool.
• AAPT2/Android Build Tools archive.
• Android SDK and NDK archives.
• OpenJDK archive and checksum file.

This phase eliminates repeated network transfers without yet requiring shared installed toolchains. NuGet, Gradle, and Maven should continue using their existing caches.

Phase 5: Migrate immutable installed tools

• Install OpenJDK under jdk/<exact-version>/<host-rid> and expose it through JavaSdkDirectory.
• Install the NDK under ndk/<exact-package-revision>/<host-rid> and expose it through AndroidNdkDirectory.
• Ensure normal clean operations never remove these versioned installations.

Most consumers already use these MSBuild properties, so moving the physical installation should not require widespread path changes.

Phase 6: Cache the .NET SDK archive while retaining local installations

• Cache the exact SDK archive by SDK version and host RID.
• Continue materializing the SDK into each worktree at bin/<Configuration>/dotnet.
• Preserve all current workload installation and mutation behavior.

This provides network savings with minimal compatibility risk, although extraction and disk usage remain per worktree.

Phase 7: Introduce a shared .NET SDK base and local facade

• Store the immutable base SDK globally by exact SDK version and host RID.
• Keep bin/<Configuration>/dotnet as a worktree-local facade.
• Share immutable host, sdk, and shared content where supported.
• Keep packs, sdk-manifests, metadata, template-packs, and library-packs local and writable.
• Copy rather than link where filesystem or platform limitations require it.
• Export DOTNET_ROOT and disable multilevel lookup for deterministic resolution.

This phase can follow the archive-only implementation after its behavior has been proven.

Phase 8: Introduce versioned Android SDK instances

• Compute a toolchain fingerprint from baseline package identities, versions, checksums, host OS, architecture, and layout schema.
• Install a complete writable Android SDK beneath that fingerprint.
• Lock sdkmanager mutations such as license acceptance and emulator image installation.
• Allow different fingerprints to coexist.

Component-level Android SDK storage and assembled views can be considered later if duplicate extracted SDK content remains significant.

Phase 9: Roll out defaults and maintenance tooling

• Dogfood the opt-in environment variable locally and in CI.
• Update CI cache keys and paths explicitly.
• After the design is proven, make the unset default repository-local as described above.
• Document migration from existing ~/android-archives and ~/android-toolchain layouts.
• Add cache inspection and explicit pruning based on stored completion manifests.

Acceptance criteria

• A developer can configure one cache directory shared by multiple Git worktrees.
• Two worktrees requiring the same tool version do not download or extract it twice.
• Debug and Release builds share immutable tool installations.
• Worktrees requiring different versions can coexist.
• Two worktrees can run preparation or builds concurrently without corrupting shared state.
• Running make clean in one worktree does not invalidate another worktree.
• Locally built workloads and packs remain isolated between branches.
• Interrupted downloads and extractions self-heal.
• Existing build commands and bin/<Configuration>/dotnet entry points continue to work.
• Windows, macOS, and Linux are supported.
• Existing package-manager caches remain usable.
• Shared content can be removed through an explicit cache-cleaning operation.

Out of scope

The following downloads were found during the audit but do not participate in normal local repository preparation and should not drive the initial design:

• Azure Pipelines artifact-transfer tasks.
• Arcade-generated tooling under eng/common.
• Cross-platform rootfs generation for FreeBSD, OpenBSD, illumos, Haiku, Tizen, or standalone CoreCLR Android builds.
• CI package-manager installation through apt, brew, dnf, or apk.
• VMR synchronization and source repository clones.
• Network behavior in runtime tests and vendored third-party CI scripts.