WIP: Feature/lockfile v7

[hunger]

Description

Fixes #{issue}

How Has This Been Tested?

AI Disclosure

• This PR contains AI-generated content.
  • I have tested any AI-generated content in my PR.
  • I take responsibility for any AI-generated content in my PR.

Tools: {e.g., Claude, Codex, GitHub Copilot, ChatGPT, etc.}

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added sufficient tests to cover my changes.
• I have verified that changes that would impact the JSON schema have been made in schema/model.py.

[uwu-420]

Hi @hunger

Just wanted to ask if this PR is already in a state that's ready for some testing? I work on quite a complex monorepo that extensively relies on pixi and editable pypi dependencies so maybe I could provide some useful feedback by testing this PR.

[baszalmstra]

We are very close to merging this. If you want to start testing that would be great!

[uwu-420]

Okay nice, this PR fixes a bug I was about to create an issue for. Consider the following small example you can try yourself. Create a new project with this pyproject.toml

[project]
dependencies = []
name = "foo"
requires-python = ">=3.14"
version = "0.1.0"

[build-system]
build-backend = "hatchling.build"
requires = ["hatchling"]

[tool.pixi.workspace]
channels = ["conda-forge"]
platforms = ["osx-arm64", "linux-64", "win-64"]

[tool.pixi.pypi-options]
no-build = true

[tool.pixi.pypi-dependencies]
foo = { path = ".", editable = true }
python-dotenv = "==1.2.1,<2"

and then run pixi install. As expected, the version for python-dotenv in the lockfile then is 1.2.1.

Now change the dependency to python-dotenv = ">=1.2.1,<2" in the lockfile and run pixi install again. The version of python-dotenv in the lockfile now jumps to 1.2.2 or whatever the latest version currently is. That's a bug in my view.

And the cool thing is that this PR fixes this behavior, the dependency version does not get bumped for no good reason anymore :) Seems to be related to [tool.pixi.pypi-options] no-build = true.

❓I think it would make sense to add a regression test for this. What do you think?

In case you're interested in what an AI agent found out about this bug

Bug: no-build = true + editable PyPI dependencies causes constant lockfile re-solve

Summary

When using no-build = true in [tool.pixi.pypi-options] together with editable path dependencies, pixi install triggers a full PyPI re-solve on every invocation, even when no files have changed. This causes unexpected version bumps of transitive PyPI dependencies (e.g., python-dotenv 1.2.1 → 1.2.2).

Root Cause

The interaction between two changes creates the bug:

1. Lockfile v6 (PR feat: stop recording PyPI editable in lock file #5106, v0.62.0) removed the editable field from the lockfile. All packages are stored with editable: false (the field is omitted entirely via skip_serializing_if). Editability is now determined from the manifest at install time.

2. The PypiNoBuildCheck in satisfiability/mod.rs:750-766 checks package_data.editable (from the lockfile) to decide whether path-based packages are exempt from the no-build restriction:

   UrlOrPath::Path(path) => {
       if package_data.editable {  // ← ALWAYS false in lockfile v6
           return Ok(());          // ← never reached
       }
       let path = Path::new(path.as_str());
       if path.is_dir() {
           Ok(DistExtension::Source(SourceDistExtension::TarGz))  // ← triggers error
       }
   }

   Since editable is always false in the lockfile, path-based packages (like ./packages/pkg1) are treated as non-binary source distributions, violating no-build = true.

3. This triggers NoBuildWithNonBinaryPackages, which is handled in outdated.rs:218-228 by setting disregard_locked_content.pypi. This flag causes all previously locked PyPI records to be discarded during re-solve.

4. Without locked records as preferences, the uv resolver uses ResolutionMode::Highest and picks the latest available version of every package.

Code Flow

pixi install
  → verify_environment_satisfiability()          (satisfiability/mod.rs:498)
    → PypiNoBuildCheck::check(package_data)       (satisfiability/mod.rs:568)
      → package_data.editable is false (lockfile v6)
      → path.is_dir() returns true for "./packages/pkg1"
      → returns DistExtension::Source(TarGz)
      → NoBuildWithNonBinaryPackages error        (satisfiability/mod.rs:774)
    → disregard_locked_content.pypi.insert(env)   (outdated.rs:225)
  → UpdateContext: locked_grouped_pypi_records = None  (update.rs:1453-1455)
  → resolve_pypi() with empty preferences         (resolve/pypi.rs:617)
  → uv picks latest versions (Highest mode)
  → lockfile rewritten with bumped versions

Why the existing fix (PR #5554, v0.65.0) doesn't work

PR #5554 moved the if package_data.editable check before the is_dir() call (to avoid issues with relative paths not resolving). However, the check still reads editable from the lockfile data, which is always false in lockfile v6. The regression test hardcodes editable: true, which never occurs in real lockfiles:

// Test at satisfiability/mod.rs:2891
editable: true,  // In real lockfile v6, this is ALWAYS false

Affected Versions

• v0.62.0+ (when lockfile v6 removed the editable field)
• v0.65.0 included an incomplete fix (PR fix: tool.pixi.pypi-options.no-build = true in combination with --locked #5554)

Reproduction

Any project with:

[tool.pixi.pypi-options]
no-build = true

[tool.pixi.pypi-dependencies]
my-package = { path = "../some-package", editable = true }

Running pixi install repeatedly will re-solve PyPI dependencies every time, potentially bumping versions.

Correct Fix

The PypiNoBuildCheck should determine editability from the manifest (where editable = true is declared), not from the lockfile data. This is already how the install step works (update.rs:703-704):

// The lock file always stores editable=false, so we apply the actual
// editability from the manifest at install time.
data.editable = is_editable_from_manifest(&manifest_pypi_deps, &data.name);

The same approach should be applied during the satisfiability check.

Workaround

Use --frozen to skip the satisfiability check:

pixi install --frozen
# or
PIXI_FROZEN=true pixi install

Lockfile v7 Branch Analysis (PR #5483 — feature/lockfile-v7)

The lockfile v7 branch fixes this bug correctly. Here's what changed:

The fix: PypiNoBuildCheck::check() now reads editability from the manifest

On main (broken):

fn check(&self, package_data: &PypiPackageData) -> Result<(), EnvironmentUnsat> {
    // ...
    if package_data.editable {  // ← reads from lockfile, always false
        return Ok(());
    }

On feature/lockfile-v7 (fixed):

fn check(
    &self,
    package_data: &PypiPackageData,
    source: Option<&PixiPypiSource>,  // ← NEW: manifest source info
) -> Result<(), EnvironmentUnsat> {
    // ...
    let is_editable = source
        .map(|source| match source {
            PixiPypiSource::Path { path: _, editable } => editable.unwrap_or_default(),
            _ => false,
        })
        .unwrap_or_default();
    if is_editable {  // ← reads from MANIFEST, correctly true
        return Ok(());
    }

The caller at verify_environment_satisfiability (line 579-583) now passes manifest info:

let pypi_source = pypi_dependencies
    .get(&package_data.name)
    .and_then(|specs| specs.last())
    .map(|spec| &spec.source);
no_build_check.check(package_data, pypi_source)?;

The regression test was also fixed

The test now correctly uses PixiPypiSource (manifest) instead of PypiPackageData.editable (lockfile):

pypi_no_build_check.check(
    &PypiPackageData { /* no editable field needed */ },
    Some(&PixiPypiSource::Path {
        path: PathBuf::from("").into(),
        editable: Some(true),  // ← from manifest, not lockfile
    }),
)

Source tree hashes still present on v7

The v7 branch still computes and stores PypiSourceTreeHashable hashes for path-based packages. This means:

• Changes to pyproject.toml/setup.py/setup.cfg of an editable package will still trigger a PyPI re-solve (expected behavior — dependency metadata may have changed)
• BUT: locked records are preserved as preferences (because SourceTreeHashMismatch is is_pypi_only() and does NOT set disregard_locked_content)
• This is much better than the current behavior where locked records are discarded entirely

PR #5174 (removing input hashes) is listed in issue #5248 as part of the v7 changes but does not appear to be merged into the v7 branch yet.

Summary: v7 behavior vs current behavior

Scenario	main (v0.65.x)	feature/lockfile-v7
no-build + editable, no changes	Re-solves every time, discards locked records	No re-solve (correct)
no-build + editable, pyproject.toml changed	Re-solves, discards locked records	Re-solves, preserves locked records as preferences
Non-editable path deps + no-build	Correctly fails (source builds not allowed)	Correctly fails

Key Files

• crates/pixi_core/src/lock_file/satisfiability/mod.rs — PypiNoBuildCheck::check() (line 720)
• crates/pixi_core/src/lock_file/outdated.rs — disregard_locked_content handling (line 218)
• crates/pixi_core/src/lock_file/update.rs — locked records filtering (line 1453)
• crates/pixi_core/src/lock_file/resolve/pypi.rs — preferences from locked records (line 617)

[uwu-420]

With the current state of the PR I get a crash when running all the install tasks in my monorepo. With current pixi 0.65.0 this does not happen. For context, my monorepo consists of a bunch of apps and packages where the apps use the packages as editable pypi dependencies. The packages in the monorepo can also depend on other packages in the monorepo.

Now with pixi built from the current state of this PR the packages do not find the other packages anymore probably due to wrong resolution of relative paths?

I hope this AI summary explains the bug I encountered

Bug: Lockfile v7 path mismatch between environment section and package definitions

Summary

On the feature/lockfile-v7 branch (PR #5483), when a PyPI editable dependency uses a relative path that differs from its canonicalized form (e.g., ../../packages/pkg2 in the manifest resolves to ../pkg2 relative to the project root), the lockfile is written with inconsistent paths between the environment section and the package definitions section. This causes a crash when the lockfile is subsequently loaded.

Error

Error: Failed to load lock file from `.../packages/pkg1/pixi.lock`
  environment default and platform linux-64 refers to a package that does not exist: ../pkg2

Root Cause

The bug is in rattler_lock/src/parse/serialize.rs on the feature/lockfile-v7 branch of the conda/rattler repo.

The Verbatim<T> type

The v7 branch introduces Verbatim<T> (rattler_lock/src/verbatim.rs), which stores both a resolved inner value and an optional "given" (user-provided) string:

pub struct Verbatim<T> {
    given: Option<String>,  // original string from manifest (e.g., "../../packages/pkg2")
    inner: T,               // resolved value (e.g., UrlOrPath::Path("../pkg2"))
}

• Serialization: Uses the given string if present, otherwise serializes inner
• Deref: Verbatim<T> implements Deref<Target=T>, so it auto-coerces to &T

The path mismatch

When a PypiPackageData is created during resolution (pixi_core/src/lock_file/resolve/pypi.rs:1137-1144):

let url_or_path = Verbatim::new_with_given(
    UrlOrPath::Path(install_path),   // inner: "../pkg2" (resolved by pathdiff)
    given.to_string(),               // given: "../../packages/pkg2" (from manifest)
);

Package definitions section (v7/pypi_package_data.rs:90) serializes Cow::Borrowed(&value.location) which is &Verbatim<UrlOrPath>. The Verbatim serializer outputs the given string: ../../packages/pkg2.

Environment section (serialize.rs:143-145) declares:

enum SerializablePackageSelector<'a> {
    Pypi {
        pypi: &'a UrlOrPath,  // <-- expects &UrlOrPath, NOT &Verbatim<UrlOrPath>
    },
}

At line 182-183:

fn from_pypi(..., package: &'a PypiPackageData, ...) -> Self {
    Self::Pypi {
        pypi: &package.location,  // &Verbatim<UrlOrPath> auto-derefs to &UrlOrPath
    }
}

Rust's Deref auto-coercion silently strips the Verbatim wrapper. The UrlOrPath serializer outputs the inner resolved path: ../pkg2.

Result in the lockfile

# Environment section (uses inner/resolved path):
packages:
  linux-64:
    - pypi: ../pkg2          # <-- from UrlOrPath (inner)

# Package definitions (uses verbatim/manifest path):
- pypi: ../../packages/pkg2  # <-- from Verbatim (given)
  name: pkg2

Crash on load

When the lockfile is loaded (deserialize.rs:186-190), the pypi_url_lookup HashMap is keyed by the package definition locations (../../packages/pkg2). The environment reference (../pkg2) doesn't match, so pypi_url_lookup.get(&pypi) returns None at line 316, triggering MissingPackage.

Reproduction

Any project where the manifest path to an editable dependency differs from the shortest relative path. For example:

project-root/
  packages/
    pkg1/          <-- workspace root
      pyproject.toml <-- has: pkg2 = { path = "../../packages/pkg2", editable = true }
    pkg2/           <-- the dependency

Here ../../packages/pkg2 (manifest) resolves to ../pkg2 (shortest relative from packages/pkg1/). These are semantically identical but different strings.

Fix

Change SerializablePackageSelector::Pypi to use &'a Verbatim<UrlOrPath> instead of &'a UrlOrPath:

enum SerializablePackageSelector<'a> {
    Pypi {
        pypi: &'a Verbatim<UrlOrPath>,  // <-- preserve verbatim string
    },
}

This ensures the environment section serializes using the same given string as the package definitions, so the paths match when the lockfile is read back.

The compare_url_by_location function used for sorting would also need to be updated to accept Verbatim<UrlOrPath>, or the comparison should use the inner value.

Affected Code

• conda/rattler repo, branch feature/lockfile-v7
  • crates/rattler_lock/src/parse/serialize.rs:143-145 — SerializablePackageSelector::Pypi type
  • crates/rattler_lock/src/parse/serialize.rs:182-183 — from_pypi() auto-deref
  • crates/rattler_lock/src/parse/deserialize.rs:186-190 — pypi_url_lookup construction
  • crates/rattler_lock/src/parse/deserialize.rs:316 — lookup that fails
  • crates/rattler_lock/src/verbatim.rs — Verbatim<T> Deref impl enabling silent coercion

[baszalmstra]

Nice! Thanks for this! @hunger can you try to implement the fix?

[uwu-420]

Also there was this debug output when running pixi install although I haven't set the output to verbose. Probably something to clean up at some point.

Debug VersionOrUrl: Ok(Requirement { name: PackageName("pydantic"), extras: [], version_or_url: Some(VersionSpecifier(VersionSpecifiers([VersionSpecifier { operator: GreaterThanEqual, version: "2.10.6" }, VersionSpecifier { operator: LessThan, version: "3" }]))), marker: true, origin: None })

[hunger]

@uwu-420: Yes, I really need to double-check I have not lettered debug output :-/ Thanks for checking.

I am currently busy trying to land another PR that decouples pixi a bit better from the lockfile data for pypi packages, which will enable us to hopefully have less info in lockfile-v7 than now -- and should fix a bug with pypi package versions that a test in Nichita's PR to remove the hash from local source packages exposed.

[hunger]

We also moved over to the "real" pixi repo in the meantime: https://github.com/prefix-dev/pixi/tree/feature/lockfile-v7

This branch is stale.

[uwu-420]

@hunger since this PR has been closed now, is there another PR where I should mention this bug? #5483 (comment)

[baszalmstra]

Yes! #5607 is the right branch