Move all yarn.lock parsing into formats/yarn and stop pinning non-registry copies

[Mikola Lysenko (mikolalysenko)]

Problem

Architecture audit items E08, B60 and B16 (theme 3.E, "JS lockfiles"):

• E08: yarn.lock had seven split("\n\n") block grammars in the hosted rewriters and restorers (patch/redirect/mod.rs x5, upstream/npm.rs x2), plus regex field edits. The layering was inverted: hosted code imported the block grammar from the vendored backends. The classic rewriter passed the artifact URL to Regex::replace unescaped, while restore escaped it. Line-ending handling also differed by path: classic hosted turned a mixed CRLF/LF lock into all-CRLF, while berry and classic restore refused it.
• B60: the yarn grammar was decided three ways:
  • a 30-line head sniff (the vendored and in-memory routers);
  • a whole-file __metadata: scan (hosted, restore, VEX, the classic gate);
  • trying both readers (the inventory fallback).
• B16: hosted yarn classic repointed file:-tarball, URL and hosted-git (codeload) copies at Socket's patched registry artifact, which silently replaced the user's fork or local build. Rollback then wrote the registry tarball back and lost the original resolved. The service-built vendored tarball did the same. The lock inventory classified codeload copies as git, but the rewriters treated them as ordinary tarballs.

No issue number tracks these exactly; the nearest are #363, #857, #921 and #939.

Change

1. Move the grammar (pure move). The block walk, field readers, key/descriptor/locator patterns and the copy classifier move from vendor::yarn_classic_lock / vendor::yarn_berry_lock into formats/yarn/{blocks,patterns,source}.rs. Every caller imports them from there now.
2. One grammar decision. sniff_grammar scans the whole file and is_berry_lock wraps it. The new grammar() reads a header-less lock as classic, which is how yarn 1 parses it. The vendored router still refuses a header-less lock, as it did before. The inventory fallback goes through grammar() instead of trying both readers.
3. Hosted writers on the shared grammar.
   • Classic rewrite and restore walk the lock with scan_blocks and splice each block in place (replace_block) through repin_classic_block. The vendored backend uses the same function.
   • Every untouched byte round-trips, so a mixed CRLF/LF lock keeps its endings. Only a bare CR is refused, by rewrite and restore alike.
   • No regex replacement is left, so a $ in the artifact URL is written as-is.
   • A classic block with no resolved is left untouched. The old code swapped its integrity.
   • Berry rewrite and restore share formats/yarn/stanzas.rs for BOM, line endings, trailing newlines and sorted re-insertion. A test asserts the stanza view and scan_blocks name the same blocks on LF, CRLF, BOM and header-comment locks.
   • The remaining grammar helpers in patch/redirect/mod.rs (classic_line_endings_supported, berry_lock_locks, berry_bin_entries, berry_npm_alias_target) move into formats/yarn/{blocks,patterns}.rs. The is_berry_lock re-export in patch::redirect is deleted; VEX discovery and the hosted engine import it from formats::yarn.
4. Berry gates on the shared grammar (after Fix yarn berry project gates drifting between modes (#628, #629) #657). Fix yarn berry project gates drifting between modes (#628, #629) #657 added formats/yarn/berry_gates.rs with its own __metadata reader (metadata_fields, scalar_field). Those are deleted: the gates read cacheKey through scan_blocks, berry_metadata and berry_field, and berry_gates::cache_key takes scanned blocks. This PR's own berry_cache_key is dropped in favour of the gates, so one copy remains.
5. CopySource classifier (B16). ClassicBlockSource::Tarball splits into Registry and RemoteTarball, using the shared npm_spec_is_registry rule. source.rs holds the per-mode policy table:
   • Hosted: skips RemoteTarball copies, named in the new warning redirect_yarn_classic_non_registry_entry_skipped (same pattern as npm's redirect_npm_non_registry_entry_skipped) and kept out of the in-run VEX. A copy an older release already pinned to this run's artifact gets redirect_yarn_classic_non_registry_legacy_pin instead: it installs Socket's build, so it is not called unpatched, and the warning points to restoring yarn.lock from version control.
   • Hosted, no resolved: a registry block with no resolved line gets redirect_yarn_classic_unresolved_entry_skipped and stays out of the in-run VEX. Before, it was silently counted as matched.
   • Hosted restore: refuses a pin that an older release wrote on such a copy.
   • Vendored: skips it with the new warning vendor_yarn_classic_non_registry_entry_skipped, and refuses with vendor_lock_entry_not_rewritable when it is the only copy. A copy an older release already wired into .socket/vendor/ stays a candidate, so an in-sync re-run is a byte-stable no-op, and is named with vendor_yarn_classic_non_registry_legacy_wiring (pointing to vendor --revert).
   • Lock inventory: drops the registry verifiers through the same rule.
   • Docs: docs/ecosystems.md is updated.

Duplicates deleted (before → after)

• split("\n\n") yarn grammars: 7 → 1. The one left is the stanza view in formats/yarn/stanzas.rs, used only by the berry writers, which re-order entries.
• Regex field replacement in yarn writers: 5 call sites → 0. All edits go through with_body_field / repin_classic_block.
• Classic pin splice (vendored rewrite_classic_block, hosted regex, restore regex): 3 → 1 (repin_classic_block).
• "Every key pattern names one package" check (yarn_classic_block_head, vendored classify_classic_block, VEX classic_block_purl): 3 → 1 (classic_key_real_name).
• Yarn grammar decision: 3 → 1.
• Berry __metadata / cacheKey readers (Fix yarn berry project gates drifting between modes (#628, #629) #657's metadata_fields + scalar_field, this branch's berry_cache_key, blocks::berry_metadata + berry_field): 3 → 1.
• Non-registry classifier for classic copies (ClassicBlockSource, inventory is_git_resolution): 2 → 1 (CopySource).
• Deleted outright: berry_sort_key, berry_entries_sorted, berry_reposition_blocks, yarn_classic_block_head, is_git_resolution. The vendor-module copies of every moved function are gone too.

Testing

Rebased on origin/main (431b8188, includes #657). All commands ran in the worktree through heavy-job.sh with CARGO_INCREMENTAL=0:

• cargo test -p socket-patch-core --lib: 5648 passed (before the stanza test was added; formats::yarn re-run after it, 24 passed).
• cargo test -p socket-patch-cli with --test in_process_redirect, in_process_vendor, in_process_rollback_hosted, hosted_memory_engine, hosted_memory_parity, in_process_get_modes, covgap_commands_vendor, covgap_commands_scan_hosted, covgap_commands_vex, covgap_commands_rollback, covgap_commands_scan_mod, e2e_vex_redirect, e2e_vex_vendor, e2e_safety_yarn_pnp, e2e_redirect_yarn_classic_build, e2e_vendor_yarn_classic_build, e2e_vendor_yarn_classic_dev_flow, cli_parse_list, global_scope_project_state, in_process_redirect_pnpm: all pass.
• mode_migration_npm: 18 pass, 1 fails (berry_vendored_then_hosted_takeover_leaves_pure_hosted), the same failure as on the pre-rebase branch and before this change: it expects a pristine package.json after a hosted berry pin, but the pin has written root resolutions by design since Fix yarn berry hosted pin leaking npm auth (#404) #465. Not touched here.
• cargo clippy --workspace --all-features -- -D warnings -A unused-variables (the CI invocation; unused-variables allowed only for the macOS-only python_crawler.rs unix_default warning on main): clean.
• redirect_golden (the fixtures shared with depscan) pins a classic edit's original/new as the split("\n\n") segment holding the block; the block-scan rewriter records that same segment again (ClassicSegments), so yarn_classic_rewrite.golden differs from main only by the unresolved-entry change.
• Scan performance: classic pins are spliced in one pass and berry's version check no longer collects every stanza's lines; local compare vs 431b8188: yarn-classic hosted +4.6%, berry hosted +1.6%.
• yarn_classic_rewrite.golden is re-blessed once more for the new unresolved-entry warning; every input digest is unchanged. (Earlier on this branch, a temporary oracle over all 400 seeds showed the new rewriter differing from the old only in the edit records' leading blank line and the unresolved-block integrity swap.)
• New tests:
  • classic mixed CRLF/LF lock keeps its untouched lines; $ in the artifact URL stays literal;
  • an unresolved block is left untouched and named (hosted);
  • file:/URL/codeload copies are skipped (hosted and vendored);
  • a legacy hosted pin on a URL-keyed copy is named, not called unpatched (hosted);
  • a legacy vendored wiring on a URL-keyed copy re-runs in sync and is named (vendored);
  • rollback refuses a hosted pin on a URL-keyed copy;
  • inventory: file:/URL fork copies carry no registry verifiers; a header-less classic lock is read as classic by the fallback;
  • the grammar sniff agrees with the readers on a marker past line 30;
  • the stanza view and scan_blocks agree on every line ending;
  • CopySource cases and the stanza round trip.
• The two inventory tests pin behaviour this branch changed; they were written after the change, so they were not observed failing against the old code.
• Not run locally: Linux, Windows and docker suites (left to CI).

Deferred

• Header-less yarn.lock: readers now treat it as classic, and the vendored router still refuses it with vendor_lockfile_version_unsupported. Whether vendor should wire it as classic is a policy call, so the current behavior is kept.
• Bundled and berry non-registry copies: berry already rewrites only npm: ranges. pnpm and bun copies belong to other workstreams (Fix 15 open pnpm issues across hosted, vendored and agent modes #1007, Fix open bun issues (#992, #861, #784, #764, #735, #635, #599, #578, #497, #443, #371) #1009).
• formats -> vendor edges: formats/yarn/{source,blocks}.rs still import vendor::npm_origin::npm_spec_is_registry and vendor::common::detect_eol, as other formats/* modules do. Moving those into a shared npm-spec/text module is a later pass.
• Already on main: Fix hosted yarn classic pins missing berry warning (#907) #917 (berry-migration warning) and Fix VEX attesting beside an unpatched same-lock copy (#935, #938, #939) #940 (same-lock unpatched-copy VEX) merged while this was in flight. This branch is rebased on both and keeps their behavior.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes yarn classic pin/restore/rollback semantics for non-registry lock entries and refactors shared lockfile parsing used across hosted, vendor, and inventory paths; behavior is heavily tested but mistakes could mis-wire or fail rollbacks.

Overview
Consolidates all yarn.lock parsing under formats/yarn (blocks, patterns, source, stanzas) so hosted rewriters, restorers, vendored backends, gates, and inventory share one block walk and field readers. Grammar detection is unified: whole-file sniff_grammar / grammar() replaces the old 30-line head sniff vs full-file split.

Classic hosted and vendored paths stop using split("\n\n") and regex field edits. They splice blocks in place via scan_blocks, repin_classic_block, and replace_block, preserving mixed CRLF/LF outside edited spans and treating $ in URLs literally. Berry writers use BerryStanzas for BOM, EOL, and sorted re-insertion.

B16 behavior change: CopySource distinguishes registry copies from remote tarballs (file: tgz, URL forks, codeload shorthands). Non-registry blocks are no longer repointed to Socket’s registry artifact; they are skipped with new warnings, excluded from VEX where applicable, and rollback refuses legacy hosted pins on those keys. Lock inventory drops registry verifiers for fork copies. Docs in docs/ecosystems.md are updated.

^{Reviewed by Cursor Bugbot for commit 209e36e. Configure here.}

---

Generated by Claude Code