Repository navigation
Merged
Conversation
…y is indexed again from its folder, nothing written (0.33.1.1) reindex rewrites the record of one artifact (or several: wildcards, tags, a repository, everything) from the meta file in its folder, with the full reindex own scanner, drops the stale aliases of its UID in the same locked write, finds a folder renamed by hand by its new name, its UID or its old name, and never touches a folder: the only record it removes is that of an artifact whose folder now holds another artifact. A missing folder, an unreadable meta, an invalid UID, another category meta or one UID in two folders is an error that leaves the index as it was. cx repo reindex indexes one registered repository again (its _cmr.yaml and its artifacts) with the incremental index of pull and plug, and refuses a repository whose folder is not there. Fixed on the way in repos.py: the incremental index kept the records of a category whose last artifact in a repository was removed; the file-system scan of a no_index category applied a wildcard only in sharded categories, crashed on a meta without category and found only the first of two spellings of an alias. Tests: tests/internal_repo_tests/test_integration_reindex_artifact.py and test_integration_repo_reindex.py (42 cases); the suite run on Windows, macOS and Ubuntu with Python 3.9, 3.12 and 3.14. The six-processes test is xfail on non-Windows: the lock file is deleted after every release, a defect of the concurrency step. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
…ck, lock files still removed after use (0.33.1.1) - cmeta.utils.files.PathLock: a byte-range lock (msvcrt) on Windows; flock, then POSIX record locks where flock is refused, then a soft lock file where both are refused, on POSIX; the file it holds is checked to be the file at the path after acquiring; the file is removed before the lock is dropped; the threads of one process are serialized per path; stale soft locks are cleared; a busy lock is polled every 1-25 ms; the timeouts, the lock file names and the functions that use them are unchanged - filelock is no longer a dependency (pyproject.toml, uv.lock, AGENTS.md, CLAUDE.md) - tests/core_tests/test_utils_files_path_lock.py: 21 tests (processes, threads, the engine's read-modify-write cycle, a killed holder, the removed-file race, the soft lock and its stale files, the record-lock fallback, timeouts); test_several_processes_reindex_at_once is no longer expected to fail on Linux - tests/benchmarks/benchmark_engine.py and its smoke test: the engine's hot paths on a home of N artifacts, --json and --compare - docs/async-and-concurrency.md: how the lock works on each platform - the new tests and the benchmark start their worker processes with an explicit null input: the inherited standard handles can be unusable under pytest's capture on Windows Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cMeta v0.34.0 - see
CHANGELOG.md(## 0.34.0).The engine's own file lock replaces the
filelocklibrary. Two processes can no longer hold one lock at once (withflock, a waiter on a removed lock file and a newcomer on the new one both acquired, and one of two concurrent index writes was lost), and lock files are still removed after use.cmeta.utils.files.PathLock: a byte-range lock on Windows;flock, then POSIX record locks whereflockis refused, then a soft lock file where both are refused, on POSIX; the file is checked after acquiring and removed before the lock is dropped; threads of one process are serialized per path; a busy lock is polled every 1-25 ms. Timeouts, lock file names and the functions that use them are unchanged.filelockis no longer a dependency. 21 new tests;test_several_processes_reindex_at_oncenow passes on every platform.cx <category> reindex [<artifact>]andcx repo reindex <repository>: the index record of an artifact, or the records of one repository, are refreshed from disk and no file is changed. Four pre-existing scanner/index issues fixed on the way (see the CHANGELOG). 42 new tests.tests/benchmarks/benchmark_engine.py: a benchmark of the engine's hot paths with--jsonand--compare, plus a smoke test in the suite.Tested with the full suite on Windows 11 (3.14), Ubuntu (3.9, 3.14, and the free-threaded 3.14t), macOS 27 arm64 (3.14), an Ubuntu 24.04 container and WSL2 (including the lock tests on a Windows drive seen through 9p).
🤖 Generated with Claude Code