Repository navigation
Conversation
…d its constants, the output decoding helpers) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
…d in the new index, two rebuilds never race for the swap (0.34.0.1) - <home>/index.lock (the sidecar of the index folder, kept through the swap) is held by the full reindex for its whole rebuild and swap, by the incremental index of pull/plug/repo reindex for its duration and by every write of a record for a moment; readers never take it - a writer that finds it held waits: a notice on stderr after 3 s naming the reindexing process (which leaves a note in the lock file), up to CMETA_INDEX_LOCK_TIMEOUT seconds (600), then an error - the first run on a home takes the lock before deciding that the index is missing, so several first runs at once build one index; the home and its local repository folder are created with exist_ok - under the lock a reindex removes every index.tmp-* / index.old-* leftover of killed rebuilds - PathLock.acquire(note=...), PathLock.write_note(), PathLock.read_note() - tests/internal_repo_tests/test_integration_index_guard.py: writers of every kind during a reindex paused in another process, two reindexes at once, four first runs on a fresh home, the leftovers, the notice and the timeout, the note - docs/async-and-concurrency.md "The index lock", AGENTS.md and docs/using-cmeta.md env lists, CHANGELOG Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
…s it, structured files are written atomically, a create is one step under the index lock, repos.json is locked by the incremental index, the reader cache keys on time, size and inode (0.34.0.1) - CMETA_LOCK_TIMEOUT (30 s) for the lock of a file: a notice on stderr after 3 s naming the holder's note, then an error naming it; timeout=None = the engine's defaults in safe_read_file, safe_write_file, safe_delete_directory, lock_path; the no-lock read retry stays 3 s; lock_timeout() and _acquire_with_notice() shared with the index lock - safe_write_file(atomic=None): JSON, YAML and pickle written next to the target and put in place with one replace (_install_file: the mode kept, a symbolic link written through, a read-only target refused); a failed write removes its temporary; text stays in place - create_ makes the folder, the meta and the record under the index lock (Repos.lock_index, add_to_index(index_lock=)); several creates of one artifact end with one artifact and "already exists"; a create whose record cannot be written removes what it made (_undo_create) - _index reads and rewrites repos.json under its lock; a new category index file is written atomically too - safe_read_file_via_cache keyed by (st_mtime_ns, st_size, st_ino), its dict updated under a lock, one stat per lookup; a file read before that is missing for a moment (a replace on a mount whose rename is not atomic, the swap of a full reindex) is retried three times before "not found" - _replace_file retried on EACCES/EPERM/EBUSY with a delay doubling 1 ms -> 100 ms until the lock timeout; a target that cannot be written fails at once - Repos.find hands out one-level copies of the records it returns (the record, its cmeta, its tags), made after its tag and match filters; find_in_index is unchanged - index files written with pickle protocol 4 - tests/core_tests/test_utils_files_writes.py, tests/internal_repo_tests/test_integration_create_guard.py, the index guard test checks that a create waits before it writes its folder - docs/async-and-concurrency.md, AGENTS.md and docs/using-cmeta.md env lists, CHANGELOG Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cMeta v0.34.1: the rest of the concurrency hardening (point 2 of the engine determinism work), on top of the engine's own path lock of 0.34.0.
The index lock. A full
cx --reindexbuilds the new index in a temporary folder and swaps it in; a record written to the live index meanwhile by another process was lost in the swap (a deletion came back, a creation stayed unindexed, an update reverted).<home>/index.lockis now held by the full reindex for its whole rebuild and swap, by the index of pulled or plugged repositories for its duration and by every write of a record for a moment; a writer that arrives during a reindex waits for it (a notice after 3 s,CMETA_INDEX_LOCK_TIMEOUT= 600 s) and then writes into the new index; two reindexes run one after the other; several first runs on a fresh home build one index; the leftovers of killed rebuilds are removed. Readers never take the lock.The write side. A writer waits up to
CMETA_LOCK_TIMEOUTseconds (30) for the lock of a file and says on stderr who holds it (instead of failing after 3 s).safe_write_filewrites JSON, YAML and pickle atomically by default (the mode kept, a symbolic link written through, a read-only target refused), so a new artifact's meta,repos.json, a new category index file and the files of the internal categories can no longer be seen half-written or left broken by a crash.createmakes the folder, the meta and the record under the index lock: a full reindex never sees a half-made artifact, several creates of one artifact at once end with one artifact and "already exists" errors, a create whose record cannot be written removes what it made. The incremental index reads and rewritesrepos.jsonunder its lock. The reader cache of the index is keyed by modification time in nanoseconds, size and inode (coarse file systems), and updated under a lock for threads. A replace whose target is held open (Windows) is retried until the lock timeout.findhands out one-level copies of each record. Index files are written with pickle protocol 4.Everything is backward compatible: no command, option or file format changes;
timeout=Noneis the new default of the file helpers, an explicit value is honoured as before.Tests:
test_integration_index_guard.py,test_integration_create_guard.py,test_utils_files_writes.py(+ the suite); verified on Windows 3.14, Ubuntu 3.14 and the free-threaded 3.14t, Ubuntu 3.9, macOS 27 arm64, an Ubuntu 24.04 container, WSL2 and its /mnt/c (9p), with a cserver load test and the engine benchmark. See CHANGELOG.md.🤖 Generated with Claude Code