Skip to content

20261009 - cMeta v0.34.1: the index lock and the write side of the concurrency hardening #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gfursin merged 4 commits into from
Oct 9, 2026
Merged

20261009 - cMeta v0.34.1: the index lock and the write side of the concurrency hardening #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e9a4ab7
docs: the API pages regenerated by the 0.34.0 docs build (PathLock an…
gfursin Oct 9, 2026
142ef77
The index lock: writes made during a full reindex wait for it and lan…
gfursin Oct 9, 2026
e1632f8
The write side: a writer waits 30 s for a busy lock and says who hold…
gfursin Oct 9, 2026
ce11b6c
cMeta v0.34.1
gfursin Oct 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,9 @@ tabulate, tqdm, packaging, psutil. (File locks are the engine's own
### Useful environment variables
`CMETA_HOME`, `CMETA_DEBUG=1`, `CMETA_LOG=DEBUG|INFO`, `CMETA_LOG_FILE=<path>`,
`CMETA_FAIL_ON_ERROR=yes`, `CMETA_VERBOSE=yes`, `CMETA_INTERNAL_REPO_PATH`,
`CMETA_PIP_INSTALL_ARGS`, and the provenance stamped on new artifacts: `CMETA_AUTHORS`, `CMETA_COPYRIGHT`,
`CMETA_PIP_INSTALL_ARGS`, `CMETA_INDEX_LOCK_TIMEOUT` (seconds a write to the index waits for a running
reindex, 600), `CMETA_LOCK_TIMEOUT` (seconds a writer waits for the lock of a file, 30), and the
provenance stamped on new artifacts: `CMETA_AUTHORS`, `CMETA_COPYRIGHT`,
`CMETA_GENERATOR` (JSON: how the artifact was made; with the `artifact_defaults` of a repository's
`_cmr.yaml` - docs/using-cmeta.md §7.6).

Expand Down
65 changes: 65 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,71 @@
All notable changes to cMeta are documented here, newest first.


## 0.34.1
- **The index lock: a full reindex no longer loses the writes made while it runs, and two rebuilds no
longer race for the swap.** A full `cx --reindex` builds the new index in a temporary folder and swaps
it in when it is complete; a record written to the live index meanwhile by another process (a
create, update, delete or reindex of an artifact) was lost in the swap, and a deletion came back as a
record whose folder was gone. The lock file `<home>/index.lock` - the sidecar of the index folder,
which the swap leaves in place - is now held by the full reindex for its whole rebuild and swap, by
the index of pulled or plugged repositories for its duration, and by every write of a record for a
moment: a writer that arrives during a reindex waits for it (a notice on stderr after 3 s naming the
reindexing process, which leaves a note in the lock file; `CMETA_INDEX_LOCK_TIMEOUT`, 600 s by
default, bounds the wait, then the write fails with an error) and then writes into the new index; two
reindexes run one after the other; several processes starting on a fresh home at once build one
index, the others wait and find it; a process that starts during the swap waits instead of
rebuilding. Under the same lock a reindex removes the `index.tmp-*` / `index.old-*` leftovers of
killed rebuilds (before, only its own). Readers never take the lock, so lookups cost what they did; a
write costs one more lock acquisition. `PathLock.acquire` takes a `note` and `PathLock.read_note`
reads it. Tests: `tests/internal_repo_tests/test_integration_index_guard.py` (writers of every kind
during a paused reindex, two reindexes at once, four first runs on a fresh home, the leftovers, the
notice and the timeout, the note). With it, the first run creates the home and its local repository
folder with `exist_ok`: two first runs on one home at once no longer fail each other with "cannot
create a file when that file already exists".
- **The write side made whole: a writer waits 30 s for a busy lock and says who holds it, every
structured file is written atomically, a create is one step under the index lock, the index of
repositories holds the lock of `repos.json`, the reader cache keys on time, size and inode, a replace
is retried until the lock timeout, `find` hands out copies, index files use pickle protocol 4.**
- A writer that found the lock of a file held (an index file, a meta, `repos.json`) waited 3 s and
failed; under load (server workers, a test matrix) a `create` had written its meta and then failed to
index it, leaving a folder the index did not know. A writer now waits up to `CMETA_LOCK_TIMEOUT`
seconds (30), prints a notice on stderr after 3 s naming the holder when it left a note in the lock
file, and then fails with an error that names the holder and the variable. `timeout=None` (the new
default of `safe_read_file`, `safe_write_file`, `safe_delete_directory`, `lock_path`) means the
engine's timeouts; an explicit value is used as before; without a lock, a read that fails is still
retried for 3 s. `cmeta.utils.files.lock_timeout()` reads the setting.
- `safe_write_file` writes a JSON, YAML or pickle file atomically by default (`<file>.tmp` + one
replace; text stays in place): a new artifact's meta, the first run's `repos.json` and local
repository meta, a new category index file and the files of the internal categories can no longer be
seen half-written or left broken by a crash (a broken meta made the next reindex skip the artifact).
A failed write removes its temporary; the mode of the target is kept; a symbolic link is written
through; a read-only target is refused as before.
- `create` makes the folder, the meta and the record under the index lock (`Repos.lock_index`,
`add_to_index(index_lock=...)`): a full reindex never sees a half-made artifact, several creates of
one artifact at once end with one artifact and "already exists" errors (return code 8), and a create
whose record cannot be written removes what it made.
- The incremental index (`pull`, `plug`, `unplug`, `repo reindex`) reads and rewrites `repos.json`
under its lock: a plug or unplug of another process at the same moment is neither lost nor read
half-written.
- The reader cache of the index (`safe_read_file_via_cache`) is keyed by the modification time in
nanoseconds, the size and the inode instead of the modification time in seconds: two writes within
one tick of a coarse file system (FAT, exFAT, some network mounts) no longer leave a long-running
process with a stale index; an index file read before that is missing for a moment (a replace on a
mount whose rename is not atomic, such as 9p, or the swap of a full reindex) is retried a few times
(75 ms at most) before "not found" - a file never seen is still not retried; its dict is updated
under a lock for the threads of one process; two `stat`s fewer per lookup.
- A replace whose target is held open (Windows: a reader of an index file, a scanner) is retried with
a delay that doubles from 1 ms to 100 ms until the lock timeout, instead of 10 times 0.1 s; a target
that cannot be written at all fails at once.
- `find` hands out one-level copies of the records it returns (the record, its `cmeta`, its `tags`;
made after its tag and match filters, so only what is returned pays for it): a caller that edits
what it gets (the aops selection adds a tag) no longer edits the cached index of its process. Deeper
values, and the records of the lower-level `find_in_index`, are shared and must be left as they are.
- Index files are written with pickle protocol 4, readable by every supported Python whatever the
default of the Python that writes. Tests: `tests/core_tests/test_utils_files_writes.py`,
`tests/internal_repo_tests/test_integration_create_guard.py`; the index guard test now checks that a
create waits before it writes its folder.

## 0.34.0
- **The engine's own path lock replaces the `filelock` library: two processes can no longer hold one
lock at once, and lock files are still removed after use.** The previous lock removed its file after
Expand Down
2 changes: 1 addition & 1 deletion CITATION.cff
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ authors:
website: "https://cTuning.ai/@gfursin"
affiliation: "cTuning Labs"
orcid: "https://orcid.org/0000-0001-7719-1624"
version: "0.34.0"
version: "0.34.1"
date-released: "2026-10-09"
license: Apache-2.0
repository-code: "https://github.com/cTuningLabs/cmeta"
Expand Down
216 changes: 136 additions & 80 deletions cmeta/category_api_v1.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1076,110 +1076,136 @@ def create_(
# The meta file of an artifact that is indexed as it is (made or copied by hand)
existing_meta_file = None

if not virtual:
if os.path.isdir(artifact_path):
if os.path.isfile(cmeta_filename_json) or os.path.isfile(cmeta_filename_yaml):
if index:
if os.path.isfile(cmeta_filename_json):
f = cmeta_filename_json
else:
f = cmeta_filename_yaml
yaml = True

# Do not use lock since it's not in the system and not in the index
r = utils.files.safe_read_file(f, fail_on_error=self.fail_on_error, logger=self.logger)
if r['return']>0: return r

meta = r['data']
existing_meta_file = f
# The folder, the meta and the record of a new artifact are made under the index lock: a full
# reindex running meanwhile never sees a half-made artifact, two creates of one artifact at once
# end with one artifact (the second finds the folder of the first), and a create whose record
# cannot be written removes what it made, so that nothing is left unindexed
index_lock = None
if not no_index:
r = self.cm.repos.lock_index('the creation of an artifact')
if r['return']>0: return r
index_lock = r['lock']

disk_path = artifact_path # where the folder and the meta go (`path` may replace artifact_path for the index)
made_folder = False # the create made the folder itself
written_meta_file = None # the meta file the create wrote
done = False

try:
if not virtual:
if os.path.isdir(artifact_path):
if os.path.isfile(cmeta_filename_json) or os.path.isfile(cmeta_filename_yaml):
if index:
if os.path.isfile(cmeta_filename_json):
f = cmeta_filename_json
else:
f = cmeta_filename_yaml
yaml = True

# Do not use lock since it's not in the system and not in the index
r = utils.files.safe_read_file(f, fail_on_error=self.fail_on_error, logger=self.logger)
if r['return']>0: return r

meta = r['data']
existing_meta_file = f

# Index the artifact under the UID of its meta (an "alias,UID" too),
# not a new one, or alias,UID references to it would not resolve
meta_uid = str(meta.get('artifact') or '').split(',')[-1].strip()
if meta_uid and utils.names.is_valid_cmeta_uid(meta_uid):
artifact_uid = meta_uid

# Index the artifact under the UID of its meta (an "alias,UID" too),
# not a new one, or alias,UID references to it would not resolve
meta_uid = str(meta.get('artifact') or '').split(',')[-1].strip()
if meta_uid and utils.names.is_valid_cmeta_uid(meta_uid):
artifact_uid = meta_uid
else:
return {'return':8, 'error':f'artifact already exists in "{artifact_path}"'}
else:
made_folder = True

else:
return {'return':8, 'error':f'artifact already exists in "{artifact_path}"'}
os.makedirs(artifact_path, exist_ok=True)

os.makedirs(artifact_path, exist_ok=True)
# Prepare meta
cmeta = copy.deepcopy(meta)

# Prepare meta
cmeta = copy.deepcopy(meta)
if 'artifact' not in cmeta:
cmeta['artifact'] = artifact_uid

if 'artifact' not in cmeta:
cmeta['artifact'] = artifact_uid
if 'category' not in cmeta:
r = utils.names.restore_cmeta_name(category, key='artifact', fail_on_error = self.fail_on_error)
if r['return'] >0: return r

if 'category' not in cmeta:
r = utils.names.restore_cmeta_name(category, key='artifact', fail_on_error = self.fail_on_error)
if r['return'] >0: return r
cmeta['category'] = r['name']

cmeta['category'] = r['name']
if tags is not None:
meta_tags = cmeta.get('tags', [])

if tags is not None:
meta_tags = cmeta.get('tags', [])
r = utils.common.normalize_tags(tags)
if r['return']>0: return r

r = utils.common.normalize_tags(tags)
if r['return']>0: return r

for t in r['tags']:
if t not in meta_tags:
meta_tags.append(t)
for t in r['tags']:
if t not in meta_tags:
meta_tags.append(t)

cmeta['tags'] = meta_tags
cmeta['tags'] = meta_tags

if existing_meta_file is None:
from datetime import datetime, timezone
if existing_meta_file is None:
from datetime import datetime, timezone

if 'creation_timestamp' not in cmeta:
cmeta['creation_timestamp'] = datetime.now(timezone.utc).isoformat()
else:
cmeta['last_update_timestamp'] = datetime.now(timezone.utc).isoformat()
if 'creation_timestamp' not in cmeta:
cmeta['creation_timestamp'] = datetime.now(timezone.utc).isoformat()
else:
cmeta['last_update_timestamp'] = datetime.now(timezone.utc).isoformat()

# Authors, copyright and generator: the meta, the environment and the repository defaults
apply_artifact_defaults(cmeta, repo_meta=repo_cmeta, cfg=self.cm.cfg, creating=True)

# Save meta
if not virtual:
if existing_meta_file is not None:
# Indexing leaves the meta file as it was written; only what the request adds (the tags,
# an identity key the file lacks) is added in place, without rewriting the rest
if cmeta != meta:
r = utils.files.safe_write_file(existing_meta_file, data=cmeta, fail_on_error = self.fail_on_error, logger=self.logger, preserve=True)
if r['return']>0: return r

# Authors, copyright and generator: the meta, the environment and the repository defaults
apply_artifact_defaults(cmeta, repo_meta=repo_cmeta, cfg=self.cm.cfg, creating=True)
else:
tmp_cmeta_filename = cmeta_filename_yaml if yaml else cmeta_filename_json
written_meta_file = tmp_cmeta_filename

# Save meta
if not virtual:
if existing_meta_file is not None:
# Indexing leaves the meta file as it was written; only what the request adds (the tags,
# an identity key the file lacks) is added in place, without rewriting the rest
if cmeta != meta:
r = utils.files.safe_write_file(existing_meta_file, data=cmeta, fail_on_error = self.fail_on_error, logger=self.logger, preserve=True)
# A new YAML meta is written in the keep style (the key order of YAML_META_KEY_ORDER,
# nothing folded, lists indented), the style `update` keeps; a JSON meta is sorted as before
r = utils.files.safe_write_file(tmp_cmeta_filename, data=cmeta, fail_on_error = self.fail_on_error, keep=True)
if r['return']>0: return r

else:
tmp_cmeta_filename = cmeta_filename_yaml if yaml else cmeta_filename_json

# A new YAML meta is written in the keep style (the key order of YAML_META_KEY_ORDER,
# nothing folded, lists indented), the style `update` keeps; a JSON meta is sorted as before
r = utils.files.safe_write_file(tmp_cmeta_filename, data=cmeta, fail_on_error = self.fail_on_error, keep=True)
if r['return']>0: return r
if path is not None:
artifact_path = path

if path is not None:
artifact_path = path
# Update index
if not no_index:
cmeta_ref_parts = {}

# Update index
if not no_index:
cmeta_ref_parts = {}
if artifact_alias is not None:
cmeta_ref_parts['artifact_alias'] = artifact_alias
artifact_alias_lowercase = artifact_alias.lower()
if artifact_alias_lowercase != artifact_alias:
cmeta_ref_parts['artifact_alias_lowercase'] = artifact_alias_lowercase

if artifact_alias is not None:
cmeta_ref_parts['artifact_alias'] = artifact_alias
artifact_alias_lowercase = artifact_alias.lower()
if artifact_alias_lowercase != artifact_alias:
cmeta_ref_parts['artifact_alias_lowercase'] = artifact_alias_lowercase
cmeta_ref_parts['artifact_uid'] = artifact_uid

cmeta_ref_parts['artifact_uid'] = artifact_uid
cmeta_ref_parts['category_alias'] = category_alias
cmeta_ref_parts['category_uid'] = category_uid

cmeta_ref_parts['category_alias'] = category_alias
cmeta_ref_parts['category_uid'] = category_uid
cmeta_ref_parts['repo_alias'] = artifact_repo_alias
cmeta_ref_parts['repo_uid'] = artifact_repo_uid

cmeta_ref_parts['repo_alias'] = artifact_repo_alias
cmeta_ref_parts['repo_uid'] = artifact_repo_uid
r = self.cm.repos.add_to_index(cmeta, cmeta_ref_parts, artifact_path, sharding_slices_num = sharding_slices_num, index_lock = index_lock)
if r['return']>0: return r

done = True

r = self.cm.repos.add_to_index(cmeta, cmeta_ref_parts, artifact_path, sharding_slices_num = sharding_slices_num)
if r['return']>0: return r
finally:
if not done and not virtual:
self._undo_create(disk_path, made_folder, written_meta_file, sharding_slices)
if index_lock is not None:
index_lock.release()

# Print artifact path
if con:
Expand All @@ -1188,6 +1214,36 @@ def create_(

return {'return':0, 'path':artifact_path, 'meta':cmeta}

############################################################
def _undo_create(
self,
artifact_path: str, # The folder of the artifact that was being created.
made_folder: bool, # True when the create made the folder itself: it goes with everything in it.
meta_file: str, # The meta file the create wrote, or None.
sharding_slices, # The sharding of the category, to remove the empty parent folders too.
):
"""
Remove what a create that failed made on disk - the folder when the create made it, else only
the meta it wrote - so that a create whose record could not be written leaves nothing unindexed.

Args:
artifact_path (str): The folder of the artifact that was being created.
made_folder (bool): True when the create made the folder itself.
meta_file (str | None): The meta file the create wrote.
sharding_slices (list | None): The sharding of the category.
"""
import shutil

if made_folder:
shutil.rmtree(artifact_path, ignore_errors=True)
utils.files.safe_delete_directory_if_empty_with_sharding(artifact_path, sharding_slices)

elif meta_file is not None:
try:
os.remove(meta_file)
except OSError:
pass

############################################################
def move_(
self,
Expand Down
Loading
Loading