From 63da68c7dbd5de5a7dd8f924f66c486c3de2bb8c Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Thu, 8 Oct 2026 18:35:17 +0200 Subject: [PATCH 1/3] cx reindex and cx repo reindex: an artifact or a repository is indexed again from its folder, nothing written (0.33.1.1) reindex rewrites the record of one artifact (or several: wildcards, tags, a repository, everything) from the meta file in its folder, with the full reindex own scanner, drops the stale aliases of its UID in the same locked write, finds a folder renamed by hand by its new name, its UID or its old name, and never touches a folder: the only record it removes is that of an artifact whose folder now holds another artifact. A missing folder, an unreadable meta, an invalid UID, another category meta or one UID in two folders is an error that leaves the index as it was. cx repo reindex indexes one registered repository again (its _cmr.yaml and its artifacts) with the incremental index of pull and plug, and refuses a repository whose folder is not there. Fixed on the way in repos.py: the incremental index kept the records of a category whose last artifact in a repository was removed; the file-system scan of a no_index category applied a wildcard only in sharded categories, crashed on a meta without category and found only the first of two spellings of an alias. Tests: tests/internal_repo_tests/test_integration_reindex_artifact.py and test_integration_repo_reindex.py (42 cases); the suite run on Windows, macOS and Ubuntu with Python 3.9, 3.12 and 3.14. The six-processes test is xfail on non-Windows: the lock file is deleted after every release, a defect of the concurrency step. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- .claude/skills/add-cserver-plugin/SKILL.md | 10 +- .claude/skills/add-plugin/SKILL.md | 13 +- .claude/skills/add-repo/SKILL.md | 11 +- .claude/skills/use-cmeta-cli/SKILL.md | 20 +- .claude/skills/use-cmeta-python/SKILL.md | 6 +- AGENTS.md | 17 +- CHANGELOG.md | 54 ++ README.md | 18 +- cmeta/category_api_v1.py | 316 +++++++ cmeta/internal-repo/category/repo/api/v1.py | 120 +++ cmeta/repos.py | 57 +- cmeta/version.py | 4 +- docs/common-commands.md | 16 +- docs/using-cmeta.md | 51 +- .../test_integration_reindex_artifact.py | 842 ++++++++++++++++++ .../test_integration_repo_reindex.py | 242 +++++ 16 files changed, 1745 insertions(+), 52 deletions(-) create mode 100644 tests/internal_repo_tests/test_integration_reindex_artifact.py create mode 100644 tests/internal_repo_tests/test_integration_repo_reindex.py diff --git a/.claude/skills/add-cserver-plugin/SKILL.md b/.claude/skills/add-cserver-plugin/SKILL.md index 3ae48c6..73ec93d 100644 --- a/.claude/skills/add-cserver-plugin/SKILL.md +++ b/.claude/skills/add-cserver-plugin/SKILL.md @@ -47,12 +47,14 @@ that one artifact instead of rebuilding the whole home: ```bash cx category index :cserver. # index a hand-made folder -cx category update :cserver. # after editing its _cmeta.* meta +cx category reindex cserver. # after editing its _cmeta.* meta (nothing written) +cx category update :cserver. # the same, but it rewrites the file too +cx repo reindex # the whole repository, when several things changed in it ``` -`cx --reindex` walks every category and is slow — keep it for moves, bulk -`git pull`, or a wiped `CMETA_HOME`. **Editing `api/`, `files/` or `src/` needs -no reindex at all**; a server restart (or `--reload`) picks those up. +`cx --reindex` walks every category and is slow — keep it for a wiped +`CMETA_HOME` or a bulk change across repositories. **Editing `api/`, `files/` or +`src/` needs no reindex at all**; a server restart (or `--reload`) picks those up. Layout: diff --git a/.claude/skills/add-plugin/SKILL.md b/.claude/skills/add-plugin/SKILL.md index 8b94fda..8f7ce7f 100644 --- a/.claude/skills/add-plugin/SKILL.md +++ b/.claude/skills/add-plugin/SKILL.md @@ -340,13 +340,15 @@ narrowest command that covers it: ```bash cx index : # you created the folder by hand (mkdir + _cmeta.*) -cx update : # you edited an existing artifact's _cmeta.* meta +cx reindex # you edited, renamed or copied in an existing artifact's folder (nothing is written) +cx repo reindex # one repository: its _cmr.yaml and its artifacts +cx update : # you edited an existing artifact's _cmeta.* meta (rewrites the file too) cx --reindex # rebuilds ALL of /index/*.pkl — slow ``` -Prefer the per-artifact forms: `--reindex` walks every category and is markedly -slower on a large home. Save it for renames/moves, a bulk `git pull`, or a wiped -`CMETA_HOME`. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no +Prefer the narrow forms: `--reindex` walks every category and is markedly +slower on a large home. Save it for a wiped `CMETA_HOME` or a bulk change across +repositories. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no reindex — the index tracks meta, not content. Category-level content caches (task results, downloaded blobs) live under the `cache` category: @@ -382,7 +384,8 @@ Before reporting the plugin as done: `TypeError: foo() takes 2 positional arguments but ...`. Rename to `foo_`. - **Command name clashes with a Python builtin** (`list`, `type`, ...). Name the method `list__` — the CLI name is still `list`. -- **New category isn't found.** The artifact index is cached; run `cx --reindex`. +- **New category isn't found.** The artifact index is cached; run + `cx repo reindex ` (or `cx --reindex`). - **Base command not exposed.** The category's `_cmeta.yaml` sets `skip_base_category_commands: true` (e.g. `utils`). Remove that flag or reimplement the command. diff --git a/.claude/skills/add-repo/SKILL.md b/.claude/skills/add-repo/SKILL.md index fbae35e..0a9b9da 100644 --- a/.claude/skills/add-repo/SKILL.md +++ b/.claude/skills/add-repo/SKILL.md @@ -147,6 +147,7 @@ cx repo find # find a repo cx repo status # git status + remote URL for git-backed repos cx repo pull # git pull; also refreshes the index cx repo update # equivalent to pull +cx repo reindex # re-read its _cmr.yaml and rescan its artifacts (files copied in by other means) cx repo checkout # git checkout branch/tag/commit cx repo space # disk usage cx repo zip # dump to cmr--YYYYMMDD-HHMMSS.zip @@ -172,10 +173,11 @@ After adding a repo, sanity-check: - [ ] `/repos.json` contains the repo's path. If a repo is registered but its artifacts don't show up in `find`/`ls`, the -fast index is stale — force a rebuild: +fast index is stale — rescan that repository (or, as a last resort, everything): ```bash -cx --reindex +cx repo reindex # its _cmr.yaml and its artifacts; other repos untouched +cx --reindex # everything ``` --- @@ -198,8 +200,9 @@ cx --reindex does. - **Renaming a repo alias.** Not supported. Reference repos by `alias,UID` in cross-repo references so alias renames wouldn't matter anyway. -- **`_cmr.yaml` edited by hand and things look off.** `cx --reindex` to - rebuild `/index/*.pkl`. +- **`_cmr.yaml` edited by hand and things look off.** `cx repo reindex ` + re-reads it and rescans the repo's artifacts (or `cx --reindex` to rebuild + `/index/*.pkl`). - **`permanent: true` in `_cmr.yaml`.** `cx repo delete` refuses. Remove the flag first if you truly want to delete. - **`subdir:` in `_cmr.yaml`.** Only that sub-tree is scanned for artifacts — diff --git a/.claude/skills/use-cmeta-cli/SKILL.md b/.claude/skills/use-cmeta-cli/SKILL.md index 0bb326d..28f5e9c 100644 --- a/.claude/skills/use-cmeta-cli/SKILL.md +++ b/.claude/skills/use-cmeta-cli/SKILL.md @@ -349,20 +349,30 @@ cx ... --dump # capture full ctx ### 10.7 Reindex when things look stale — narrowest command first `cx --reindex` rebuilds *every* category pickle and gets slow as a home grows. -Reach for the per-artifact commands first; they touch one index entry: +Reach for the narrow commands first; they touch one index entry or one repository: ```bash cx index : # register ONE hand-made (mkdir-ed) artifact -cx update : # refresh entry after editing its _cmeta.* meta +cx reindex # refresh ONE record from its folder: a meta edited by hand, a folder + # renamed/moved/copied in; nothing is written on disk +cx reindex # every artifact of the category, hand-made folders included +cx repo reindex # ONE repository: its _cmr.yaml and its artifacts; other repos untouched +cx update : # refresh entry after editing its _cmeta.* meta (rewrites the file too) cx --reindex # whole-home rebuild — last resort ``` - `cx add :` already indexes as it creates — nothing - extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`). + extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`); + `reindex` takes those too, and an artifact the index knows already. +- `reindex` never touches a folder; the only record it removes is that of an + artifact whose folder now holds another artifact, and it says so. A folder it + cannot find, a meta it cannot read, or one UID in two folders is an error that + changes nothing. After a rename by hand the new name, the UID and the old name + all find the folder, and the old alias is dropped. - Editing only an artifact's **payload** (`api/`, `files/`, `src/`, `_desc.yaml`) needs no reindex at all — the index tracks meta, not content. -- Keep `--reindex` for the broad cases: bulk `git pull`, moved/renamed folders, - `CMETA_HOME` wipe, or an index that looks generally wrong. Safe, idempotent. +- Keep `--reindex` for the broad cases: a wiped `CMETA_HOME`, a bulk change + across repositories, or an index that looks generally wrong. Safe, idempotent. ## 11. Scripting the CLI diff --git a/.claude/skills/use-cmeta-python/SKILL.md b/.claude/skills/use-cmeta-python/SKILL.md index 80b6b63..054b43d 100644 --- a/.claude/skills/use-cmeta-python/SKILL.md +++ b/.claude/skills/use-cmeta-python/SKILL.md @@ -289,7 +289,11 @@ loaded = r['artifact']['loaded_files'] ### 6.5 Reindex on-demand ```python -cm.repos.reindex(con=False) # or: cm.access({'reindex': True}) +cm.repos.reindex(con=False) # or: cm.access({'reindex': True}) - everything, slow + +# One artifact's record from its folder (nothing written), one repository +r = cm.access({'category': 'note', 'command': 'reindex', 'arg1': 'my-note'}) # r['artifacts'], r['dropped_aliases'] +r = cm.access({'category': 'repo', 'command': 'reindex', 'arg1': 'myrepo'}) # its _cmr.yaml and its artifacts ``` ## 6.6 Use `cm.utils.*` from category code — **recommended** diff --git a/AGENTS.md b/AGENTS.md index fb72ea4..e1547d3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,7 +75,7 @@ tabulate, tqdm, filelock, packaging, psutil. | `core.py` | `CMeta` class + `access()` — the one dispatch every request funnels through. Resolves the category artifact, loads its API module, calls the matching command function. | | `core_async.py` | Async variant of `access()`. | | `category.py` | `InitCategory` base class every category API subclasses. Provides `_prepare_input_from_params/ctx` helpers used to re-enter `cm.access()`. | -| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`info`/`tags`/`get`/`set`/`index`/`test`. | +| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`migrate`/`info`/`tags`/`get`/`set`/`index`/`reindex`/`test`. | | `repos.py` | `Repos` — index and resolve content repositories + artifacts (by alias / UID / tags / wildcards). Owns the per-category `.pkl` index files. | | `packages.py` | `Packages` — detect installed tools/versions and auto-install Python packages. | | `config.py` | Global `cfg` dict, param descriptors, env-var handling, command aliases (`add→create`, `rm→delete`, `ls→list`, `search→find`, `mv→move`, ...). | @@ -141,10 +141,17 @@ tabulate, tqdm, filelock, packaging, psutil. refreshes automatically on `create`/`update`/`delete` and on `cx repo` changes. If artifacts were touched outside cMeta, reindex with the narrowest command: `cx index :` registers a single folder you - created by hand (`mkdir` + `_cmeta.*`); `cx update :` - refreshes the entry after you edit an existing artifact's `_cmeta.*` meta; - `cx --reindex` rebuilds every pickle and is slow — keep it for manual moves, - bulk `git pull`, or a cleared `CMETA_HOME`. Payload-only edits (`api/`, + created by hand (`mkdir` + `_cmeta.*`); `cx reindex ` + rewrites the record of one artifact (or of several, with wildcards) from the + meta file in its folder without writing anything — a meta edited by hand, a + folder renamed, moved or copied in (its stale aliases are dropped, and the + record of a folder that now holds another artifact; a folder it cannot find or + one UID in two folders is an error that changes nothing); + `cx update :` refreshes the entry too but rewrites + the meta file (a new timestamp); `cx repo reindex ` rescans one + repository (its `_cmr.yaml` and its artifacts) and leaves the others alone; + `cx --reindex` rebuilds every pickle and is slow — keep it for a cleared + `CMETA_HOME` or a bulk change across repositories. Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) need no reindex. Categories with `no_index: true` in `_cmeta.yaml` are always found by filesystem scan. - **Content-addressed task caching / reproducibility.** Workflow caches keyed diff --git a/CHANGELOG.md b/CHANGELOG.md index 00ba4d0..5a61051 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,60 @@ All notable changes to cMeta are documented here, newest first. +## 0.33.1.1 (in development) +- **`cx reindex []`: the index record of one artifact (or of several) is refreshed + from its folder, and no file is changed.** The meta file is read from the artifact's folder on disk + and the record in the category's index is rewritten from it: the alias is the folder's name, every + other alias that still pointed at the artifact's UID is dropped, and the path, the meta and the + sharding depth are what a full `cx --reindex` records. Use it after a meta was edited by hand, after + a folder was renamed, moved or copied in from another machine (bisync, a USB disk), or when a lookup + looks stale - instead of the whole `cx --reindex`, and instead of an empty `cx update`, + which refreshes the record but also rewrites the meta file (a new timestamp and a generator record, + which a bisynced folder then carries back). The artifact is named by its alias, UID, alias,UID or + repo:alias, with wildcards; `--tags` prunes; nothing names every artifact of the category. A folder + the index does not know yet (made or renamed by hand) is found on disk in the repositories; an + artifact whose indexed folder is gone is looked for by its UID in its repository, so the new name, + the UID or the old name all work after a rename by hand. No folder is touched, and the only record + `reindex` ever removes is that of an artifact whose folder, still there, holds another artifact now + (replaced by hand) - a record a `delete` would act on by removing the other artifact's folder - and + it says so. A folder that cannot be found, a meta that cannot be read or has no valid `artifact` + UID, a meta of another category, or one UID found in two existing folders is an error that leaves + the index as it was (`--ignore_errors` reindexes the other artifacts and reports these). `index`, + `update`, `move` and `migrate` keep their behaviour. Tests: `tests/internal_repo_tests/test_integration_reindex_artifact.py` + (a hand-edited meta, YAML and JSON, the record of a full reindex, renames by hand found by the new + name, the UID or the old name, a rename twice, a case-only rename, two spellings on a case-sensitive + file system, a folder replaced by another artifact, a folder moved and its old name taken, one UID + in two folders, a missing folder, an unreadable meta, an invalid UID, an upper-case UID, another + category's folder, a non-ASCII alias, wildcards, tags, a repository, a repository with `subdir`, + everything, a UID-named folder, a sharded category and a rename inside it, a `no_index` category, a + category without an index file yet, the record of a category itself, the shipped internal repository, + a migration stub, the CLI, several processes at once). +- **`cx repo reindex `: one registered repository is indexed again.** Its `_cmr.yaml` is + re-read and its artifacts are rescanned with the incremental index that `pull`, `plug` and `unzip` + use, and the records of the other repositories stay as they are. The repository is named by its + alias, UID or alias,UID (wildcards allowed), by the path of its folder (or of a folder inside it), + or by `.` for the repository of the current directory. A repository whose folder or `_cmr.yaml` is + not there (a detached drive) is refused and nothing is changed. `cx --reindex` stays the full + rebuild of every repository. Tests: `tests/internal_repo_tests/test_integration_repo_reindex.py`. +- **The incremental index drops the records of a category of which a repository holds no artifact + any more.** When a repository was indexed again (`pull`, `plug`, `unzip`, now `repo reindex`), the + records of its deleted artifacts were removed only from the index files of the categories in which + the repository still had an artifact: when its last artifact of a category was gone, that record + stayed and the lookup kept answering with a folder that did not exist. Every existing category + index now takes part in the removal. The full `cx --reindex` is unchanged. +- **A wildcard search in a `no_index` category returns only the folders the pattern names.** The scan + that finds the artifacts of such a category (and that `reindex` uses to find folders on disk) + applied the pattern only in sharded categories; in a plain one, `cx find "name-*"` + returned every artifact. The pattern is now matched against the folder names, case-insensitively, + as the sharded scan does; and an exact alias now finds every folder that spells it (two spellings + can coexist on a case-sensitive file system), not only the first. The same scan no longer stops + with a `KeyError` on a meta file that has no `category` key: the folder is skipped with a warning, + as a folder without an `artifact` key is. +- `Repos.add_to_index` takes `clean_aliases=True` to drop, in the same locked write, every other + alias of the alias index that still lists the UID being written; its result carries them as + `dropped_aliases`. The default is unchanged. + + ## 0.33.1 - **A command's output is decoded on any machine in any language, and never fails.** `utils.sys.run` captured output in the text mode of `subprocess`, which decodes with the locale's encoding and diff --git a/README.md b/README.md index 804ea11..d6ca90a 100644 --- a/README.md +++ b/README.md @@ -560,16 +560,22 @@ filesystem. The framework rebuilds the index automatically when repos are added/removed via `cx repo`, and updates it in-place on `create`/`update`/`delete`. -If you edited artifacts outside cMeta (moved folders, edited `_cmeta.yaml` by -hand, `git pull`-ed a repo, cleared `CMETA_HOME`, or things simply look -inconsistent), force a full rebuild: +If you edited artifacts outside cMeta (moved or renamed folders, edited +`_cmeta.yaml` by hand, copied artifacts in from another machine, `git pull`-ed a +repo), refresh the narrowest thing — one artifact, one repository — or, after a +cleared `CMETA_HOME` or when things simply look inconsistent, rebuild everything: ```bash -cx --reindex +cx reindex # one artifact's record, from its folder; nothing is written on disk +cx repo reindex # one repository: its _cmr.yaml and its artifacts; the others untouched +cx --reindex # everything ``` -This is safe, idempotent, and typically fast. Categories that opt out with -`no_index: true` in `_cmeta.yaml` are found by scanning the filesystem instead. +All three are safe and idempotent; `reindex` never touches a folder, and the only +record it removes is that of an artifact whose folder now holds another artifact +(a folder it cannot find or one UID in two folders is an error that changes +nothing). Categories that opt out with `no_index: true` in `_cmeta.yaml` are +found by scanning the filesystem instead. Task caches produced by workflows live under the `cache` category and can be inspected/pruned with `cx cache show` / `cx cache clean` / `cx cache delete`. diff --git a/cmeta/category_api_v1.py b/cmeta/category_api_v1.py index d2df2f9..1d4e595 100644 --- a/cmeta/category_api_v1.py +++ b/cmeta/category_api_v1.py @@ -1683,6 +1683,322 @@ def index( return self.cm.access(p) + ############################################################ + def reindex_( + self, + ctx: dict, # cMeta context. + arg1: str = None, # Artifact alias, UID, alias,UID or repo:alias (wildcards allowed); nothing names every artifact of the category. + tags: str = None, # Prune the artifacts by tags, as in find (a tag with a leading "-" excludes). + sort: bool = True, # Sort the artifacts by alias and UID. + skip_uids: bool = False, # Skip UIDs when using wildcards. + ignore_errors: bool = False, # Go on past an artifact that cannot be reindexed and report it at the end. + ): + """ + Refresh the index record(s) of artifact(s) from their folders, changing no file. + + The meta file of each artifact is read from its folder on disk and the artifact's record in the + index of the category is rewritten from it: the alias is the folder's name, every other alias + that still pointed at the artifact's UID is dropped, and the path, the meta and the sharding + depth are what a full `cx --reindex` records. The meta file is read, never written (an empty + `update` rewrites it). Use it after a meta file was edited by hand, after a folder was renamed, + moved or copied in from another machine, or when a lookup looks stale - instead of the whole + `cx --reindex`. + + The artifact may be named by its alias, UID, alias,UID or repo:alias, with wildcards; nothing + names every artifact of the category. A folder the index does not know yet (made or renamed by + hand) is found on disk in the repositories; an artifact whose indexed folder is gone is looked + for by its UID in its repository. No folder is touched, and the only record ever removed is + that of an artifact whose folder, still there, holds another artifact now (replaced by hand) - + said in the output. A folder that cannot be found, a meta without a valid `artifact` UID, or a + UID found in two existing folders is an error, and the index is left as it was + (`--ignore_errors` reindexes the other artifacts and reports these). + + Args: + ctx (dict): cMeta context. + arg1 (str | None): Artifact alias, UID, alias,UID or repo:alias (wildcards allowed); nothing + names every artifact of the category. + tags (str | list | None): Prune the artifacts by tags, as in find (a leading "-" excludes). + sort (bool): Sort the artifacts by alias and UID. + skip_uids (bool): Skip UIDs when using wildcards. + ignore_errors (bool): Go on past an artifact that cannot be reindexed and report it at the end. + + Returns: + dict: A cMeta dictionary with the following keys: + - **return** (int): 0 if success, >0 if error (16: nothing found). + - **error** (str): Error message if `return > 0`. + - **artifacts** (list): The reindexed artifacts, as their new index records. + - **dropped_aliases** (dict): Per UID, the (lowercase) aliases dropped because they were not the folder's name. + - **dropped_records** (list): The records removed because their folder holds another artifact now (`ref`, `path`, `now`). + - **errors** (list): With `ignore_errors`, one dict per artifact that could not be reindexed (`ref`, `path`, `error`). + + Raises: + Exception: Propagated runtime errors, if any. + """ + + con = ctx.get('control',{}).get('con', False) + + ctx['control']['con'] = False + + category_cmeta = ctx['category_artifact']['cmeta'] + category_cmeta_ref_parts = ctx['category_artifact']['cmeta_ref_parts'] + + category_alias = category_cmeta_ref_parts['artifact_alias'] + category_uid = category_cmeta_ref_parts['artifact_uid'] + + if category_cmeta.get('no_index', False): + return {'return':1, 'error':f'category "{category_alias}" is not indexed (no_index) - there is nothing to reindex'} + + # The reference: an alias (with wildcards), a UID, both, a repository + artifact_obj_parts = {} + + if arg1 is not None and str(arg1).strip() != '': + r = utils.names.parse_cmeta_obj(arg1, key = "artifact", fail_on_error = self.fail_on_error) + if r['return'] >0: return r + + artifact_obj_parts = r['obj_parts'] + + artifact_alias = artifact_obj_parts.get('artifact_alias') + artifact_uid = artifact_obj_parts.get('artifact_uid') + artifact_repo_alias = artifact_obj_parts.get('artifact_repo_alias') + artifact_repo_uid = artifact_obj_parts.get('artifact_repo_uid') + + if tags is not None: + r = utils.common.normalize_tags(tags, fail_on_error = self.fail_on_error) + if r['return']>0: return r + + tags = r['tags'] + + # The repositories to look in: the one named, else all of them + repo_uids = None + + if (artifact_repo_alias is not None and artifact_repo_alias != '') or (artifact_repo_uid is not None and artifact_repo_uid != ''): + r = self.cm.repos.find_in_index('repo', self.cm.cfg['category_repo_uid'], artifact_repo_alias, artifact_repo_uid, only_uids=True, skip_uids=skip_uids) + if r['return']>0: return r + + repo_uids = r['artifact_uids'] + + def _norm(path): + return os.path.normcase(os.path.normpath(os.path.abspath(path))) + + def _same_folder(path1, path2): + # The same folder: the same path, or two spellings of one folder on a case-insensitive file system + if _norm(path1) == _norm(path2): + return True + try: + return os.path.isdir(path1) and os.path.isdir(path2) and os.path.samefile(path1, path2) + except OSError: + return False + + def _ref(parts): + alias = parts.get('artifact_alias') + uid = parts.get('artifact_uid', '') + return f'"{alias}" ({uid})' if alias else uid + + def _uid_in_folder(path): + # The UID the meta file of a folder names, or None (no folder, no meta, no valid UID) + if not os.path.isdir(path): + return None + r = utils.files.safe_read_yaml_or_json(os.path.join(path, self.cm.cfg['meta_filename_base']), fail_on_error=False, logger=self.logger) + if r['return']>0 or not isinstance(r.get('data'), dict): + return None + r = utils.names.parse_cmeta_name(r['data'].get('artifact')) + if r['return']>0: + return None + uid = r['name'].get('uid') + return uid if uid is not None and utils.names.is_valid_cmeta_uid(uid) else None + + # 1. The folders on disk that the reference names: the real folder name and the meta as it is now, + # built as the full reindex builds its records + r = self.cm.repos.find_in_file_system(category_cmeta, category_alias, category_uid, artifact_alias, artifact_uid, repo_uids = repo_uids) + if r['return']>0: return r + + entries = [e for e in r['artifacts'] if self._match_tags(e['cmeta'].get('tags', []), tags)] + + # 2. The records the index holds for the reference (a migration stub is reindexed as itself) + r = self.find_(ctx, arg1, tags, sort=False, add_index_file=True, skip_uids=skip_uids, follow_migrated=False) + if r['return']>0 and r['return']!=16: return r + + indexed = r.get('artifacts', []) if r['return'] == 0 else [] + + if len(entries) == 0 and len(indexed) == 0: + x = f'"{arg1}"' if arg1 is not None and str(arg1).strip() != '' else 'artifacts' + return {'return':16, 'error':f'{category_alias} {x} not found in the index or on disk'} + + seen_paths = {} + seen_uids = {} + + for e in entries: + seen_paths[_norm(e['path'])] = e + seen_uids.setdefault(e['cmeta_ref_parts']['artifact_uid'], []).append(e) + + errors = [] + stale = [] + + # An indexed artifact that was not found on disk by the reference: its folder was renamed or moved + # by hand, replaced by another artifact, or it is gone - look for its UID in its repository + for a in indexed: + parts = a['cmeta_ref_parts'] + uid = parts['artifact_uid'] + + if uid in seen_uids: + continue + + xrepo_uids = [parts['repo_uid']] if parts.get('repo_uid') else None + + r = self.cm.repos.find_in_file_system(category_cmeta, category_alias, category_uid, None, uid, repo_uids = xrepo_uids) + if r['return']>0: return r + + found = r['artifacts'] + + if len(found) == 0: + other = seen_paths.get(_norm(a['path'])) + + if other is not None: + # The folder is still there but holds another artifact now (replaced by hand): the record + # is provably stale, and a "delete" of it would remove the other artifact's folder - it is + # dropped, as a full reindex would drop it, and said so + stale.append({'ref': _ref(parts), 'path': a['path'], 'uid': uid, + 'index_file': a.get('index_file'), + 'alias_lowercase': parts.get('artifact_alias_lowercase', parts.get('artifact_alias')), + 'other': _ref(other['cmeta_ref_parts'])}) + elif os.path.isdir(a['path']): + # The folder is there, but its meta file cannot be read or has no valid artifact UID + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{a["path"]}", and the folder is there, but its meta file ' + f'cannot be read or has no valid "artifact" UID - fix the file, then reindex'}) + else: + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{a["path"]}" but no folder with its UID was found in its repository - ' + f'if it was moved elsewhere, reindex it by its new name; if it was deleted, remove it from the index with "delete"'}) + continue + + if len(found) > 1: + paths = ', '.join('"' + f['path'] + '"' for f in found) + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is in {len(found)} folders with the same UID: {paths} - give all but one a new UID first'}) + continue + + e = found[0] + + seen_paths[_norm(e['path'])] = e + seen_uids.setdefault(uid, []).append(e) + entries.append(e) + + # 3. Check everything before writing anything: one folder per UID, and a UID that the index holds at + # another folder that still exists is left alone (two folders with one UID) + r = self.cm.repos.find_in_index(category_alias, category_uid, only_uids=True, skip_uids=True) + if r['return']>0: return r + + index_uids = r.get('index', {}).get(self.cm.repos.KEY_INDEX_UIDS, {}) + + to_write = [] + + for uid in seen_uids: + uid_entries = seen_uids[uid] + parts = uid_entries[0]['cmeta_ref_parts'] + + if len(uid_entries) > 1: + paths = ', '.join('"' + x['path'] + '"' for x in uid_entries) + errors.append({'ref': _ref(parts), 'path': uid_entries[0]['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is in {len(uid_entries)} folders with the same UID: {paths} - give all but one a new UID first'}) + continue + + e = uid_entries[0] + + record = index_uids.get(uid) + if record is not None: + old_path = record.get('path', '') + # A conflict only when the folder the index knows still holds this very UID + if old_path and not _same_folder(old_path, e['path']) and _uid_in_folder(old_path) == uid: + errors.append({'ref': _ref(parts), 'path': e['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{old_path}", which still exists, while the same UID is in "{e["path"]}" - ' + f'two folders with one UID: give one of them a new UID (or remove it), then reindex'}) + continue + + to_write.append(e) + + if len(errors) > 0 and not ignore_errors: + if len(errors) == 1: + return {'return':1, 'error': errors[0]['error']} + return {'return':1, 'error': 'nothing was reindexed:\n' + '\n'.join('* ' + x['error'] for x in errors)} + + if sort: + to_write.sort(key=lambda e: (str(e['cmeta_ref_parts'].get('artifact_alias', '')).lower(), e['cmeta_ref_parts']['artifact_uid'])) + + # 4. Rewrite the records, each in one locked write that also drops the stale aliases of its UID + reindexed = [] + dropped_aliases = {} + + for e in to_write: + parts = e['cmeta_ref_parts'] + uid = parts['artifact_uid'] + + r = self.cm.repos.add_to_index(e['cmeta'], parts, e['path'], original_uid = uid, + sharding_slices_num = e.get('sharding_slices_num', 0), clean_aliases = True) + if r['return']>0: return r + + record = {'path': e['path'], 'cmeta': e['cmeta'], 'cmeta_ref_parts': parts} + if e.get('sharding_slices_num'): + record['sharding_slices_num'] = e['sharding_slices_num'] + + reindexed.append(record) + + if len(r.get('dropped_aliases', [])) > 0: + dropped_aliases[uid] = r['dropped_aliases'] + + if con: + print (f'Reindexed {category_alias} {_ref(parts)}: {e["path"]}') + for x in r.get('dropped_aliases', []): + print (f' dropped the alias "{x}"') + + # 5. The records of artifacts whose folder now holds another artifact + dropped_records = [] + + for x in stale: + if x['index_file']: + r = self.cm.repos.remove_from_index(x['index_file'], x['uid'], x['alias_lowercase']) + if r['return']>0: return r + + dropped_records.append({'ref': x['ref'], 'path': x['path'], 'now': x['other']}) + + if con: + print (f'Dropped the record of {category_alias} {x["ref"]}: its folder {x["path"]} now holds {x["other"]}') + + if con: + for x in errors: + print (f'Skipped: {x["error"]}') + + return {'return':0, 'artifacts': reindexed, 'dropped_aliases': dropped_aliases, 'dropped_records': dropped_records, 'errors': errors} + + ############################################################ + def _match_tags( + self, + ctags, # The tags of an artifact's meta. + tags, # Normalized tags to match (a leading "-" excludes); None or empty matches everything. + ): + """ + Match the tags of an artifact as `find` does: every tag given must be present, none of the + excluded ones (a leading "-"), case-insensitively. + """ + + if tags is None or len(tags) == 0: + return True + + if type(ctags) != list: + return False + + ctags_lower = [str(t).lower() for t in ctags] + + for tag in tags: + tag = str(tag) + if tag.startswith('-'): + if tag[1:].lower() in ctags_lower: + return False + elif tag.lower() not in ctags_lower: + return False + + return True + ############################################################ def get_( self, diff --git a/cmeta/internal-repo/category/repo/api/v1.py b/cmeta/internal-repo/category/repo/api/v1.py index e37b3e6..cb3fb1e 100644 --- a/cmeta/internal-repo/category/repo/api/v1.py +++ b/cmeta/internal-repo/category/repo/api/v1.py @@ -1457,6 +1457,126 @@ def unplug( return self.delete(p) + ############################################################################################### + def reindex_( + self, + ctx: dict, # cMeta context. + arg1: str = None, # The repository: alias, UID or alias,UID (wildcards allowed), the path of its folder, or "." for the repository of the current directory. + ): + """ + Reindex registered repositories: re-read their `_cmr.yaml` and index their artifacts again, + as `pull`, `plug` and `unzip` do when they bring a repository in, leaving the records of the + other repositories as they are. Use it after files were copied or pulled into a repository by + other means, after its `_cmr.yaml` was edited, or when its artifacts look stale - instead of + `cx --reindex`, which rebuilds the index of every repository. + + A repository whose folder is not there (a detached drive) is refused and nothing is changed. + + Args: + ctx (dict): cMeta context. + arg1 (str): The repository: alias, UID or alias,UID (wildcards allowed), the path of its + folder, or "." for the repository of the current directory. Required. + + Returns: + dict: A cMeta dictionary with the following keys: + - **return** (int): 0 if success, >0 if error (16: no such repository). + - **error** (str): Error message if `return > 0`. + - **artifacts** (list): The repositories reindexed (their records as they were before). + - **paths** (list): Their registered paths. + + Raises: + Exception: Propagated runtime errors, if any. + """ + + con = ctx.get('control',{}).get('con', False) + verbose = ctx.get('control',{}).get('verbose', False) + + if arg1 is None or str(arg1).strip() == '': + return {'return':1, 'error':'name the repository to reindex: its alias, UID or path, or "." for the repository of the current directory ' + '(to rebuild the index of every repository, run "cx --reindex")'} + + arg1 = str(arg1).strip() + + repo_artifacts = [] + + if arg1 == '.' or os.path.isdir(arg1): + # A folder: the registered repository that holds it (the deepest one, for nested repositories) + path = os.path.abspath(os.path.normpath(os.getcwd() if arg1 == '.' else arg1)) + + r = self.cm.repos.find_in_index('repo', self.cm.cfg['category_repo_uid']) + if r['return']>0: return r + + best = None + best_path = None + + for repo in r['artifacts']: + repo_path = os.path.abspath(os.path.normpath(repo['path'])) + + if utils.files.is_path_within(repo_path, path): + if best is None or len(repo_path) > len(best_path): + best = repo + best_path = repo_path + + if best is None: + return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + + repo_artifacts = [best] + + else: + p = {'category': ctx['category'], + 'command': 'find', + 'arg1': arg1, + 'sort': False, + 'base': True, + 'con': False} + + r = self.cm.access(p) + if r['return']>0: return r + + repo_artifacts = r['artifacts'] + + # Only a registered path can be reindexed, and only one whose folder is there: the incremental + # index drops a repository whose descriptor it cannot read (unless the repository is kept) + repos_config_path = self.cm.repos_config_path + + r = utils.files.safe_read_file(repos_config_path, lock=False, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + repos_paths = r['data'] + + paths = [] + + for repo in repo_artifacts: + repo_cmeta_ref_parts = repo['cmeta_ref_parts'] + repo_alias = repo_cmeta_ref_parts.get('artifact_alias', repo_cmeta_ref_parts['artifact_uid']) + + path = repo['path'] + + if path not in repos_paths: + caller = ctx.get('origin',{}).get('cli',{}).get('caller','') + return {'return':1, 'error':f'repository "{repo_alias}" is indexed with the path "{path}" that is not in {repos_config_path} - try "{caller} --reindex"'} + + if not os.path.isdir(path) or not os.path.isfile(os.path.join(path, self.cm.cfg['repo_meta_desc'])): + return {'return':1, 'error':f'the folder of repository "{repo_alias}" or its {self.cm.cfg["repo_meta_desc"]} is not there: {path} (a detached drive?) - nothing was reindexed'} + + paths.append(path) + + if con: + print ('') + + r = self.cm.repos.index(clean=False, con=con, verbose=verbose, add_repo_paths=paths) + if r['return']>0: return r + + if con: + for repo, path in zip(repo_artifacts, paths): + repo_cmeta_ref_parts = repo['cmeta_ref_parts'] + repo_alias = repo_cmeta_ref_parts.get('artifact_alias', '') + repo_uid = repo_cmeta_ref_parts['artifact_uid'] + + print (f'Reindexed repository "{repo_alias}" ({repo_uid}): {path}') + + return {'return':0, 'artifacts': repo_artifacts, 'paths': paths} + ############################################################################################### def space( self, diff --git a/cmeta/repos.py b/cmeta/repos.py index 6bb7aa6..eb7ae58 100644 --- a/cmeta/repos.py +++ b/cmeta/repos.py @@ -158,6 +158,7 @@ def add_to_index( original_alias = None, # Previous artifact alias before update. original_uid = None, # Previous artifact UID before update. sharding_slices_num = 0, # Number of sharding slices + clean_aliases = False, # If True, drop every other alias of this UID from the alias index (a folder renamed by hand), in the same locked write. ): """ Insert or update an artifact record in the category index. @@ -169,8 +170,11 @@ def add_to_index( original_alias: Previous artifact alias before update. original_uid: Previous artifact UID before update. sharding_slices_num (int): Number of UID slices used to shard artifact paths. + clean_aliases (bool): If True, every alias of the alias index that still lists this UID + and is not the alias of the record is dropped (the stale aliases a folder renamed by + hand leaves behind), in the same locked write. Aliases of other UIDs are not touched. Returns: - dict: Operation result. + dict: Operation result, with 'dropped_aliases': the (lowercase) aliases dropped by `clean_aliases`. Raises: Exception: Propagated runtime errors, if any. """ @@ -215,7 +219,9 @@ def add_to_index( uids[artifact_uid] = record - if artifact_alias is not None: + artifact_alias_lowercase = None + + if artifact_alias is not None: artifact_alias_lowercase = artifact_alias.lower() lowercase_alias_uids = lowercase_aliases.get(artifact_alias_lowercase, []) @@ -224,6 +230,24 @@ def add_to_index( lowercase_alias_uids.append(artifact_uid) lowercase_aliases[artifact_alias_lowercase] = lowercase_alias_uids + # The other aliases that still point at this UID (a folder renamed by hand keeps its old + # alias in the index): only this UID is removed from them, other UIDs under the same alias stay + dropped_aliases = [] + + if clean_aliases: + for lowercase_alias in list(lowercase_aliases.keys()): + if lowercase_alias == artifact_alias_lowercase: + continue + + lowercase_alias_uids = lowercase_aliases[lowercase_alias] + + if artifact_uid in lowercase_alias_uids: + lowercase_alias_uids.remove(artifact_uid) + dropped_aliases.append(lowercase_alias) + + if len(lowercase_alias_uids) == 0: + del(lowercase_aliases[lowercase_alias]) + # Use atomic write to avoid corrupting large index files r = utils.files.safe_write_file(index_file, index_data, file_lock=index_file_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r @@ -231,7 +255,7 @@ def add_to_index( # r = utils.files.safe_write_file(os.path.splitext(index_file)[0] + ".json", index_data, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger) # if r['return']>0: return r - return {'return':0} + return {'return':0, 'dropped_aliases': dropped_aliases} @@ -1311,13 +1335,26 @@ def _index( path_to_category = os.path.join(repo_full_path, category) - r = self._find_artifacts(repo_meta, repo_alias, repo_uid, category_meta, category_alias, category_uid, path_to_category, + r = self._find_artifacts(repo_meta, repo_alias, repo_uid, category_meta, category_alias, category_uid, path_to_category, con, conx, index_artifacts, artifact_num) if r['return'] >0: return r an = r['artifact_num'] artifact_num = an + if not clean: + # The records of the repositories being indexed again are removed from every existing + # category index below, also from the index of a category of which such a repository + # holds no artifact any more (otherwise the records of its deleted artifacts would stay) + for category_mix in categories: + category = category_mix['category'] + + if category in ['category', 'repo'] or category_mix['meta'].get('no_index', False): + continue + + if category not in index_artifacts and os.path.isfile(os.path.join(index_path, category + self.index_extension)): + index_artifacts[category] = {self.KEY_INDEX_UIDS:{}, self.KEY_INDEX_LOWERCASE_ALIASES:{}} + else: for category_mix in categories: index_artifacts[category_mix['category']] = {} @@ -1491,7 +1528,8 @@ def _find_artifacts( if sharding_slices is not None: tmp_artifact_dirs = _get_artifacts_from_sharded_path(path_to_category, sharding_slices, artifact_alias=artifact_alias) else: - tmp_artifact_dirs = os.listdir(path_to_category) + # Only the folders the pattern names (case-insensitively, as the sharded search does) + tmp_artifact_dirs = [d for d in os.listdir(path_to_category) if fnmatch.fnmatch(d.lower(), artifact_alias_lowercase)] else: last_artifact_dirs_parent = None @@ -1507,11 +1545,11 @@ def _find_artifacts( path_to_category_with_shard = os.path.join(path_to_category, last_artifact_dirs_parent) if os.path.isdir(path_to_category_with_shard): + # Every folder that spells the alias (two spellings can coexist on a case-sensitive file system) for artifact_dir in os.listdir(path_to_category_with_shard): if artifact_dir.lower() == artifact_alias_lowercase: tmp_dir = artifact_dir if last_artifact_dirs_parent is None else os.path.join(last_artifact_dirs_parent, artifact_dir) tmp_artifact_dirs.append(tmp_dir) - break for long_artifact in tmp_artifact_dirs: @@ -1550,6 +1588,13 @@ def _find_artifacts( if artifact_meta: if index_artifacts is None: + # A meta without a category (or with one that does not parse) is not an artifact of this category + if 'category' not in artifact_meta: + print ('', flush=True) + print (f" Warning: {artifact} doesn't have 'category' key in {path_to_artifact}") + self.logger.error (f" Warning: {artifact} doesn't have proper 'category' key in {path_to_artifact}") + continue + r = utils.names.parse_cmeta_name(artifact_meta['category']) if r['return']>0: return r diff --git a/cmeta/version.py b/cmeta/version.py index 40f23e5..f296d88 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.33.1" -__release_date__ = "2026-10-07" +__version__ = "0.33.1.1" +__release_date__ = "2026-10-08" diff --git a/docs/common-commands.md b/docs/common-commands.md index 898864b..cd477f4 100644 --- a/docs/common-commands.md +++ b/docs/common-commands.md @@ -99,6 +99,7 @@ cx update --meta.description="..." cx tags --add=t1,t2 --remove=t3 cx mv cx migrate # the same, plus a stub: the old alias keeps working +cx reindex # refresh its index record from its folder (nothing is written) cx rm ``` @@ -141,6 +142,7 @@ than with `cx add`. Afterwards, register it so it enters the index: ```bash cx index : +cx reindex # the same, and it also refreshes an artifact the index knows already ``` --- @@ -157,6 +159,7 @@ cx repo get --path= --local # register an existing folder cx repo pull cx repo status cx repo unplug # detach without deleting +cx repo reindex # re-read its _cmr.yaml and rescan its artifacts; other repos untouched cx repo space # disk usage ``` @@ -181,7 +184,11 @@ Common ones: `default` (framework defaults), `task` (caches, version checks), ```bash cx index : # register ONE hand-made artifact -cx update : # refresh index after editing its meta +cx reindex # refresh ONE record from its folder: a meta edited by hand, a folder + # renamed or copied in; nothing is written, stale aliases are dropped +cx reindex # every artifact of the category (hand-made folders included) +cx repo reindex # ONE repository: its _cmr.yaml and its artifacts; the others untouched +cx update : # refresh after editing its meta (rewrites the file: a new timestamp) cx --reindex # rebuild everything (slow — last resort) cx cache show @@ -189,8 +196,11 @@ cx cache clean cx cache delete ``` -Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) need **no** reindex -— the index tracks meta, not content. +`reindex` never touches a folder, and the only record it removes is that of an +artifact whose folder now holds another artifact (it says so): a folder it cannot +find, a meta it cannot read, or one UID in two folders is an error that leaves +the index as it was. Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) +need **no** reindex — the index tracks meta, not content. Categories holding very many artifacts can spread them into sub-directories (`sharding_slices`) and/or skip the index entirely (`no_index: true`) — see diff --git a/docs/using-cmeta.md b/docs/using-cmeta.md index 73c2b36..131d0bc 100644 --- a/docs/using-cmeta.md +++ b/docs/using-cmeta.md @@ -68,7 +68,7 @@ any command unless noted. |------|------|--------| | `--help`, `-h` | bool | Show help. On a category or command, prints its usage. | | `--version`, `-V` | bool | Print cMeta version and check for a newer release. | -| `--reindex` | bool | Clean and rebuild the fast index (`/index/*.pkl`). Use after manual edits, `git pull`, cleared `CMETA_HOME`, or when lookups look stale. | +| `--reindex` | bool | Clean and rebuild the fast index (`/index/*.pkl`): every repository, every category. Use after a cleared `CMETA_HOME` or a bulk change; for one artifact or one repository, `cx reindex ` and `cx repo reindex ` are the narrower forms (§11). | | `--verbose`, `-v` | bool | Verbose output — extra progress info from repo ops and package detection. | | `--quiet`, `-q` | bool | Auto-accept the default answer for any interactive prompt (safe for scripts). | | `--repro`, `-r` | bool | Write `cmeta-repro-input.json` (+ `-input2.json`, `-output.json`) for reproducibility of the call. | @@ -171,6 +171,7 @@ cx ... -v --con # noisy output for troubleshoot cx ... --repro # capture inputs/outputs for later replay cx ... -j --jf=out.json # machine-readable result to stdout + file cx --reindex # rebuild the fast index +cx reindex # refresh one artifact's index record from its folder cx --home=D:\work\cmeta-home # one-shot alternative CMETA_HOME ``` @@ -460,7 +461,8 @@ cx utils uid --clipboard- --jf=uid.json # {"return": 0, "uid": "29576481b258 From Python the underlying helper is `cm.utils.names.generate_cmeta_uid()`. Remember to register a hand-made artifact afterwards so it enters the index: -`cx index :` (see §11). +`cx index :`, or `cx reindex `, +which also refreshes an artifact the index knows already (see §11). Two neighbours of these generators exist for credentials. `cx utils api_key` prints random keys from `secrets.token_urlsafe` together with the @@ -1533,17 +1535,44 @@ the filesystem. The framework refreshes it automatically: - when repos are added/removed via `cx repo` operations, - on first launch if the index is missing. -Force a full clean rebuild when things look inconsistent — after a manual move, -a hand-edit of `_cmeta.yaml`, a `git pull` inside a repo, a `CMETA_HOME` wipe, -or if `cx find ` returns something surprising: +When artifacts were touched outside cMeta, refresh the narrowest thing: ```bash -cx --reindex -``` - -It's safe, idempotent and typically fast. Categories that set `no_index: true` -in their `_cmeta.yaml` are always found by filesystem scan and are never -written to the index. +cx reindex # one artifact: its record is rewritten from the meta file in its folder +cx reindex # every artifact of the category, folders made by hand included +cx repo reindex # one repository: its _cmr.yaml re-read, its artifacts rescanned +cx --reindex # everything: a clean rebuild of every index file +``` + +`cx reindex ` reads the artifact's `_cmeta.*` from its folder +and rewrites its index record, and writes nothing on disk — after a meta edited by +hand, a folder renamed, moved or copied in from another machine, or when +`cx find ` returns something surprising. The artifact is named +by its alias, UID, `alias,UID` or `repo:alias`, with wildcards (`--tags` prunes); +a folder the index does not know yet is found on disk in the repositories, and an +artifact whose indexed folder is gone is looked for by its UID in its repository, +so after a rename by hand the new name, the UID and the old name all work, and +the old alias is dropped from the index. No folder is touched, and the only record +ever removed is that of an artifact whose folder, still there, holds another +artifact now (replaced by hand) — said in the output. A folder that cannot be +found, a meta that cannot be read or has no valid `artifact` UID, a meta of +another category, or one UID found in two existing folders is an error that +leaves the index as it was (`--ignore_errors` reindexes the other artifacts and +reports these). An empty `cx update ` +refreshes the record too, but it also rewrites the meta file (a new +`last_update_timestamp` and a generator record): use `reindex` when the file must +stay as it is, for instance in a folder that another machine syncs. + +`cx repo reindex ` — by alias, UID, the path of its folder, or `.` from +inside it — does what `cx repo pull` and `cx repo plug` do once a repository is +in place: its `_cmr.yaml` is re-read and its artifacts are indexed again, and the +records of the other repositories stay as they are. A repository whose folder is +not there (a detached drive) is refused and nothing changes. + +`cx --reindex` rebuilds every index file from scratch — after a `CMETA_HOME` +wipe, a bulk change across repositories, or when in doubt. It's safe, idempotent +and typically fast. Categories that set `no_index: true` in their `_cmeta.yaml` +are always found by filesystem scan and are never written to the index. Task-level content caches produced by workflows live under the `cache` category and are managed separately: diff --git a/tests/internal_repo_tests/test_integration_reindex_artifact.py b/tests/internal_repo_tests/test_integration_reindex_artifact.py new file mode 100644 index 0000000..d908b89 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_reindex_artifact.py @@ -0,0 +1,842 @@ +""" +Integration tests for `cx reindex []`: the index record of an artifact is +rewritten from the meta file in its folder without any file being changed - after a meta edited by +hand, a folder renamed or copied in by hand, a stale lookup - and nothing is ever removed from the +index or touched on disk. The record is the one a full `cx --reindex` records. + +Each test runs against a fresh in tmp_path so it does not touch the user's real +cMeta state. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import os +import pickle +import shutil +import subprocess +import sys + +import pytest + +from cmeta import CMeta +from cmeta import repos as cmeta_repos + +CATEGORY_UID = 'fedcba9876543210' + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + monkeypatch.setattr(cmeta_repos, '_migrated_notices', set()) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make(cm, alias, category='log', repo='local', yaml=False, **meta): + """An artifact made by cMeta; returns its UID.""" + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=yaml) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def reindex(cm, ref=None, category='log', **params): + return access(cm, category=category, command='reindex', arg1=ref, **params) + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def record(cm, ref, category='log'): + """The index record of one artifact as `find` returns it (path, cmeta, cmeta_ref_parts, sharding).""" + r = find(cm, ref, category=category) + assert r['return'] == 0, r.get('error') + assert len(r['artifacts']) == 1, r['artifacts'] + a = r['artifacts'][0] + return {k: a[k] for k in a if k != 'index_file'} + + +def index_data(tmp_path, category='log'): + """The pickled index of a category as it is on disk.""" + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def index_bytes(tmp_path, category='log'): + return (tmp_path / 'index' / f'{category}.pkl').read_bytes() + + +def meta_file(folder): + for name in ('_cmeta.yaml', '_cmeta.json'): + if (folder / name).is_file(): + return folder / name + raise AssertionError(f'no meta file in {folder}') + + +def file_state(path): + return (path.stat().st_mtime_ns, path.read_bytes()) + + +def hand_edit(folder, text): + """Append text to the meta file as a user would, keeping its format: YAML lines or a JSON key.""" + f = meta_file(folder) + if f.suffix == '.yaml': + f.write_text(f.read_text(encoding='utf-8') + text, encoding='utf-8') + else: + import json + data = json.loads(f.read_text(encoding='utf-8')) + for line in text.strip().splitlines(): + key, value = line.split(':', 1) + data[key.strip()] = value.strip() + f.write_text(json.dumps(data, indent=2, sort_keys=True) + '\n', encoding='utf-8') + return f + + +def make_repo(tmp_path, name, repo_uid): + """A repository folder with its descriptor only (its artifacts use the categories of the internal repo).""" + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def case_insensitive_fs(tmp_path): + probe = tmp_path / 'CaseProbe' + probe.mkdir(exist_ok=True) + return (tmp_path / 'caseprobe').is_dir() + + +########################################################################################## +# A meta edited by hand + +def test_reindex_reads_a_hand_edited_meta_and_leaves_the_file_alone(cm, tmp_path): + uid = make(cm, 'edited', yaml=True, tags=['before']) + folder = tmp_path / 'repos' / 'local' / 'log' / 'edited' + f = hand_edit(folder, 'owner: me\n') + before = file_state(f) + + assert find(cm, 'edited', tags='before')['return'] == 0 + assert 'owner' not in record(cm, 'edited')['cmeta'] + + r = reindex(cm, 'edited') + assert r['return'] == 0, r.get('error') + assert [a['cmeta_ref_parts']['artifact_uid'] for a in r['artifacts']] == [uid] + assert r['dropped_aliases'] == {} and r['errors'] == [] + + # The index knows the edit, the file is exactly as the user left it + assert record(cm, 'edited')['cmeta']['owner'] == 'me' + assert record(cm, uid)['cmeta']['tags'] == ['before'] + assert file_state(f) == before + + +def test_reindex_of_a_json_meta(cm, tmp_path): + uid = make(cm, 'json-one', payload='x') + folder = tmp_path / 'repos' / 'local' / 'log' / 'json-one' + f = hand_edit(folder, 'owner: me\n') + before = file_state(f) + + r = reindex(cm, 'json-one') + assert r['return'] == 0, r.get('error') + assert record(cm, 'json-one')['cmeta'] == {**record(cm, uid)['cmeta']} + assert record(cm, uid)['cmeta']['owner'] == 'me' and record(cm, uid)['cmeta']['payload'] == 'x' + assert file_state(f) == before + + +def test_the_record_is_the_one_of_a_full_reindex(cm, tmp_path): + make(cm, 'yaml-art', yaml=True, tags=['t1']) + make(cm, 'json-art', tags=['t2']) + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'yaml-art', 'owner: me\n') + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'json-art', 'owner: you\n') + + # A folder made by hand, unknown to the index, is indexed by reindex as well + hand_made = tmp_path / 'repos' / 'local' / 'log' / 'By-Hand' + hand_made.mkdir() + (hand_made / '_cmeta.yaml').write_text('artifact: By-Hand,0123456789abcdef\ncategory: log,487a7639093a4685\ntags:\n- hand\n', encoding='utf-8') + + r = reindex(cm) # every artifact of the category + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['By-Hand', 'json-art', 'yaml-art'] + + after_reindex = {ref: record(cm, ref) for ref in ('yaml-art', 'json-art', 'By-Hand', 'by-hand', '0123456789abcdef')} + data_reindex = index_data(tmp_path) + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + + for ref in after_reindex: + assert record(cm, ref) == after_reindex[ref], ref + + data_full = index_data(tmp_path) + assert data_full['uids'] == data_reindex['uids'] + assert data_full['lowercase_aliases'] == data_reindex['lowercase_aliases'] + # The alias keeps the case of the folder, the lowercase form is recorded when it differs + parts = after_reindex['By-Hand']['cmeta_ref_parts'] + assert parts['artifact_alias'] == 'By-Hand' and parts['artifact_alias_lowercase'] == 'by-hand' + + +def test_a_second_reindex_changes_nothing(cm, tmp_path): + make(cm, 'stable', yaml=True) + assert reindex(cm, 'stable')['return'] == 0 + data = index_data(tmp_path) + r = reindex(cm, 'stable') + assert r['return'] == 0 and r['dropped_aliases'] == {} + assert index_data(tmp_path) == data + + +########################################################################################## +# Folders renamed by hand + +def test_a_folder_renamed_by_hand_is_found_by_its_new_name(cm, tmp_path): + uid = make(cm, 'old-name', yaml=True, payload='x') + other = make(cm, 'other', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'old-name').rename(base / 'new-name') + + # Before: the index still answers with the old name and a folder that is gone + assert uids(find(cm, 'old-name')) == [uid] + assert find(cm, 'new-name')['return'] == 16 + + r = reindex(cm, 'new-name') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['old-name']} + + assert uids(find(cm, 'new-name')) == [uid] + assert find(cm, 'old-name')['return'] == 16 + assert same(record(cm, uid)['path']) == same(base / 'new-name') + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias'] == 'new-name' + assert record(cm, uid)['cmeta']['payload'] == 'x' + assert 'old-name' not in index_data(tmp_path)['lowercase_aliases'] + # alias,UID with the old alias still resolves: the UID is authoritative + assert uids(find(cm, f'old-name,{uid}')) == [uid] + # The other artifact is untouched + assert uids(find(cm, 'other')) == [other] + + +def test_a_folder_renamed_by_hand_is_found_by_its_uid_or_its_old_name(cm, tmp_path): + base = tmp_path / 'repos' / 'local' / 'log' + + uid1 = make(cm, 'by-uid', yaml=True) + (base / 'by-uid').rename(base / 'by-uid-renamed') + r = reindex(cm, uid1) + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid1: ['by-uid']} + assert uids(find(cm, 'by-uid-renamed')) == [uid1] and find(cm, 'by-uid')['return'] == 16 + + uid2 = make(cm, 'by-old', yaml=True) + (base / 'by-old').rename(base / 'by-old-renamed') + r = reindex(cm, 'by-old') # the stale alias: the folder is looked for by its UID + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid2: ['by-old']} + assert uids(find(cm, 'by-old-renamed')) == [uid2] and find(cm, 'by-old')['return'] == 16 + + uid3 = make(cm, 'by-both', yaml=True) + (base / 'by-both').rename(base / 'by-both-renamed') + r = reindex(cm, f'by-both,{uid3}') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-both-renamed')) == [uid3] and find(cm, 'by-both')['return'] == 16 + + data = index_data(tmp_path) + assert set(data['lowercase_aliases']) == {'by-uid-renamed', 'by-old-renamed', 'by-both-renamed'} + assert set(data['uids']) == {uid1, uid2, uid3} + + +def test_a_folder_renamed_twice_loses_both_old_aliases(cm, tmp_path): + uid = make(cm, 'first', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'first').rename(base / 'second') + # The plain "index" of the renamed folder keeps the old alias (today's behaviour, unchanged) + r = access(cm, category='log', command='index', arg1='local:second') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'first')) == [uid] and uids(find(cm, 'second')) == [uid] + (base / 'second').rename(base / 'third') + + r = reindex(cm, 'third') + assert r['return'] == 0, r.get('error') + assert sorted(r['dropped_aliases'][uid]) == ['first', 'second'] + assert find(cm, 'first')['return'] == 16 and find(cm, 'second')['return'] == 16 + assert uids(find(cm, 'third')) == [uid] + + +def test_a_case_only_rename_follows_the_folder(cm, tmp_path): + uid = make(cm, 'MixedCase', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias'] == 'MixedCase' + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias_lowercase'] == 'mixedcase' + + os.rename(str(base / 'MixedCase'), str(base / 'mixedcase')) + assert sorted(p.name for p in base.iterdir()) == ['mixedcase'] + + r = reindex(cm, 'mixedcase') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {} # the same lowercase alias: nothing to drop + + parts = record(cm, uid)['cmeta_ref_parts'] + assert parts['artifact_alias'] == 'mixedcase' + assert 'artifact_alias_lowercase' not in parts + assert os.path.basename(record(cm, uid)['path']) == 'mixedcase' + # Lookups are case-insensitive either way + assert uids(find(cm, 'MixedCase')) == [uid] and uids(find(cm, 'MIXEDCASE')) == [uid] + + +def test_two_spellings_on_a_case_sensitive_file_system(cm, tmp_path): + if case_insensitive_fs(tmp_path): + pytest.skip('the file system folds case: two spellings cannot coexist') + base = tmp_path / 'repos' / 'local' / 'log' + uid1 = make(cm, 'Spell', yaml=True) + # cMeta itself refuses the second spelling (the index is case-insensitive): it can only be made by hand + r = access(cm, category='log', command='create', arg1='local:spell', yaml=True) + assert r['return'] == 8 + (base / 'spell').mkdir() + (base / 'spell' / '_cmeta.yaml').write_text('artifact: spell,dddddddddddddddd\ncategory: log,487a7639093a4685\nowner: two\n', encoding='utf-8') + hand_edit(base / 'Spell', 'owner: one\n') + + # Both folders are found, whichever spelling is given, and both UIDs sit under the one lowercase alias + r = reindex(cm, 'spell') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([uid1, 'dddddddddddddddd']) + assert record(cm, uid1)['cmeta']['owner'] == 'one' and record(cm, 'dddddddddddddddd')['cmeta']['owner'] == 'two' + assert record(cm, uid1)['cmeta_ref_parts']['artifact_alias'] == 'Spell' + assert record(cm, 'dddddddddddddddd')['cmeta_ref_parts']['artifact_alias'] == 'spell' + assert sorted(index_data(tmp_path)['lowercase_aliases']['spell']) == sorted([uid1, 'dddddddddddddddd']) + # A lookup by the alias is ambiguous, as after a full reindex + assert uids(find(cm, 'Spell')) == sorted([uid1, 'dddddddddddddddd']) + both = {uid: record(cm, uid) for uid in (uid1, 'dddddddddddddddd')} + assert access(cm, reindex=True)['return'] == 0 + for uid in both: + assert record(cm, uid) == both[uid] + + +def test_a_json_meta_renamed_by_hand(cm, tmp_path): + uid = make(cm, 'js-old', payload='j') # a JSON meta + base = tmp_path / 'repos' / 'local' / 'log' + assert (base / 'js-old' / '_cmeta.json').is_file() + (base / 'js-old').rename(base / 'js-new') + f = base / 'js-new' / '_cmeta.json' + before = file_state(f) + + r = reindex(cm, 'js-new') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['js-old']} + assert uids(find(cm, 'js-new')) == [uid] and find(cm, 'js-old')['return'] == 16 + assert record(cm, uid)['cmeta']['payload'] == 'j' + assert file_state(f) == before + + +def test_an_upper_case_uid_in_a_meta(cm, tmp_path): + make(cm, 'seed', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'upper').mkdir() + (base / 'upper' / '_cmeta.yaml').write_text('artifact: upper,ABCDEF0123456789\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'upper') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['abcdef0123456789'] + assert uids(find(cm, 'ABCDEF0123456789')) == ['abcdef0123456789'] + assert uids(find(cm, 'abcdef0123456789')) == ['abcdef0123456789'] + rec = record(cm, 'upper') + assert access(cm, reindex=True)['return'] == 0 + assert record(cm, 'upper') == rec + + +def test_an_unreadable_meta_file(cm, tmp_path): + uid = make(cm, 'broken', yaml=True) + other = make(cm, 'fine', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + f = base / 'broken' / '_cmeta.yaml' + f.write_text('artifact: [unclosed\n', encoding='utf-8') + before = index_bytes(tmp_path) + state = file_state(f) + + for ref in ('broken', uid): + r = reindex(cm, ref) + assert r['return'] == 1 and 'cannot be read' in r['error'], r + assert index_bytes(tmp_path) == before + assert file_state(f) == state # the broken file is not touched either + + hand_edit(base / 'fine', 'owner: me\n') + r = reindex(cm, ignore_errors=True) + assert r['return'] == 0 and uids(r) == [other] and len(r['errors']) == 1 + assert uids(find(cm, 'broken')) == [uid] # its old record is kept + + +def test_a_non_ascii_alias(cm, tmp_path): + r = access(cm, category='log', command='create', arg1='local:café-ünïcode', yaml=True, meta={'payload': 'é'}) + if r['return'] != 0: + pytest.skip(f'non-ASCII aliases are refused here: {r.get("error")}') + uid = r['meta']['artifact'] + base = tmp_path / 'repos' / 'local' / 'log' + hand_edit(base / 'café-ünïcode', 'owner: moi\n') + r = reindex(cm, 'café-ünïcode') + assert r['return'] == 0, r.get('error') + assert record(cm, uid)['cmeta']['owner'] == 'moi' + + (base / 'café-ünïcode').rename(base / 'naïve-änderung') + r = reindex(cm, 'naïve-änderung') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['café-ünïcode']} + assert uids(find(cm, 'naïve-änderung')) == [uid] and uids(find(cm, 'NAÏVE-ÄNDERUNG')) == [uid] + assert find(cm, 'café-ünïcode')['return'] == 16 + + +def test_a_rename_by_hand_inside_a_sharded_category(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', 'sharding_slices:\n- 2\n') + plug(cm, repo) + uid = make(cm, 'abcdef', category='shard', repo='shard', yaml=True) + folder = pytest.importorskip('pathlib').Path(find(cm, 'abcdef', category='shard')['artifacts'][0]['path']) + # Renamed in place: the folder stays in the "ab" shard although the new name would belong to "xy" + folder.rename(folder.parent / 'xyzdef') + + r = reindex(cm, uid, category='shard') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['abcdef']} + rec = record(cm, uid, category='shard') + assert rec['cmeta_ref_parts']['artifact_alias'] == 'xyzdef' and rec['sharding_slices_num'] == 1 + assert same(rec['path']) == same(folder.parent / 'xyzdef') + assert uids(find(cm, 'xyzdef', category='shard')) == [uid] + # The same record as the full reindex, which also finds the folder where it is + assert access(cm, reindex=True)['return'] == 0 + assert record(cm, uid, category='shard') == rec + + +def test_a_category_without_an_index_file_yet(cm, tmp_path): + repo = make_category_repo(tmp_path, 'fresh', '5555555555555555', '') + plug(cm, repo) + assert not (tmp_path / 'index' / 'fresh.pkl').is_file() + folder = repo / 'fresh' / 'first' + folder.mkdir(parents=True) + (folder / '_cmeta.yaml').write_text(f'artifact: first,eeeeeeeeeeeeeeee\ncategory: fresh,{CATEGORY_UID}\n', encoding='utf-8') + + r = reindex(cm, category='fresh') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['eeeeeeeeeeeeeeee'] + assert (tmp_path / 'index' / 'fresh.pkl').is_file() + assert uids(find(cm, 'first', category='fresh')) == ['eeeeeeeeeeeeeeee'] + + +def test_a_repository_with_a_subdir(cm, tmp_path): + repo = tmp_path / 'extra' / 'withsub' + (repo / 'content').mkdir(parents=True) + (repo / '_cmr.yaml').write_text('artifact: withsub,6666666666666666\ncategory: repo,f4f792ab40c7498f\nsubdir: content\n', encoding='utf-8') + plug(cm, repo) + uid = make(cm, 'in-sub', repo='withsub', yaml=True) + folder = repo / 'content' / 'log' / 'in-sub' + assert folder.is_dir() + hand_edit(folder, 'owner: me\n') + + r = reindex(cm, 'withsub:in-sub') + assert r['return'] == 0, r.get('error') + assert record(cm, uid)['cmeta']['owner'] == 'me' + assert same(record(cm, uid)['path']) == same(folder) + + (folder.parent / 'in-sub').rename(folder.parent / 'in-sub-2') + r = reindex(cm, uid) + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'in-sub-2')) == [uid] and find(cm, 'in-sub')['return'] == 16 + + +def test_the_shipped_internal_repository_is_only_read(cm): + before = record(cm, 'log', category='category') + f = pytest.importorskip('pathlib').Path(before['path']) / '_cmeta.yaml' + state = file_state(f) + + r = reindex(cm, 'log', category='category') + assert r['return'] == 0, r.get('error') + assert record(cm, 'log', category='category') == before + assert file_state(f) == state + + +@pytest.mark.xfail(sys.platform != 'win32', strict=False, + reason='the index lock file is deleted after every release (utils.files._release_lock -> ' + '_cleanup_lock_file): with flock a process waiting on the old inode and a newcomer that ' + 'creates a new lock file both acquire, two writers enter add_to_index and one update is ' + 'lost (seen on WSL2 and in a container, 2026-10-08; Windows cannot delete an open file). ' + 'The fix belongs to the concurrency step of the engine hardening') +def test_several_processes_reindex_at_once(cm, tmp_path): + base = tmp_path / 'repos' / 'local' / 'log' + names = [f'par-{i}' for i in range(6)] + made = {name: make(cm, name, yaml=True) for name in names} + for name in names: + hand_edit(base / name, f'owner: {name}\n') + + env = {k: v for k, v in os.environ.items() if not k.startswith('CMETA_') and k not in ('VIRTUAL_ENV', 'CONDA_PREFIX')} + env['CMETA_HOME'] = str(tmp_path) + env['PYTHONIOENCODING'] = 'utf-8' + + procs = [subprocess.Popen([sys.executable, '-m', 'cmeta', 'log', 'reindex', name], env=env, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) for name in names] + outputs = [p.communicate(timeout=180)[0] for p in procs] + assert all(p.returncode == 0 for p in procs), outputs + + data = index_data(tmp_path) + for name in names: + assert data['uids'][made[name]]['cmeta']['owner'] == name + assert data['lowercase_aliases'][name] == [made[name]] + assert len(data['uids']) == len(names) + + +########################################################################################## +# Nothing is lost: the errors leave the index as it was + +def test_two_folders_with_one_uid_is_an_error_that_changes_nothing(cm, tmp_path): + uid = make(cm, 'original', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + shutil.copytree(base / 'original', base / 'copied') + before = index_bytes(tmp_path) + + for ref in ('copied', uid, None): + r = reindex(cm, ref) + assert r['return'] == 1, (ref, r) + assert 'same UID' in r['error'] or 'two folders with one UID' in r['error'], r['error'] + assert index_bytes(tmp_path) == before, ref + + # With --ignore_errors the other artifacts are reindexed and the conflict is reported + other = make(cm, 'fine', yaml=True) + hand_edit(base / 'fine', 'owner: me\n') + r = reindex(cm, ignore_errors=True) + assert r['return'] == 0, r.get('error') + assert uids(r) == [other] + assert len(r['errors']) == 1 and uid in r['errors'][0]['error'] + assert record(cm, other)['cmeta']['owner'] == 'me' + assert same(record(cm, uid)['path']) == same(base / 'original') + + # A new UID for the copy resolves it + (base / 'copied' / '_cmeta.yaml').write_text('artifact: copied,aaaaaaaaaaaaaaaa\ncategory: log,487a7639093a4685\n', encoding='utf-8') + r = reindex(cm, 'copied') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'copied')) == ['aaaaaaaaaaaaaaaa'] and uids(find(cm, 'original')) == [uid] + + +def test_a_missing_folder_is_an_error_that_changes_nothing(cm, tmp_path): + uid = make(cm, 'gone', yaml=True) + shutil.rmtree(tmp_path / 'repos' / 'local' / 'log' / 'gone') + before = index_bytes(tmp_path) + + for ref in ('gone', uid): + r = reindex(cm, ref) + assert r['return'] == 1 and 'no folder with its UID' in r['error'], r + assert index_bytes(tmp_path) == before + + # delete, as the error says, removes the record of the missing folder + r = access(cm, category='log', command='delete', arg1='gone', force=True) + assert r['return'] == 0, r.get('error') + assert find(cm, 'gone')['return'] == 16 + + +def test_a_folder_replaced_by_another_artifact(cm, tmp_path): + # The folder of an artifact was removed and another artifact was put under the same name by hand + uid = make(cm, 'slot', yaml=True) + other = make(cm, 'other', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + shutil.rmtree(base / 'slot') + (base / 'slot').mkdir() + (base / 'slot' / '_cmeta.yaml').write_text('artifact: slot,aaaaaaaaaaaaaaaa\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'slot') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['aaaaaaaaaaaaaaaa'] + # The record of the artifact that is not there any more is dropped, and said + assert len(r['dropped_records']) == 1 and uid in r['dropped_records'][0]['ref'] and 'aaaaaaaaaaaaaaaa' in r['dropped_records'][0]['now'] + assert uids(find(cm, 'slot')) == ['aaaaaaaaaaaaaaaa'] + assert find(cm, uid)['return'] == 16 + assert uid not in index_data(tmp_path)['uids'] + assert index_data(tmp_path)['lowercase_aliases']['slot'] == ['aaaaaaaaaaaaaaaa'] + assert uids(find(cm, 'other')) == [other] + # The new artifact's folder is untouched + assert (base / 'slot' / '_cmeta.yaml').is_file() + + +def test_a_folder_moved_and_its_old_name_taken(cm, tmp_path): + # The artifact was renamed by hand and another artifact took its old name + uid1 = make(cm, 'taken', yaml=True, payload='one') + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'taken').rename(base / 'moved-away') + (base / 'taken').mkdir() + (base / 'taken' / '_cmeta.yaml').write_text('artifact: taken,bbbbbbbbbbbbbbbb\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'taken') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([uid1, 'bbbbbbbbbbbbbbbb']) + assert r['dropped_records'] == [] + assert r['dropped_aliases'] == {uid1: ['taken']} + assert uids(find(cm, 'taken')) == ['bbbbbbbbbbbbbbbb'] + assert uids(find(cm, 'moved-away')) == [uid1] + assert record(cm, uid1)['cmeta']['payload'] == 'one' + + +def test_an_unknown_artifact_is_not_found(cm, tmp_path): + make(cm, 'exists', yaml=True) + r = reindex(cm, 'does-not-exist') + assert r['return'] == 16 and 'not found' in r['error'] + r = reindex(cm, 'ffffffffffffffff') + assert r['return'] == 16 + r = reindex(cm, 'no-such-repo:exists') + assert r['return'] == 16 + + +def test_a_hand_made_folder_without_a_valid_uid_is_not_indexed(cm, tmp_path): + make(cm, 'exists', yaml=True) + before = index_bytes(tmp_path) + bad = tmp_path / 'repos' / 'local' / 'log' / 'bad-uid' + bad.mkdir() + (bad / '_cmeta.yaml').write_text('artifact: bad-uid\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'bad-uid') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + no_key = tmp_path / 'repos' / 'local' / 'log' / 'no-key' + no_key.mkdir() + (no_key / '_cmeta.yaml').write_text('tags:\n- x\n', encoding='utf-8') + r = reindex(cm, 'no-key') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + +def test_a_folder_of_another_category_is_not_taken(cm, tmp_path): + # A folder copied into the wrong category folder: its meta names another category + make(cm, 'exists', yaml=True) + before = index_bytes(tmp_path) + wrong = tmp_path / 'repos' / 'local' / 'log' / 'an-asset' + wrong.mkdir() + (wrong / '_cmeta.yaml').write_text('artifact: an-asset,bbbbbbbbbbbbbbbb\ncategory: asset,e0982f19227743c8\n', encoding='utf-8') + r = reindex(cm, 'an-asset') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + +def test_other_records_are_untouched(cm, tmp_path): + uid_a = make(cm, 'art-a', yaml=True, tags=['a']) + uid_b = make(cm, 'art-b', yaml=True, tags=['b']) + uid_c = make(cm, 'art-c', tags=['c']) + data = index_data(tmp_path) + + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'art-b', 'owner: me\n') + r = reindex(cm, 'art-b') + assert r['return'] == 0, r.get('error') + + after = index_data(tmp_path) + for uid in (uid_a, uid_c): + assert after['uids'][uid] == data['uids'][uid] + assert after['uids'][uid_b]['cmeta']['owner'] == 'me' + assert after['lowercase_aliases'] == data['lowercase_aliases'] + + +########################################################################################## +# Selections: wildcards, tags, a repository, everything + +def test_wildcards_tags_and_a_repository(cm, tmp_path): + repo = make_repo(tmp_path, 'second', '2222222222222222') + plug(cm, repo) + base_local = tmp_path / 'repos' / 'local' / 'log' + base_second = repo / 'log' + + a1 = make(cm, 'wild-1', yaml=True, tags=['odd']) + a2 = make(cm, 'wild-2', yaml=True, tags=['even']) + s1 = make(cm, 'wild-1', repo='second', yaml=True, tags=['odd', 'second']) + other = make(cm, 'other', yaml=True) + for folder in (base_local / 'wild-1', base_local / 'wild-2', base_second / 'wild-1', base_local / 'other'): + hand_edit(folder, 'owner: me\n') + + r = reindex(cm, 'wild-*') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([a1, a2, s1]) + assert 'owner' not in record(cm, other)['cmeta'] + + r = reindex(cm, 'other', tags='nope') + assert r['return'] == 16 + + hand_edit(base_local / 'wild-1', 'owner2: me\n') + hand_edit(base_second / 'wild-1', 'owner2: me\n') + r = reindex(cm, 'second:wild-1') # the one in the second repository only + assert r['return'] == 0, r.get('error') + assert uids(r) == [s1] + assert 'owner2' in record(cm, s1)['cmeta'] and 'owner2' not in record(cm, a1)['cmeta'] + # The same alias in two repositories: both UIDs stay under the alias + assert sorted(index_data(tmp_path)['lowercase_aliases']['wild-1']) == sorted([a1, s1]) + + r = reindex(cm, 'wild-*', tags='odd,-second') + assert r['return'] == 0, r.get('error') + assert uids(r) == [a1] + + r = reindex(cm, 'local:') # everything of the category in one repository + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([a1, a2, other]) + + +def test_everything_of_the_category_takes_the_folders_made_by_hand(cm, tmp_path): + make(cm, 'made', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + for i in range(3): + folder = base / f'hand-{i}' + folder.mkdir() + (folder / '_cmeta.yaml').write_text(f'artifact: hand-{i},{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm) + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['hand-0', 'hand-1', 'hand-2', 'made'] + assert uids(find(cm, 'hand-1')) == ['1111111111111111'] + + +def test_a_folder_named_by_its_uid(cm, tmp_path): + base = tmp_path / 'repos' / 'local' / 'log' + make(cm, 'seed', yaml=True) # the index exists + folder = base / 'cccccccccccccccc' + folder.mkdir() + (folder / '_cmeta.yaml').write_text('artifact: cccccccccccccccc\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'cccccccccccccccc') + assert r['return'] == 0, r.get('error') + parts = record(cm, 'cccccccccccccccc')['cmeta_ref_parts'] + assert parts['artifact_uid'] == 'cccccccccccccccc' + assert parts.get('artifact_alias') == 'cccccccccccccccc' # the folder's name, as the full reindex records it + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, 'cccccccccccccccc')['cmeta_ref_parts'] == parts + + +########################################################################################## +# Categories with special layouts + +def make_category_repo(tmp_path, name, repo_uid, category_meta_extra): + """A repository with a category of its own (`shard` or `noidx`) and nothing else.""" + repo = tmp_path / 'extra' / name + category = repo / 'category' / name + (category / 'api').mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + (category / '_cmeta.yaml').write_text(f'artifact: {name},{CATEGORY_UID}\ncategory: category,dd9ea50e7f76467f\n' + 'last_api_version: 1\nbase_category_default_api_versions:\n "1": 1\n' + category_meta_extra, + encoding='utf-8') + (category / 'api' / 'v1.py').write_text('from cmeta.category import InitCategory\n\nclass Category(InitCategory):\n' + ' def __init__(self, *args, **kwargs):\n' + ' super().__init__(*args, module_file_path = __file__, **kwargs)\n', encoding='utf-8') + return repo + + +def test_a_sharded_category(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', 'sharding_slices:\n- 2\n') + plug(cm, repo) + + uid = make(cm, 'abcdef', category='shard', repo='shard', yaml=True) + r = find(cm, 'abcdef', category='shard') + folder = r['artifacts'][0]['path'] + assert r['artifacts'][0]['sharding_slices_num'] == 1 + assert os.path.basename(os.path.dirname(folder)) == 'ab' # the shard + + hand_edit(pytest.importorskip('pathlib').Path(folder), 'owner: me\n') + r = reindex(cm, 'abcdef', category='shard') + assert r['return'] == 0, r.get('error') + assert r['artifacts'][0]['sharding_slices_num'] == 1 + after_reindex = record(cm, uid, category='shard') + assert after_reindex['cmeta']['owner'] == 'me' and after_reindex['sharding_slices_num'] == 1 + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, uid, category='shard') == after_reindex + + +def test_a_no_index_category_is_refused(cm, tmp_path): + repo = make_category_repo(tmp_path, 'noidx', '4444444444444444', 'no_index: true\n') + plug(cm, repo) + scanned = make(cm, 'scanned', category='noidx', repo='noidx', yaml=True) + make(cm, 'other', category='noidx', repo='noidx', yaml=True) + r = reindex(cm, 'scanned', category='noidx') + assert r['return'] == 1 and 'no_index' in r['error'] + + # The scan that finds the artifacts of such a category honours a wildcard (it used to return every folder) + assert uids(find(cm, 'sc*', category='noidx')) == [scanned] + assert uids(find(cm, 'SCAN?ED', category='noidx')) == [scanned] + assert find(cm, 'nothing-*', category='noidx')['return'] == 16 + assert len(find(cm, '*', category='noidx')['artifacts']) == 2 + + +def test_the_record_of_a_category_itself(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', '') + plug(cm, repo) + assert 'extra' not in record(cm, 'shard', category='category')['cmeta'] + + f = repo / 'category' / 'shard' / '_cmeta.yaml' + f.write_text(f.read_text(encoding='utf-8') + 'extra: 1\n', encoding='utf-8') + before = file_state(f) + + r = reindex(cm, 'shard', category='category') + assert r['return'] == 0, r.get('error') + assert record(cm, 'shard', category='category')['cmeta']['extra'] == 1 + assert file_state(f) == before + after_reindex = record(cm, 'shard', category='category') + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, 'shard', category='category') == after_reindex + + +def test_a_migration_stub_is_reindexed_as_itself(cm, tmp_path): + uid = make(cm, 'old-name', yaml=True) + r = access(cm, category='log', command='migrate', arg1='old-name', arg2='new-name') + assert r['return'] == 0, r.get('error') + stub_uid = r['stub_meta']['artifact'] + + r = reindex(cm, 'old-name') + assert r['return'] == 0, r.get('error') + assert uids(r) == [stub_uid] # the stub, not the artifact it points to + assert record(cm, stub_uid)['cmeta']['migrated_to'] == f'new-name,{uid}' + assert uids(find(cm, 'old-name')) == [uid] # the old alias alone still follows the stub + + +########################################################################################## +# The commands that exist keep their answers + +def test_index_and_update_are_unchanged(cm, tmp_path): + uid = make(cm, 'kept', yaml=True) + # "index" still refuses an artifact that is indexed + r = access(cm, category='log', command='index', arg1='local:kept') + assert r['return'] == 8 + # an empty "update" still refreshes the record and rewrites the file (its timestamp) + f = meta_file(tmp_path / 'repos' / 'local' / 'log' / 'kept') + before = file_state(f) + r = access(cm, category='log', command='update', arg1='kept') + assert r['return'] == 0, r.get('error') + assert file_state(f) != before + assert 'last_update_timestamp' in record(cm, uid)['cmeta'] + + +def test_the_cli_route(cm, tmp_path): + uid = make(cm, 'via-cli', yaml=True) + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'via-cli', 'owner: me\n') + + env = {k: v for k, v in os.environ.items() if not k.startswith('CMETA_') and k not in ('VIRTUAL_ENV', 'CONDA_PREFIX')} + env['CMETA_HOME'] = str(tmp_path) + env['PYTHONIOENCODING'] = 'utf-8' + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'via-cli'], capture_output=True, text=True, env=env) + assert r.returncode == 0, r.stdout + r.stderr + assert f'Reindexed log "via-cli" ({uid})' in r.stdout + assert record(cm, uid)['cmeta']['owner'] == 'me' + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'nothing-here'], capture_output=True, text=True, env=env) + assert r.returncode == 16, r.stdout + r.stderr + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', '--help'], capture_output=True, text=True, env=env) + assert r.returncode == 0 and 'ignore_errors' in r.stdout, r.stdout + r.stderr diff --git a/tests/internal_repo_tests/test_integration_repo_reindex.py b/tests/internal_repo_tests/test_integration_repo_reindex.py new file mode 100644 index 0000000..cd31dd7 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_reindex.py @@ -0,0 +1,242 @@ +""" +Integration tests for `cx repo reindex `: one registered repository is indexed again +(its `_cmr.yaml` re-read, its artifacts rescanned) while the records of the other repositories +stay as they are; a repository whose folder is not there is refused. Also the incremental index +that pull, plug and unzip use: the records of a category of which the repository holds no +artifact any more are dropped. + +Each test runs against a fresh in tmp_path so it does not touch the user's real +cMeta state. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import pickle +import shutil + +import pytest + +from cmeta import CMeta + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid): + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def make(cm, alias, repo, category='log', **meta): + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=True) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def hand_made(repo, alias, uid, category='log', category_ref='log,487a7639093a4685'): + folder = repo / category / alias + folder.mkdir(parents=True) + (folder / '_cmeta.yaml').write_text(f'artifact: {alias},{uid}\ncategory: {category_ref}\n', encoding='utf-8') + return folder + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def repo_record(cm, ref): + r = access(cm, category='repo', command='find', arg1=ref) + assert r['return'] == 0, r.get('error') + assert len(r['artifacts']) == 1 + return r['artifacts'][0] + + +def index_data(tmp_path, category): + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def repos_json(tmp_path): + return json.loads((tmp_path / 'repos.json').read_text(encoding='utf-8')) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def test_repo_reindex_rereads_the_descriptor_and_rescans_the_artifacts(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + local_uid = make(cm, 'in-local', 'local', tags=['local']) + kept = make(cm, 'kept', 'mine') + removed = make(cm, 'removed', 'mine') + local_before = index_data(tmp_path, 'log')['uids'][local_uid] + + # Changes made behind cMeta's back: the descriptor, a new folder, a folder removed + f = repo / '_cmr.yaml' + f.write_text(f.read_text(encoding='utf-8') + 'owner: me\n', encoding='utf-8') + hand_made(repo, 'added', 'aaaaaaaaaaaaaaaa') + shutil.rmtree(repo / 'log' / 'removed') + + assert 'owner' not in repo_record(cm, 'mine')['cmeta'] + assert find(cm, 'added')['return'] == 16 + assert uids(find(cm, 'removed')) == [removed] + + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + assert [a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']] == ['mine'] + assert [same(p) for p in r['paths']] == [same(repo)] + + assert repo_record(cm, 'mine')['cmeta']['owner'] == 'me' + assert uids(find(cm, 'added')) == ['aaaaaaaaaaaaaaaa'] + assert uids(find(cm, 'kept')) == [kept] + assert find(cm, 'removed')['return'] == 16 + assert 'removed' not in index_data(tmp_path, 'log')['lowercase_aliases'] + # The other repository's records are as they were + assert index_data(tmp_path, 'log')['uids'][local_uid] == local_before + assert repos_json(tmp_path) == repos_json(tmp_path) # still readable + assert same(repo) in [same(p) for p in repos_json(tmp_path)] + + +def test_repo_reindex_by_uid_path_and_dot(cm, tmp_path, monkeypatch): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + make(cm, 'seed', 'mine') + + hand_made(repo, 'by-uid', 'aaaaaaaaaaaaaaaa') + r = access(cm, category='repo', command='reindex', arg1='1111111111111111') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-uid')) == ['aaaaaaaaaaaaaaaa'] + + hand_made(repo, 'by-path', 'bbbbbbbbbbbbbbbb') + r = access(cm, category='repo', command='reindex', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-path')) == ['bbbbbbbbbbbbbbbb'] + + hand_made(repo, 'by-sub-path', 'cccccccccccccccc') + r = access(cm, category='repo', command='reindex', arg1=str(repo / 'log')) # a folder inside the repository + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-sub-path')) == ['cccccccccccccccc'] + + hand_made(repo, 'by-dot', 'dddddddddddddddd') + monkeypatch.chdir(repo / 'log' / 'by-dot') + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-dot')) == ['dddddddddddddddd'] + + monkeypatch.chdir(tmp_path) + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 16 and 'plug' in r['error'] + + +def test_repo_reindex_refuses_what_it_cannot_do_safely(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'seed', 'mine') + before = index_data(tmp_path, 'log') + repos_before = repos_json(tmp_path) + + r = access(cm, category='repo', command='reindex') + assert r['return'] == 1 and 'cx --reindex' in r['error'] + + r = access(cm, category='repo', command='reindex', arg1='no-such-repo') + assert r['return'] == 16 + + # The repository's folder is not there (a detached drive): refused, nothing changed, the + # registration and the records stay + moved = tmp_path / 'extra' / 'mine-away' + (tmp_path / 'extra' / 'mine').rename(moved) + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 1 and 'not there' in r['error'], r + assert index_data(tmp_path, 'log') == before + assert repos_json(tmp_path) == repos_before + assert uids(find(cm, 'seed')) == [uid] + + moved.rename(tmp_path / 'extra' / 'mine') + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'seed')) == [uid] + + +def test_repo_reindex_after_the_descriptor_renamed_the_repository(cm, tmp_path): + repo = make_repo(tmp_path, 'old-repo', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'art', 'old-repo') + + (repo / '_cmr.yaml').write_text('artifact: new-repo,1111111111111111\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + + r = access(cm, category='repo', command='reindex', arg1='1111111111111111') + assert r['return'] == 0, r.get('error') + assert repo_record(cm, 'new-repo')['cmeta_ref_parts']['artifact_uid'] == '1111111111111111' + assert access(cm, category='repo', command='find', arg1='old-repo')['return'] == 16 + assert find(cm, 'art')['artifacts'][0]['cmeta_ref_parts']['repo_alias'] == 'new-repo' + assert uids(find(cm, 'new-repo:art')) == [uid] + + +def test_repo_reindex_with_a_wildcard_and_several_repositories(cm, tmp_path): + repo_a = make_repo(tmp_path, 'mine-a', '1111111111111111') + repo_b = make_repo(tmp_path, 'mine-b', '2222222222222222') + other = make_repo(tmp_path, 'other', '3333333333333333') + for repo in (repo_a, repo_b, other): + plug(cm, repo) + hand_made(repo_a, 'in-a', 'aaaaaaaaaaaaaaaa') + hand_made(repo_b, 'in-b', 'bbbbbbbbbbbbbbbb') + hand_made(other, 'in-other', 'cccccccccccccccc') + + r = access(cm, category='repo', command='reindex', arg1='mine-*') + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['mine-a', 'mine-b'] + assert uids(find(cm, 'in-a')) == ['aaaaaaaaaaaaaaaa'] and uids(find(cm, 'in-b')) == ['bbbbbbbbbbbbbbbb'] + assert find(cm, 'in-other')['return'] == 16 # not asked for, not scanned + + +def test_the_incremental_index_drops_the_records_of_an_emptied_category(cm, tmp_path): + # The index that pull, plug and reindex share: a repository whose last artifact of a category + # was removed by hand loses that record too + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + local_uid = make(cm, 'in-local', 'local') + only = make(cm, 'only-one', 'mine') + asset = make(cm, 'an-asset', 'mine', category='asset') + + shutil.rmtree(repo / 'log' / 'only-one') + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + + assert find(cm, 'only-one')['return'] == 16 + assert only not in index_data(tmp_path, 'log')['uids'] + assert uids(find(cm, 'in-local')) == [local_uid] + assert uids(find(cm, 'an-asset', category='asset')) == [asset] + + # The same through plug of a second copy: plugging indexes incrementally as well + other = make_repo(tmp_path, 'other', '2222222222222222') + hand_made(other, 'in-other', 'eeeeeeeeeeeeeeee') + plug(cm, other) + assert uids(find(cm, 'in-other')) == ['eeeeeeeeeeeeeeee'] + assert uids(find(cm, 'in-local')) == [local_uid] From 66c5eb13c83104df594a38ba56fed02fb85d9ed0 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Fri, 9 Oct 2026 09:42:39 +0200 Subject: [PATCH 2/3] The engine's own PathLock replaces filelock: no two holders of one lock, lock files still removed after use (0.33.1.1) - cmeta.utils.files.PathLock: a byte-range lock (msvcrt) on Windows; flock, then POSIX record locks where flock is refused, then a soft lock file where both are refused, on POSIX; the file it holds is checked to be the file at the path after acquiring; the file is removed before the lock is dropped; the threads of one process are serialized per path; stale soft locks are cleared; a busy lock is polled every 1-25 ms; the timeouts, the lock file names and the functions that use them are unchanged - filelock is no longer a dependency (pyproject.toml, uv.lock, AGENTS.md, CLAUDE.md) - tests/core_tests/test_utils_files_path_lock.py: 21 tests (processes, threads, the engine's read-modify-write cycle, a killed holder, the removed-file race, the soft lock and its stale files, the record-lock fallback, timeouts); test_several_processes_reindex_at_once is no longer expected to fail on Linux - tests/benchmarks/benchmark_engine.py and its smoke test: the engine's hot paths on a home of N artifacts, --json and --compare - docs/async-and-concurrency.md: how the lock works on each platform - the new tests and the benchmark start their worker processes with an explicit null input: the inherited standard handles can be unusable under pytest's capture on Windows Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- AGENTS.md | 3 +- CHANGELOG.md | 26 + CLAUDE.md | 3 +- cmeta/utils/files.py | 565 +++++++++++++++-- docs/async-and-concurrency.md | 42 +- pyproject.toml | 1 - tests/benchmarks/benchmark_engine.py | 567 ++++++++++++++++++ tests/benchmarks/test_benchmark_engine.py | 68 +++ .../core_tests/test_utils_files_path_lock.py | 520 ++++++++++++++++ .../test_integration_reindex_artifact.py | 18 +- uv.lock | 11 - 11 files changed, 1741 insertions(+), 83 deletions(-) create mode 100644 tests/benchmarks/benchmark_engine.py create mode 100644 tests/benchmarks/test_benchmark_engine.py create mode 100644 tests/core_tests/test_utils_files_path_lock.py diff --git a/AGENTS.md b/AGENTS.md index e1547d3..e0cc7c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,8 @@ The commands above are cross-platform truth. Any Windows `.bat` wrappers around them are the author's local workflow and are not tracked in this repository. Runtime deps are intentionally minimal: pyyaml, requests, setuptools, wheel, -tabulate, tqdm, filelock, packaging, psutil. +tabulate, tqdm, packaging, psutil. (File locks are the engine's own +`cmeta.utils.files.PathLock` since 0.34.0; `filelock` is no longer used.) ### Useful environment variables `CMETA_HOME`, `CMETA_DEBUG=1`, `CMETA_LOG=DEBUG|INFO`, `CMETA_LOG_FILE=`, diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a61051..6a599df 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,32 @@ All notable changes to cMeta are documented here, newest first. ## 0.33.1.1 (in development) +- **The engine's own path lock replaces the `filelock` library: two processes can no longer hold one + lock at once, and lock files are still removed after use.** The previous lock removed its file after + releasing it; with `flock` on Linux and macOS a process waiting on the removed file and a newcomer + that created a new one then both acquired, and one of two concurrent index writes was lost (seen with + six `cx log reindex` at once on WSL2 and in a container). `cmeta.utils.files.PathLock` removes the + file BEFORE releasing (after closing it on Windows, where an open file cannot be removed) and checks + after acquiring that the file it holds is still the file at the path, trying again otherwise. It takes + a byte-range lock (`msvcrt.locking`) on Windows, `fcntl.flock` on POSIX, POSIX record locks + (`fcntl.lockf`) where the file system refuses `flock` (NFS and some network mounts), and a soft lock + file (`pid host time`, created exclusively; a stale one - its process dead on this host, or older than + ten minutes when left by another host - is removed by the waiter that finds it) where the file system + refuses both (some FUSE, 9p and SMB mounts). The threads of one process are serialized on the path + first, so the lock is safe for threads as well as processes in every mode. A busy lock is polled every + 1 to 25 ms instead of every 50 ms. The timeouts, the lock file names and the functions that use them + (`safe_read_file(lock=True)`, `safe_write_file`, `safe_delete_directory`, `lock_path` / `unlock_path`) + are unchanged; `filelock` is no longer a dependency. Tests: `tests/core_tests/test_utils_files_path_lock.py` + (one holder at a time, the file removed on release, eight processes and sixteen threads through one + counter, the engine's read-modify-write cycle under contention, a killed holder, the removed-file race + replayed deterministically, a lock file that is not its own left alone, the soft lock and its stale + files, the POSIX record-lock fallback, timeouts, `safe_delete_directory` and `safe_write_file`); + `test_several_processes_reindex_at_once` passes on every platform (it was expected to fail on Linux). +- **`tests/benchmarks/benchmark_engine.py`: a benchmark of the engine's hot paths** - the index reads + behind `find`, the locked writes behind `create`, `update`, `delete` and `reindex`, the full reindex, the + reload after a change, the primitives (stat, lock, record copy, pickle, atomic write), concurrent creates + and the `cx` command line - on a home of its own with N hand-made artifacts; `--json` writes the numbers + and `--compare ` prints the ratio against an earlier run. A smoke test runs it in the suite. - **`cx reindex []`: the index record of one artifact (or of several) is refreshed from its folder, and no file is changed.** The meta file is read from the artifact's folder on disk and the record in the category's index is rewritten from it: the alias is the folder's name, every diff --git a/CLAUDE.md b/CLAUDE.md index 7144df0..3d78007 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,7 +48,8 @@ separate content repos (e.g. `cmeta-aops`). ## Conventions worth repeating - Runtime deps are intentionally minimal (pyyaml, requests, setuptools, wheel, - tabulate, tqdm, filelock, packaging, psutil). Justify any new one. + tabulate, tqdm, packaging, psutil). Justify any new one. File locks are the + engine's own `cmeta.utils.files.PathLock` (no `filelock`). - Preserve module docstring/copyright headers — never strip or rewrite them. New files get the standard header; new artifacts/categories record `authors` and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and diff --git a/cmeta/utils/files.py b/cmeta/utils/files.py index 20ae9aa..8f00e6f 100644 --- a/cmeta/utils/files.py +++ b/cmeta/utils/files.py @@ -18,6 +18,10 @@ import hashlib import fnmatch import re +import errno +import logging +import random +import threading from .common import _error @@ -27,11 +31,24 @@ yaml = None # YAML is optional try: - from filelock import FileLock + import fcntl # POSIX: flock and POSIX record locks except ImportError: - raise ImportError("filelock library is required. Install with: pip install filelock") + fcntl = None + +try: + import msvcrt # Windows: byte-range locks +except ImportError: + msvcrt = None LOCK_SUFFIX = ".lock" +LOCK_POLL_START = 0.001 # the first wait when a lock is busy; it doubles up to LOCK_POLL_MAX +LOCK_POLL_MAX = 0.025 +LOCK_SOFT_STALE_SECONDS = 600 # a soft lock file older than this, whose holder is on another host, is stale +LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file older than this is a leftover (a holder writes its content at once) +LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock file of a dead pid younger than this is a release in flight, not a crash +LOCK_SOFT_CHECK_SECONDS = 0.25 # how often a waiter on a soft lock reads the file to check for a stale holder +LOCK_SOFT_REMOVE_SECONDS = 2.0 # how long a releasing soft lock retries the removal of its file (Windows: a reader may hold it) +LOCK_IDENTITY_RETRIES = 50 # consecutive identity mismatches after which a path is locked with the soft lock ERROR_CODE_FILE_NOT_FOUND = 16 RETRY_DELAY = 0.1 RETRY_NOT_FOUND_FILE = 10 @@ -108,6 +125,488 @@ def is_path_within( return common == base +########################################################################################## +# The path lock +# +# One exclusive lock per path for processes and threads alike, through the sidecar file `.lock`: +# the lock file is created for the operation and removed by the process that releases it, so a repository +# never keeps lock files (a crash leaves at most the one of the operation in flight, which the next holder +# reuses and removes - every lock dies with its process). +# +# Why not a library: a lock file that is removed after release is only correct when the holder removes +# it BEFORE releasing and every newcomer checks, after acquiring, that the file it holds is still the +# file at the path (otherwise a waiter blocked on the removed file and a newcomer that created a new +# one both hold "the lock" - the lost updates seen on Linux with the previous library + removal). That +# needs the lock's own descriptor, which libraries do not expose. +# +# The ladder, per platform: +# Windows: msvcrt.locking (a byte-range lock, per handle, mandatory) on the lock file; an open file +# cannot be removed there, so the identity check is not needed - a newcomer that finds the +# file "being deleted" (PermissionError) simply tries again. +# POSIX: fcntl.flock (per open file description, so threads of one process exclude each other too), +# then fcntl.lockf (POSIX record locks: NFS and some network mounts refuse flock), each with +# the identity check (fstat of the descriptor vs stat of the path, inode and device); +# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time") where the file +# system refuses both OS locks (some FUSE, 9p and SMB mounts) or the identity check keeps +# failing - a stale soft lock (its pid dead on this host, or older than LOCK_SOFT_STALE_SECONDS +# when left by another host) is renamed away and removed by the waiter that finds it. +# In every mode the threads of one process are serialized on the path first (a threading.Lock per path), +# which POSIX record locks and soft locks need (they are per process), and the busy wait polls with a +# delay that doubles from LOCK_POLL_START to LOCK_POLL_MAX until the timeout. + +_LOCK_BUSY_ERRNOS = tuple(e for e in (getattr(errno, name, None) for name in ('EWOULDBLOCK', 'EAGAIN', 'EACCES', 'EDEADLK', 'EDEADLOCK')) if e is not None) +_LOCK_UNSUPPORTED_ERRNOS = tuple(e for e in (getattr(errno, name, None) for name in ('ENOLCK', 'ENOSYS', 'EOPNOTSUPP', 'ENOTSUP', 'EINVAL', 'ENOTTY', 'EPERM', 'EBADF')) if e is not None) + +_lock_registry = {} # normalized lock file path -> threading.Lock (the threads of this process) +_lock_registry_guard = threading.Lock() +_lock_soft_paths = set() # lock files locked with the soft lock from now on (the OS lock was refused) +_lock_no_flock_dirs = set() # folders where flock was refused (POSIX record locks are tried first there) +_lock_host = None + + +def _lock_thread_lock(lock_file): + key = os.path.normcase(os.path.abspath(lock_file)) + with _lock_registry_guard: + lock = _lock_registry.get(key) + if lock is None: + lock = threading.Lock() + _lock_registry[key] = lock + return lock + + +def _lock_host_name(): + global _lock_host + if _lock_host is None: + try: + import platform + _lock_host = platform.node() or 'unknown-host' + except Exception: + _lock_host = 'unknown-host' + return _lock_host + + +def _lock_pid_alive(pid): + try: + import psutil + return psutil.pid_exists(pid) + except Exception: + return True # unknown: never treat it as dead + + +class _LockUnsupported(Exception): + pass + + +class PathLock: + """ + An exclusive lock on a path for processes and threads, through the sidecar file `.lock` + (see the notes above). Use `acquire(timeout)` / `release()` or the context manager; `is_locked` + says whether this object holds the lock; `mode` is how it was taken ('windows', 'flock', 'lockf' + or 'soft'); `identity_retries` counts the times a replaced lock file was detected and retried. + """ + + def __init__( + self, + lock_file: str, # The lock file (normally `.lock`, see _get_lockfile_path). + logger = None, # Optional logger for debug and warning messages. + ): + self.lock_file = lock_file + self.logger = logger if logger is not None else logging.getLogger('cmeta.utils.files') + self.is_locked = False + self.mode = None + self.identity_retries = 0 + self._fd = None + self._ident = None + self._thread_lock = None + self._mismatches = 0 + self._last_error = None + self._next_stale_check = 0.0 + + def __enter__(self): + self.acquire() + return self + + def __exit__(self, exc_type, exc, tb): + self.release() + + def __del__(self): + try: + if self.is_locked: + self.release() + except Exception: + pass + + def acquire( + self, + timeout: float = 3, # Seconds to wait for the lock (0: one try). + ): + """Take the lock or raise TimeoutError; returns self.""" + timeout = max(float(timeout), 0.0) + thread_lock = _lock_thread_lock(self.lock_file) + if not thread_lock.acquire(timeout=timeout): + raise TimeoutError(f"the lock {self.lock_file} is held by another thread of this process (waited {timeout} s)") + + try: + deadline = time.monotonic() + timeout + delay = LOCK_POLL_START + while True: + if self._try(): + self._thread_lock = thread_lock + self.is_locked = True + self.logger.debug(f"utils.files.PathLock - acquired {self.lock_file} ({self.mode})") + return self + + remaining = deadline - time.monotonic() + if remaining <= 0: + why = f" ({self._last_error})" if self._last_error is not None else "" + raise TimeoutError(f"the lock {self.lock_file} is held by another process (waited {timeout} s){why}") + + time.sleep(min(delay, remaining)) + delay = min(delay * 2, LOCK_POLL_MAX) + + except BaseException: + thread_lock.release() + raise + + def release(self): + """Release the lock and remove its file (when the file at the path is still the one this lock holds).""" + if not self.is_locked: + return + + try: + if self.mode == 'windows': + try: + msvcrt.locking(self._fd, msvcrt.LK_UNLCK, 1) + except OSError: + pass + os.close(self._fd) + self._unlink_own_file() + + elif self.mode in ('flock', 'lockf'): + self._unlink_own_file() + try: + self._posix_unlock(self._fd, self.mode) + except OSError: + pass + os.close(self._fd) + + elif self.mode == 'soft': + self._remove_soft_file() + + finally: + self._fd = None + self._ident = None + self.is_locked = False + self.mode = None + thread_lock = self._thread_lock + self._thread_lock = None + if thread_lock is not None: + thread_lock.release() + + # ---- one attempt ---------------------------------------------------------------------- + + def _try(self): + if self.lock_file in _lock_soft_paths: + return self._try_soft() + if msvcrt is not None: + return self._try_windows() + if fcntl is not None: + return self._try_posix() + return self._try_soft() + + def _folder_is_there(self): + return os.path.isdir(os.path.dirname(self.lock_file) or '.') + + def _open_failed_for_the_moment(self, e): + """An open of the lock file that failed for a moment: the file being removed by the process that + just released it (Windows: PermissionError; 9p and other network mounts: ENOENT with the folder + there) or held by a scanner. A missing folder is a real error.""" + if isinstance(e, PermissionError): + self._last_error = e + return True + if isinstance(e, FileNotFoundError) and self._folder_is_there(): + self._last_error = e + return True + return False + + def _try_windows(self): + try: + fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT, 0o644) + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + except OSError as e: + os.close(fd) + if e.errno in _LOCK_BUSY_ERRNOS: + self._last_error = e + return False + self._fall_back_to_soft(e) + return self._try_soft() + + try: + self._ident = self._identity(os.fstat(fd)) + except OSError: + self._ident = None + self._fd = fd + self.mode = 'windows' + return True + + def _try_posix(self): + try: + fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT | getattr(os, 'O_CLOEXEC', 0), 0o644) + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + kind = self._posix_lock(fd) + except _LockUnsupported as e: + os.close(fd) + self._fall_back_to_soft(e) + return self._try_soft() + except BaseException: + os.close(fd) + raise + + if kind is None: + os.close(fd) + return False + + # The file this descriptor holds must still be the file at the path: a holder that removed + # the file and released it leaves a waiter with a lock on a file that is not there any more + try: + path_ident = self._identity(os.stat(self.lock_file)) + except OSError: + path_ident = None + fd_ident = self._identity(os.fstat(fd)) + + if fd_ident is not None and (path_ident is None or path_ident != fd_ident): + try: + self._posix_unlock(fd, kind) + except OSError: + pass + os.close(fd) + self.identity_retries += 1 + self._mismatches += 1 + if self._mismatches >= LOCK_IDENTITY_RETRIES: + self._fall_back_to_soft(f"the identity of the lock file could not be confirmed {self._mismatches} times") + return self._try_soft() + return False + + self._mismatches = 0 + self._fd = fd + self._ident = fd_ident + self.mode = kind + return True + + def _posix_lock(self, fd): + """flock, else POSIX record lock; 'flock' | 'lockf' when taken, None when busy, _LockUnsupported when refused.""" + folder = os.path.dirname(self.lock_file) + + if folder not in _lock_no_flock_dirs: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + return 'flock' + except OSError as e: + if e.errno in _LOCK_BUSY_ERRNOS or e.errno == errno.EINTR: + self._last_error = e + return None + if e.errno not in _LOCK_UNSUPPORTED_ERRNOS: + raise + _lock_no_flock_dirs.add(folder) + self.logger.info(f"utils.files.PathLock - flock is not supported for {self.lock_file} ({e}): POSIX record locks are used in that folder") + + try: + fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + return 'lockf' + except OSError as e: + if e.errno in _LOCK_BUSY_ERRNOS or e.errno == errno.EINTR: + self._last_error = e + return None + if e.errno in _LOCK_UNSUPPORTED_ERRNOS: + raise _LockUnsupported(str(e)) + raise + + @staticmethod + def _posix_unlock(fd, kind): + if kind == 'flock': + fcntl.flock(fd, fcntl.LOCK_UN) + else: + fcntl.lockf(fd, fcntl.LOCK_UN) + + def _try_soft(self): + try: + fd = os.open(self.lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) + except FileExistsError as e: + self._last_error = e + now = time.monotonic() + if now >= self._next_stale_check: + self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS + self._clear_stale_soft_lock() + return False + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + os.write(fd, (f"{os.getpid()} {_lock_host_name()} {time.time():.3f}\n").encode('utf-8', 'replace')) + finally: + os.close(fd) + + try: + self._ident = self._identity(os.stat(self.lock_file)) + except OSError: + self._ident = None + self.mode = 'soft' + return True + + def _clear_stale_soft_lock(self): + """A soft lock file whose holder is dead (this host; the file at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS + old, so that a holder that just released and exited is not mistaken for a crash), that was marked + released by a holder that could not remove it, that is empty and old (a leftover) or too old (another + host) is renamed away and removed - by one waiter only (the rename is atomic), and only when the file + at the path is still the file that was judged (a fresh lock file of a live holder moved by mistake in + the instant between the check and the rename is put back).""" + try: + fd = os.open(self.lock_file, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + except OSError: + return + try: + st = os.fstat(fd) + words = os.read(fd, 256).decode('utf-8', 'replace').split() if st.st_size > 0 else [] + except OSError: + return + finally: + os.close(fd) + + ident = self._identity(st) + age = time.time() - st.st_mtime + pid = int(words[0]) if len(words) > 0 and words[0].isdigit() else None + host = words[1] if len(words) > 1 else None + + if not words: + stale = age > LOCK_SOFT_EMPTY_STALE_SECONDS + elif words[0] == 'released': + stale = True + elif pid is not None and host == _lock_host_name(): + stale = age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) + else: + stale = age > LOCK_SOFT_STALE_SECONDS + + if not stale: + return + + try: + if ident is not None and self._identity(os.stat(self.lock_file)) != ident: + return # the file changed under us: judge it again next time + except OSError: + return + + moved = f"{self.lock_file}.stale-{os.getpid()}-{random.getrandbits(32):08x}" + try: + os.rename(self.lock_file, moved) # atomic: only one waiter gets to remove it + except OSError: + return + + if ident is not None: + try: + moved_ident = self._identity(os.stat(moved)) + except OSError: + return + if moved_ident != ident: + # Not the file judged stale but a fresh one made in between: back where it was, without + # clobbering (a hard link fails where a file exists already), then the extra name goes + try: + os.link(moved, self.lock_file) + except OSError: + try: + if not os.path.exists(self.lock_file): + os.rename(moved, self.lock_file) + except OSError: + pass + return + try: + os.unlink(moved) + except OSError: + pass + return + + try: + os.unlink(moved) + except OSError: + pass + self.logger.warning(f"utils.files.PathLock - removed the stale lock file {self.lock_file} (left by pid {pid} on {host})") + + def _fall_back_to_soft(self, why): + if self.lock_file not in _lock_soft_paths: + _lock_soft_paths.add(self.lock_file) + self.logger.warning(f"utils.files.PathLock - the file system refuses OS locks for {self.lock_file} ({why}): a soft lock file is used instead (best effort between hosts)") + # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its + # content at once): out of the way, or the soft lock could never be taken + try: + if os.path.getsize(self.lock_file) == 0: + os.unlink(self.lock_file) + except OSError: + pass + + def _remove_soft_file(self): + """Remove the soft lock file; on Windows a waiter reading it holds it open for a moment, so the removal + is retried, and a file that cannot be removed in time is marked released for the waiters to clear.""" + deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS + while True: + if self._ident is not None: + try: + if self._identity(os.stat(self.lock_file)) != self._ident: + return # not ours any more + except OSError: + return + try: + os.unlink(self.lock_file) + return + except FileNotFoundError: + return + except PermissionError: + if time.monotonic() >= deadline: + break + time.sleep(0.002) + except OSError: + break + + try: + with open(self.lock_file, 'w') as f: + f.write(f"released {os.getpid()}\n") + except OSError: + pass + self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") + + @staticmethod + def _identity(st): + """The (inode, device) of a stat result; None where the file system reports no inode (some FAT drivers).""" + if not st.st_ino: + return None + return (st.st_ino, st.st_dev) + + def _unlink_own_file(self): + """Remove the lock file, unless the file at the path is no longer the one this lock took (someone + removed it under us and another process created a new one: that one is theirs).""" + if self._ident is not None: + try: + if self._identity(os.stat(self.lock_file)) != self._ident: + return + except OSError: + return + try: + os.unlink(self.lock_file) + except OSError: + pass + + ########################################################################################## def _acquire_lock( filepath: str, # Path to the file to lock. @@ -115,10 +614,8 @@ def _acquire_lock( logger = None, # Optional logger for debug messages. ): """ - Acquire a file lock for cross-platform thread/process-safe file operations. - - Uses FileLock library to create and acquire a lock file. Blocks until lock - is acquired or timeout expires. + Acquire the lock of a path (`.lock`, see PathLock) for process- and thread-safe + file operations. Blocks until the lock is acquired or the timeout expires. Args: filepath (str): Path to the file to lock. @@ -126,7 +623,7 @@ def _acquire_lock( logger: Optional logger for debug messages. Returns: - FileLock: Acquired lock object that must be released later. + PathLock: Acquired lock object that must be released later (`_release_lock`). Raises: TimeoutError: If lock cannot be acquired within timeout period. @@ -134,8 +631,8 @@ def _acquire_lock( lockfile = _get_lockfile_path(filepath) if logger is not None: logger.debug(f"utils.files._acquire_lock - attempting to create {lockfile} ...") - - file_lock = FileLock(lockfile, timeout=timeout) + + file_lock = PathLock(lockfile, logger=logger) try: file_lock.acquire(timeout=timeout) if logger is not None: @@ -149,7 +646,7 @@ def _acquire_lock( ########################################################################################## def _check_lock( filepath: str, # Path to the locked file. - file_lock, # FileLock object to check. + file_lock, # PathLock object to check. logger = None, # Optional logger for debug messages. ): """ @@ -157,7 +654,7 @@ def _check_lock( Args: filepath (str): Path to the locked file. - file_lock: FileLock object to check. + file_lock: PathLock object to check. logger: Optional logger for debug messages. Raises: @@ -176,15 +673,15 @@ def _check_lock( ########################################################################################## def _release_lock( filepath: str, # Path to the locked file. - file_lock, # FileLock object to release. + file_lock, # PathLock object to release. logger = None, # Optional logger for debug messages. ): """ - Release a previously acquired file lock. + Release a previously acquired file lock; the lock file is removed by the lock itself. Args: filepath (str): Path to the locked file. - file_lock: FileLock object to release. + file_lock: PathLock object to release. logger: Optional logger for debug messages. Returns: @@ -193,7 +690,7 @@ def _release_lock( Exception: Propagated runtime errors, if any. """ lockfile = _get_lockfile_path(filepath) - + try: if file_lock.is_locked: file_lock.release() @@ -202,48 +699,12 @@ def _release_lock( else: if logger is not None: logger.debug(f"utils.files._release_lock - lock {lockfile} doesn't exist - weird but keep running ...") - - # Explicitly clean up lock file - filelock doesn't always do this on Linux - _cleanup_lock_file(lockfile, logger) - + except Exception as e: if logger is not None: logger.debug(f"utils.files._release_lock - error releasing lock {lockfile} ({str(e)})") - - # Still try to cleanup lock file even if release failed - _cleanup_lock_file(lockfile, logger) raise e -########################################################################################## -def _cleanup_lock_file( - lockfile_path: str, # Path to the lock file to remove. - logger = None, # Optional logger for debug messages. -): - """ - Manually remove lock file if it exists. - - Ensures cleanup on systems where filelock doesn't auto-cleanup. - Errors are logged but not raised since this is a cleanup operation. - - Args: - lockfile_path (str): Path to the lock file to remove. - logger: Optional logger for debug messages. - - Returns: - dict: Operation result. - Raises: - Exception: Propagated runtime errors, if any. - """ - try: - if os.path.exists(lockfile_path): - os.remove(lockfile_path) - if logger is not None: - logger.debug(f"utils.files._cleanup_lock_file - manually removed {lockfile_path}") - except Exception as e: - if logger is not None: - logger.debug(f"utils.files._cleanup_lock_file - failed to remove {lockfile_path}: {str(e)}") - # Don't raise - this is a cleanup operation - ########################################################################################## def _detect_file_format( filepath: str, # Path to the file. diff --git a/docs/async-and-concurrency.md b/docs/async-and-concurrency.md index da86920..5f9b3de 100644 --- a/docs/async-and-concurrency.md +++ b/docs/async-and-concurrency.md @@ -209,29 +209,57 @@ r = utils.files.safe_write_file(index_file, index_data, file_lock=index_file_lock, atomic=True, ...) ``` -- **File locks** come from the `filelock` package (a runtime dependency), using - a sidecar lock file, so they work cross-platform and *across processes* — - not just across threads. Default acquisition timeout is **3 seconds**; - exceeding it produces an error dict rather than a corrupt file. +- **File locks** are the engine's own `cmeta.utils.files.PathLock` (no library), + using the sidecar file `.lock`, so they work cross-platform, *across + processes* and across the threads of one process. Default acquisition timeout + is **3 seconds**; exceeding it produces an error dict rather than a corrupt + file. - **`keep_locked=True`** hands the lock back with the data so the caller can modify and write without ever dropping it — this is what prevents lost updates. - **`atomic=True`** writes to `.tmp` and then `os.replace()`s it into position (retried up to 10 times), so a reader never observes a partially written file, and a crash mid-write cannot truncate the original. -- Locks are released in a `finally:` block, and stale lock files are cleaned up - explicitly (`filelock` does not always do so on Linux). +- Locks are released in a `finally:` block, and the lock file is removed by the + process that releases it, so repositories never keep lock files. A crash + leaves at most the lock file of the operation in flight; it blocks nobody + (the OS lock dies with the process) and the next holder reuses and removes it. The same mechanism protects `/index/*.pkl`, `repos.json`, and each artifact's `_cmeta.yaml` / `_cmeta.json`. +### How the lock works on each platform + +A lock file that is removed after use is correct only if the holder removes it +*before* releasing the OS lock and every newcomer checks, after acquiring, that +the file it holds is still the file at the path (otherwise a waiter blocked on +the removed file and a newcomer that created a new one both "hold" the lock). +`PathLock` does both. On **Windows** it takes a byte-range lock +(`msvcrt.locking`) on the lock file; an open file cannot be removed there, so a +newcomer that meets a file being deleted simply tries again. On **POSIX** it +takes `fcntl.flock` (per open file description, so threads exclude each other +too), or POSIX record locks (`fcntl.lockf`) where the file system refuses +`flock` (NFS and some network mounts), each with the identity check. Where the +file system refuses both (some FUSE, 9p and SMB mounts) it falls back to a +**soft lock**: the file is created exclusively and holds `pid host time`; a +stale one (its process dead on this host, or older than 10 minutes when left +by another host) is removed by the waiter that finds it. In every mode the +threads of one process are serialized on the path first. A busy lock is polled +with a delay that doubles from 1 ms to 25 ms until the timeout. A folder used +by two operating systems at once (a Windows tree plugged from WSL) is best +effort: each side locks with its own mechanism. + ### Measured behaviour Ten `cx note add` processes run simultaneously against one `` all exit `0`, and both the index and the directory listing show all ten artifacts. Eight processes updating **the same artifact's** metadata at the same time also all exit `0`, and **all eight fields are present afterwards** — no lost -updates, and no stale `.lock` files left behind. +updates, and no stale `.lock` files left behind. The lock's own tests +(`tests/core_tests/test_utils_files_path_lock.py`) run eight processes and +sixteen threads through one counter, replay the removed-file race +deterministically, kill a holder, and exercise the soft lock and its stale +files; `tests/benchmarks/benchmark_engine.py` measures the cost. ### What is *not* concurrency-safe diff --git a/pyproject.toml b/pyproject.toml index 468e3e0..aafefed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,7 +42,6 @@ dependencies = [ "wheel", "tabulate", "tqdm", - "filelock", "packaging", "psutil", ] diff --git a/tests/benchmarks/benchmark_engine.py b/tests/benchmarks/benchmark_engine.py new file mode 100644 index 0000000..51a2867 --- /dev/null +++ b/tests/benchmarks/benchmark_engine.py @@ -0,0 +1,567 @@ +""" +A benchmark of the cMeta engine's hot paths: the index reads behind `find`, the locked index writes behind +`create`, `update`, `delete` and `reindex`, the full reindex, the reload of a changed index, and the primitives +they are built on (stat, file lock, record copy, pickle, atomic replace). It makes a home of its own with N +hand-made artifacts in one category and prints one table; with --json it also writes the numbers, and with +--compare it prints every number next to the one of an earlier run (the before/after check of an engine +change). With --procs P it also lets P processes create artifacts in the same category at once and reports +the throughput and whether every record made it into the index. + + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json before.json + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json after.json --compare before.json + +Nothing outside --home is touched: CMETA_HOME is set to it for this process and its workers. The folder is +removed first when it holds the marker of an earlier run and refused otherwise. The pytest module next to +this file runs it with a small N as a smoke test; the numbers themselves are not asserted (the speed of a +CI runner varies). + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import argparse +import copy +import json +import os +import pickle +import platform +import random +import shutil +import statistics +import subprocess +import sys +import time + +MARKER = '.cmeta-benchmark-home' +CATEGORY = 'log' +CATEGORY_REF = 'log,487a7639093a4685' +REPO_ALIAS = 'bench-repo' +REPO_UID = 'b0c4e5f6a7b8c9d0' +ENV_TO_DROP = ('CMETA_HOME', 'CMETA_HOME2', 'CMETA_VERBOSE', 'CMETA_DEBUG', 'CMETA_FAIL_ON_ERROR', + 'CMETA_INTERNAL_REPO_PATH', 'VIRTUAL_ENV', 'CONDA_PREFIX') + + +########################################################################################## +# Timing helpers + +def timed_calls(fn, repeat): + """Every call timed on its own; returns the list of durations in seconds.""" + times = [] + for _ in range(repeat): + t0 = time.perf_counter() + fn() + times.append(time.perf_counter() - t0) + return times + + +class Results: + """The measurements of one run: name -> median, min, max, unit, repeats, note.""" + + UNITS = {'us': 1e6, 'ms': 1e3, 's': 1.0} + + def __init__(self): + self.rows = {} + self.order = [] + + def add(self, name, times, unit, note=''): + scale = self.UNITS[unit] + row = {'median': statistics.median(times) * scale, 'min': min(times) * scale, 'max': max(times) * scale, + 'unit': unit, 'n': len(times), 'note': note} + self.rows[name] = row + self.order.append(name) + print(' %-44s %12.2f %-3s (min %10.2f, max %10.2f, n=%d) %s' % ( + name, row['median'], unit, row['min'], row['max'], row['n'], note)) + return row + + def value(self, name, value, unit, note=''): + row = {'median': value, 'min': value, 'max': value, 'unit': unit, 'n': 1, 'note': note} + self.rows[name] = row + self.order.append(name) + print(' %-44s %12.2f %-3s %s' % (name, value, unit, note)) + return row + + +def section(title): + print('') + print('=== ' + title) + + +########################################################################################## +# The home with N hand-made artifacts + +def prepare_home(home): + if os.path.exists(home): + if not os.path.isfile(os.path.join(home, MARKER)): + sys.exit('refusing to remove %s: it is not the home of an earlier benchmark run (no %s)' % (home, MARKER)) + shutil.rmtree(home) + os.makedirs(home) + with open(os.path.join(home, MARKER), 'w') as f: + f.write('made by tests/benchmarks/benchmark_engine.py - safe to delete\n') + + +def make_artifacts(home, n, seed=20261008): + """A repository folder with N artifacts of the log category, written by hand (no engine call).""" + rnd = random.Random(seed) + repo = os.path.join(home, 'bench', REPO_ALIAS) + cat_dir = os.path.join(repo, CATEGORY) + os.makedirs(cat_dir) + with open(os.path.join(repo, '_cmr.yaml'), 'w', encoding='utf-8') as f: + f.write('artifact: %s,%s\ncategory: repo,f4f792ab40c7498f\n' % (REPO_ALIAS, REPO_UID)) + uids = [] + for i in range(n): + alias = 'art-%06d' % i + uid = '%016x' % rnd.getrandbits(64) + uids.append(uid) + folder = os.path.join(cat_dir, alias) + os.mkdir(folder) + with open(os.path.join(folder, '_cmeta.yaml'), 'w', encoding='utf-8') as f: + f.write('artifact: %s,%s\ncategory: %s\ntags:\n - bench\n - g%d\n' + 'desc: benchmark artifact number %d of the log category\nowner: benchmark\n' + 'generator: script\n' % (alias, uid, CATEGORY_REF, i % 10, i)) + return repo, uids + + +########################################################################################## +# The lock candidates of the concurrency work (measured here, not used by the engine) + +def candidate_lock(path, keep_file): + """A lock that keeps the file at `path` the one every process locks: acquire, then check that the file + opened is still the file at the path (it is not when another process removed it meanwhile); on release + the file is removed BEFORE the lock is dropped (POSIX) or after the handle is closed (Windows, where a + file still open elsewhere cannot be removed). With keep_file the lock file is never removed.""" + if os.name == 'nt': + import msvcrt + while True: + try: + fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644) + except PermissionError: + time.sleep(0.001) + continue + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + return fd + except OSError: + os.close(fd) + time.sleep(0.001) + else: + import fcntl + while True: + fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644) + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + os.close(fd) + time.sleep(0.001) + continue + try: + st = os.stat(path) + except FileNotFoundError: + st = None + fst = os.fstat(fd) + if st is None or (st.st_ino, st.st_dev) != (fst.st_ino, fst.st_dev): + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + continue + return fd + + +def candidate_unlock(path, fd, keep_file): + if os.name == 'nt': + import msvcrt + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + os.close(fd) + if not keep_file: + try: + os.unlink(path) + except OSError: + pass + else: + import fcntl + if not keep_file: + try: + os.unlink(path) + except OSError: + pass + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + + +########################################################################################## +# The worker of the concurrency measurement + +def worker_create(home, prefix, count): + for v in ENV_TO_DROP: + os.environ.pop(v, None) + os.environ['CMETA_HOME'] = home + from cmeta import CMeta + cm = CMeta(home=home) + times = [] + errors = [] + for i in range(count): + t0 = time.perf_counter() + r = cm.access({'category': CATEGORY, 'command': 'create', 'arg1': '%s:%s-%04d' % (REPO_ALIAS, prefix, i), + 'meta': {'tags': ['concurrent']}, 'yaml': True, 'con': False}) + times.append(time.perf_counter() - t0) + if r['return'] > 0: + errors.append(r.get('error', '?')) + print(json.dumps({'times': times, 'errors': errors})) + + +def concurrent_creates(home, procs, per, results): + section('Concurrent creates: %d processes x %d artifacts in the same category at once' % (procs, per)) + env = dict(os.environ) + env['CMETA_HOME'] = home + t0 = time.perf_counter() + children = [] + for p in range(procs): + children.append(subprocess.Popen([sys.executable, os.path.abspath(__file__), '--worker-create', 'cw%d' % p, + str(per), '--home', home], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)) + times = [] + errors = [] + for child in children: + out, err = child.communicate() + try: + data = json.loads(out.decode('utf-8', 'replace').strip().splitlines()[-1]) + except Exception: + errors.append('worker output unreadable: %s %s' % (out[-300:], err[-300:])) + continue + times.extend(data['times']) + errors.extend(data['errors']) + wall = time.perf_counter() - t0 + results.value('concurrent creates: wall time', wall, 's', '%d creates' % (procs * per)) + if times: + results.value('concurrent creates: creates per second', len(times) / wall, 's') + results.add('concurrent create latency', times, 'ms') + return errors + + +def check_index(home, cm, procs, per, results): + """Every record the workers made is in the index, and the alias map agrees with the records.""" + r = cm.access({'category': CATEGORY, 'command': 'find', 'arg1': 'cw*', 'con': False}) + found = len(r.get('artifacts', [])) if r['return'] == 0 else 0 + expected = procs * per + results.value('concurrent creates: records lost', expected - found, 's', 'of %d (0 = every write kept)' % expected) + with open(os.path.join(home, 'index', CATEGORY + '.pkl'), 'rb') as f: + index = pickle.load(f) + uids = index['uids'] + bad = 0 + for alias, alias_uids in index['lowercase_aliases'].items(): + for uid in alias_uids: + if uid not in uids: + bad += 1 + for uid, rec in uids.items(): + alias = rec['cmeta_ref_parts'].get('artifact_alias_lowercase', rec['cmeta_ref_parts'].get('artifact_alias', '')).lower() + if alias and uid not in index['lowercase_aliases'].get(alias, []): + bad += 1 + results.value('index consistency: broken alias links', bad, 's', '0 = aliases and records agree') + on_disk = 0 + cat_dir = os.path.join(home, 'bench', REPO_ALIAS, CATEGORY) + for name in os.listdir(cat_dir): + if name.startswith('cw'): + on_disk += 1 + results.value('concurrent creates: folders on disk', on_disk, 's', 'of %d' % expected) + + +########################################################################################## +# CLI timings + +def cli_timings(home, cx, results): + section('CLI (a new process per call, as a user runs cx)') + env = dict(os.environ) + env['CMETA_HOME'] = home + + def run(args): + r = subprocess.run([cx] + args, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if r.returncode != 0: + raise RuntimeError('%s failed: %s' % (' '.join(args), r.stderr.decode('utf-8', 'replace')[-500:])) + + try: + run(['--version']) + except Exception as e: + print(' cx not usable (%s), the CLI timings are skipped' % e) + return + results.add('cx --version', timed_calls(lambda: run(['--version']), 5), 'ms', 'process start + engine init') + results.add('cx log find ', timed_calls(lambda: run([CATEGORY, 'find', 'art-000001']), 5), 'ms') + results.add('cx log reindex ', timed_calls(lambda: run([CATEGORY, 'reindex', 'art-000001']), 3), 'ms') + results.add('cx --reindex (full)', timed_calls(lambda: run(['--reindex']), 1), 's') + + +########################################################################################## +# Comparison with an earlier run + +def compare(results, before_path): + with open(before_path) as f: + before = json.load(f)['results'] + section('Compared with ' + before_path + ' (ratio = now / before, by the medians)') + print(' %-44s %12s %12s %8s' % ('measurement', 'before', 'now', 'ratio')) + for name in results.order: + now = results.rows[name] + old = before.get(name) + if old is None or old['unit'] != now['unit']: + continue + if old['median'] > 0: + ratio = now['median'] / old['median'] + if 'per second' in name: # a rate: more is better + flag = ' faster' if ratio > 1.25 else (' slower' if ratio < 0.8 else '') + elif name.endswith(('lost', 'links', 'on disk', 'index', 'size')): # counts, not times + flag = '' + else: + flag = ' slower' if ratio > 1.25 else (' faster' if ratio < 0.8 else '') + print(' %-44s %12.2f %12.2f %8.2f%s' % (name, old['median'], now['median'], ratio, flag)) + else: + print(' %-44s %12.2f %12.2f %8s' % (name, old['median'], now['median'], '-')) + + +########################################################################################## + +def main(): + parser = argparse.ArgumentParser(description='benchmark of the cMeta engine hot paths') + parser.add_argument('--home', required=True, help='an empty folder (or the home of an earlier run) used as CMETA_HOME') + parser.add_argument('--n', type=int, default=10000, help='artifacts in the category (default 10000)') + parser.add_argument('--writes', type=int, default=100, help='creates/updates/reindexes/deletes timed (default 100)') + parser.add_argument('--procs', type=int, default=4, help='processes of the concurrent-create measurement (0 = skip)') + parser.add_argument('--per', type=int, default=25, help='creates per process in that measurement') + parser.add_argument('--cli', action='store_true', help='also time the cx command line') + parser.add_argument('--cx', default='cx', help='the cx executable for --cli') + parser.add_argument('--json', help='write the numbers to this file') + parser.add_argument('--compare', help='an earlier --json file to print the ratios against') + parser.add_argument('--keep', action='store_true', help='keep the home after the run') + parser.add_argument('--worker-create', nargs=2, metavar=('PREFIX', 'COUNT'), help=argparse.SUPPRESS) + args = parser.parse_args() + + home = os.path.abspath(args.home) + + if args.worker_create: + worker_create(home, args.worker_create[0], int(args.worker_create[1])) + return + + for v in ENV_TO_DROP: + os.environ.pop(v, None) + os.environ['CMETA_HOME'] = home + + import cmeta + from cmeta import CMeta + from cmeta import utils + + results = Results() + meta = {'python': sys.version.split()[0], 'platform': platform.platform(), 'machine': platform.machine(), + 'cmeta': cmeta.__version__, 'cmeta_path': os.path.dirname(cmeta.__file__), 'home': home, 'n': args.n, + 'pickle_default_protocol': pickle.DEFAULT_PROTOCOL, 'date': time.strftime('%Y-%m-%d %H:%M:%S')} + section('Benchmark of the cMeta engine') + for k in ('python', 'platform', 'cmeta', 'cmeta_path', 'home', 'n', 'pickle_default_protocol'): + print(' %-24s %s' % (k, meta[k])) + + prepare_home(home) + + section('Engine start') + t0 = time.perf_counter() + cm = CMeta(home=home) + results.value('CMeta() first run of a home', time.perf_counter() - t0, 'ms', 'the home and its index are made') + results.add('CMeta() warm', timed_calls(lambda: CMeta(home=home), 5), 'ms', 'a new engine object on an existing home') + + section('Index build of %d hand-made artifacts' % args.n) + t0 = time.perf_counter() + repo, uids = make_artifacts(home, args.n) + results.value('writing %d artifact folders by hand' % args.n, time.perf_counter() - t0, 's', 'not the engine') + t0 = time.perf_counter() + r = cm.access({'category': 'repo', 'command': 'plug', 'arg1': repo, 'con': False}) + assert r['return'] == 0, r.get('error') + results.value('repo plug (incremental index of the repo)', time.perf_counter() - t0, 's') + results.add('full reindex (cx --reindex)', timed_calls(lambda: cm.repos.reindex(con=False), 2), 's') + index_file = os.path.join(home, 'index', CATEGORY + '.pkl') + results.value('index file size', os.path.getsize(index_file) / 1024.0, 's', 'KB (unit column: KB)') + with open(index_file, 'rb') as f: + index_data = pickle.load(f) + results.value('records in the index', len(index_data['uids']), 's', '(unit column: records)') + + section('Reads with a warm cache (the index already loaded in this process)') + alias = 'art-%06d' % (args.n // 2) + uid = uids[args.n // 2] + + def find(**p): + p.setdefault('category', CATEGORY) + p.setdefault('command', 'find') + p.setdefault('con', False) + return cm.access(p) + + r = find(arg1=alias) + assert r['return'] == 0 and len(r['artifacts']) == 1, r + results.add('find by alias (1 hit)', timed_calls(lambda: find(arg1=alias), 2000), 'us') + results.add('find by UID (1 hit)', timed_calls(lambda: find(arg1=uid), 2000), 'us') + results.add('find a missing alias (error 16)', timed_calls(lambda: find(arg1='no-such-artifact'), 2000), 'us') + results.add('find by alias + tag filter', timed_calls(lambda: find(arg1=alias, tags='bench'), 1000), 'us') + r = find(arg1='art-00*') + hits = len(r.get('artifacts', [])) + results.add('find wildcard art-00* (%d hits)' % hits, timed_calls(lambda: find(arg1='art-00*'), 20), 'ms') + r = find() + hits = len(r.get('artifacts', [])) + results.add('find all (%d hits)' % hits, timed_calls(lambda: find(), 5), 'ms') + r = find(tags='g3') + hits = len(r.get('artifacts', [])) + results.add('find by tag g3 (%d hits)' % hits, timed_calls(lambda: find(tags='g3'), 5), 'ms', 'every record checked') + + section('Reads after the index changed (the reload a write by another process causes)') + + def bump_and_find(): + st = os.stat(index_file) + os.utime(index_file, ns=(st.st_atime_ns, st.st_mtime_ns + 1000000000)) + r = find(arg1=alias) + assert r['return'] == 0 + + results.add('find by alias after a changed index', timed_calls(bump_and_find, 5), 'ms', 'the pickle load + 1 find') + results.add('first find of a new engine object', timed_calls(lambda: CMeta(home=home).access( + {'category': CATEGORY, 'command': 'find', 'arg1': alias, 'con': False}), 5), 'ms', 'CMeta() + load + find') + + section('Writes (each one rewrites the whole index file of the category under its lock)') + w = args.writes + + def create(i): + r = cm.access({'category': CATEGORY, 'command': 'create', 'arg1': '%s:new-%04d' % (REPO_ALIAS, i), + 'meta': {'tags': ['new']}, 'yaml': True, 'con': False}) + assert r['return'] == 0, r.get('error') + + def update(i): + r = cm.access({'category': CATEGORY, 'command': 'update', 'arg1': 'new-0000', 'meta': {'counter': i}, 'con': False}) + assert r['return'] == 0, r.get('error') + + def reindex_one(): + r = cm.access({'category': CATEGORY, 'command': 'reindex', 'arg1': alias, 'con': False}) + assert r['return'] == 0, r.get('error') + + def delete(i): + r = cm.access({'category': CATEGORY, 'command': 'delete', 'arg1': 'new-%04d' % i, 'force': True, 'con': False}) + assert r['return'] == 0, r.get('error') + + times = [] + for i in range(w): + t0 = time.perf_counter() + create(i) + times.append(time.perf_counter() - t0) + results.add('create (meta file + index write)', times, 'ms') + times = [] + for i in range(w): + t0 = time.perf_counter() + update(i) + times.append(time.perf_counter() - t0) + results.add('update (meta edit in place + index write)', times, 'ms') + results.add('reindex one artifact', timed_calls(reindex_one, w), 'ms', 'folder scan + index write') + times = [] + for i in range(w): + t0 = time.perf_counter() + delete(i) + times.append(time.perf_counter() - t0) + results.add('delete (index write + folder removal)', times, 'ms') + + section('Primitives (what one index read or write is made of)') + results.add('os.stat of the index file', timed_calls(lambda: os.stat(index_file), 10000), 'us', + '3 per find today (is_dir, exists, stat)') + lock_target = os.path.join(home, 'index', 'lock-benchmark.bin') + with open(lock_target, 'wb') as f: + f.write(b'x') + + def engine_lock(): + lock = utils.files._acquire_lock(lock_target) + utils.files._release_lock(lock_target, lock) + + results.add('engine lock acquire+release', timed_calls(engine_lock, 500), 'us', 'filelock + the lock file removed') + try: + from filelock import FileLock + + def library_lock(): + lock = FileLock(lock_target + '.lock') + lock.acquire() + lock.release() + + results.add('filelock acquire+release alone', timed_calls(library_lock, 500), 'us', 'the library by itself') + except ImportError: + pass + cand = lock_target + '.cand' + + def cand_lock(keep): + fd = candidate_lock(cand, keep) + candidate_unlock(cand, fd, keep) + + results.add('candidate lock, file removed on release', timed_calls(lambda: cand_lock(False), 500), 'us', + 'verified identity, removed before release') + results.add('candidate lock, file kept', timed_calls(lambda: cand_lock(True), 500), 'us', 'verified identity, never removed') + try: + os.unlink(cand) + except OSError: + pass + + records = list(index_data['uids'].values()) + + def copy_level1(rec): + c = dict(rec) + c['cmeta'] = dict(rec['cmeta']) + tags = c['cmeta'].get('tags') + if isinstance(tags, list): + c['cmeta']['tags'] = list(tags) + return c + + def all_records(fn): + for rec in records: + fn(rec) + + results.add('copy of %d records: record.copy() (today)' % len(records), timed_calls(lambda: all_records(dict.copy), 5), 'ms') + results.add('copy of %d records: record + cmeta + tags' % len(records), timed_calls(lambda: all_records(copy_level1), 5), 'ms', + 'the cheap copy that protects the cache') + results.add('copy of %d records: deepcopy' % len(records), timed_calls(lambda: all_records(copy.deepcopy), 3), 'ms') + + for proto in sorted(set([4, 5, pickle.HIGHEST_PROTOCOL])): + if proto > pickle.HIGHEST_PROTOCOL: + continue + data = pickle.dumps(index_data, protocol=proto) + results.add('pickle dump of the index, protocol %d' % proto, timed_calls(lambda: pickle.dumps(index_data, protocol=proto), 5), + 'ms', '%d KB' % (len(data) // 1024)) + results.add('pickle load of the index, protocol %d' % proto, timed_calls(lambda: pickle.loads(data), 5), 'ms') + + small = os.path.join(home, 'index', 'write-benchmark.yaml') + payload = ('key%03d: value %d\n' % (i, i) for i in range(40)) + payload = ''.join(payload) + + def plain_write(): + with open(small, 'w', encoding='utf-8') as f: + f.write(payload) + + def atomic_write(): + tmp = small + '.tmp' + with open(tmp, 'w', encoding='utf-8') as f: + f.write(payload) + os.replace(tmp, small) + + results.add('write a 1 KB file in place', timed_calls(plain_write, 500), 'us') + results.add('write a 1 KB file atomically', timed_calls(atomic_write, 500), 'us', 'temp file + os.replace') + for p in (small, lock_target): + try: + os.unlink(p) + except OSError: + pass + + errors = [] + if args.procs > 0: + errors = concurrent_creates(home, args.procs, args.per, results) + check_index(home, cm, args.procs, args.per, results) + if errors: + print(' worker errors (%d): %s' % (len(errors), errors[:3])) + + if args.cli: + cli_timings(home, args.cx, results) + + if args.compare: + compare(results, args.compare) + + if args.json: + with open(args.json, 'w') as f: + json.dump({'meta': meta, 'results': {k: results.rows[k] for k in results.order}, 'worker_errors': errors}, f, indent=1) + print('') + print('numbers written to ' + args.json) + + if not args.keep: + shutil.rmtree(home, ignore_errors=True) + + +if __name__ == '__main__': + main() diff --git a/tests/benchmarks/test_benchmark_engine.py b/tests/benchmarks/test_benchmark_engine.py new file mode 100644 index 0000000..4d5b630 --- /dev/null +++ b/tests/benchmarks/test_benchmark_engine.py @@ -0,0 +1,68 @@ +""" +The engine benchmark next to this file (benchmark_engine.py) must keep running on every platform and Python +the suite covers: this runs it on a small home and checks that every section reported its numbers and that +the index it built is consistent. The numbers themselves are not asserted (the speed of a CI runner varies). +The benchmark proper is run by hand before and after an engine change: + + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json before.json + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json after.json --compare before.json + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +SCRIPT = os.path.join(HERE, 'benchmark_engine.py') + +EXPECTED = ( + 'CMeta() warm', + 'repo plug (incremental index of the repo)', + 'full reindex (cx --reindex)', + 'records in the index', + 'find by alias (1 hit)', + 'find by UID (1 hit)', + 'find a missing alias (error 16)', + 'find by alias after a changed index', + 'create (meta file + index write)', + 'update (meta edit in place + index write)', + 'reindex one artifact', + 'delete (index write + folder removal)', + 'os.stat of the index file', + 'engine lock acquire+release', + 'candidate lock, file removed on release', + 'write a 1 KB file atomically', + 'concurrent creates: wall time', + 'concurrent creates: records lost', + 'concurrent creates: folders on disk', + 'index consistency: broken alias links', +) + + +def test_benchmark_runs_and_reports_every_section(tmp_path): + out = tmp_path / 'bench.json' + env = dict(os.environ) + for v in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + env.pop(v, None) + + r = subprocess.run([sys.executable, SCRIPT, '--home', str(tmp_path / 'home'), '--n', '50', '--writes', '3', + '--procs', '2', '--per', '3', '--json', str(out)], + env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=900) + assert r.returncode == 0, r.stdout.decode('utf-8', 'replace')[-4000:] + + data = json.loads(out.read_text(encoding='utf-8')) + results = data['results'] + for name in EXPECTED: + assert name in results, name + assert results[name]['n'] >= 1 + + assert results['records in the index']['median'] == 50 + assert results['concurrent creates: folders on disk']['median'] == 6 + assert results['index consistency: broken alias links']['median'] == 0 + assert data['worker_errors'] == [] + assert data['meta']['n'] == 50 diff --git a/tests/core_tests/test_utils_files_path_lock.py b/tests/core_tests/test_utils_files_path_lock.py new file mode 100644 index 0000000..bae050a --- /dev/null +++ b/tests/core_tests/test_utils_files_path_lock.py @@ -0,0 +1,520 @@ +""" +Tests of the engine's own path lock (`cmeta.utils.files.PathLock`, behind `_acquire_lock` / `_release_lock`, +`safe_read_file(lock=True, keep_locked=True)`, `safe_write_file`, `safe_delete_directory`, `lock_path`): +one holder at a time across processes and across the threads of one process on every platform; the lock +file is created for the operation and removed by the process that releases it; a lock file removed under a +waiter is detected (the identity check) and never leads to two holders; a killed holder blocks nobody; the +soft lock used where the file system refuses OS locks, with its stale files; timeouts; the engine's own +read-modify-write cycle under contention. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import subprocess +import sys +import threading +import time + +import pytest + +from cmeta.utils import files + +POSIX = sys.platform != 'win32' + + +########################################################################################## +# Workers run as separate processes + +COUNTER_WORKER = r''' +import os, sys, time +from cmeta.utils import files +mode, target, log_path, n = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]) +if len(sys.argv) > 5 and sys.argv[5] == 'soft': + files._lock_soft_paths.add(target + '.lock') +pid = os.getpid() +for i in range(n): + if mode == 'pathlock': + lock = files.PathLock(target + '.lock').acquire(timeout=120) + try: + try: + with open(target) as f: + v = int(f.read().strip() or '0') + except FileNotFoundError: + v = 0 + time.sleep(0.0005) + with open(target, 'w') as f: + f.write(str(v + 1)) + with open(log_path, 'a') as f: + f.write('%d %d %s\n' % (pid, i, lock.mode)) + finally: + lock.release() + else: + r = files.safe_read_file(target, lock=True, keep_locked=True, timeout=120) + if r['return'] > 0: + print('ERROR read', r['error']); sys.exit(3) + data = r['data'] + data['n'] = data.get('n', 0) + 1 + time.sleep(0.0005) + r = files.safe_write_file(target, data, file_lock=r['file_lock'], atomic=True, timeout=120) + if r['return'] > 0: + print('ERROR write', r['error']); sys.exit(4) +print('OK', pid) +''' + +HOLDER = r''' +import sys, time +from cmeta.utils import files +lock = files.PathLock(sys.argv[1]).acquire(timeout=60) +print('LOCKED', lock.mode, flush=True) +time.sleep(float(sys.argv[2]) if len(sys.argv) > 2 else 60) +lock.release() +print('RELEASED', flush=True) +''' + +HOOKED_WAITER = r''' +import os, sys, time +from cmeta.utils import files +lock_file, marker, go, out = sys.argv[1:5] +real = files.fcntl.flock +state = {'first': True} +def hooked(fd, op): + # The first lock attempt: the file is open (the old file), tell the test and wait for its go + if state['first'] and (op & files.fcntl.LOCK_NB): + state['first'] = False + open(marker, 'w').close() + while not os.path.exists(go): + time.sleep(0.002) + return real(fd, op) +files.fcntl.flock = hooked +lock = files.PathLock(lock_file) +lock.acquire(timeout=60) +same = files.PathLock._identity(os.fstat(lock._fd)) == files.PathLock._identity(os.stat(lock_file)) +with open(out, 'w') as f: + f.write('%f %d %s %s\n' % (time.time(), lock.identity_retries, lock.mode, same)) +lock.release() +''' + + +def run_workers(script, args_list, timeout=600): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + procs = [subprocess.Popen([sys.executable, '-c', script] + [str(a) for a in args], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env) + for args in args_list] + outputs = [] + for p in procs: + out, _ = p.communicate(timeout=timeout) + outputs.append((p.returncode, out.decode('utf-8', 'replace'))) + return outputs + + +def start(script, *args): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + # The workers never read their input: an explicit null input keeps them independent of the + # parent's standard-input handle (which some agent and service environments leave unusable) + return subprocess.Popen([sys.executable, '-c', script] + [str(a) for a in args], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env) + + +def wait_for(path, seconds=30): + deadline = time.time() + seconds + while not os.path.exists(path): + if time.time() > deadline: + raise AssertionError(f'{path} did not appear in {seconds} s') + time.sleep(0.005) + + +########################################################################################## +# The lock object + +def test_one_holder_at_a_time_and_the_file_is_removed_on_release(tmp_path): + lock_file = str(tmp_path / 'thing.lock') + first = files.PathLock(lock_file).acquire(timeout=5) + assert first.is_locked and first.mode in ('windows', 'flock', 'lockf', 'soft') + assert os.path.isfile(lock_file) + + second = files.PathLock(lock_file) + t0 = time.monotonic() + with pytest.raises(TimeoutError): + second.acquire(timeout=0.3) + assert 0.25 <= time.monotonic() - t0 < 3 + assert not second.is_locked + + first.release() + assert not first.is_locked + assert not os.path.exists(lock_file) + + second.acquire(timeout=5) + assert second.is_locked + second.release() + assert not os.path.exists(lock_file) + + +def test_context_manager_double_release_and_check(tmp_path): + target = str(tmp_path / 'file.json') + with files.PathLock(target + '.lock') as lock: + assert lock.is_locked + files._check_lock(target, lock) + assert not lock.is_locked + lock.release() # a second release is harmless + with pytest.raises(TimeoutError): + files._check_lock(target, lock) + assert not os.path.exists(target + '.lock') + + +def test_lock_path_and_unlock_path(tmp_path): + target = str(tmp_path / 'folder') + r = files.lock_path(target, timeout=5) + assert r['return'] == 0 and r['file_lock'].is_locked + assert os.path.isfile(target + '.lock') + r2 = files.lock_path(target, timeout=0.2) + assert r2['return'] > 0 + assert files.unlock_path(target, r['file_lock'])['return'] == 0 + assert not os.path.exists(target + '.lock') + + +def test_a_missing_folder_is_an_error_not_a_wait(tmp_path): + t0 = time.monotonic() + with pytest.raises(TimeoutError): + files._acquire_lock(str(tmp_path / 'no-such-folder' / 'file'), timeout=5) + assert time.monotonic() - t0 < 2 + + +def test_threads_of_one_process_exclude_each_other(tmp_path): + target = str(tmp_path / 'counter.txt') + state = {'value': 0, 'inside': 0, 'overlap': 0, 'modes': set()} + + def work(): + for _ in range(50): + lock = files.PathLock(target + '.lock').acquire(timeout=60) + try: + state['inside'] += 1 + if state['inside'] > 1: + state['overlap'] += 1 + v = state['value'] + time.sleep(0.0002) + state['value'] = v + 1 + state['modes'].add(lock.mode) + state['inside'] -= 1 + finally: + lock.release() + + threads = [threading.Thread(target=work) for _ in range(16)] + for t in threads: + t.start() + for t in threads: + t.join() + assert state['value'] == 16 * 50 + assert state['overlap'] == 0 + assert not os.path.exists(target + '.lock') + + +def test_a_thread_waiting_on_another_thread_times_out_instead_of_hanging(tmp_path): + lock_file = str(tmp_path / 'held.lock') + held = files.PathLock(lock_file).acquire(timeout=5) + result = {} + + def waiter(): + t0 = time.monotonic() + try: + files.PathLock(lock_file).acquire(timeout=0.3) + result['got'] = True + except TimeoutError as e: + result['error'] = str(e) + result['seconds'] = time.monotonic() - t0 + + t = threading.Thread(target=waiter) + t.start() + t.join(10) + assert 'error' in result and 'thread' in result['error'] + assert 0.25 <= result['seconds'] < 3 + held.release() + + +########################################################################################## +# Processes + +def test_processes_exclude_each_other_through_pathlock(tmp_path): + target = str(tmp_path / 'counter.txt') + log_path = str(tmp_path / 'log.txt') + outputs = run_workers(COUNTER_WORKER, [('pathlock', target, log_path, 60)] * 8) + for rc, out in outputs: + assert rc == 0 and out.startswith('OK'), out + with open(target) as f: + assert int(f.read().strip()) == 8 * 60 + with open(log_path) as f: + lines = f.read().splitlines() + assert len(lines) == 8 * 60 and len(set(lines)) == 8 * 60 + assert not os.path.exists(target + '.lock') + + +def test_the_engine_read_modify_write_cycle_under_contention(tmp_path): + target = str(tmp_path / 'index.json') + with open(target, 'w') as f: + json.dump({'n': 0}, f) + outputs = run_workers(COUNTER_WORKER, [('engine', target, '', 40)] * 6) + for rc, out in outputs: + assert rc == 0 and out.startswith('OK'), out + with open(target) as f: + assert json.load(f)['n'] == 6 * 40 + assert not os.path.exists(target + '.lock') + assert not os.path.exists(target + '.tmp') + + +def test_a_killed_holder_blocks_nobody(tmp_path): + lock_file = str(tmp_path / 'crash.lock') + holder = start(HOLDER, lock_file, 60) + line = holder.stdout.readline().decode('utf-8', 'replace') + assert line.startswith('LOCKED'), line + assert os.path.isfile(lock_file) + + with pytest.raises(TimeoutError): + files.PathLock(lock_file).acquire(timeout=0.3) + + holder.kill() + holder.wait(30) + + t0 = time.monotonic() + lock = files.PathLock(lock_file).acquire(timeout=10) + assert time.monotonic() - t0 < 5 + lock.release() + assert not os.path.exists(lock_file) + + +@pytest.mark.skipif(POSIX, reason='Windows: a lock file held by another process cannot be removed') +def test_windows_keeps_a_held_lock_file_from_being_removed(tmp_path): + lock_file = str(tmp_path / 'held.lock') + holder = start(HOLDER, lock_file, 60) + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('LOCKED') + with pytest.raises(PermissionError): + os.remove(lock_file) + holder.kill() + holder.wait(30) + + +@pytest.mark.skipif(not POSIX, reason='the identity check is a POSIX matter (flock on a removed file)') +def test_a_lock_file_removed_under_a_waiter_never_gives_two_holders(tmp_path): + """The race of the previous lock, replayed deterministically: a waiter has opened the lock file (A) and is + about to flock it when the holder removes A and releases, and a newcomer creates B and holds it. The waiter's + flock on A succeeds (nobody holds A any more) - and the identity check sends it back to try B.""" + lock_file = str(tmp_path / 'race.lock') + marker, go, out = str(tmp_path / 'marker'), str(tmp_path / 'go'), str(tmp_path / 'out') + + holder = files.PathLock(lock_file).acquire(timeout=5) + waiter = start(HOOKED_WAITER, lock_file, marker, go, out) + wait_for(marker) # the waiter has opened A + + holder.release() # A is removed, then released + assert not os.path.exists(lock_file) + newcomer = files.PathLock(lock_file).acquire(timeout=5) # B + open(go, 'w').close() + + time.sleep(0.5) + assert not os.path.exists(out), 'the waiter acquired while the newcomer held B' + + released_at = time.time() + newcomer.release() + waiter.wait(60) + assert waiter.returncode == 0, waiter.stdout.read().decode('utf-8', 'replace') + with open(out) as f: + acquired_at, retries, mode, same = f.read().split() + assert float(acquired_at) >= released_at - 0.001 + assert int(retries) >= 1 + assert mode == 'flock' and same == 'True' + assert not os.path.exists(lock_file) + + +@pytest.mark.skipif(not POSIX, reason='on Windows the file cannot be removed while it is held') +def test_release_leaves_a_lock_file_that_is_not_its_own(tmp_path): + lock_file = str(tmp_path / 'mine.lock') + mine = files.PathLock(lock_file).acquire(timeout=5) + os.unlink(lock_file) # a third party removed it ... + with open(lock_file, 'w') as f: # ... and another process made a new one + f.write('theirs') + mine.release() + assert os.path.isfile(lock_file) # theirs is kept + with open(lock_file) as f: + assert f.read() == 'theirs' + os.unlink(lock_file) + + +########################################################################################## +# The soft lock (where the file system refuses OS locks) + +@pytest.fixture() +def soft(monkeypatch, tmp_path): + lock_file = str(tmp_path / 'soft.lock') + monkeypatch.setattr(files, '_lock_soft_paths', {lock_file}) + return lock_file + + +def test_soft_lock_holds_pid_and_host_and_is_removed_on_release(soft, caplog): + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + with open(soft) as f: + pid, host, when = f.read().split() + assert int(pid) == os.getpid() and host == files._lock_host_name() + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + lock.release() + assert not os.path.exists(soft) + + +def test_soft_lock_excludes_processes(tmp_path): + target = str(tmp_path / 'counter.txt') + log_path = str(tmp_path / 'log.txt') + outputs = run_workers(COUNTER_WORKER, [('pathlock', target, log_path, 30, 'soft')] * 4) + for rc, out in outputs: + assert rc == 0 and 'OK' in out, out + assert 'stale' not in out, out # every holder is alive here: a "stale" removal would be a misjudgement + with open(target) as f: + assert int(f.read().strip()) == 4 * 30 + with open(log_path) as f: + lines = f.read().splitlines() + assert len(lines) == 4 * 30 and all(line.endswith(' soft') for line in lines) + assert not os.path.exists(target + '.lock') + + +def test_a_stale_soft_lock_of_a_dead_process_is_removed(soft, caplog, monkeypatch): + gone = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + gone.wait() + with open(soft, 'w') as f: + f.write(f'{gone.pid} {files._lock_host_name()} {time.time():.3f}\n') + caplog.set_level('WARNING', logger='cmeta.utils.files') + + # Fresh, the file of a dead pid is a release in flight: waited for + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + + old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + assert any('stale' in rec.getMessage() for rec in caplog.records) + lock.release() + + +def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkeypatch): + # A holder that could not remove its file (a reader held it open on Windows for too long) marks it released + with open(soft, 'w') as f: + f.write('released 12345\n') + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + # A fresh lock file of a live holder is never taken for a stale one, whatever the check says + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its content at once): + # fresh, it is waited for; old, it is cleared + open(soft, 'w').close() + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 30.0) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def test_a_soft_lock_of_another_host_is_stale_only_after_the_age(soft, monkeypatch): + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def test_soft_lock_is_used_where_the_os_lock_is_refused(tmp_path, monkeypatch, caplog): + lock_file = str(tmp_path / 'refused.lock') + err = OSError(files.errno.ENOLCK, 'No locks available') + + def refuse(*args, **kwargs): + raise err + + if POSIX: + monkeypatch.setattr(files.fcntl, 'flock', refuse) + monkeypatch.setattr(files.fcntl, 'lockf', refuse) + else: + monkeypatch.setattr(files.msvcrt, 'locking', refuse) + monkeypatch.setattr(files, '_lock_soft_paths', set()) + monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) + caplog.set_level('WARNING', logger='cmeta.utils.files') + + lock = files.PathLock(lock_file).acquire(timeout=5) + assert lock.mode == 'soft' + assert lock_file in files._lock_soft_paths + assert any('soft lock' in rec.getMessage() for rec in caplog.records) + lock.release() + assert not os.path.exists(lock_file) + + again = files.PathLock(lock_file).acquire(timeout=5) # remembered: soft at once + assert again.mode == 'soft' + again.release() + + +@pytest.mark.skipif(not POSIX, reason='POSIX record locks are the POSIX fallback') +def test_posix_record_locks_are_used_where_flock_is_refused(tmp_path, monkeypatch): + lock_file = str(tmp_path / 'nfs-like.lock') + + def refuse(*args, **kwargs): + raise OSError(files.errno.ENOLCK, 'No locks available') + + monkeypatch.setattr(files.fcntl, 'flock', refuse) + monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) + lock = files.PathLock(lock_file).acquire(timeout=5) + assert lock.mode == 'lockf' + assert os.path.dirname(lock_file) in files._lock_no_flock_dirs + lock.release() + assert not os.path.exists(lock_file) + + +########################################################################################## +# The engine's functions + +def test_safe_delete_directory_takes_and_removes_its_lock(tmp_path): + folder = tmp_path / 'artifact' + folder.mkdir() + (folder / 'f.txt').write_text('x') + r = files.safe_delete_directory(str(folder)) + assert r['return'] == 0 + assert not folder.exists() + assert not os.path.exists(str(folder) + '.lock') + + +def test_safe_write_file_releases_the_lock_it_took(tmp_path): + target = str(tmp_path / 'data.json') + r = files.safe_write_file(target, {'a': 1}, atomic=True) + assert r['return'] == 0 + assert not os.path.exists(target + '.lock') and not os.path.exists(target + '.tmp') + r = files.safe_read_file(target, lock=True) + assert r['return'] == 0 and r['data'] == {'a': 1} + assert not os.path.exists(target + '.lock') diff --git a/tests/internal_repo_tests/test_integration_reindex_artifact.py b/tests/internal_repo_tests/test_integration_reindex_artifact.py index d908b89..da5a221 100644 --- a/tests/internal_repo_tests/test_integration_reindex_artifact.py +++ b/tests/internal_repo_tests/test_integration_reindex_artifact.py @@ -467,13 +467,11 @@ def test_the_shipped_internal_repository_is_only_read(cm): assert file_state(f) == state -@pytest.mark.xfail(sys.platform != 'win32', strict=False, - reason='the index lock file is deleted after every release (utils.files._release_lock -> ' - '_cleanup_lock_file): with flock a process waiting on the old inode and a newcomer that ' - 'creates a new lock file both acquire, two writers enter add_to_index and one update is ' - 'lost (seen on WSL2 and in a container, 2026-10-08; Windows cannot delete an open file). ' - 'The fix belongs to the concurrency step of the engine hardening') def test_several_processes_reindex_at_once(cm, tmp_path): + # Six processes write the same index file at once: with the previous lock (its file removed after the + # release) a waiter and a newcomer could both acquire on Linux and one record was lost; the engine's own + # PathLock (the file removed before the release, the identity of the file checked after acquiring) keeps + # every record on every platform base = tmp_path / 'repos' / 'local' / 'log' names = [f'par-{i}' for i in range(6)] made = {name: make(cm, name, yaml=True) for name in names} @@ -485,7 +483,7 @@ def test_several_processes_reindex_at_once(cm, tmp_path): env['PYTHONIOENCODING'] = 'utf-8' procs = [subprocess.Popen([sys.executable, '-m', 'cmeta', 'log', 'reindex', name], env=env, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) for name in names] + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) for name in names] outputs = [p.communicate(timeout=180)[0] for p in procs] assert all(p.returncode == 0 for p in procs), outputs @@ -830,13 +828,13 @@ def test_the_cli_route(cm, tmp_path): env['CMETA_HOME'] = str(tmp_path) env['PYTHONIOENCODING'] = 'utf-8' - r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'via-cli'], capture_output=True, text=True, env=env) + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'via-cli'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) assert r.returncode == 0, r.stdout + r.stderr assert f'Reindexed log "via-cli" ({uid})' in r.stdout assert record(cm, uid)['cmeta']['owner'] == 'me' - r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'nothing-here'], capture_output=True, text=True, env=env) + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'nothing-here'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) assert r.returncode == 16, r.stdout + r.stderr - r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', '--help'], capture_output=True, text=True, env=env) + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', '--help'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) assert r.returncode == 0 and 'ignore_errors' in r.stdout, r.stdout + r.stderr diff --git a/uv.lock b/uv.lock index 567a818..95531da 100644 --- a/uv.lock +++ b/uv.lock @@ -164,7 +164,6 @@ wheels = [ name = "cmeta" source = { editable = "." } dependencies = [ - { name = "filelock" }, { name = "packaging" }, { name = "psutil" }, { name = "pyyaml" }, @@ -206,7 +205,6 @@ requires-dist = [ { name = "cmeta", extras = ["async"], marker = "extra == 'server'" }, { name = "cmeta", extras = ["async", "dev", "server"], marker = "extra == 'all'" }, { name = "fastapi", marker = "extra == 'server'" }, - { name = "filelock" }, { name = "flake8", marker = "extra == 'dev'" }, { name = "httpx", marker = "extra == 'dev'" }, { name = "itsdangerous", marker = "extra == 'server'" }, @@ -378,15 +376,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3e/57/aa70121b5008f44031be645a61a7c4abc24e0e888ad3fc8fda916f4d188e/fastapi-0.124.4-py3-none-any.whl", hash = "sha256:6d1e703698443ccb89e50abe4893f3c84d9d6689c0cf1ca4fad6d3c15cf69f15", size = 113281, upload-time = "2025-12-12T15:00:42.44Z" }, ] -[[package]] -name = "filelock" -version = "3.19.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/40/bb/0ab3e58d22305b6f5440629d20683af28959bf793d98d11950e305c1c326/filelock-3.19.1.tar.gz", hash = "sha256:66eda1888b0171c998b35be2bcc0f6d75c388a7ce20c3f3f37aa8e96c2dddf58", size = 17687, upload-time = "2025-08-14T16:56:03.016Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/42/14/42b2651a2f46b022ccd948bca9f2d5af0fd8929c4eec235b8d6d844fbe67/filelock-3.19.1-py3-none-any.whl", hash = "sha256:d38e30481def20772f5baf097c122c3babc4fcdb7e14e57049eb9d88c6dc017d", size = 15988, upload-time = "2025-08-14T16:56:01.633Z" }, -] - [[package]] name = "flake8" version = "7.3.0" From e17ed091b7a92eceeb0c70697bb12fdb1c610280 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Fri, 9 Oct 2026 09:42:50 +0200 Subject: [PATCH 3/3] cMeta v0.34.0 Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- CHANGELOG.md | 2 +- CITATION.cff | 4 ++-- cmeta/internal-repo/_cmr.yaml | 4 ++-- cmeta/version.py | 4 ++-- cmeta/versions.json | 4 ++++ docs/en/api/index.rst | 4 ++-- docs/en/index.rst | 4 ++-- docs/history.md | 2 +- docs/installation.md | 4 ++-- 9 files changed, 18 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a599df..200682f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ All notable changes to cMeta are documented here, newest first. -## 0.33.1.1 (in development) +## 0.34.0 - **The engine's own path lock replaces the `filelock` library: two processes can no longer hold one lock at once, and lock files are still removed after use.** The previous lock removed its file after releasing it; with `flock` on Linux and macOS a process waiting on the removed file and a newcomer diff --git a/CITATION.cff b/CITATION.cff index 2df391d..6225fc3 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -13,8 +13,8 @@ authors: website: "https://cTuning.ai/@gfursin" affiliation: "cTuning Labs" orcid: "https://orcid.org/0000-0001-7719-1624" -version: "0.33.1" -date-released: "2026-10-07" +version: "0.34.0" +date-released: "2026-10-09" license: Apache-2.0 repository-code: "https://github.com/cTuningLabs/cmeta" url: "https://cTuning.ai" diff --git a/cmeta/internal-repo/_cmr.yaml b/cmeta/internal-repo/_cmr.yaml index 9e1ec50..f91330f 100644 --- a/cmeta/internal-repo/_cmr.yaml +++ b/cmeta/internal-repo/_cmr.yaml @@ -3,5 +3,5 @@ authors: "[Grigori Fursin](https://cTuning.ai/@gfursin)" category: repo,f4f792ab40c7498f copyright: "2025-2026 Grigori Fursin and cTuning Labs. See the cMeta COPYRIGHT and LICENSE files for details." permanent: True -version: '0.33.1' -version_date: '20261007' +version: '0.34.0' +version_date: '20261009' diff --git a/cmeta/version.py b/cmeta/version.py index f296d88..c3b0502 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.33.1.1" -__release_date__ = "2026-10-08" +__version__ = "0.34.0" +__release_date__ = "2026-10-09" diff --git a/cmeta/versions.json b/cmeta/versions.json index 1e76bbe..09868e6 100644 --- a/cmeta/versions.json +++ b/cmeta/versions.json @@ -89,6 +89,10 @@ { "version": "0.33.1", "version_date": "20261007" + }, + { + "version": "0.34.0", + "version_date": "20261009" } ] } diff --git a/docs/en/api/index.rst b/docs/en/api/index.rst index bba00dc..c409e27 100644 --- a/docs/en/api/index.rst +++ b/docs/en/api/index.rst @@ -1,7 +1,7 @@ -API Reference (v0.33.1) +API Reference (v0.34.0) ======================= -This section contains the complete API reference for cMeta v0.33.1. +This section contains the complete API reference for cMeta v0.34.0. .. toctree:: :maxdepth: 2 diff --git a/docs/en/index.rst b/docs/en/index.rst index 925ace5..3811c7f 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -1,11 +1,11 @@ cMeta Documentation =================== -Welcome to cMeta v0.33.1 documentation! +Welcome to cMeta v0.34.0 documentation! .. toctree:: :maxdepth: 2 - :caption: cMeta v0.33.1 Contents: + :caption: cMeta v0.34.0 Contents: home diff --git a/docs/history.md b/docs/history.md index 026edfd..1628e9a 100644 --- a/docs/history.md +++ b/docs/history.md @@ -82,7 +82,7 @@ produces APA and BibTeX automatically. The BibTeX entry: author = {Fursin, Grigori}, title = {{cMeta (Common Meta Framework)}}, year = {2026}, - version = {0.33.1}, + version = {0.34.0}, license = {Apache-2.0}, url = {https://github.com/cTuningLabs/cmeta}, note = {cTuning Labs} diff --git a/docs/installation.md b/docs/installation.md index 7d4edf2..44e62ea 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -157,8 +157,8 @@ repositories) into these commands. **A specific version**, to pin or to go back: ```bash -uv tool install --force "cmeta[server]==0.33.1" -pip install "cmeta==0.33.1" +uv tool install --force "cmeta[server]==0.34.0" +pip install "cmeta==0.34.0" ``` **Moving an install from git to PyPI** (or back) is a reinstall from the other