diff --git a/.claude/skills/add-cserver-plugin/SKILL.md b/.claude/skills/add-cserver-plugin/SKILL.md index 3ae48c6..73ec93d 100644 --- a/.claude/skills/add-cserver-plugin/SKILL.md +++ b/.claude/skills/add-cserver-plugin/SKILL.md @@ -47,12 +47,14 @@ that one artifact instead of rebuilding the whole home: ```bash cx category index :cserver. # index a hand-made folder -cx category update :cserver. # after editing its _cmeta.* meta +cx category reindex cserver. # after editing its _cmeta.* meta (nothing written) +cx category update :cserver. # the same, but it rewrites the file too +cx repo reindex # the whole repository, when several things changed in it ``` -`cx --reindex` walks every category and is slow — keep it for moves, bulk -`git pull`, or a wiped `CMETA_HOME`. **Editing `api/`, `files/` or `src/` needs -no reindex at all**; a server restart (or `--reload`) picks those up. +`cx --reindex` walks every category and is slow — keep it for a wiped +`CMETA_HOME` or a bulk change across repositories. **Editing `api/`, `files/` or +`src/` needs no reindex at all**; a server restart (or `--reload`) picks those up. Layout: diff --git a/.claude/skills/add-plugin/SKILL.md b/.claude/skills/add-plugin/SKILL.md index 8b94fda..8f7ce7f 100644 --- a/.claude/skills/add-plugin/SKILL.md +++ b/.claude/skills/add-plugin/SKILL.md @@ -340,13 +340,15 @@ narrowest command that covers it: ```bash cx index : # you created the folder by hand (mkdir + _cmeta.*) -cx update : # you edited an existing artifact's _cmeta.* meta +cx reindex # you edited, renamed or copied in an existing artifact's folder (nothing is written) +cx repo reindex # one repository: its _cmr.yaml and its artifacts +cx update : # you edited an existing artifact's _cmeta.* meta (rewrites the file too) cx --reindex # rebuilds ALL of /index/*.pkl — slow ``` -Prefer the per-artifact forms: `--reindex` walks every category and is markedly -slower on a large home. Save it for renames/moves, a bulk `git pull`, or a wiped -`CMETA_HOME`. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no +Prefer the narrow forms: `--reindex` walks every category and is markedly +slower on a large home. Save it for a wiped `CMETA_HOME` or a bulk change across +repositories. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no reindex — the index tracks meta, not content. Category-level content caches (task results, downloaded blobs) live under the `cache` category: @@ -382,7 +384,8 @@ Before reporting the plugin as done: `TypeError: foo() takes 2 positional arguments but ...`. Rename to `foo_`. - **Command name clashes with a Python builtin** (`list`, `type`, ...). Name the method `list__` — the CLI name is still `list`. -- **New category isn't found.** The artifact index is cached; run `cx --reindex`. +- **New category isn't found.** The artifact index is cached; run + `cx repo reindex ` (or `cx --reindex`). - **Base command not exposed.** The category's `_cmeta.yaml` sets `skip_base_category_commands: true` (e.g. `utils`). Remove that flag or reimplement the command. diff --git a/.claude/skills/add-repo/SKILL.md b/.claude/skills/add-repo/SKILL.md index fbae35e..0a9b9da 100644 --- a/.claude/skills/add-repo/SKILL.md +++ b/.claude/skills/add-repo/SKILL.md @@ -147,6 +147,7 @@ cx repo find # find a repo cx repo status # git status + remote URL for git-backed repos cx repo pull # git pull; also refreshes the index cx repo update # equivalent to pull +cx repo reindex # re-read its _cmr.yaml and rescan its artifacts (files copied in by other means) cx repo checkout # git checkout branch/tag/commit cx repo space # disk usage cx repo zip # dump to cmr--YYYYMMDD-HHMMSS.zip @@ -172,10 +173,11 @@ After adding a repo, sanity-check: - [ ] `/repos.json` contains the repo's path. If a repo is registered but its artifacts don't show up in `find`/`ls`, the -fast index is stale — force a rebuild: +fast index is stale — rescan that repository (or, as a last resort, everything): ```bash -cx --reindex +cx repo reindex # its _cmr.yaml and its artifacts; other repos untouched +cx --reindex # everything ``` --- @@ -198,8 +200,9 @@ cx --reindex does. - **Renaming a repo alias.** Not supported. Reference repos by `alias,UID` in cross-repo references so alias renames wouldn't matter anyway. -- **`_cmr.yaml` edited by hand and things look off.** `cx --reindex` to - rebuild `/index/*.pkl`. +- **`_cmr.yaml` edited by hand and things look off.** `cx repo reindex ` + re-reads it and rescans the repo's artifacts (or `cx --reindex` to rebuild + `/index/*.pkl`). - **`permanent: true` in `_cmr.yaml`.** `cx repo delete` refuses. Remove the flag first if you truly want to delete. - **`subdir:` in `_cmr.yaml`.** Only that sub-tree is scanned for artifacts — diff --git a/.claude/skills/use-cmeta-cli/SKILL.md b/.claude/skills/use-cmeta-cli/SKILL.md index 0bb326d..28f5e9c 100644 --- a/.claude/skills/use-cmeta-cli/SKILL.md +++ b/.claude/skills/use-cmeta-cli/SKILL.md @@ -349,20 +349,30 @@ cx ... --dump # capture full ctx ### 10.7 Reindex when things look stale — narrowest command first `cx --reindex` rebuilds *every* category pickle and gets slow as a home grows. -Reach for the per-artifact commands first; they touch one index entry: +Reach for the narrow commands first; they touch one index entry or one repository: ```bash cx index : # register ONE hand-made (mkdir-ed) artifact -cx update : # refresh entry after editing its _cmeta.* meta +cx reindex # refresh ONE record from its folder: a meta edited by hand, a folder + # renamed/moved/copied in; nothing is written on disk +cx reindex # every artifact of the category, hand-made folders included +cx repo reindex # ONE repository: its _cmr.yaml and its artifacts; other repos untouched +cx update : # refresh entry after editing its _cmeta.* meta (rewrites the file too) cx --reindex # whole-home rebuild — last resort ``` - `cx add :` already indexes as it creates — nothing - extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`). + extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`); + `reindex` takes those too, and an artifact the index knows already. +- `reindex` never touches a folder; the only record it removes is that of an + artifact whose folder now holds another artifact, and it says so. A folder it + cannot find, a meta it cannot read, or one UID in two folders is an error that + changes nothing. After a rename by hand the new name, the UID and the old name + all find the folder, and the old alias is dropped. - Editing only an artifact's **payload** (`api/`, `files/`, `src/`, `_desc.yaml`) needs no reindex at all — the index tracks meta, not content. -- Keep `--reindex` for the broad cases: bulk `git pull`, moved/renamed folders, - `CMETA_HOME` wipe, or an index that looks generally wrong. Safe, idempotent. +- Keep `--reindex` for the broad cases: a wiped `CMETA_HOME`, a bulk change + across repositories, or an index that looks generally wrong. Safe, idempotent. ## 11. Scripting the CLI diff --git a/.claude/skills/use-cmeta-python/SKILL.md b/.claude/skills/use-cmeta-python/SKILL.md index 80b6b63..054b43d 100644 --- a/.claude/skills/use-cmeta-python/SKILL.md +++ b/.claude/skills/use-cmeta-python/SKILL.md @@ -289,7 +289,11 @@ loaded = r['artifact']['loaded_files'] ### 6.5 Reindex on-demand ```python -cm.repos.reindex(con=False) # or: cm.access({'reindex': True}) +cm.repos.reindex(con=False) # or: cm.access({'reindex': True}) - everything, slow + +# One artifact's record from its folder (nothing written), one repository +r = cm.access({'category': 'note', 'command': 'reindex', 'arg1': 'my-note'}) # r['artifacts'], r['dropped_aliases'] +r = cm.access({'category': 'repo', 'command': 'reindex', 'arg1': 'myrepo'}) # its _cmr.yaml and its artifacts ``` ## 6.6 Use `cm.utils.*` from category code — **recommended** diff --git a/AGENTS.md b/AGENTS.md index fb72ea4..e0cc7c5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,8 @@ The commands above are cross-platform truth. Any Windows `.bat` wrappers around them are the author's local workflow and are not tracked in this repository. Runtime deps are intentionally minimal: pyyaml, requests, setuptools, wheel, -tabulate, tqdm, filelock, packaging, psutil. +tabulate, tqdm, packaging, psutil. (File locks are the engine's own +`cmeta.utils.files.PathLock` since 0.34.0; `filelock` is no longer used.) ### Useful environment variables `CMETA_HOME`, `CMETA_DEBUG=1`, `CMETA_LOG=DEBUG|INFO`, `CMETA_LOG_FILE=`, @@ -75,7 +76,7 @@ tabulate, tqdm, filelock, packaging, psutil. | `core.py` | `CMeta` class + `access()` — the one dispatch every request funnels through. Resolves the category artifact, loads its API module, calls the matching command function. | | `core_async.py` | Async variant of `access()`. | | `category.py` | `InitCategory` base class every category API subclasses. Provides `_prepare_input_from_params/ctx` helpers used to re-enter `cm.access()`. | -| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`info`/`tags`/`get`/`set`/`index`/`test`. | +| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`migrate`/`info`/`tags`/`get`/`set`/`index`/`reindex`/`test`. | | `repos.py` | `Repos` — index and resolve content repositories + artifacts (by alias / UID / tags / wildcards). Owns the per-category `.pkl` index files. | | `packages.py` | `Packages` — detect installed tools/versions and auto-install Python packages. | | `config.py` | Global `cfg` dict, param descriptors, env-var handling, command aliases (`add→create`, `rm→delete`, `ls→list`, `search→find`, `mv→move`, ...). | @@ -141,10 +142,17 @@ tabulate, tqdm, filelock, packaging, psutil. refreshes automatically on `create`/`update`/`delete` and on `cx repo` changes. If artifacts were touched outside cMeta, reindex with the narrowest command: `cx index :` registers a single folder you - created by hand (`mkdir` + `_cmeta.*`); `cx update :` - refreshes the entry after you edit an existing artifact's `_cmeta.*` meta; - `cx --reindex` rebuilds every pickle and is slow — keep it for manual moves, - bulk `git pull`, or a cleared `CMETA_HOME`. Payload-only edits (`api/`, + created by hand (`mkdir` + `_cmeta.*`); `cx reindex ` + rewrites the record of one artifact (or of several, with wildcards) from the + meta file in its folder without writing anything — a meta edited by hand, a + folder renamed, moved or copied in (its stale aliases are dropped, and the + record of a folder that now holds another artifact; a folder it cannot find or + one UID in two folders is an error that changes nothing); + `cx update :` refreshes the entry too but rewrites + the meta file (a new timestamp); `cx repo reindex ` rescans one + repository (its `_cmr.yaml` and its artifacts) and leaves the others alone; + `cx --reindex` rebuilds every pickle and is slow — keep it for a cleared + `CMETA_HOME` or a bulk change across repositories. Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) need no reindex. Categories with `no_index: true` in `_cmeta.yaml` are always found by filesystem scan. - **Content-addressed task caching / reproducibility.** Workflow caches keyed diff --git a/CHANGELOG.md b/CHANGELOG.md index 00ba4d0..200682f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,86 @@ All notable changes to cMeta are documented here, newest first. +## 0.34.0 +- **The engine's own path lock replaces the `filelock` library: two processes can no longer hold one + lock at once, and lock files are still removed after use.** The previous lock removed its file after + releasing it; with `flock` on Linux and macOS a process waiting on the removed file and a newcomer + that created a new one then both acquired, and one of two concurrent index writes was lost (seen with + six `cx log reindex` at once on WSL2 and in a container). `cmeta.utils.files.PathLock` removes the + file BEFORE releasing (after closing it on Windows, where an open file cannot be removed) and checks + after acquiring that the file it holds is still the file at the path, trying again otherwise. It takes + a byte-range lock (`msvcrt.locking`) on Windows, `fcntl.flock` on POSIX, POSIX record locks + (`fcntl.lockf`) where the file system refuses `flock` (NFS and some network mounts), and a soft lock + file (`pid host time`, created exclusively; a stale one - its process dead on this host, or older than + ten minutes when left by another host - is removed by the waiter that finds it) where the file system + refuses both (some FUSE, 9p and SMB mounts). The threads of one process are serialized on the path + first, so the lock is safe for threads as well as processes in every mode. A busy lock is polled every + 1 to 25 ms instead of every 50 ms. The timeouts, the lock file names and the functions that use them + (`safe_read_file(lock=True)`, `safe_write_file`, `safe_delete_directory`, `lock_path` / `unlock_path`) + are unchanged; `filelock` is no longer a dependency. Tests: `tests/core_tests/test_utils_files_path_lock.py` + (one holder at a time, the file removed on release, eight processes and sixteen threads through one + counter, the engine's read-modify-write cycle under contention, a killed holder, the removed-file race + replayed deterministically, a lock file that is not its own left alone, the soft lock and its stale + files, the POSIX record-lock fallback, timeouts, `safe_delete_directory` and `safe_write_file`); + `test_several_processes_reindex_at_once` passes on every platform (it was expected to fail on Linux). +- **`tests/benchmarks/benchmark_engine.py`: a benchmark of the engine's hot paths** - the index reads + behind `find`, the locked writes behind `create`, `update`, `delete` and `reindex`, the full reindex, the + reload after a change, the primitives (stat, lock, record copy, pickle, atomic write), concurrent creates + and the `cx` command line - on a home of its own with N hand-made artifacts; `--json` writes the numbers + and `--compare ` prints the ratio against an earlier run. A smoke test runs it in the suite. +- **`cx reindex []`: the index record of one artifact (or of several) is refreshed + from its folder, and no file is changed.** The meta file is read from the artifact's folder on disk + and the record in the category's index is rewritten from it: the alias is the folder's name, every + other alias that still pointed at the artifact's UID is dropped, and the path, the meta and the + sharding depth are what a full `cx --reindex` records. Use it after a meta was edited by hand, after + a folder was renamed, moved or copied in from another machine (bisync, a USB disk), or when a lookup + looks stale - instead of the whole `cx --reindex`, and instead of an empty `cx update`, + which refreshes the record but also rewrites the meta file (a new timestamp and a generator record, + which a bisynced folder then carries back). The artifact is named by its alias, UID, alias,UID or + repo:alias, with wildcards; `--tags` prunes; nothing names every artifact of the category. A folder + the index does not know yet (made or renamed by hand) is found on disk in the repositories; an + artifact whose indexed folder is gone is looked for by its UID in its repository, so the new name, + the UID or the old name all work after a rename by hand. No folder is touched, and the only record + `reindex` ever removes is that of an artifact whose folder, still there, holds another artifact now + (replaced by hand) - a record a `delete` would act on by removing the other artifact's folder - and + it says so. A folder that cannot be found, a meta that cannot be read or has no valid `artifact` + UID, a meta of another category, or one UID found in two existing folders is an error that leaves + the index as it was (`--ignore_errors` reindexes the other artifacts and reports these). `index`, + `update`, `move` and `migrate` keep their behaviour. Tests: `tests/internal_repo_tests/test_integration_reindex_artifact.py` + (a hand-edited meta, YAML and JSON, the record of a full reindex, renames by hand found by the new + name, the UID or the old name, a rename twice, a case-only rename, two spellings on a case-sensitive + file system, a folder replaced by another artifact, a folder moved and its old name taken, one UID + in two folders, a missing folder, an unreadable meta, an invalid UID, an upper-case UID, another + category's folder, a non-ASCII alias, wildcards, tags, a repository, a repository with `subdir`, + everything, a UID-named folder, a sharded category and a rename inside it, a `no_index` category, a + category without an index file yet, the record of a category itself, the shipped internal repository, + a migration stub, the CLI, several processes at once). +- **`cx repo reindex `: one registered repository is indexed again.** Its `_cmr.yaml` is + re-read and its artifacts are rescanned with the incremental index that `pull`, `plug` and `unzip` + use, and the records of the other repositories stay as they are. The repository is named by its + alias, UID or alias,UID (wildcards allowed), by the path of its folder (or of a folder inside it), + or by `.` for the repository of the current directory. A repository whose folder or `_cmr.yaml` is + not there (a detached drive) is refused and nothing is changed. `cx --reindex` stays the full + rebuild of every repository. Tests: `tests/internal_repo_tests/test_integration_repo_reindex.py`. +- **The incremental index drops the records of a category of which a repository holds no artifact + any more.** When a repository was indexed again (`pull`, `plug`, `unzip`, now `repo reindex`), the + records of its deleted artifacts were removed only from the index files of the categories in which + the repository still had an artifact: when its last artifact of a category was gone, that record + stayed and the lookup kept answering with a folder that did not exist. Every existing category + index now takes part in the removal. The full `cx --reindex` is unchanged. +- **A wildcard search in a `no_index` category returns only the folders the pattern names.** The scan + that finds the artifacts of such a category (and that `reindex` uses to find folders on disk) + applied the pattern only in sharded categories; in a plain one, `cx find "name-*"` + returned every artifact. The pattern is now matched against the folder names, case-insensitively, + as the sharded scan does; and an exact alias now finds every folder that spells it (two spellings + can coexist on a case-sensitive file system), not only the first. The same scan no longer stops + with a `KeyError` on a meta file that has no `category` key: the folder is skipped with a warning, + as a folder without an `artifact` key is. +- `Repos.add_to_index` takes `clean_aliases=True` to drop, in the same locked write, every other + alias of the alias index that still lists the UID being written; its result carries them as + `dropped_aliases`. The default is unchanged. + + ## 0.33.1 - **A command's output is decoded on any machine in any language, and never fails.** `utils.sys.run` captured output in the text mode of `subprocess`, which decodes with the locale's encoding and diff --git a/CITATION.cff b/CITATION.cff index 2df391d..6225fc3 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -13,8 +13,8 @@ authors: website: "https://cTuning.ai/@gfursin" affiliation: "cTuning Labs" orcid: "https://orcid.org/0000-0001-7719-1624" -version: "0.33.1" -date-released: "2026-10-07" +version: "0.34.0" +date-released: "2026-10-09" license: Apache-2.0 repository-code: "https://github.com/cTuningLabs/cmeta" url: "https://cTuning.ai" diff --git a/CLAUDE.md b/CLAUDE.md index 7144df0..3d78007 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,7 +48,8 @@ separate content repos (e.g. `cmeta-aops`). ## Conventions worth repeating - Runtime deps are intentionally minimal (pyyaml, requests, setuptools, wheel, - tabulate, tqdm, filelock, packaging, psutil). Justify any new one. + tabulate, tqdm, packaging, psutil). Justify any new one. File locks are the + engine's own `cmeta.utils.files.PathLock` (no `filelock`). - Preserve module docstring/copyright headers — never strip or rewrite them. New files get the standard header; new artifacts/categories record `authors` and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and diff --git a/README.md b/README.md index 804ea11..d6ca90a 100644 --- a/README.md +++ b/README.md @@ -560,16 +560,22 @@ filesystem. The framework rebuilds the index automatically when repos are added/removed via `cx repo`, and updates it in-place on `create`/`update`/`delete`. -If you edited artifacts outside cMeta (moved folders, edited `_cmeta.yaml` by -hand, `git pull`-ed a repo, cleared `CMETA_HOME`, or things simply look -inconsistent), force a full rebuild: +If you edited artifacts outside cMeta (moved or renamed folders, edited +`_cmeta.yaml` by hand, copied artifacts in from another machine, `git pull`-ed a +repo), refresh the narrowest thing — one artifact, one repository — or, after a +cleared `CMETA_HOME` or when things simply look inconsistent, rebuild everything: ```bash -cx --reindex +cx reindex # one artifact's record, from its folder; nothing is written on disk +cx repo reindex # one repository: its _cmr.yaml and its artifacts; the others untouched +cx --reindex # everything ``` -This is safe, idempotent, and typically fast. Categories that opt out with -`no_index: true` in `_cmeta.yaml` are found by scanning the filesystem instead. +All three are safe and idempotent; `reindex` never touches a folder, and the only +record it removes is that of an artifact whose folder now holds another artifact +(a folder it cannot find or one UID in two folders is an error that changes +nothing). Categories that opt out with `no_index: true` in `_cmeta.yaml` are +found by scanning the filesystem instead. Task caches produced by workflows live under the `cache` category and can be inspected/pruned with `cx cache show` / `cx cache clean` / `cx cache delete`. diff --git a/cmeta/category_api_v1.py b/cmeta/category_api_v1.py index d2df2f9..1d4e595 100644 --- a/cmeta/category_api_v1.py +++ b/cmeta/category_api_v1.py @@ -1683,6 +1683,322 @@ def index( return self.cm.access(p) + ############################################################ + def reindex_( + self, + ctx: dict, # cMeta context. + arg1: str = None, # Artifact alias, UID, alias,UID or repo:alias (wildcards allowed); nothing names every artifact of the category. + tags: str = None, # Prune the artifacts by tags, as in find (a tag with a leading "-" excludes). + sort: bool = True, # Sort the artifacts by alias and UID. + skip_uids: bool = False, # Skip UIDs when using wildcards. + ignore_errors: bool = False, # Go on past an artifact that cannot be reindexed and report it at the end. + ): + """ + Refresh the index record(s) of artifact(s) from their folders, changing no file. + + The meta file of each artifact is read from its folder on disk and the artifact's record in the + index of the category is rewritten from it: the alias is the folder's name, every other alias + that still pointed at the artifact's UID is dropped, and the path, the meta and the sharding + depth are what a full `cx --reindex` records. The meta file is read, never written (an empty + `update` rewrites it). Use it after a meta file was edited by hand, after a folder was renamed, + moved or copied in from another machine, or when a lookup looks stale - instead of the whole + `cx --reindex`. + + The artifact may be named by its alias, UID, alias,UID or repo:alias, with wildcards; nothing + names every artifact of the category. A folder the index does not know yet (made or renamed by + hand) is found on disk in the repositories; an artifact whose indexed folder is gone is looked + for by its UID in its repository. No folder is touched, and the only record ever removed is + that of an artifact whose folder, still there, holds another artifact now (replaced by hand) - + said in the output. A folder that cannot be found, a meta without a valid `artifact` UID, or a + UID found in two existing folders is an error, and the index is left as it was + (`--ignore_errors` reindexes the other artifacts and reports these). + + Args: + ctx (dict): cMeta context. + arg1 (str | None): Artifact alias, UID, alias,UID or repo:alias (wildcards allowed); nothing + names every artifact of the category. + tags (str | list | None): Prune the artifacts by tags, as in find (a leading "-" excludes). + sort (bool): Sort the artifacts by alias and UID. + skip_uids (bool): Skip UIDs when using wildcards. + ignore_errors (bool): Go on past an artifact that cannot be reindexed and report it at the end. + + Returns: + dict: A cMeta dictionary with the following keys: + - **return** (int): 0 if success, >0 if error (16: nothing found). + - **error** (str): Error message if `return > 0`. + - **artifacts** (list): The reindexed artifacts, as their new index records. + - **dropped_aliases** (dict): Per UID, the (lowercase) aliases dropped because they were not the folder's name. + - **dropped_records** (list): The records removed because their folder holds another artifact now (`ref`, `path`, `now`). + - **errors** (list): With `ignore_errors`, one dict per artifact that could not be reindexed (`ref`, `path`, `error`). + + Raises: + Exception: Propagated runtime errors, if any. + """ + + con = ctx.get('control',{}).get('con', False) + + ctx['control']['con'] = False + + category_cmeta = ctx['category_artifact']['cmeta'] + category_cmeta_ref_parts = ctx['category_artifact']['cmeta_ref_parts'] + + category_alias = category_cmeta_ref_parts['artifact_alias'] + category_uid = category_cmeta_ref_parts['artifact_uid'] + + if category_cmeta.get('no_index', False): + return {'return':1, 'error':f'category "{category_alias}" is not indexed (no_index) - there is nothing to reindex'} + + # The reference: an alias (with wildcards), a UID, both, a repository + artifact_obj_parts = {} + + if arg1 is not None and str(arg1).strip() != '': + r = utils.names.parse_cmeta_obj(arg1, key = "artifact", fail_on_error = self.fail_on_error) + if r['return'] >0: return r + + artifact_obj_parts = r['obj_parts'] + + artifact_alias = artifact_obj_parts.get('artifact_alias') + artifact_uid = artifact_obj_parts.get('artifact_uid') + artifact_repo_alias = artifact_obj_parts.get('artifact_repo_alias') + artifact_repo_uid = artifact_obj_parts.get('artifact_repo_uid') + + if tags is not None: + r = utils.common.normalize_tags(tags, fail_on_error = self.fail_on_error) + if r['return']>0: return r + + tags = r['tags'] + + # The repositories to look in: the one named, else all of them + repo_uids = None + + if (artifact_repo_alias is not None and artifact_repo_alias != '') or (artifact_repo_uid is not None and artifact_repo_uid != ''): + r = self.cm.repos.find_in_index('repo', self.cm.cfg['category_repo_uid'], artifact_repo_alias, artifact_repo_uid, only_uids=True, skip_uids=skip_uids) + if r['return']>0: return r + + repo_uids = r['artifact_uids'] + + def _norm(path): + return os.path.normcase(os.path.normpath(os.path.abspath(path))) + + def _same_folder(path1, path2): + # The same folder: the same path, or two spellings of one folder on a case-insensitive file system + if _norm(path1) == _norm(path2): + return True + try: + return os.path.isdir(path1) and os.path.isdir(path2) and os.path.samefile(path1, path2) + except OSError: + return False + + def _ref(parts): + alias = parts.get('artifact_alias') + uid = parts.get('artifact_uid', '') + return f'"{alias}" ({uid})' if alias else uid + + def _uid_in_folder(path): + # The UID the meta file of a folder names, or None (no folder, no meta, no valid UID) + if not os.path.isdir(path): + return None + r = utils.files.safe_read_yaml_or_json(os.path.join(path, self.cm.cfg['meta_filename_base']), fail_on_error=False, logger=self.logger) + if r['return']>0 or not isinstance(r.get('data'), dict): + return None + r = utils.names.parse_cmeta_name(r['data'].get('artifact')) + if r['return']>0: + return None + uid = r['name'].get('uid') + return uid if uid is not None and utils.names.is_valid_cmeta_uid(uid) else None + + # 1. The folders on disk that the reference names: the real folder name and the meta as it is now, + # built as the full reindex builds its records + r = self.cm.repos.find_in_file_system(category_cmeta, category_alias, category_uid, artifact_alias, artifact_uid, repo_uids = repo_uids) + if r['return']>0: return r + + entries = [e for e in r['artifacts'] if self._match_tags(e['cmeta'].get('tags', []), tags)] + + # 2. The records the index holds for the reference (a migration stub is reindexed as itself) + r = self.find_(ctx, arg1, tags, sort=False, add_index_file=True, skip_uids=skip_uids, follow_migrated=False) + if r['return']>0 and r['return']!=16: return r + + indexed = r.get('artifacts', []) if r['return'] == 0 else [] + + if len(entries) == 0 and len(indexed) == 0: + x = f'"{arg1}"' if arg1 is not None and str(arg1).strip() != '' else 'artifacts' + return {'return':16, 'error':f'{category_alias} {x} not found in the index or on disk'} + + seen_paths = {} + seen_uids = {} + + for e in entries: + seen_paths[_norm(e['path'])] = e + seen_uids.setdefault(e['cmeta_ref_parts']['artifact_uid'], []).append(e) + + errors = [] + stale = [] + + # An indexed artifact that was not found on disk by the reference: its folder was renamed or moved + # by hand, replaced by another artifact, or it is gone - look for its UID in its repository + for a in indexed: + parts = a['cmeta_ref_parts'] + uid = parts['artifact_uid'] + + if uid in seen_uids: + continue + + xrepo_uids = [parts['repo_uid']] if parts.get('repo_uid') else None + + r = self.cm.repos.find_in_file_system(category_cmeta, category_alias, category_uid, None, uid, repo_uids = xrepo_uids) + if r['return']>0: return r + + found = r['artifacts'] + + if len(found) == 0: + other = seen_paths.get(_norm(a['path'])) + + if other is not None: + # The folder is still there but holds another artifact now (replaced by hand): the record + # is provably stale, and a "delete" of it would remove the other artifact's folder - it is + # dropped, as a full reindex would drop it, and said so + stale.append({'ref': _ref(parts), 'path': a['path'], 'uid': uid, + 'index_file': a.get('index_file'), + 'alias_lowercase': parts.get('artifact_alias_lowercase', parts.get('artifact_alias')), + 'other': _ref(other['cmeta_ref_parts'])}) + elif os.path.isdir(a['path']): + # The folder is there, but its meta file cannot be read or has no valid artifact UID + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{a["path"]}", and the folder is there, but its meta file ' + f'cannot be read or has no valid "artifact" UID - fix the file, then reindex'}) + else: + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{a["path"]}" but no folder with its UID was found in its repository - ' + f'if it was moved elsewhere, reindex it by its new name; if it was deleted, remove it from the index with "delete"'}) + continue + + if len(found) > 1: + paths = ', '.join('"' + f['path'] + '"' for f in found) + errors.append({'ref': _ref(parts), 'path': a['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is in {len(found)} folders with the same UID: {paths} - give all but one a new UID first'}) + continue + + e = found[0] + + seen_paths[_norm(e['path'])] = e + seen_uids.setdefault(uid, []).append(e) + entries.append(e) + + # 3. Check everything before writing anything: one folder per UID, and a UID that the index holds at + # another folder that still exists is left alone (two folders with one UID) + r = self.cm.repos.find_in_index(category_alias, category_uid, only_uids=True, skip_uids=True) + if r['return']>0: return r + + index_uids = r.get('index', {}).get(self.cm.repos.KEY_INDEX_UIDS, {}) + + to_write = [] + + for uid in seen_uids: + uid_entries = seen_uids[uid] + parts = uid_entries[0]['cmeta_ref_parts'] + + if len(uid_entries) > 1: + paths = ', '.join('"' + x['path'] + '"' for x in uid_entries) + errors.append({'ref': _ref(parts), 'path': uid_entries[0]['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is in {len(uid_entries)} folders with the same UID: {paths} - give all but one a new UID first'}) + continue + + e = uid_entries[0] + + record = index_uids.get(uid) + if record is not None: + old_path = record.get('path', '') + # A conflict only when the folder the index knows still holds this very UID + if old_path and not _same_folder(old_path, e['path']) and _uid_in_folder(old_path) == uid: + errors.append({'ref': _ref(parts), 'path': e['path'], + 'error': f'{category_alias} artifact {_ref(parts)} is indexed at "{old_path}", which still exists, while the same UID is in "{e["path"]}" - ' + f'two folders with one UID: give one of them a new UID (or remove it), then reindex'}) + continue + + to_write.append(e) + + if len(errors) > 0 and not ignore_errors: + if len(errors) == 1: + return {'return':1, 'error': errors[0]['error']} + return {'return':1, 'error': 'nothing was reindexed:\n' + '\n'.join('* ' + x['error'] for x in errors)} + + if sort: + to_write.sort(key=lambda e: (str(e['cmeta_ref_parts'].get('artifact_alias', '')).lower(), e['cmeta_ref_parts']['artifact_uid'])) + + # 4. Rewrite the records, each in one locked write that also drops the stale aliases of its UID + reindexed = [] + dropped_aliases = {} + + for e in to_write: + parts = e['cmeta_ref_parts'] + uid = parts['artifact_uid'] + + r = self.cm.repos.add_to_index(e['cmeta'], parts, e['path'], original_uid = uid, + sharding_slices_num = e.get('sharding_slices_num', 0), clean_aliases = True) + if r['return']>0: return r + + record = {'path': e['path'], 'cmeta': e['cmeta'], 'cmeta_ref_parts': parts} + if e.get('sharding_slices_num'): + record['sharding_slices_num'] = e['sharding_slices_num'] + + reindexed.append(record) + + if len(r.get('dropped_aliases', [])) > 0: + dropped_aliases[uid] = r['dropped_aliases'] + + if con: + print (f'Reindexed {category_alias} {_ref(parts)}: {e["path"]}') + for x in r.get('dropped_aliases', []): + print (f' dropped the alias "{x}"') + + # 5. The records of artifacts whose folder now holds another artifact + dropped_records = [] + + for x in stale: + if x['index_file']: + r = self.cm.repos.remove_from_index(x['index_file'], x['uid'], x['alias_lowercase']) + if r['return']>0: return r + + dropped_records.append({'ref': x['ref'], 'path': x['path'], 'now': x['other']}) + + if con: + print (f'Dropped the record of {category_alias} {x["ref"]}: its folder {x["path"]} now holds {x["other"]}') + + if con: + for x in errors: + print (f'Skipped: {x["error"]}') + + return {'return':0, 'artifacts': reindexed, 'dropped_aliases': dropped_aliases, 'dropped_records': dropped_records, 'errors': errors} + + ############################################################ + def _match_tags( + self, + ctags, # The tags of an artifact's meta. + tags, # Normalized tags to match (a leading "-" excludes); None or empty matches everything. + ): + """ + Match the tags of an artifact as `find` does: every tag given must be present, none of the + excluded ones (a leading "-"), case-insensitively. + """ + + if tags is None or len(tags) == 0: + return True + + if type(ctags) != list: + return False + + ctags_lower = [str(t).lower() for t in ctags] + + for tag in tags: + tag = str(tag) + if tag.startswith('-'): + if tag[1:].lower() in ctags_lower: + return False + elif tag.lower() not in ctags_lower: + return False + + return True + ############################################################ def get_( self, diff --git a/cmeta/internal-repo/_cmr.yaml b/cmeta/internal-repo/_cmr.yaml index 9e1ec50..f91330f 100644 --- a/cmeta/internal-repo/_cmr.yaml +++ b/cmeta/internal-repo/_cmr.yaml @@ -3,5 +3,5 @@ authors: "[Grigori Fursin](https://cTuning.ai/@gfursin)" category: repo,f4f792ab40c7498f copyright: "2025-2026 Grigori Fursin and cTuning Labs. See the cMeta COPYRIGHT and LICENSE files for details." permanent: True -version: '0.33.1' -version_date: '20261007' +version: '0.34.0' +version_date: '20261009' diff --git a/cmeta/internal-repo/category/repo/api/v1.py b/cmeta/internal-repo/category/repo/api/v1.py index e37b3e6..cb3fb1e 100644 --- a/cmeta/internal-repo/category/repo/api/v1.py +++ b/cmeta/internal-repo/category/repo/api/v1.py @@ -1457,6 +1457,126 @@ def unplug( return self.delete(p) + ############################################################################################### + def reindex_( + self, + ctx: dict, # cMeta context. + arg1: str = None, # The repository: alias, UID or alias,UID (wildcards allowed), the path of its folder, or "." for the repository of the current directory. + ): + """ + Reindex registered repositories: re-read their `_cmr.yaml` and index their artifacts again, + as `pull`, `plug` and `unzip` do when they bring a repository in, leaving the records of the + other repositories as they are. Use it after files were copied or pulled into a repository by + other means, after its `_cmr.yaml` was edited, or when its artifacts look stale - instead of + `cx --reindex`, which rebuilds the index of every repository. + + A repository whose folder is not there (a detached drive) is refused and nothing is changed. + + Args: + ctx (dict): cMeta context. + arg1 (str): The repository: alias, UID or alias,UID (wildcards allowed), the path of its + folder, or "." for the repository of the current directory. Required. + + Returns: + dict: A cMeta dictionary with the following keys: + - **return** (int): 0 if success, >0 if error (16: no such repository). + - **error** (str): Error message if `return > 0`. + - **artifacts** (list): The repositories reindexed (their records as they were before). + - **paths** (list): Their registered paths. + + Raises: + Exception: Propagated runtime errors, if any. + """ + + con = ctx.get('control',{}).get('con', False) + verbose = ctx.get('control',{}).get('verbose', False) + + if arg1 is None or str(arg1).strip() == '': + return {'return':1, 'error':'name the repository to reindex: its alias, UID or path, or "." for the repository of the current directory ' + '(to rebuild the index of every repository, run "cx --reindex")'} + + arg1 = str(arg1).strip() + + repo_artifacts = [] + + if arg1 == '.' or os.path.isdir(arg1): + # A folder: the registered repository that holds it (the deepest one, for nested repositories) + path = os.path.abspath(os.path.normpath(os.getcwd() if arg1 == '.' else arg1)) + + r = self.cm.repos.find_in_index('repo', self.cm.cfg['category_repo_uid']) + if r['return']>0: return r + + best = None + best_path = None + + for repo in r['artifacts']: + repo_path = os.path.abspath(os.path.normpath(repo['path'])) + + if utils.files.is_path_within(repo_path, path): + if best is None or len(repo_path) > len(best_path): + best = repo + best_path = repo_path + + if best is None: + return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + + repo_artifacts = [best] + + else: + p = {'category': ctx['category'], + 'command': 'find', + 'arg1': arg1, + 'sort': False, + 'base': True, + 'con': False} + + r = self.cm.access(p) + if r['return']>0: return r + + repo_artifacts = r['artifacts'] + + # Only a registered path can be reindexed, and only one whose folder is there: the incremental + # index drops a repository whose descriptor it cannot read (unless the repository is kept) + repos_config_path = self.cm.repos_config_path + + r = utils.files.safe_read_file(repos_config_path, lock=False, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + repos_paths = r['data'] + + paths = [] + + for repo in repo_artifacts: + repo_cmeta_ref_parts = repo['cmeta_ref_parts'] + repo_alias = repo_cmeta_ref_parts.get('artifact_alias', repo_cmeta_ref_parts['artifact_uid']) + + path = repo['path'] + + if path not in repos_paths: + caller = ctx.get('origin',{}).get('cli',{}).get('caller','') + return {'return':1, 'error':f'repository "{repo_alias}" is indexed with the path "{path}" that is not in {repos_config_path} - try "{caller} --reindex"'} + + if not os.path.isdir(path) or not os.path.isfile(os.path.join(path, self.cm.cfg['repo_meta_desc'])): + return {'return':1, 'error':f'the folder of repository "{repo_alias}" or its {self.cm.cfg["repo_meta_desc"]} is not there: {path} (a detached drive?) - nothing was reindexed'} + + paths.append(path) + + if con: + print ('') + + r = self.cm.repos.index(clean=False, con=con, verbose=verbose, add_repo_paths=paths) + if r['return']>0: return r + + if con: + for repo, path in zip(repo_artifacts, paths): + repo_cmeta_ref_parts = repo['cmeta_ref_parts'] + repo_alias = repo_cmeta_ref_parts.get('artifact_alias', '') + repo_uid = repo_cmeta_ref_parts['artifact_uid'] + + print (f'Reindexed repository "{repo_alias}" ({repo_uid}): {path}') + + return {'return':0, 'artifacts': repo_artifacts, 'paths': paths} + ############################################################################################### def space( self, diff --git a/cmeta/repos.py b/cmeta/repos.py index 6bb7aa6..eb7ae58 100644 --- a/cmeta/repos.py +++ b/cmeta/repos.py @@ -158,6 +158,7 @@ def add_to_index( original_alias = None, # Previous artifact alias before update. original_uid = None, # Previous artifact UID before update. sharding_slices_num = 0, # Number of sharding slices + clean_aliases = False, # If True, drop every other alias of this UID from the alias index (a folder renamed by hand), in the same locked write. ): """ Insert or update an artifact record in the category index. @@ -169,8 +170,11 @@ def add_to_index( original_alias: Previous artifact alias before update. original_uid: Previous artifact UID before update. sharding_slices_num (int): Number of UID slices used to shard artifact paths. + clean_aliases (bool): If True, every alias of the alias index that still lists this UID + and is not the alias of the record is dropped (the stale aliases a folder renamed by + hand leaves behind), in the same locked write. Aliases of other UIDs are not touched. Returns: - dict: Operation result. + dict: Operation result, with 'dropped_aliases': the (lowercase) aliases dropped by `clean_aliases`. Raises: Exception: Propagated runtime errors, if any. """ @@ -215,7 +219,9 @@ def add_to_index( uids[artifact_uid] = record - if artifact_alias is not None: + artifact_alias_lowercase = None + + if artifact_alias is not None: artifact_alias_lowercase = artifact_alias.lower() lowercase_alias_uids = lowercase_aliases.get(artifact_alias_lowercase, []) @@ -224,6 +230,24 @@ def add_to_index( lowercase_alias_uids.append(artifact_uid) lowercase_aliases[artifact_alias_lowercase] = lowercase_alias_uids + # The other aliases that still point at this UID (a folder renamed by hand keeps its old + # alias in the index): only this UID is removed from them, other UIDs under the same alias stay + dropped_aliases = [] + + if clean_aliases: + for lowercase_alias in list(lowercase_aliases.keys()): + if lowercase_alias == artifact_alias_lowercase: + continue + + lowercase_alias_uids = lowercase_aliases[lowercase_alias] + + if artifact_uid in lowercase_alias_uids: + lowercase_alias_uids.remove(artifact_uid) + dropped_aliases.append(lowercase_alias) + + if len(lowercase_alias_uids) == 0: + del(lowercase_aliases[lowercase_alias]) + # Use atomic write to avoid corrupting large index files r = utils.files.safe_write_file(index_file, index_data, file_lock=index_file_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r @@ -231,7 +255,7 @@ def add_to_index( # r = utils.files.safe_write_file(os.path.splitext(index_file)[0] + ".json", index_data, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger) # if r['return']>0: return r - return {'return':0} + return {'return':0, 'dropped_aliases': dropped_aliases} @@ -1311,13 +1335,26 @@ def _index( path_to_category = os.path.join(repo_full_path, category) - r = self._find_artifacts(repo_meta, repo_alias, repo_uid, category_meta, category_alias, category_uid, path_to_category, + r = self._find_artifacts(repo_meta, repo_alias, repo_uid, category_meta, category_alias, category_uid, path_to_category, con, conx, index_artifacts, artifact_num) if r['return'] >0: return r an = r['artifact_num'] artifact_num = an + if not clean: + # The records of the repositories being indexed again are removed from every existing + # category index below, also from the index of a category of which such a repository + # holds no artifact any more (otherwise the records of its deleted artifacts would stay) + for category_mix in categories: + category = category_mix['category'] + + if category in ['category', 'repo'] or category_mix['meta'].get('no_index', False): + continue + + if category not in index_artifacts and os.path.isfile(os.path.join(index_path, category + self.index_extension)): + index_artifacts[category] = {self.KEY_INDEX_UIDS:{}, self.KEY_INDEX_LOWERCASE_ALIASES:{}} + else: for category_mix in categories: index_artifacts[category_mix['category']] = {} @@ -1491,7 +1528,8 @@ def _find_artifacts( if sharding_slices is not None: tmp_artifact_dirs = _get_artifacts_from_sharded_path(path_to_category, sharding_slices, artifact_alias=artifact_alias) else: - tmp_artifact_dirs = os.listdir(path_to_category) + # Only the folders the pattern names (case-insensitively, as the sharded search does) + tmp_artifact_dirs = [d for d in os.listdir(path_to_category) if fnmatch.fnmatch(d.lower(), artifact_alias_lowercase)] else: last_artifact_dirs_parent = None @@ -1507,11 +1545,11 @@ def _find_artifacts( path_to_category_with_shard = os.path.join(path_to_category, last_artifact_dirs_parent) if os.path.isdir(path_to_category_with_shard): + # Every folder that spells the alias (two spellings can coexist on a case-sensitive file system) for artifact_dir in os.listdir(path_to_category_with_shard): if artifact_dir.lower() == artifact_alias_lowercase: tmp_dir = artifact_dir if last_artifact_dirs_parent is None else os.path.join(last_artifact_dirs_parent, artifact_dir) tmp_artifact_dirs.append(tmp_dir) - break for long_artifact in tmp_artifact_dirs: @@ -1550,6 +1588,13 @@ def _find_artifacts( if artifact_meta: if index_artifacts is None: + # A meta without a category (or with one that does not parse) is not an artifact of this category + if 'category' not in artifact_meta: + print ('', flush=True) + print (f" Warning: {artifact} doesn't have 'category' key in {path_to_artifact}") + self.logger.error (f" Warning: {artifact} doesn't have proper 'category' key in {path_to_artifact}") + continue + r = utils.names.parse_cmeta_name(artifact_meta['category']) if r['return']>0: return r diff --git a/cmeta/utils/files.py b/cmeta/utils/files.py index 20ae9aa..8f00e6f 100644 --- a/cmeta/utils/files.py +++ b/cmeta/utils/files.py @@ -18,6 +18,10 @@ import hashlib import fnmatch import re +import errno +import logging +import random +import threading from .common import _error @@ -27,11 +31,24 @@ yaml = None # YAML is optional try: - from filelock import FileLock + import fcntl # POSIX: flock and POSIX record locks except ImportError: - raise ImportError("filelock library is required. Install with: pip install filelock") + fcntl = None + +try: + import msvcrt # Windows: byte-range locks +except ImportError: + msvcrt = None LOCK_SUFFIX = ".lock" +LOCK_POLL_START = 0.001 # the first wait when a lock is busy; it doubles up to LOCK_POLL_MAX +LOCK_POLL_MAX = 0.025 +LOCK_SOFT_STALE_SECONDS = 600 # a soft lock file older than this, whose holder is on another host, is stale +LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file older than this is a leftover (a holder writes its content at once) +LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock file of a dead pid younger than this is a release in flight, not a crash +LOCK_SOFT_CHECK_SECONDS = 0.25 # how often a waiter on a soft lock reads the file to check for a stale holder +LOCK_SOFT_REMOVE_SECONDS = 2.0 # how long a releasing soft lock retries the removal of its file (Windows: a reader may hold it) +LOCK_IDENTITY_RETRIES = 50 # consecutive identity mismatches after which a path is locked with the soft lock ERROR_CODE_FILE_NOT_FOUND = 16 RETRY_DELAY = 0.1 RETRY_NOT_FOUND_FILE = 10 @@ -108,6 +125,488 @@ def is_path_within( return common == base +########################################################################################## +# The path lock +# +# One exclusive lock per path for processes and threads alike, through the sidecar file `.lock`: +# the lock file is created for the operation and removed by the process that releases it, so a repository +# never keeps lock files (a crash leaves at most the one of the operation in flight, which the next holder +# reuses and removes - every lock dies with its process). +# +# Why not a library: a lock file that is removed after release is only correct when the holder removes +# it BEFORE releasing and every newcomer checks, after acquiring, that the file it holds is still the +# file at the path (otherwise a waiter blocked on the removed file and a newcomer that created a new +# one both hold "the lock" - the lost updates seen on Linux with the previous library + removal). That +# needs the lock's own descriptor, which libraries do not expose. +# +# The ladder, per platform: +# Windows: msvcrt.locking (a byte-range lock, per handle, mandatory) on the lock file; an open file +# cannot be removed there, so the identity check is not needed - a newcomer that finds the +# file "being deleted" (PermissionError) simply tries again. +# POSIX: fcntl.flock (per open file description, so threads of one process exclude each other too), +# then fcntl.lockf (POSIX record locks: NFS and some network mounts refuse flock), each with +# the identity check (fstat of the descriptor vs stat of the path, inode and device); +# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time") where the file +# system refuses both OS locks (some FUSE, 9p and SMB mounts) or the identity check keeps +# failing - a stale soft lock (its pid dead on this host, or older than LOCK_SOFT_STALE_SECONDS +# when left by another host) is renamed away and removed by the waiter that finds it. +# In every mode the threads of one process are serialized on the path first (a threading.Lock per path), +# which POSIX record locks and soft locks need (they are per process), and the busy wait polls with a +# delay that doubles from LOCK_POLL_START to LOCK_POLL_MAX until the timeout. + +_LOCK_BUSY_ERRNOS = tuple(e for e in (getattr(errno, name, None) for name in ('EWOULDBLOCK', 'EAGAIN', 'EACCES', 'EDEADLK', 'EDEADLOCK')) if e is not None) +_LOCK_UNSUPPORTED_ERRNOS = tuple(e for e in (getattr(errno, name, None) for name in ('ENOLCK', 'ENOSYS', 'EOPNOTSUPP', 'ENOTSUP', 'EINVAL', 'ENOTTY', 'EPERM', 'EBADF')) if e is not None) + +_lock_registry = {} # normalized lock file path -> threading.Lock (the threads of this process) +_lock_registry_guard = threading.Lock() +_lock_soft_paths = set() # lock files locked with the soft lock from now on (the OS lock was refused) +_lock_no_flock_dirs = set() # folders where flock was refused (POSIX record locks are tried first there) +_lock_host = None + + +def _lock_thread_lock(lock_file): + key = os.path.normcase(os.path.abspath(lock_file)) + with _lock_registry_guard: + lock = _lock_registry.get(key) + if lock is None: + lock = threading.Lock() + _lock_registry[key] = lock + return lock + + +def _lock_host_name(): + global _lock_host + if _lock_host is None: + try: + import platform + _lock_host = platform.node() or 'unknown-host' + except Exception: + _lock_host = 'unknown-host' + return _lock_host + + +def _lock_pid_alive(pid): + try: + import psutil + return psutil.pid_exists(pid) + except Exception: + return True # unknown: never treat it as dead + + +class _LockUnsupported(Exception): + pass + + +class PathLock: + """ + An exclusive lock on a path for processes and threads, through the sidecar file `.lock` + (see the notes above). Use `acquire(timeout)` / `release()` or the context manager; `is_locked` + says whether this object holds the lock; `mode` is how it was taken ('windows', 'flock', 'lockf' + or 'soft'); `identity_retries` counts the times a replaced lock file was detected and retried. + """ + + def __init__( + self, + lock_file: str, # The lock file (normally `.lock`, see _get_lockfile_path). + logger = None, # Optional logger for debug and warning messages. + ): + self.lock_file = lock_file + self.logger = logger if logger is not None else logging.getLogger('cmeta.utils.files') + self.is_locked = False + self.mode = None + self.identity_retries = 0 + self._fd = None + self._ident = None + self._thread_lock = None + self._mismatches = 0 + self._last_error = None + self._next_stale_check = 0.0 + + def __enter__(self): + self.acquire() + return self + + def __exit__(self, exc_type, exc, tb): + self.release() + + def __del__(self): + try: + if self.is_locked: + self.release() + except Exception: + pass + + def acquire( + self, + timeout: float = 3, # Seconds to wait for the lock (0: one try). + ): + """Take the lock or raise TimeoutError; returns self.""" + timeout = max(float(timeout), 0.0) + thread_lock = _lock_thread_lock(self.lock_file) + if not thread_lock.acquire(timeout=timeout): + raise TimeoutError(f"the lock {self.lock_file} is held by another thread of this process (waited {timeout} s)") + + try: + deadline = time.monotonic() + timeout + delay = LOCK_POLL_START + while True: + if self._try(): + self._thread_lock = thread_lock + self.is_locked = True + self.logger.debug(f"utils.files.PathLock - acquired {self.lock_file} ({self.mode})") + return self + + remaining = deadline - time.monotonic() + if remaining <= 0: + why = f" ({self._last_error})" if self._last_error is not None else "" + raise TimeoutError(f"the lock {self.lock_file} is held by another process (waited {timeout} s){why}") + + time.sleep(min(delay, remaining)) + delay = min(delay * 2, LOCK_POLL_MAX) + + except BaseException: + thread_lock.release() + raise + + def release(self): + """Release the lock and remove its file (when the file at the path is still the one this lock holds).""" + if not self.is_locked: + return + + try: + if self.mode == 'windows': + try: + msvcrt.locking(self._fd, msvcrt.LK_UNLCK, 1) + except OSError: + pass + os.close(self._fd) + self._unlink_own_file() + + elif self.mode in ('flock', 'lockf'): + self._unlink_own_file() + try: + self._posix_unlock(self._fd, self.mode) + except OSError: + pass + os.close(self._fd) + + elif self.mode == 'soft': + self._remove_soft_file() + + finally: + self._fd = None + self._ident = None + self.is_locked = False + self.mode = None + thread_lock = self._thread_lock + self._thread_lock = None + if thread_lock is not None: + thread_lock.release() + + # ---- one attempt ---------------------------------------------------------------------- + + def _try(self): + if self.lock_file in _lock_soft_paths: + return self._try_soft() + if msvcrt is not None: + return self._try_windows() + if fcntl is not None: + return self._try_posix() + return self._try_soft() + + def _folder_is_there(self): + return os.path.isdir(os.path.dirname(self.lock_file) or '.') + + def _open_failed_for_the_moment(self, e): + """An open of the lock file that failed for a moment: the file being removed by the process that + just released it (Windows: PermissionError; 9p and other network mounts: ENOENT with the folder + there) or held by a scanner. A missing folder is a real error.""" + if isinstance(e, PermissionError): + self._last_error = e + return True + if isinstance(e, FileNotFoundError) and self._folder_is_there(): + self._last_error = e + return True + return False + + def _try_windows(self): + try: + fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT, 0o644) + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + except OSError as e: + os.close(fd) + if e.errno in _LOCK_BUSY_ERRNOS: + self._last_error = e + return False + self._fall_back_to_soft(e) + return self._try_soft() + + try: + self._ident = self._identity(os.fstat(fd)) + except OSError: + self._ident = None + self._fd = fd + self.mode = 'windows' + return True + + def _try_posix(self): + try: + fd = os.open(self.lock_file, os.O_RDWR | os.O_CREAT | getattr(os, 'O_CLOEXEC', 0), 0o644) + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + kind = self._posix_lock(fd) + except _LockUnsupported as e: + os.close(fd) + self._fall_back_to_soft(e) + return self._try_soft() + except BaseException: + os.close(fd) + raise + + if kind is None: + os.close(fd) + return False + + # The file this descriptor holds must still be the file at the path: a holder that removed + # the file and released it leaves a waiter with a lock on a file that is not there any more + try: + path_ident = self._identity(os.stat(self.lock_file)) + except OSError: + path_ident = None + fd_ident = self._identity(os.fstat(fd)) + + if fd_ident is not None and (path_ident is None or path_ident != fd_ident): + try: + self._posix_unlock(fd, kind) + except OSError: + pass + os.close(fd) + self.identity_retries += 1 + self._mismatches += 1 + if self._mismatches >= LOCK_IDENTITY_RETRIES: + self._fall_back_to_soft(f"the identity of the lock file could not be confirmed {self._mismatches} times") + return self._try_soft() + return False + + self._mismatches = 0 + self._fd = fd + self._ident = fd_ident + self.mode = kind + return True + + def _posix_lock(self, fd): + """flock, else POSIX record lock; 'flock' | 'lockf' when taken, None when busy, _LockUnsupported when refused.""" + folder = os.path.dirname(self.lock_file) + + if folder not in _lock_no_flock_dirs: + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + return 'flock' + except OSError as e: + if e.errno in _LOCK_BUSY_ERRNOS or e.errno == errno.EINTR: + self._last_error = e + return None + if e.errno not in _LOCK_UNSUPPORTED_ERRNOS: + raise + _lock_no_flock_dirs.add(folder) + self.logger.info(f"utils.files.PathLock - flock is not supported for {self.lock_file} ({e}): POSIX record locks are used in that folder") + + try: + fcntl.lockf(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + return 'lockf' + except OSError as e: + if e.errno in _LOCK_BUSY_ERRNOS or e.errno == errno.EINTR: + self._last_error = e + return None + if e.errno in _LOCK_UNSUPPORTED_ERRNOS: + raise _LockUnsupported(str(e)) + raise + + @staticmethod + def _posix_unlock(fd, kind): + if kind == 'flock': + fcntl.flock(fd, fcntl.LOCK_UN) + else: + fcntl.lockf(fd, fcntl.LOCK_UN) + + def _try_soft(self): + try: + fd = os.open(self.lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) + except FileExistsError as e: + self._last_error = e + now = time.monotonic() + if now >= self._next_stale_check: + self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS + self._clear_stale_soft_lock() + return False + except OSError as e: + if self._open_failed_for_the_moment(e): + return False + raise + + try: + os.write(fd, (f"{os.getpid()} {_lock_host_name()} {time.time():.3f}\n").encode('utf-8', 'replace')) + finally: + os.close(fd) + + try: + self._ident = self._identity(os.stat(self.lock_file)) + except OSError: + self._ident = None + self.mode = 'soft' + return True + + def _clear_stale_soft_lock(self): + """A soft lock file whose holder is dead (this host; the file at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS + old, so that a holder that just released and exited is not mistaken for a crash), that was marked + released by a holder that could not remove it, that is empty and old (a leftover) or too old (another + host) is renamed away and removed - by one waiter only (the rename is atomic), and only when the file + at the path is still the file that was judged (a fresh lock file of a live holder moved by mistake in + the instant between the check and the rename is put back).""" + try: + fd = os.open(self.lock_file, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + except OSError: + return + try: + st = os.fstat(fd) + words = os.read(fd, 256).decode('utf-8', 'replace').split() if st.st_size > 0 else [] + except OSError: + return + finally: + os.close(fd) + + ident = self._identity(st) + age = time.time() - st.st_mtime + pid = int(words[0]) if len(words) > 0 and words[0].isdigit() else None + host = words[1] if len(words) > 1 else None + + if not words: + stale = age > LOCK_SOFT_EMPTY_STALE_SECONDS + elif words[0] == 'released': + stale = True + elif pid is not None and host == _lock_host_name(): + stale = age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) + else: + stale = age > LOCK_SOFT_STALE_SECONDS + + if not stale: + return + + try: + if ident is not None and self._identity(os.stat(self.lock_file)) != ident: + return # the file changed under us: judge it again next time + except OSError: + return + + moved = f"{self.lock_file}.stale-{os.getpid()}-{random.getrandbits(32):08x}" + try: + os.rename(self.lock_file, moved) # atomic: only one waiter gets to remove it + except OSError: + return + + if ident is not None: + try: + moved_ident = self._identity(os.stat(moved)) + except OSError: + return + if moved_ident != ident: + # Not the file judged stale but a fresh one made in between: back where it was, without + # clobbering (a hard link fails where a file exists already), then the extra name goes + try: + os.link(moved, self.lock_file) + except OSError: + try: + if not os.path.exists(self.lock_file): + os.rename(moved, self.lock_file) + except OSError: + pass + return + try: + os.unlink(moved) + except OSError: + pass + return + + try: + os.unlink(moved) + except OSError: + pass + self.logger.warning(f"utils.files.PathLock - removed the stale lock file {self.lock_file} (left by pid {pid} on {host})") + + def _fall_back_to_soft(self, why): + if self.lock_file not in _lock_soft_paths: + _lock_soft_paths.add(self.lock_file) + self.logger.warning(f"utils.files.PathLock - the file system refuses OS locks for {self.lock_file} ({why}): a soft lock file is used instead (best effort between hosts)") + # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its + # content at once): out of the way, or the soft lock could never be taken + try: + if os.path.getsize(self.lock_file) == 0: + os.unlink(self.lock_file) + except OSError: + pass + + def _remove_soft_file(self): + """Remove the soft lock file; on Windows a waiter reading it holds it open for a moment, so the removal + is retried, and a file that cannot be removed in time is marked released for the waiters to clear.""" + deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS + while True: + if self._ident is not None: + try: + if self._identity(os.stat(self.lock_file)) != self._ident: + return # not ours any more + except OSError: + return + try: + os.unlink(self.lock_file) + return + except FileNotFoundError: + return + except PermissionError: + if time.monotonic() >= deadline: + break + time.sleep(0.002) + except OSError: + break + + try: + with open(self.lock_file, 'w') as f: + f.write(f"released {os.getpid()}\n") + except OSError: + pass + self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") + + @staticmethod + def _identity(st): + """The (inode, device) of a stat result; None where the file system reports no inode (some FAT drivers).""" + if not st.st_ino: + return None + return (st.st_ino, st.st_dev) + + def _unlink_own_file(self): + """Remove the lock file, unless the file at the path is no longer the one this lock took (someone + removed it under us and another process created a new one: that one is theirs).""" + if self._ident is not None: + try: + if self._identity(os.stat(self.lock_file)) != self._ident: + return + except OSError: + return + try: + os.unlink(self.lock_file) + except OSError: + pass + + ########################################################################################## def _acquire_lock( filepath: str, # Path to the file to lock. @@ -115,10 +614,8 @@ def _acquire_lock( logger = None, # Optional logger for debug messages. ): """ - Acquire a file lock for cross-platform thread/process-safe file operations. - - Uses FileLock library to create and acquire a lock file. Blocks until lock - is acquired or timeout expires. + Acquire the lock of a path (`.lock`, see PathLock) for process- and thread-safe + file operations. Blocks until the lock is acquired or the timeout expires. Args: filepath (str): Path to the file to lock. @@ -126,7 +623,7 @@ def _acquire_lock( logger: Optional logger for debug messages. Returns: - FileLock: Acquired lock object that must be released later. + PathLock: Acquired lock object that must be released later (`_release_lock`). Raises: TimeoutError: If lock cannot be acquired within timeout period. @@ -134,8 +631,8 @@ def _acquire_lock( lockfile = _get_lockfile_path(filepath) if logger is not None: logger.debug(f"utils.files._acquire_lock - attempting to create {lockfile} ...") - - file_lock = FileLock(lockfile, timeout=timeout) + + file_lock = PathLock(lockfile, logger=logger) try: file_lock.acquire(timeout=timeout) if logger is not None: @@ -149,7 +646,7 @@ def _acquire_lock( ########################################################################################## def _check_lock( filepath: str, # Path to the locked file. - file_lock, # FileLock object to check. + file_lock, # PathLock object to check. logger = None, # Optional logger for debug messages. ): """ @@ -157,7 +654,7 @@ def _check_lock( Args: filepath (str): Path to the locked file. - file_lock: FileLock object to check. + file_lock: PathLock object to check. logger: Optional logger for debug messages. Raises: @@ -176,15 +673,15 @@ def _check_lock( ########################################################################################## def _release_lock( filepath: str, # Path to the locked file. - file_lock, # FileLock object to release. + file_lock, # PathLock object to release. logger = None, # Optional logger for debug messages. ): """ - Release a previously acquired file lock. + Release a previously acquired file lock; the lock file is removed by the lock itself. Args: filepath (str): Path to the locked file. - file_lock: FileLock object to release. + file_lock: PathLock object to release. logger: Optional logger for debug messages. Returns: @@ -193,7 +690,7 @@ def _release_lock( Exception: Propagated runtime errors, if any. """ lockfile = _get_lockfile_path(filepath) - + try: if file_lock.is_locked: file_lock.release() @@ -202,48 +699,12 @@ def _release_lock( else: if logger is not None: logger.debug(f"utils.files._release_lock - lock {lockfile} doesn't exist - weird but keep running ...") - - # Explicitly clean up lock file - filelock doesn't always do this on Linux - _cleanup_lock_file(lockfile, logger) - + except Exception as e: if logger is not None: logger.debug(f"utils.files._release_lock - error releasing lock {lockfile} ({str(e)})") - - # Still try to cleanup lock file even if release failed - _cleanup_lock_file(lockfile, logger) raise e -########################################################################################## -def _cleanup_lock_file( - lockfile_path: str, # Path to the lock file to remove. - logger = None, # Optional logger for debug messages. -): - """ - Manually remove lock file if it exists. - - Ensures cleanup on systems where filelock doesn't auto-cleanup. - Errors are logged but not raised since this is a cleanup operation. - - Args: - lockfile_path (str): Path to the lock file to remove. - logger: Optional logger for debug messages. - - Returns: - dict: Operation result. - Raises: - Exception: Propagated runtime errors, if any. - """ - try: - if os.path.exists(lockfile_path): - os.remove(lockfile_path) - if logger is not None: - logger.debug(f"utils.files._cleanup_lock_file - manually removed {lockfile_path}") - except Exception as e: - if logger is not None: - logger.debug(f"utils.files._cleanup_lock_file - failed to remove {lockfile_path}: {str(e)}") - # Don't raise - this is a cleanup operation - ########################################################################################## def _detect_file_format( filepath: str, # Path to the file. diff --git a/cmeta/version.py b/cmeta/version.py index 40f23e5..c3b0502 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.33.1" -__release_date__ = "2026-10-07" +__version__ = "0.34.0" +__release_date__ = "2026-10-09" diff --git a/cmeta/versions.json b/cmeta/versions.json index 1e76bbe..09868e6 100644 --- a/cmeta/versions.json +++ b/cmeta/versions.json @@ -89,6 +89,10 @@ { "version": "0.33.1", "version_date": "20261007" + }, + { + "version": "0.34.0", + "version_date": "20261009" } ] } diff --git a/docs/async-and-concurrency.md b/docs/async-and-concurrency.md index da86920..5f9b3de 100644 --- a/docs/async-and-concurrency.md +++ b/docs/async-and-concurrency.md @@ -209,29 +209,57 @@ r = utils.files.safe_write_file(index_file, index_data, file_lock=index_file_lock, atomic=True, ...) ``` -- **File locks** come from the `filelock` package (a runtime dependency), using - a sidecar lock file, so they work cross-platform and *across processes* — - not just across threads. Default acquisition timeout is **3 seconds**; - exceeding it produces an error dict rather than a corrupt file. +- **File locks** are the engine's own `cmeta.utils.files.PathLock` (no library), + using the sidecar file `.lock`, so they work cross-platform, *across + processes* and across the threads of one process. Default acquisition timeout + is **3 seconds**; exceeding it produces an error dict rather than a corrupt + file. - **`keep_locked=True`** hands the lock back with the data so the caller can modify and write without ever dropping it — this is what prevents lost updates. - **`atomic=True`** writes to `.tmp` and then `os.replace()`s it into position (retried up to 10 times), so a reader never observes a partially written file, and a crash mid-write cannot truncate the original. -- Locks are released in a `finally:` block, and stale lock files are cleaned up - explicitly (`filelock` does not always do so on Linux). +- Locks are released in a `finally:` block, and the lock file is removed by the + process that releases it, so repositories never keep lock files. A crash + leaves at most the lock file of the operation in flight; it blocks nobody + (the OS lock dies with the process) and the next holder reuses and removes it. The same mechanism protects `/index/*.pkl`, `repos.json`, and each artifact's `_cmeta.yaml` / `_cmeta.json`. +### How the lock works on each platform + +A lock file that is removed after use is correct only if the holder removes it +*before* releasing the OS lock and every newcomer checks, after acquiring, that +the file it holds is still the file at the path (otherwise a waiter blocked on +the removed file and a newcomer that created a new one both "hold" the lock). +`PathLock` does both. On **Windows** it takes a byte-range lock +(`msvcrt.locking`) on the lock file; an open file cannot be removed there, so a +newcomer that meets a file being deleted simply tries again. On **POSIX** it +takes `fcntl.flock` (per open file description, so threads exclude each other +too), or POSIX record locks (`fcntl.lockf`) where the file system refuses +`flock` (NFS and some network mounts), each with the identity check. Where the +file system refuses both (some FUSE, 9p and SMB mounts) it falls back to a +**soft lock**: the file is created exclusively and holds `pid host time`; a +stale one (its process dead on this host, or older than 10 minutes when left +by another host) is removed by the waiter that finds it. In every mode the +threads of one process are serialized on the path first. A busy lock is polled +with a delay that doubles from 1 ms to 25 ms until the timeout. A folder used +by two operating systems at once (a Windows tree plugged from WSL) is best +effort: each side locks with its own mechanism. + ### Measured behaviour Ten `cx note add` processes run simultaneously against one `` all exit `0`, and both the index and the directory listing show all ten artifacts. Eight processes updating **the same artifact's** metadata at the same time also all exit `0`, and **all eight fields are present afterwards** — no lost -updates, and no stale `.lock` files left behind. +updates, and no stale `.lock` files left behind. The lock's own tests +(`tests/core_tests/test_utils_files_path_lock.py`) run eight processes and +sixteen threads through one counter, replay the removed-file race +deterministically, kill a holder, and exercise the soft lock and its stale +files; `tests/benchmarks/benchmark_engine.py` measures the cost. ### What is *not* concurrency-safe diff --git a/docs/common-commands.md b/docs/common-commands.md index 898864b..cd477f4 100644 --- a/docs/common-commands.md +++ b/docs/common-commands.md @@ -99,6 +99,7 @@ cx update --meta.description="..." cx tags --add=t1,t2 --remove=t3 cx mv cx migrate # the same, plus a stub: the old alias keeps working +cx reindex # refresh its index record from its folder (nothing is written) cx rm ``` @@ -141,6 +142,7 @@ than with `cx add`. Afterwards, register it so it enters the index: ```bash cx index : +cx reindex # the same, and it also refreshes an artifact the index knows already ``` --- @@ -157,6 +159,7 @@ cx repo get --path= --local # register an existing folder cx repo pull cx repo status cx repo unplug # detach without deleting +cx repo reindex # re-read its _cmr.yaml and rescan its artifacts; other repos untouched cx repo space # disk usage ``` @@ -181,7 +184,11 @@ Common ones: `default` (framework defaults), `task` (caches, version checks), ```bash cx index : # register ONE hand-made artifact -cx update : # refresh index after editing its meta +cx reindex # refresh ONE record from its folder: a meta edited by hand, a folder + # renamed or copied in; nothing is written, stale aliases are dropped +cx reindex # every artifact of the category (hand-made folders included) +cx repo reindex # ONE repository: its _cmr.yaml and its artifacts; the others untouched +cx update : # refresh after editing its meta (rewrites the file: a new timestamp) cx --reindex # rebuild everything (slow — last resort) cx cache show @@ -189,8 +196,11 @@ cx cache clean cx cache delete ``` -Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) need **no** reindex -— the index tracks meta, not content. +`reindex` never touches a folder, and the only record it removes is that of an +artifact whose folder now holds another artifact (it says so): a folder it cannot +find, a meta it cannot read, or one UID in two folders is an error that leaves +the index as it was. Payload-only edits (`api/`, `files/`, `src/`, `_desc.yaml`) +need **no** reindex — the index tracks meta, not content. Categories holding very many artifacts can spread them into sub-directories (`sharding_slices`) and/or skip the index entirely (`no_index: true`) — see diff --git a/docs/en/api/index.rst b/docs/en/api/index.rst index bba00dc..c409e27 100644 --- a/docs/en/api/index.rst +++ b/docs/en/api/index.rst @@ -1,7 +1,7 @@ -API Reference (v0.33.1) +API Reference (v0.34.0) ======================= -This section contains the complete API reference for cMeta v0.33.1. +This section contains the complete API reference for cMeta v0.34.0. .. toctree:: :maxdepth: 2 diff --git a/docs/en/index.rst b/docs/en/index.rst index 925ace5..3811c7f 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -1,11 +1,11 @@ cMeta Documentation =================== -Welcome to cMeta v0.33.1 documentation! +Welcome to cMeta v0.34.0 documentation! .. toctree:: :maxdepth: 2 - :caption: cMeta v0.33.1 Contents: + :caption: cMeta v0.34.0 Contents: home diff --git a/docs/history.md b/docs/history.md index 026edfd..1628e9a 100644 --- a/docs/history.md +++ b/docs/history.md @@ -82,7 +82,7 @@ produces APA and BibTeX automatically. The BibTeX entry: author = {Fursin, Grigori}, title = {{cMeta (Common Meta Framework)}}, year = {2026}, - version = {0.33.1}, + version = {0.34.0}, license = {Apache-2.0}, url = {https://github.com/cTuningLabs/cmeta}, note = {cTuning Labs} diff --git a/docs/installation.md b/docs/installation.md index 7d4edf2..44e62ea 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -157,8 +157,8 @@ repositories) into these commands. **A specific version**, to pin or to go back: ```bash -uv tool install --force "cmeta[server]==0.33.1" -pip install "cmeta==0.33.1" +uv tool install --force "cmeta[server]==0.34.0" +pip install "cmeta==0.34.0" ``` **Moving an install from git to PyPI** (or back) is a reinstall from the other diff --git a/docs/using-cmeta.md b/docs/using-cmeta.md index 73c2b36..131d0bc 100644 --- a/docs/using-cmeta.md +++ b/docs/using-cmeta.md @@ -68,7 +68,7 @@ any command unless noted. |------|------|--------| | `--help`, `-h` | bool | Show help. On a category or command, prints its usage. | | `--version`, `-V` | bool | Print cMeta version and check for a newer release. | -| `--reindex` | bool | Clean and rebuild the fast index (`/index/*.pkl`). Use after manual edits, `git pull`, cleared `CMETA_HOME`, or when lookups look stale. | +| `--reindex` | bool | Clean and rebuild the fast index (`/index/*.pkl`): every repository, every category. Use after a cleared `CMETA_HOME` or a bulk change; for one artifact or one repository, `cx reindex ` and `cx repo reindex ` are the narrower forms (§11). | | `--verbose`, `-v` | bool | Verbose output — extra progress info from repo ops and package detection. | | `--quiet`, `-q` | bool | Auto-accept the default answer for any interactive prompt (safe for scripts). | | `--repro`, `-r` | bool | Write `cmeta-repro-input.json` (+ `-input2.json`, `-output.json`) for reproducibility of the call. | @@ -171,6 +171,7 @@ cx ... -v --con # noisy output for troubleshoot cx ... --repro # capture inputs/outputs for later replay cx ... -j --jf=out.json # machine-readable result to stdout + file cx --reindex # rebuild the fast index +cx reindex # refresh one artifact's index record from its folder cx --home=D:\work\cmeta-home # one-shot alternative CMETA_HOME ``` @@ -460,7 +461,8 @@ cx utils uid --clipboard- --jf=uid.json # {"return": 0, "uid": "29576481b258 From Python the underlying helper is `cm.utils.names.generate_cmeta_uid()`. Remember to register a hand-made artifact afterwards so it enters the index: -`cx index :` (see §11). +`cx index :`, or `cx reindex `, +which also refreshes an artifact the index knows already (see §11). Two neighbours of these generators exist for credentials. `cx utils api_key` prints random keys from `secrets.token_urlsafe` together with the @@ -1533,17 +1535,44 @@ the filesystem. The framework refreshes it automatically: - when repos are added/removed via `cx repo` operations, - on first launch if the index is missing. -Force a full clean rebuild when things look inconsistent — after a manual move, -a hand-edit of `_cmeta.yaml`, a `git pull` inside a repo, a `CMETA_HOME` wipe, -or if `cx find ` returns something surprising: +When artifacts were touched outside cMeta, refresh the narrowest thing: ```bash -cx --reindex -``` - -It's safe, idempotent and typically fast. Categories that set `no_index: true` -in their `_cmeta.yaml` are always found by filesystem scan and are never -written to the index. +cx reindex # one artifact: its record is rewritten from the meta file in its folder +cx reindex # every artifact of the category, folders made by hand included +cx repo reindex # one repository: its _cmr.yaml re-read, its artifacts rescanned +cx --reindex # everything: a clean rebuild of every index file +``` + +`cx reindex ` reads the artifact's `_cmeta.*` from its folder +and rewrites its index record, and writes nothing on disk — after a meta edited by +hand, a folder renamed, moved or copied in from another machine, or when +`cx find ` returns something surprising. The artifact is named +by its alias, UID, `alias,UID` or `repo:alias`, with wildcards (`--tags` prunes); +a folder the index does not know yet is found on disk in the repositories, and an +artifact whose indexed folder is gone is looked for by its UID in its repository, +so after a rename by hand the new name, the UID and the old name all work, and +the old alias is dropped from the index. No folder is touched, and the only record +ever removed is that of an artifact whose folder, still there, holds another +artifact now (replaced by hand) — said in the output. A folder that cannot be +found, a meta that cannot be read or has no valid `artifact` UID, a meta of +another category, or one UID found in two existing folders is an error that +leaves the index as it was (`--ignore_errors` reindexes the other artifacts and +reports these). An empty `cx update ` +refreshes the record too, but it also rewrites the meta file (a new +`last_update_timestamp` and a generator record): use `reindex` when the file must +stay as it is, for instance in a folder that another machine syncs. + +`cx repo reindex ` — by alias, UID, the path of its folder, or `.` from +inside it — does what `cx repo pull` and `cx repo plug` do once a repository is +in place: its `_cmr.yaml` is re-read and its artifacts are indexed again, and the +records of the other repositories stay as they are. A repository whose folder is +not there (a detached drive) is refused and nothing changes. + +`cx --reindex` rebuilds every index file from scratch — after a `CMETA_HOME` +wipe, a bulk change across repositories, or when in doubt. It's safe, idempotent +and typically fast. Categories that set `no_index: true` in their `_cmeta.yaml` +are always found by filesystem scan and are never written to the index. Task-level content caches produced by workflows live under the `cache` category and are managed separately: diff --git a/pyproject.toml b/pyproject.toml index 468e3e0..aafefed 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,7 +42,6 @@ dependencies = [ "wheel", "tabulate", "tqdm", - "filelock", "packaging", "psutil", ] diff --git a/tests/benchmarks/benchmark_engine.py b/tests/benchmarks/benchmark_engine.py new file mode 100644 index 0000000..51a2867 --- /dev/null +++ b/tests/benchmarks/benchmark_engine.py @@ -0,0 +1,567 @@ +""" +A benchmark of the cMeta engine's hot paths: the index reads behind `find`, the locked index writes behind +`create`, `update`, `delete` and `reindex`, the full reindex, the reload of a changed index, and the primitives +they are built on (stat, file lock, record copy, pickle, atomic replace). It makes a home of its own with N +hand-made artifacts in one category and prints one table; with --json it also writes the numbers, and with +--compare it prints every number next to the one of an earlier run (the before/after check of an engine +change). With --procs P it also lets P processes create artifacts in the same category at once and reports +the throughput and whether every record made it into the index. + + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json before.json + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json after.json --compare before.json + +Nothing outside --home is touched: CMETA_HOME is set to it for this process and its workers. The folder is +removed first when it holds the marker of an earlier run and refused otherwise. The pytest module next to +this file runs it with a small N as a smoke test; the numbers themselves are not asserted (the speed of a +CI runner varies). + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import argparse +import copy +import json +import os +import pickle +import platform +import random +import shutil +import statistics +import subprocess +import sys +import time + +MARKER = '.cmeta-benchmark-home' +CATEGORY = 'log' +CATEGORY_REF = 'log,487a7639093a4685' +REPO_ALIAS = 'bench-repo' +REPO_UID = 'b0c4e5f6a7b8c9d0' +ENV_TO_DROP = ('CMETA_HOME', 'CMETA_HOME2', 'CMETA_VERBOSE', 'CMETA_DEBUG', 'CMETA_FAIL_ON_ERROR', + 'CMETA_INTERNAL_REPO_PATH', 'VIRTUAL_ENV', 'CONDA_PREFIX') + + +########################################################################################## +# Timing helpers + +def timed_calls(fn, repeat): + """Every call timed on its own; returns the list of durations in seconds.""" + times = [] + for _ in range(repeat): + t0 = time.perf_counter() + fn() + times.append(time.perf_counter() - t0) + return times + + +class Results: + """The measurements of one run: name -> median, min, max, unit, repeats, note.""" + + UNITS = {'us': 1e6, 'ms': 1e3, 's': 1.0} + + def __init__(self): + self.rows = {} + self.order = [] + + def add(self, name, times, unit, note=''): + scale = self.UNITS[unit] + row = {'median': statistics.median(times) * scale, 'min': min(times) * scale, 'max': max(times) * scale, + 'unit': unit, 'n': len(times), 'note': note} + self.rows[name] = row + self.order.append(name) + print(' %-44s %12.2f %-3s (min %10.2f, max %10.2f, n=%d) %s' % ( + name, row['median'], unit, row['min'], row['max'], row['n'], note)) + return row + + def value(self, name, value, unit, note=''): + row = {'median': value, 'min': value, 'max': value, 'unit': unit, 'n': 1, 'note': note} + self.rows[name] = row + self.order.append(name) + print(' %-44s %12.2f %-3s %s' % (name, value, unit, note)) + return row + + +def section(title): + print('') + print('=== ' + title) + + +########################################################################################## +# The home with N hand-made artifacts + +def prepare_home(home): + if os.path.exists(home): + if not os.path.isfile(os.path.join(home, MARKER)): + sys.exit('refusing to remove %s: it is not the home of an earlier benchmark run (no %s)' % (home, MARKER)) + shutil.rmtree(home) + os.makedirs(home) + with open(os.path.join(home, MARKER), 'w') as f: + f.write('made by tests/benchmarks/benchmark_engine.py - safe to delete\n') + + +def make_artifacts(home, n, seed=20261008): + """A repository folder with N artifacts of the log category, written by hand (no engine call).""" + rnd = random.Random(seed) + repo = os.path.join(home, 'bench', REPO_ALIAS) + cat_dir = os.path.join(repo, CATEGORY) + os.makedirs(cat_dir) + with open(os.path.join(repo, '_cmr.yaml'), 'w', encoding='utf-8') as f: + f.write('artifact: %s,%s\ncategory: repo,f4f792ab40c7498f\n' % (REPO_ALIAS, REPO_UID)) + uids = [] + for i in range(n): + alias = 'art-%06d' % i + uid = '%016x' % rnd.getrandbits(64) + uids.append(uid) + folder = os.path.join(cat_dir, alias) + os.mkdir(folder) + with open(os.path.join(folder, '_cmeta.yaml'), 'w', encoding='utf-8') as f: + f.write('artifact: %s,%s\ncategory: %s\ntags:\n - bench\n - g%d\n' + 'desc: benchmark artifact number %d of the log category\nowner: benchmark\n' + 'generator: script\n' % (alias, uid, CATEGORY_REF, i % 10, i)) + return repo, uids + + +########################################################################################## +# The lock candidates of the concurrency work (measured here, not used by the engine) + +def candidate_lock(path, keep_file): + """A lock that keeps the file at `path` the one every process locks: acquire, then check that the file + opened is still the file at the path (it is not when another process removed it meanwhile); on release + the file is removed BEFORE the lock is dropped (POSIX) or after the handle is closed (Windows, where a + file still open elsewhere cannot be removed). With keep_file the lock file is never removed.""" + if os.name == 'nt': + import msvcrt + while True: + try: + fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644) + except PermissionError: + time.sleep(0.001) + continue + try: + msvcrt.locking(fd, msvcrt.LK_NBLCK, 1) + return fd + except OSError: + os.close(fd) + time.sleep(0.001) + else: + import fcntl + while True: + fd = os.open(path, os.O_RDWR | os.O_CREAT, 0o644) + try: + fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + except OSError: + os.close(fd) + time.sleep(0.001) + continue + try: + st = os.stat(path) + except FileNotFoundError: + st = None + fst = os.fstat(fd) + if st is None or (st.st_ino, st.st_dev) != (fst.st_ino, fst.st_dev): + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + continue + return fd + + +def candidate_unlock(path, fd, keep_file): + if os.name == 'nt': + import msvcrt + msvcrt.locking(fd, msvcrt.LK_UNLCK, 1) + os.close(fd) + if not keep_file: + try: + os.unlink(path) + except OSError: + pass + else: + import fcntl + if not keep_file: + try: + os.unlink(path) + except OSError: + pass + fcntl.flock(fd, fcntl.LOCK_UN) + os.close(fd) + + +########################################################################################## +# The worker of the concurrency measurement + +def worker_create(home, prefix, count): + for v in ENV_TO_DROP: + os.environ.pop(v, None) + os.environ['CMETA_HOME'] = home + from cmeta import CMeta + cm = CMeta(home=home) + times = [] + errors = [] + for i in range(count): + t0 = time.perf_counter() + r = cm.access({'category': CATEGORY, 'command': 'create', 'arg1': '%s:%s-%04d' % (REPO_ALIAS, prefix, i), + 'meta': {'tags': ['concurrent']}, 'yaml': True, 'con': False}) + times.append(time.perf_counter() - t0) + if r['return'] > 0: + errors.append(r.get('error', '?')) + print(json.dumps({'times': times, 'errors': errors})) + + +def concurrent_creates(home, procs, per, results): + section('Concurrent creates: %d processes x %d artifacts in the same category at once' % (procs, per)) + env = dict(os.environ) + env['CMETA_HOME'] = home + t0 = time.perf_counter() + children = [] + for p in range(procs): + children.append(subprocess.Popen([sys.executable, os.path.abspath(__file__), '--worker-create', 'cw%d' % p, + str(per), '--home', home], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)) + times = [] + errors = [] + for child in children: + out, err = child.communicate() + try: + data = json.loads(out.decode('utf-8', 'replace').strip().splitlines()[-1]) + except Exception: + errors.append('worker output unreadable: %s %s' % (out[-300:], err[-300:])) + continue + times.extend(data['times']) + errors.extend(data['errors']) + wall = time.perf_counter() - t0 + results.value('concurrent creates: wall time', wall, 's', '%d creates' % (procs * per)) + if times: + results.value('concurrent creates: creates per second', len(times) / wall, 's') + results.add('concurrent create latency', times, 'ms') + return errors + + +def check_index(home, cm, procs, per, results): + """Every record the workers made is in the index, and the alias map agrees with the records.""" + r = cm.access({'category': CATEGORY, 'command': 'find', 'arg1': 'cw*', 'con': False}) + found = len(r.get('artifacts', [])) if r['return'] == 0 else 0 + expected = procs * per + results.value('concurrent creates: records lost', expected - found, 's', 'of %d (0 = every write kept)' % expected) + with open(os.path.join(home, 'index', CATEGORY + '.pkl'), 'rb') as f: + index = pickle.load(f) + uids = index['uids'] + bad = 0 + for alias, alias_uids in index['lowercase_aliases'].items(): + for uid in alias_uids: + if uid not in uids: + bad += 1 + for uid, rec in uids.items(): + alias = rec['cmeta_ref_parts'].get('artifact_alias_lowercase', rec['cmeta_ref_parts'].get('artifact_alias', '')).lower() + if alias and uid not in index['lowercase_aliases'].get(alias, []): + bad += 1 + results.value('index consistency: broken alias links', bad, 's', '0 = aliases and records agree') + on_disk = 0 + cat_dir = os.path.join(home, 'bench', REPO_ALIAS, CATEGORY) + for name in os.listdir(cat_dir): + if name.startswith('cw'): + on_disk += 1 + results.value('concurrent creates: folders on disk', on_disk, 's', 'of %d' % expected) + + +########################################################################################## +# CLI timings + +def cli_timings(home, cx, results): + section('CLI (a new process per call, as a user runs cx)') + env = dict(os.environ) + env['CMETA_HOME'] = home + + def run(args): + r = subprocess.run([cx] + args, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + if r.returncode != 0: + raise RuntimeError('%s failed: %s' % (' '.join(args), r.stderr.decode('utf-8', 'replace')[-500:])) + + try: + run(['--version']) + except Exception as e: + print(' cx not usable (%s), the CLI timings are skipped' % e) + return + results.add('cx --version', timed_calls(lambda: run(['--version']), 5), 'ms', 'process start + engine init') + results.add('cx log find ', timed_calls(lambda: run([CATEGORY, 'find', 'art-000001']), 5), 'ms') + results.add('cx log reindex ', timed_calls(lambda: run([CATEGORY, 'reindex', 'art-000001']), 3), 'ms') + results.add('cx --reindex (full)', timed_calls(lambda: run(['--reindex']), 1), 's') + + +########################################################################################## +# Comparison with an earlier run + +def compare(results, before_path): + with open(before_path) as f: + before = json.load(f)['results'] + section('Compared with ' + before_path + ' (ratio = now / before, by the medians)') + print(' %-44s %12s %12s %8s' % ('measurement', 'before', 'now', 'ratio')) + for name in results.order: + now = results.rows[name] + old = before.get(name) + if old is None or old['unit'] != now['unit']: + continue + if old['median'] > 0: + ratio = now['median'] / old['median'] + if 'per second' in name: # a rate: more is better + flag = ' faster' if ratio > 1.25 else (' slower' if ratio < 0.8 else '') + elif name.endswith(('lost', 'links', 'on disk', 'index', 'size')): # counts, not times + flag = '' + else: + flag = ' slower' if ratio > 1.25 else (' faster' if ratio < 0.8 else '') + print(' %-44s %12.2f %12.2f %8.2f%s' % (name, old['median'], now['median'], ratio, flag)) + else: + print(' %-44s %12.2f %12.2f %8s' % (name, old['median'], now['median'], '-')) + + +########################################################################################## + +def main(): + parser = argparse.ArgumentParser(description='benchmark of the cMeta engine hot paths') + parser.add_argument('--home', required=True, help='an empty folder (or the home of an earlier run) used as CMETA_HOME') + parser.add_argument('--n', type=int, default=10000, help='artifacts in the category (default 10000)') + parser.add_argument('--writes', type=int, default=100, help='creates/updates/reindexes/deletes timed (default 100)') + parser.add_argument('--procs', type=int, default=4, help='processes of the concurrent-create measurement (0 = skip)') + parser.add_argument('--per', type=int, default=25, help='creates per process in that measurement') + parser.add_argument('--cli', action='store_true', help='also time the cx command line') + parser.add_argument('--cx', default='cx', help='the cx executable for --cli') + parser.add_argument('--json', help='write the numbers to this file') + parser.add_argument('--compare', help='an earlier --json file to print the ratios against') + parser.add_argument('--keep', action='store_true', help='keep the home after the run') + parser.add_argument('--worker-create', nargs=2, metavar=('PREFIX', 'COUNT'), help=argparse.SUPPRESS) + args = parser.parse_args() + + home = os.path.abspath(args.home) + + if args.worker_create: + worker_create(home, args.worker_create[0], int(args.worker_create[1])) + return + + for v in ENV_TO_DROP: + os.environ.pop(v, None) + os.environ['CMETA_HOME'] = home + + import cmeta + from cmeta import CMeta + from cmeta import utils + + results = Results() + meta = {'python': sys.version.split()[0], 'platform': platform.platform(), 'machine': platform.machine(), + 'cmeta': cmeta.__version__, 'cmeta_path': os.path.dirname(cmeta.__file__), 'home': home, 'n': args.n, + 'pickle_default_protocol': pickle.DEFAULT_PROTOCOL, 'date': time.strftime('%Y-%m-%d %H:%M:%S')} + section('Benchmark of the cMeta engine') + for k in ('python', 'platform', 'cmeta', 'cmeta_path', 'home', 'n', 'pickle_default_protocol'): + print(' %-24s %s' % (k, meta[k])) + + prepare_home(home) + + section('Engine start') + t0 = time.perf_counter() + cm = CMeta(home=home) + results.value('CMeta() first run of a home', time.perf_counter() - t0, 'ms', 'the home and its index are made') + results.add('CMeta() warm', timed_calls(lambda: CMeta(home=home), 5), 'ms', 'a new engine object on an existing home') + + section('Index build of %d hand-made artifacts' % args.n) + t0 = time.perf_counter() + repo, uids = make_artifacts(home, args.n) + results.value('writing %d artifact folders by hand' % args.n, time.perf_counter() - t0, 's', 'not the engine') + t0 = time.perf_counter() + r = cm.access({'category': 'repo', 'command': 'plug', 'arg1': repo, 'con': False}) + assert r['return'] == 0, r.get('error') + results.value('repo plug (incremental index of the repo)', time.perf_counter() - t0, 's') + results.add('full reindex (cx --reindex)', timed_calls(lambda: cm.repos.reindex(con=False), 2), 's') + index_file = os.path.join(home, 'index', CATEGORY + '.pkl') + results.value('index file size', os.path.getsize(index_file) / 1024.0, 's', 'KB (unit column: KB)') + with open(index_file, 'rb') as f: + index_data = pickle.load(f) + results.value('records in the index', len(index_data['uids']), 's', '(unit column: records)') + + section('Reads with a warm cache (the index already loaded in this process)') + alias = 'art-%06d' % (args.n // 2) + uid = uids[args.n // 2] + + def find(**p): + p.setdefault('category', CATEGORY) + p.setdefault('command', 'find') + p.setdefault('con', False) + return cm.access(p) + + r = find(arg1=alias) + assert r['return'] == 0 and len(r['artifacts']) == 1, r + results.add('find by alias (1 hit)', timed_calls(lambda: find(arg1=alias), 2000), 'us') + results.add('find by UID (1 hit)', timed_calls(lambda: find(arg1=uid), 2000), 'us') + results.add('find a missing alias (error 16)', timed_calls(lambda: find(arg1='no-such-artifact'), 2000), 'us') + results.add('find by alias + tag filter', timed_calls(lambda: find(arg1=alias, tags='bench'), 1000), 'us') + r = find(arg1='art-00*') + hits = len(r.get('artifacts', [])) + results.add('find wildcard art-00* (%d hits)' % hits, timed_calls(lambda: find(arg1='art-00*'), 20), 'ms') + r = find() + hits = len(r.get('artifacts', [])) + results.add('find all (%d hits)' % hits, timed_calls(lambda: find(), 5), 'ms') + r = find(tags='g3') + hits = len(r.get('artifacts', [])) + results.add('find by tag g3 (%d hits)' % hits, timed_calls(lambda: find(tags='g3'), 5), 'ms', 'every record checked') + + section('Reads after the index changed (the reload a write by another process causes)') + + def bump_and_find(): + st = os.stat(index_file) + os.utime(index_file, ns=(st.st_atime_ns, st.st_mtime_ns + 1000000000)) + r = find(arg1=alias) + assert r['return'] == 0 + + results.add('find by alias after a changed index', timed_calls(bump_and_find, 5), 'ms', 'the pickle load + 1 find') + results.add('first find of a new engine object', timed_calls(lambda: CMeta(home=home).access( + {'category': CATEGORY, 'command': 'find', 'arg1': alias, 'con': False}), 5), 'ms', 'CMeta() + load + find') + + section('Writes (each one rewrites the whole index file of the category under its lock)') + w = args.writes + + def create(i): + r = cm.access({'category': CATEGORY, 'command': 'create', 'arg1': '%s:new-%04d' % (REPO_ALIAS, i), + 'meta': {'tags': ['new']}, 'yaml': True, 'con': False}) + assert r['return'] == 0, r.get('error') + + def update(i): + r = cm.access({'category': CATEGORY, 'command': 'update', 'arg1': 'new-0000', 'meta': {'counter': i}, 'con': False}) + assert r['return'] == 0, r.get('error') + + def reindex_one(): + r = cm.access({'category': CATEGORY, 'command': 'reindex', 'arg1': alias, 'con': False}) + assert r['return'] == 0, r.get('error') + + def delete(i): + r = cm.access({'category': CATEGORY, 'command': 'delete', 'arg1': 'new-%04d' % i, 'force': True, 'con': False}) + assert r['return'] == 0, r.get('error') + + times = [] + for i in range(w): + t0 = time.perf_counter() + create(i) + times.append(time.perf_counter() - t0) + results.add('create (meta file + index write)', times, 'ms') + times = [] + for i in range(w): + t0 = time.perf_counter() + update(i) + times.append(time.perf_counter() - t0) + results.add('update (meta edit in place + index write)', times, 'ms') + results.add('reindex one artifact', timed_calls(reindex_one, w), 'ms', 'folder scan + index write') + times = [] + for i in range(w): + t0 = time.perf_counter() + delete(i) + times.append(time.perf_counter() - t0) + results.add('delete (index write + folder removal)', times, 'ms') + + section('Primitives (what one index read or write is made of)') + results.add('os.stat of the index file', timed_calls(lambda: os.stat(index_file), 10000), 'us', + '3 per find today (is_dir, exists, stat)') + lock_target = os.path.join(home, 'index', 'lock-benchmark.bin') + with open(lock_target, 'wb') as f: + f.write(b'x') + + def engine_lock(): + lock = utils.files._acquire_lock(lock_target) + utils.files._release_lock(lock_target, lock) + + results.add('engine lock acquire+release', timed_calls(engine_lock, 500), 'us', 'filelock + the lock file removed') + try: + from filelock import FileLock + + def library_lock(): + lock = FileLock(lock_target + '.lock') + lock.acquire() + lock.release() + + results.add('filelock acquire+release alone', timed_calls(library_lock, 500), 'us', 'the library by itself') + except ImportError: + pass + cand = lock_target + '.cand' + + def cand_lock(keep): + fd = candidate_lock(cand, keep) + candidate_unlock(cand, fd, keep) + + results.add('candidate lock, file removed on release', timed_calls(lambda: cand_lock(False), 500), 'us', + 'verified identity, removed before release') + results.add('candidate lock, file kept', timed_calls(lambda: cand_lock(True), 500), 'us', 'verified identity, never removed') + try: + os.unlink(cand) + except OSError: + pass + + records = list(index_data['uids'].values()) + + def copy_level1(rec): + c = dict(rec) + c['cmeta'] = dict(rec['cmeta']) + tags = c['cmeta'].get('tags') + if isinstance(tags, list): + c['cmeta']['tags'] = list(tags) + return c + + def all_records(fn): + for rec in records: + fn(rec) + + results.add('copy of %d records: record.copy() (today)' % len(records), timed_calls(lambda: all_records(dict.copy), 5), 'ms') + results.add('copy of %d records: record + cmeta + tags' % len(records), timed_calls(lambda: all_records(copy_level1), 5), 'ms', + 'the cheap copy that protects the cache') + results.add('copy of %d records: deepcopy' % len(records), timed_calls(lambda: all_records(copy.deepcopy), 3), 'ms') + + for proto in sorted(set([4, 5, pickle.HIGHEST_PROTOCOL])): + if proto > pickle.HIGHEST_PROTOCOL: + continue + data = pickle.dumps(index_data, protocol=proto) + results.add('pickle dump of the index, protocol %d' % proto, timed_calls(lambda: pickle.dumps(index_data, protocol=proto), 5), + 'ms', '%d KB' % (len(data) // 1024)) + results.add('pickle load of the index, protocol %d' % proto, timed_calls(lambda: pickle.loads(data), 5), 'ms') + + small = os.path.join(home, 'index', 'write-benchmark.yaml') + payload = ('key%03d: value %d\n' % (i, i) for i in range(40)) + payload = ''.join(payload) + + def plain_write(): + with open(small, 'w', encoding='utf-8') as f: + f.write(payload) + + def atomic_write(): + tmp = small + '.tmp' + with open(tmp, 'w', encoding='utf-8') as f: + f.write(payload) + os.replace(tmp, small) + + results.add('write a 1 KB file in place', timed_calls(plain_write, 500), 'us') + results.add('write a 1 KB file atomically', timed_calls(atomic_write, 500), 'us', 'temp file + os.replace') + for p in (small, lock_target): + try: + os.unlink(p) + except OSError: + pass + + errors = [] + if args.procs > 0: + errors = concurrent_creates(home, args.procs, args.per, results) + check_index(home, cm, args.procs, args.per, results) + if errors: + print(' worker errors (%d): %s' % (len(errors), errors[:3])) + + if args.cli: + cli_timings(home, args.cx, results) + + if args.compare: + compare(results, args.compare) + + if args.json: + with open(args.json, 'w') as f: + json.dump({'meta': meta, 'results': {k: results.rows[k] for k in results.order}, 'worker_errors': errors}, f, indent=1) + print('') + print('numbers written to ' + args.json) + + if not args.keep: + shutil.rmtree(home, ignore_errors=True) + + +if __name__ == '__main__': + main() diff --git a/tests/benchmarks/test_benchmark_engine.py b/tests/benchmarks/test_benchmark_engine.py new file mode 100644 index 0000000..4d5b630 --- /dev/null +++ b/tests/benchmarks/test_benchmark_engine.py @@ -0,0 +1,68 @@ +""" +The engine benchmark next to this file (benchmark_engine.py) must keep running on every platform and Python +the suite covers: this runs it on a small home and checks that every section reported its numbers and that +the index it built is consistent. The numbers themselves are not asserted (the speed of a CI runner varies). +The benchmark proper is run by hand before and after an engine change: + + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json before.json + python tests/benchmarks/benchmark_engine.py --home --n 10000 --json after.json --compare before.json + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +SCRIPT = os.path.join(HERE, 'benchmark_engine.py') + +EXPECTED = ( + 'CMeta() warm', + 'repo plug (incremental index of the repo)', + 'full reindex (cx --reindex)', + 'records in the index', + 'find by alias (1 hit)', + 'find by UID (1 hit)', + 'find a missing alias (error 16)', + 'find by alias after a changed index', + 'create (meta file + index write)', + 'update (meta edit in place + index write)', + 'reindex one artifact', + 'delete (index write + folder removal)', + 'os.stat of the index file', + 'engine lock acquire+release', + 'candidate lock, file removed on release', + 'write a 1 KB file atomically', + 'concurrent creates: wall time', + 'concurrent creates: records lost', + 'concurrent creates: folders on disk', + 'index consistency: broken alias links', +) + + +def test_benchmark_runs_and_reports_every_section(tmp_path): + out = tmp_path / 'bench.json' + env = dict(os.environ) + for v in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + env.pop(v, None) + + r = subprocess.run([sys.executable, SCRIPT, '--home', str(tmp_path / 'home'), '--n', '50', '--writes', '3', + '--procs', '2', '--per', '3', '--json', str(out)], + env=env, stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=900) + assert r.returncode == 0, r.stdout.decode('utf-8', 'replace')[-4000:] + + data = json.loads(out.read_text(encoding='utf-8')) + results = data['results'] + for name in EXPECTED: + assert name in results, name + assert results[name]['n'] >= 1 + + assert results['records in the index']['median'] == 50 + assert results['concurrent creates: folders on disk']['median'] == 6 + assert results['index consistency: broken alias links']['median'] == 0 + assert data['worker_errors'] == [] + assert data['meta']['n'] == 50 diff --git a/tests/core_tests/test_utils_files_path_lock.py b/tests/core_tests/test_utils_files_path_lock.py new file mode 100644 index 0000000..bae050a --- /dev/null +++ b/tests/core_tests/test_utils_files_path_lock.py @@ -0,0 +1,520 @@ +""" +Tests of the engine's own path lock (`cmeta.utils.files.PathLock`, behind `_acquire_lock` / `_release_lock`, +`safe_read_file(lock=True, keep_locked=True)`, `safe_write_file`, `safe_delete_directory`, `lock_path`): +one holder at a time across processes and across the threads of one process on every platform; the lock +file is created for the operation and removed by the process that releases it; a lock file removed under a +waiter is detected (the identity check) and never leads to two holders; a killed holder blocks nobody; the +soft lock used where the file system refuses OS locks, with its stale files; timeouts; the engine's own +read-modify-write cycle under contention. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import subprocess +import sys +import threading +import time + +import pytest + +from cmeta.utils import files + +POSIX = sys.platform != 'win32' + + +########################################################################################## +# Workers run as separate processes + +COUNTER_WORKER = r''' +import os, sys, time +from cmeta.utils import files +mode, target, log_path, n = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4]) +if len(sys.argv) > 5 and sys.argv[5] == 'soft': + files._lock_soft_paths.add(target + '.lock') +pid = os.getpid() +for i in range(n): + if mode == 'pathlock': + lock = files.PathLock(target + '.lock').acquire(timeout=120) + try: + try: + with open(target) as f: + v = int(f.read().strip() or '0') + except FileNotFoundError: + v = 0 + time.sleep(0.0005) + with open(target, 'w') as f: + f.write(str(v + 1)) + with open(log_path, 'a') as f: + f.write('%d %d %s\n' % (pid, i, lock.mode)) + finally: + lock.release() + else: + r = files.safe_read_file(target, lock=True, keep_locked=True, timeout=120) + if r['return'] > 0: + print('ERROR read', r['error']); sys.exit(3) + data = r['data'] + data['n'] = data.get('n', 0) + 1 + time.sleep(0.0005) + r = files.safe_write_file(target, data, file_lock=r['file_lock'], atomic=True, timeout=120) + if r['return'] > 0: + print('ERROR write', r['error']); sys.exit(4) +print('OK', pid) +''' + +HOLDER = r''' +import sys, time +from cmeta.utils import files +lock = files.PathLock(sys.argv[1]).acquire(timeout=60) +print('LOCKED', lock.mode, flush=True) +time.sleep(float(sys.argv[2]) if len(sys.argv) > 2 else 60) +lock.release() +print('RELEASED', flush=True) +''' + +HOOKED_WAITER = r''' +import os, sys, time +from cmeta.utils import files +lock_file, marker, go, out = sys.argv[1:5] +real = files.fcntl.flock +state = {'first': True} +def hooked(fd, op): + # The first lock attempt: the file is open (the old file), tell the test and wait for its go + if state['first'] and (op & files.fcntl.LOCK_NB): + state['first'] = False + open(marker, 'w').close() + while not os.path.exists(go): + time.sleep(0.002) + return real(fd, op) +files.fcntl.flock = hooked +lock = files.PathLock(lock_file) +lock.acquire(timeout=60) +same = files.PathLock._identity(os.fstat(lock._fd)) == files.PathLock._identity(os.stat(lock_file)) +with open(out, 'w') as f: + f.write('%f %d %s %s\n' % (time.time(), lock.identity_retries, lock.mode, same)) +lock.release() +''' + + +def run_workers(script, args_list, timeout=600): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + procs = [subprocess.Popen([sys.executable, '-c', script] + [str(a) for a in args], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env) + for args in args_list] + outputs = [] + for p in procs: + out, _ = p.communicate(timeout=timeout) + outputs.append((p.returncode, out.decode('utf-8', 'replace'))) + return outputs + + +def start(script, *args): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + # The workers never read their input: an explicit null input keeps them independent of the + # parent's standard-input handle (which some agent and service environments leave unusable) + return subprocess.Popen([sys.executable, '-c', script] + [str(a) for a in args], + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, env=env) + + +def wait_for(path, seconds=30): + deadline = time.time() + seconds + while not os.path.exists(path): + if time.time() > deadline: + raise AssertionError(f'{path} did not appear in {seconds} s') + time.sleep(0.005) + + +########################################################################################## +# The lock object + +def test_one_holder_at_a_time_and_the_file_is_removed_on_release(tmp_path): + lock_file = str(tmp_path / 'thing.lock') + first = files.PathLock(lock_file).acquire(timeout=5) + assert first.is_locked and first.mode in ('windows', 'flock', 'lockf', 'soft') + assert os.path.isfile(lock_file) + + second = files.PathLock(lock_file) + t0 = time.monotonic() + with pytest.raises(TimeoutError): + second.acquire(timeout=0.3) + assert 0.25 <= time.monotonic() - t0 < 3 + assert not second.is_locked + + first.release() + assert not first.is_locked + assert not os.path.exists(lock_file) + + second.acquire(timeout=5) + assert second.is_locked + second.release() + assert not os.path.exists(lock_file) + + +def test_context_manager_double_release_and_check(tmp_path): + target = str(tmp_path / 'file.json') + with files.PathLock(target + '.lock') as lock: + assert lock.is_locked + files._check_lock(target, lock) + assert not lock.is_locked + lock.release() # a second release is harmless + with pytest.raises(TimeoutError): + files._check_lock(target, lock) + assert not os.path.exists(target + '.lock') + + +def test_lock_path_and_unlock_path(tmp_path): + target = str(tmp_path / 'folder') + r = files.lock_path(target, timeout=5) + assert r['return'] == 0 and r['file_lock'].is_locked + assert os.path.isfile(target + '.lock') + r2 = files.lock_path(target, timeout=0.2) + assert r2['return'] > 0 + assert files.unlock_path(target, r['file_lock'])['return'] == 0 + assert not os.path.exists(target + '.lock') + + +def test_a_missing_folder_is_an_error_not_a_wait(tmp_path): + t0 = time.monotonic() + with pytest.raises(TimeoutError): + files._acquire_lock(str(tmp_path / 'no-such-folder' / 'file'), timeout=5) + assert time.monotonic() - t0 < 2 + + +def test_threads_of_one_process_exclude_each_other(tmp_path): + target = str(tmp_path / 'counter.txt') + state = {'value': 0, 'inside': 0, 'overlap': 0, 'modes': set()} + + def work(): + for _ in range(50): + lock = files.PathLock(target + '.lock').acquire(timeout=60) + try: + state['inside'] += 1 + if state['inside'] > 1: + state['overlap'] += 1 + v = state['value'] + time.sleep(0.0002) + state['value'] = v + 1 + state['modes'].add(lock.mode) + state['inside'] -= 1 + finally: + lock.release() + + threads = [threading.Thread(target=work) for _ in range(16)] + for t in threads: + t.start() + for t in threads: + t.join() + assert state['value'] == 16 * 50 + assert state['overlap'] == 0 + assert not os.path.exists(target + '.lock') + + +def test_a_thread_waiting_on_another_thread_times_out_instead_of_hanging(tmp_path): + lock_file = str(tmp_path / 'held.lock') + held = files.PathLock(lock_file).acquire(timeout=5) + result = {} + + def waiter(): + t0 = time.monotonic() + try: + files.PathLock(lock_file).acquire(timeout=0.3) + result['got'] = True + except TimeoutError as e: + result['error'] = str(e) + result['seconds'] = time.monotonic() - t0 + + t = threading.Thread(target=waiter) + t.start() + t.join(10) + assert 'error' in result and 'thread' in result['error'] + assert 0.25 <= result['seconds'] < 3 + held.release() + + +########################################################################################## +# Processes + +def test_processes_exclude_each_other_through_pathlock(tmp_path): + target = str(tmp_path / 'counter.txt') + log_path = str(tmp_path / 'log.txt') + outputs = run_workers(COUNTER_WORKER, [('pathlock', target, log_path, 60)] * 8) + for rc, out in outputs: + assert rc == 0 and out.startswith('OK'), out + with open(target) as f: + assert int(f.read().strip()) == 8 * 60 + with open(log_path) as f: + lines = f.read().splitlines() + assert len(lines) == 8 * 60 and len(set(lines)) == 8 * 60 + assert not os.path.exists(target + '.lock') + + +def test_the_engine_read_modify_write_cycle_under_contention(tmp_path): + target = str(tmp_path / 'index.json') + with open(target, 'w') as f: + json.dump({'n': 0}, f) + outputs = run_workers(COUNTER_WORKER, [('engine', target, '', 40)] * 6) + for rc, out in outputs: + assert rc == 0 and out.startswith('OK'), out + with open(target) as f: + assert json.load(f)['n'] == 6 * 40 + assert not os.path.exists(target + '.lock') + assert not os.path.exists(target + '.tmp') + + +def test_a_killed_holder_blocks_nobody(tmp_path): + lock_file = str(tmp_path / 'crash.lock') + holder = start(HOLDER, lock_file, 60) + line = holder.stdout.readline().decode('utf-8', 'replace') + assert line.startswith('LOCKED'), line + assert os.path.isfile(lock_file) + + with pytest.raises(TimeoutError): + files.PathLock(lock_file).acquire(timeout=0.3) + + holder.kill() + holder.wait(30) + + t0 = time.monotonic() + lock = files.PathLock(lock_file).acquire(timeout=10) + assert time.monotonic() - t0 < 5 + lock.release() + assert not os.path.exists(lock_file) + + +@pytest.mark.skipif(POSIX, reason='Windows: a lock file held by another process cannot be removed') +def test_windows_keeps_a_held_lock_file_from_being_removed(tmp_path): + lock_file = str(tmp_path / 'held.lock') + holder = start(HOLDER, lock_file, 60) + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('LOCKED') + with pytest.raises(PermissionError): + os.remove(lock_file) + holder.kill() + holder.wait(30) + + +@pytest.mark.skipif(not POSIX, reason='the identity check is a POSIX matter (flock on a removed file)') +def test_a_lock_file_removed_under_a_waiter_never_gives_two_holders(tmp_path): + """The race of the previous lock, replayed deterministically: a waiter has opened the lock file (A) and is + about to flock it when the holder removes A and releases, and a newcomer creates B and holds it. The waiter's + flock on A succeeds (nobody holds A any more) - and the identity check sends it back to try B.""" + lock_file = str(tmp_path / 'race.lock') + marker, go, out = str(tmp_path / 'marker'), str(tmp_path / 'go'), str(tmp_path / 'out') + + holder = files.PathLock(lock_file).acquire(timeout=5) + waiter = start(HOOKED_WAITER, lock_file, marker, go, out) + wait_for(marker) # the waiter has opened A + + holder.release() # A is removed, then released + assert not os.path.exists(lock_file) + newcomer = files.PathLock(lock_file).acquire(timeout=5) # B + open(go, 'w').close() + + time.sleep(0.5) + assert not os.path.exists(out), 'the waiter acquired while the newcomer held B' + + released_at = time.time() + newcomer.release() + waiter.wait(60) + assert waiter.returncode == 0, waiter.stdout.read().decode('utf-8', 'replace') + with open(out) as f: + acquired_at, retries, mode, same = f.read().split() + assert float(acquired_at) >= released_at - 0.001 + assert int(retries) >= 1 + assert mode == 'flock' and same == 'True' + assert not os.path.exists(lock_file) + + +@pytest.mark.skipif(not POSIX, reason='on Windows the file cannot be removed while it is held') +def test_release_leaves_a_lock_file_that_is_not_its_own(tmp_path): + lock_file = str(tmp_path / 'mine.lock') + mine = files.PathLock(lock_file).acquire(timeout=5) + os.unlink(lock_file) # a third party removed it ... + with open(lock_file, 'w') as f: # ... and another process made a new one + f.write('theirs') + mine.release() + assert os.path.isfile(lock_file) # theirs is kept + with open(lock_file) as f: + assert f.read() == 'theirs' + os.unlink(lock_file) + + +########################################################################################## +# The soft lock (where the file system refuses OS locks) + +@pytest.fixture() +def soft(monkeypatch, tmp_path): + lock_file = str(tmp_path / 'soft.lock') + monkeypatch.setattr(files, '_lock_soft_paths', {lock_file}) + return lock_file + + +def test_soft_lock_holds_pid_and_host_and_is_removed_on_release(soft, caplog): + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + with open(soft) as f: + pid, host, when = f.read().split() + assert int(pid) == os.getpid() and host == files._lock_host_name() + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + lock.release() + assert not os.path.exists(soft) + + +def test_soft_lock_excludes_processes(tmp_path): + target = str(tmp_path / 'counter.txt') + log_path = str(tmp_path / 'log.txt') + outputs = run_workers(COUNTER_WORKER, [('pathlock', target, log_path, 30, 'soft')] * 4) + for rc, out in outputs: + assert rc == 0 and 'OK' in out, out + assert 'stale' not in out, out # every holder is alive here: a "stale" removal would be a misjudgement + with open(target) as f: + assert int(f.read().strip()) == 4 * 30 + with open(log_path) as f: + lines = f.read().splitlines() + assert len(lines) == 4 * 30 and all(line.endswith(' soft') for line in lines) + assert not os.path.exists(target + '.lock') + + +def test_a_stale_soft_lock_of_a_dead_process_is_removed(soft, caplog, monkeypatch): + gone = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + gone.wait() + with open(soft, 'w') as f: + f.write(f'{gone.pid} {files._lock_host_name()} {time.time():.3f}\n') + caplog.set_level('WARNING', logger='cmeta.utils.files') + + # Fresh, the file of a dead pid is a release in flight: waited for + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + + old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + assert any('stale' in rec.getMessage() for rec in caplog.records) + lock.release() + + +def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkeypatch): + # A holder that could not remove its file (a reader held it open on Windows for too long) marks it released + with open(soft, 'w') as f: + f.write('released 12345\n') + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + # A fresh lock file of a live holder is never taken for a stale one, whatever the check says + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its content at once): + # fresh, it is waited for; old, it is cleared + open(soft, 'w').close() + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 30.0) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def test_a_soft_lock_of_another_host_is_stale_only_after_the_age(soft, monkeypatch): + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + assert os.path.isfile(soft) + + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def test_soft_lock_is_used_where_the_os_lock_is_refused(tmp_path, monkeypatch, caplog): + lock_file = str(tmp_path / 'refused.lock') + err = OSError(files.errno.ENOLCK, 'No locks available') + + def refuse(*args, **kwargs): + raise err + + if POSIX: + monkeypatch.setattr(files.fcntl, 'flock', refuse) + monkeypatch.setattr(files.fcntl, 'lockf', refuse) + else: + monkeypatch.setattr(files.msvcrt, 'locking', refuse) + monkeypatch.setattr(files, '_lock_soft_paths', set()) + monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) + caplog.set_level('WARNING', logger='cmeta.utils.files') + + lock = files.PathLock(lock_file).acquire(timeout=5) + assert lock.mode == 'soft' + assert lock_file in files._lock_soft_paths + assert any('soft lock' in rec.getMessage() for rec in caplog.records) + lock.release() + assert not os.path.exists(lock_file) + + again = files.PathLock(lock_file).acquire(timeout=5) # remembered: soft at once + assert again.mode == 'soft' + again.release() + + +@pytest.mark.skipif(not POSIX, reason='POSIX record locks are the POSIX fallback') +def test_posix_record_locks_are_used_where_flock_is_refused(tmp_path, monkeypatch): + lock_file = str(tmp_path / 'nfs-like.lock') + + def refuse(*args, **kwargs): + raise OSError(files.errno.ENOLCK, 'No locks available') + + monkeypatch.setattr(files.fcntl, 'flock', refuse) + monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) + lock = files.PathLock(lock_file).acquire(timeout=5) + assert lock.mode == 'lockf' + assert os.path.dirname(lock_file) in files._lock_no_flock_dirs + lock.release() + assert not os.path.exists(lock_file) + + +########################################################################################## +# The engine's functions + +def test_safe_delete_directory_takes_and_removes_its_lock(tmp_path): + folder = tmp_path / 'artifact' + folder.mkdir() + (folder / 'f.txt').write_text('x') + r = files.safe_delete_directory(str(folder)) + assert r['return'] == 0 + assert not folder.exists() + assert not os.path.exists(str(folder) + '.lock') + + +def test_safe_write_file_releases_the_lock_it_took(tmp_path): + target = str(tmp_path / 'data.json') + r = files.safe_write_file(target, {'a': 1}, atomic=True) + assert r['return'] == 0 + assert not os.path.exists(target + '.lock') and not os.path.exists(target + '.tmp') + r = files.safe_read_file(target, lock=True) + assert r['return'] == 0 and r['data'] == {'a': 1} + assert not os.path.exists(target + '.lock') diff --git a/tests/internal_repo_tests/test_integration_reindex_artifact.py b/tests/internal_repo_tests/test_integration_reindex_artifact.py new file mode 100644 index 0000000..da5a221 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_reindex_artifact.py @@ -0,0 +1,840 @@ +""" +Integration tests for `cx reindex []`: the index record of an artifact is +rewritten from the meta file in its folder without any file being changed - after a meta edited by +hand, a folder renamed or copied in by hand, a stale lookup - and nothing is ever removed from the +index or touched on disk. The record is the one a full `cx --reindex` records. + +Each test runs against a fresh in tmp_path so it does not touch the user's real +cMeta state. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import os +import pickle +import shutil +import subprocess +import sys + +import pytest + +from cmeta import CMeta +from cmeta import repos as cmeta_repos + +CATEGORY_UID = 'fedcba9876543210' + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + monkeypatch.setattr(cmeta_repos, '_migrated_notices', set()) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make(cm, alias, category='log', repo='local', yaml=False, **meta): + """An artifact made by cMeta; returns its UID.""" + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=yaml) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def reindex(cm, ref=None, category='log', **params): + return access(cm, category=category, command='reindex', arg1=ref, **params) + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def record(cm, ref, category='log'): + """The index record of one artifact as `find` returns it (path, cmeta, cmeta_ref_parts, sharding).""" + r = find(cm, ref, category=category) + assert r['return'] == 0, r.get('error') + assert len(r['artifacts']) == 1, r['artifacts'] + a = r['artifacts'][0] + return {k: a[k] for k in a if k != 'index_file'} + + +def index_data(tmp_path, category='log'): + """The pickled index of a category as it is on disk.""" + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def index_bytes(tmp_path, category='log'): + return (tmp_path / 'index' / f'{category}.pkl').read_bytes() + + +def meta_file(folder): + for name in ('_cmeta.yaml', '_cmeta.json'): + if (folder / name).is_file(): + return folder / name + raise AssertionError(f'no meta file in {folder}') + + +def file_state(path): + return (path.stat().st_mtime_ns, path.read_bytes()) + + +def hand_edit(folder, text): + """Append text to the meta file as a user would, keeping its format: YAML lines or a JSON key.""" + f = meta_file(folder) + if f.suffix == '.yaml': + f.write_text(f.read_text(encoding='utf-8') + text, encoding='utf-8') + else: + import json + data = json.loads(f.read_text(encoding='utf-8')) + for line in text.strip().splitlines(): + key, value = line.split(':', 1) + data[key.strip()] = value.strip() + f.write_text(json.dumps(data, indent=2, sort_keys=True) + '\n', encoding='utf-8') + return f + + +def make_repo(tmp_path, name, repo_uid): + """A repository folder with its descriptor only (its artifacts use the categories of the internal repo).""" + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def case_insensitive_fs(tmp_path): + probe = tmp_path / 'CaseProbe' + probe.mkdir(exist_ok=True) + return (tmp_path / 'caseprobe').is_dir() + + +########################################################################################## +# A meta edited by hand + +def test_reindex_reads_a_hand_edited_meta_and_leaves_the_file_alone(cm, tmp_path): + uid = make(cm, 'edited', yaml=True, tags=['before']) + folder = tmp_path / 'repos' / 'local' / 'log' / 'edited' + f = hand_edit(folder, 'owner: me\n') + before = file_state(f) + + assert find(cm, 'edited', tags='before')['return'] == 0 + assert 'owner' not in record(cm, 'edited')['cmeta'] + + r = reindex(cm, 'edited') + assert r['return'] == 0, r.get('error') + assert [a['cmeta_ref_parts']['artifact_uid'] for a in r['artifacts']] == [uid] + assert r['dropped_aliases'] == {} and r['errors'] == [] + + # The index knows the edit, the file is exactly as the user left it + assert record(cm, 'edited')['cmeta']['owner'] == 'me' + assert record(cm, uid)['cmeta']['tags'] == ['before'] + assert file_state(f) == before + + +def test_reindex_of_a_json_meta(cm, tmp_path): + uid = make(cm, 'json-one', payload='x') + folder = tmp_path / 'repos' / 'local' / 'log' / 'json-one' + f = hand_edit(folder, 'owner: me\n') + before = file_state(f) + + r = reindex(cm, 'json-one') + assert r['return'] == 0, r.get('error') + assert record(cm, 'json-one')['cmeta'] == {**record(cm, uid)['cmeta']} + assert record(cm, uid)['cmeta']['owner'] == 'me' and record(cm, uid)['cmeta']['payload'] == 'x' + assert file_state(f) == before + + +def test_the_record_is_the_one_of_a_full_reindex(cm, tmp_path): + make(cm, 'yaml-art', yaml=True, tags=['t1']) + make(cm, 'json-art', tags=['t2']) + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'yaml-art', 'owner: me\n') + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'json-art', 'owner: you\n') + + # A folder made by hand, unknown to the index, is indexed by reindex as well + hand_made = tmp_path / 'repos' / 'local' / 'log' / 'By-Hand' + hand_made.mkdir() + (hand_made / '_cmeta.yaml').write_text('artifact: By-Hand,0123456789abcdef\ncategory: log,487a7639093a4685\ntags:\n- hand\n', encoding='utf-8') + + r = reindex(cm) # every artifact of the category + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['By-Hand', 'json-art', 'yaml-art'] + + after_reindex = {ref: record(cm, ref) for ref in ('yaml-art', 'json-art', 'By-Hand', 'by-hand', '0123456789abcdef')} + data_reindex = index_data(tmp_path) + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + + for ref in after_reindex: + assert record(cm, ref) == after_reindex[ref], ref + + data_full = index_data(tmp_path) + assert data_full['uids'] == data_reindex['uids'] + assert data_full['lowercase_aliases'] == data_reindex['lowercase_aliases'] + # The alias keeps the case of the folder, the lowercase form is recorded when it differs + parts = after_reindex['By-Hand']['cmeta_ref_parts'] + assert parts['artifact_alias'] == 'By-Hand' and parts['artifact_alias_lowercase'] == 'by-hand' + + +def test_a_second_reindex_changes_nothing(cm, tmp_path): + make(cm, 'stable', yaml=True) + assert reindex(cm, 'stable')['return'] == 0 + data = index_data(tmp_path) + r = reindex(cm, 'stable') + assert r['return'] == 0 and r['dropped_aliases'] == {} + assert index_data(tmp_path) == data + + +########################################################################################## +# Folders renamed by hand + +def test_a_folder_renamed_by_hand_is_found_by_its_new_name(cm, tmp_path): + uid = make(cm, 'old-name', yaml=True, payload='x') + other = make(cm, 'other', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'old-name').rename(base / 'new-name') + + # Before: the index still answers with the old name and a folder that is gone + assert uids(find(cm, 'old-name')) == [uid] + assert find(cm, 'new-name')['return'] == 16 + + r = reindex(cm, 'new-name') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['old-name']} + + assert uids(find(cm, 'new-name')) == [uid] + assert find(cm, 'old-name')['return'] == 16 + assert same(record(cm, uid)['path']) == same(base / 'new-name') + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias'] == 'new-name' + assert record(cm, uid)['cmeta']['payload'] == 'x' + assert 'old-name' not in index_data(tmp_path)['lowercase_aliases'] + # alias,UID with the old alias still resolves: the UID is authoritative + assert uids(find(cm, f'old-name,{uid}')) == [uid] + # The other artifact is untouched + assert uids(find(cm, 'other')) == [other] + + +def test_a_folder_renamed_by_hand_is_found_by_its_uid_or_its_old_name(cm, tmp_path): + base = tmp_path / 'repos' / 'local' / 'log' + + uid1 = make(cm, 'by-uid', yaml=True) + (base / 'by-uid').rename(base / 'by-uid-renamed') + r = reindex(cm, uid1) + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid1: ['by-uid']} + assert uids(find(cm, 'by-uid-renamed')) == [uid1] and find(cm, 'by-uid')['return'] == 16 + + uid2 = make(cm, 'by-old', yaml=True) + (base / 'by-old').rename(base / 'by-old-renamed') + r = reindex(cm, 'by-old') # the stale alias: the folder is looked for by its UID + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid2: ['by-old']} + assert uids(find(cm, 'by-old-renamed')) == [uid2] and find(cm, 'by-old')['return'] == 16 + + uid3 = make(cm, 'by-both', yaml=True) + (base / 'by-both').rename(base / 'by-both-renamed') + r = reindex(cm, f'by-both,{uid3}') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-both-renamed')) == [uid3] and find(cm, 'by-both')['return'] == 16 + + data = index_data(tmp_path) + assert set(data['lowercase_aliases']) == {'by-uid-renamed', 'by-old-renamed', 'by-both-renamed'} + assert set(data['uids']) == {uid1, uid2, uid3} + + +def test_a_folder_renamed_twice_loses_both_old_aliases(cm, tmp_path): + uid = make(cm, 'first', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'first').rename(base / 'second') + # The plain "index" of the renamed folder keeps the old alias (today's behaviour, unchanged) + r = access(cm, category='log', command='index', arg1='local:second') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'first')) == [uid] and uids(find(cm, 'second')) == [uid] + (base / 'second').rename(base / 'third') + + r = reindex(cm, 'third') + assert r['return'] == 0, r.get('error') + assert sorted(r['dropped_aliases'][uid]) == ['first', 'second'] + assert find(cm, 'first')['return'] == 16 and find(cm, 'second')['return'] == 16 + assert uids(find(cm, 'third')) == [uid] + + +def test_a_case_only_rename_follows_the_folder(cm, tmp_path): + uid = make(cm, 'MixedCase', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias'] == 'MixedCase' + assert record(cm, uid)['cmeta_ref_parts']['artifact_alias_lowercase'] == 'mixedcase' + + os.rename(str(base / 'MixedCase'), str(base / 'mixedcase')) + assert sorted(p.name for p in base.iterdir()) == ['mixedcase'] + + r = reindex(cm, 'mixedcase') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {} # the same lowercase alias: nothing to drop + + parts = record(cm, uid)['cmeta_ref_parts'] + assert parts['artifact_alias'] == 'mixedcase' + assert 'artifact_alias_lowercase' not in parts + assert os.path.basename(record(cm, uid)['path']) == 'mixedcase' + # Lookups are case-insensitive either way + assert uids(find(cm, 'MixedCase')) == [uid] and uids(find(cm, 'MIXEDCASE')) == [uid] + + +def test_two_spellings_on_a_case_sensitive_file_system(cm, tmp_path): + if case_insensitive_fs(tmp_path): + pytest.skip('the file system folds case: two spellings cannot coexist') + base = tmp_path / 'repos' / 'local' / 'log' + uid1 = make(cm, 'Spell', yaml=True) + # cMeta itself refuses the second spelling (the index is case-insensitive): it can only be made by hand + r = access(cm, category='log', command='create', arg1='local:spell', yaml=True) + assert r['return'] == 8 + (base / 'spell').mkdir() + (base / 'spell' / '_cmeta.yaml').write_text('artifact: spell,dddddddddddddddd\ncategory: log,487a7639093a4685\nowner: two\n', encoding='utf-8') + hand_edit(base / 'Spell', 'owner: one\n') + + # Both folders are found, whichever spelling is given, and both UIDs sit under the one lowercase alias + r = reindex(cm, 'spell') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([uid1, 'dddddddddddddddd']) + assert record(cm, uid1)['cmeta']['owner'] == 'one' and record(cm, 'dddddddddddddddd')['cmeta']['owner'] == 'two' + assert record(cm, uid1)['cmeta_ref_parts']['artifact_alias'] == 'Spell' + assert record(cm, 'dddddddddddddddd')['cmeta_ref_parts']['artifact_alias'] == 'spell' + assert sorted(index_data(tmp_path)['lowercase_aliases']['spell']) == sorted([uid1, 'dddddddddddddddd']) + # A lookup by the alias is ambiguous, as after a full reindex + assert uids(find(cm, 'Spell')) == sorted([uid1, 'dddddddddddddddd']) + both = {uid: record(cm, uid) for uid in (uid1, 'dddddddddddddddd')} + assert access(cm, reindex=True)['return'] == 0 + for uid in both: + assert record(cm, uid) == both[uid] + + +def test_a_json_meta_renamed_by_hand(cm, tmp_path): + uid = make(cm, 'js-old', payload='j') # a JSON meta + base = tmp_path / 'repos' / 'local' / 'log' + assert (base / 'js-old' / '_cmeta.json').is_file() + (base / 'js-old').rename(base / 'js-new') + f = base / 'js-new' / '_cmeta.json' + before = file_state(f) + + r = reindex(cm, 'js-new') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['js-old']} + assert uids(find(cm, 'js-new')) == [uid] and find(cm, 'js-old')['return'] == 16 + assert record(cm, uid)['cmeta']['payload'] == 'j' + assert file_state(f) == before + + +def test_an_upper_case_uid_in_a_meta(cm, tmp_path): + make(cm, 'seed', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'upper').mkdir() + (base / 'upper' / '_cmeta.yaml').write_text('artifact: upper,ABCDEF0123456789\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'upper') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['abcdef0123456789'] + assert uids(find(cm, 'ABCDEF0123456789')) == ['abcdef0123456789'] + assert uids(find(cm, 'abcdef0123456789')) == ['abcdef0123456789'] + rec = record(cm, 'upper') + assert access(cm, reindex=True)['return'] == 0 + assert record(cm, 'upper') == rec + + +def test_an_unreadable_meta_file(cm, tmp_path): + uid = make(cm, 'broken', yaml=True) + other = make(cm, 'fine', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + f = base / 'broken' / '_cmeta.yaml' + f.write_text('artifact: [unclosed\n', encoding='utf-8') + before = index_bytes(tmp_path) + state = file_state(f) + + for ref in ('broken', uid): + r = reindex(cm, ref) + assert r['return'] == 1 and 'cannot be read' in r['error'], r + assert index_bytes(tmp_path) == before + assert file_state(f) == state # the broken file is not touched either + + hand_edit(base / 'fine', 'owner: me\n') + r = reindex(cm, ignore_errors=True) + assert r['return'] == 0 and uids(r) == [other] and len(r['errors']) == 1 + assert uids(find(cm, 'broken')) == [uid] # its old record is kept + + +def test_a_non_ascii_alias(cm, tmp_path): + r = access(cm, category='log', command='create', arg1='local:café-ünïcode', yaml=True, meta={'payload': 'é'}) + if r['return'] != 0: + pytest.skip(f'non-ASCII aliases are refused here: {r.get("error")}') + uid = r['meta']['artifact'] + base = tmp_path / 'repos' / 'local' / 'log' + hand_edit(base / 'café-ünïcode', 'owner: moi\n') + r = reindex(cm, 'café-ünïcode') + assert r['return'] == 0, r.get('error') + assert record(cm, uid)['cmeta']['owner'] == 'moi' + + (base / 'café-ünïcode').rename(base / 'naïve-änderung') + r = reindex(cm, 'naïve-änderung') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['café-ünïcode']} + assert uids(find(cm, 'naïve-änderung')) == [uid] and uids(find(cm, 'NAÏVE-ÄNDERUNG')) == [uid] + assert find(cm, 'café-ünïcode')['return'] == 16 + + +def test_a_rename_by_hand_inside_a_sharded_category(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', 'sharding_slices:\n- 2\n') + plug(cm, repo) + uid = make(cm, 'abcdef', category='shard', repo='shard', yaml=True) + folder = pytest.importorskip('pathlib').Path(find(cm, 'abcdef', category='shard')['artifacts'][0]['path']) + # Renamed in place: the folder stays in the "ab" shard although the new name would belong to "xy" + folder.rename(folder.parent / 'xyzdef') + + r = reindex(cm, uid, category='shard') + assert r['return'] == 0, r.get('error') + assert r['dropped_aliases'] == {uid: ['abcdef']} + rec = record(cm, uid, category='shard') + assert rec['cmeta_ref_parts']['artifact_alias'] == 'xyzdef' and rec['sharding_slices_num'] == 1 + assert same(rec['path']) == same(folder.parent / 'xyzdef') + assert uids(find(cm, 'xyzdef', category='shard')) == [uid] + # The same record as the full reindex, which also finds the folder where it is + assert access(cm, reindex=True)['return'] == 0 + assert record(cm, uid, category='shard') == rec + + +def test_a_category_without_an_index_file_yet(cm, tmp_path): + repo = make_category_repo(tmp_path, 'fresh', '5555555555555555', '') + plug(cm, repo) + assert not (tmp_path / 'index' / 'fresh.pkl').is_file() + folder = repo / 'fresh' / 'first' + folder.mkdir(parents=True) + (folder / '_cmeta.yaml').write_text(f'artifact: first,eeeeeeeeeeeeeeee\ncategory: fresh,{CATEGORY_UID}\n', encoding='utf-8') + + r = reindex(cm, category='fresh') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['eeeeeeeeeeeeeeee'] + assert (tmp_path / 'index' / 'fresh.pkl').is_file() + assert uids(find(cm, 'first', category='fresh')) == ['eeeeeeeeeeeeeeee'] + + +def test_a_repository_with_a_subdir(cm, tmp_path): + repo = tmp_path / 'extra' / 'withsub' + (repo / 'content').mkdir(parents=True) + (repo / '_cmr.yaml').write_text('artifact: withsub,6666666666666666\ncategory: repo,f4f792ab40c7498f\nsubdir: content\n', encoding='utf-8') + plug(cm, repo) + uid = make(cm, 'in-sub', repo='withsub', yaml=True) + folder = repo / 'content' / 'log' / 'in-sub' + assert folder.is_dir() + hand_edit(folder, 'owner: me\n') + + r = reindex(cm, 'withsub:in-sub') + assert r['return'] == 0, r.get('error') + assert record(cm, uid)['cmeta']['owner'] == 'me' + assert same(record(cm, uid)['path']) == same(folder) + + (folder.parent / 'in-sub').rename(folder.parent / 'in-sub-2') + r = reindex(cm, uid) + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'in-sub-2')) == [uid] and find(cm, 'in-sub')['return'] == 16 + + +def test_the_shipped_internal_repository_is_only_read(cm): + before = record(cm, 'log', category='category') + f = pytest.importorskip('pathlib').Path(before['path']) / '_cmeta.yaml' + state = file_state(f) + + r = reindex(cm, 'log', category='category') + assert r['return'] == 0, r.get('error') + assert record(cm, 'log', category='category') == before + assert file_state(f) == state + + +def test_several_processes_reindex_at_once(cm, tmp_path): + # Six processes write the same index file at once: with the previous lock (its file removed after the + # release) a waiter and a newcomer could both acquire on Linux and one record was lost; the engine's own + # PathLock (the file removed before the release, the identity of the file checked after acquiring) keeps + # every record on every platform + base = tmp_path / 'repos' / 'local' / 'log' + names = [f'par-{i}' for i in range(6)] + made = {name: make(cm, name, yaml=True) for name in names} + for name in names: + hand_edit(base / name, f'owner: {name}\n') + + env = {k: v for k, v in os.environ.items() if not k.startswith('CMETA_') and k not in ('VIRTUAL_ENV', 'CONDA_PREFIX')} + env['CMETA_HOME'] = str(tmp_path) + env['PYTHONIOENCODING'] = 'utf-8' + + procs = [subprocess.Popen([sys.executable, '-m', 'cmeta', 'log', 'reindex', name], env=env, + stdin=subprocess.DEVNULL, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) for name in names] + outputs = [p.communicate(timeout=180)[0] for p in procs] + assert all(p.returncode == 0 for p in procs), outputs + + data = index_data(tmp_path) + for name in names: + assert data['uids'][made[name]]['cmeta']['owner'] == name + assert data['lowercase_aliases'][name] == [made[name]] + assert len(data['uids']) == len(names) + + +########################################################################################## +# Nothing is lost: the errors leave the index as it was + +def test_two_folders_with_one_uid_is_an_error_that_changes_nothing(cm, tmp_path): + uid = make(cm, 'original', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + shutil.copytree(base / 'original', base / 'copied') + before = index_bytes(tmp_path) + + for ref in ('copied', uid, None): + r = reindex(cm, ref) + assert r['return'] == 1, (ref, r) + assert 'same UID' in r['error'] or 'two folders with one UID' in r['error'], r['error'] + assert index_bytes(tmp_path) == before, ref + + # With --ignore_errors the other artifacts are reindexed and the conflict is reported + other = make(cm, 'fine', yaml=True) + hand_edit(base / 'fine', 'owner: me\n') + r = reindex(cm, ignore_errors=True) + assert r['return'] == 0, r.get('error') + assert uids(r) == [other] + assert len(r['errors']) == 1 and uid in r['errors'][0]['error'] + assert record(cm, other)['cmeta']['owner'] == 'me' + assert same(record(cm, uid)['path']) == same(base / 'original') + + # A new UID for the copy resolves it + (base / 'copied' / '_cmeta.yaml').write_text('artifact: copied,aaaaaaaaaaaaaaaa\ncategory: log,487a7639093a4685\n', encoding='utf-8') + r = reindex(cm, 'copied') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'copied')) == ['aaaaaaaaaaaaaaaa'] and uids(find(cm, 'original')) == [uid] + + +def test_a_missing_folder_is_an_error_that_changes_nothing(cm, tmp_path): + uid = make(cm, 'gone', yaml=True) + shutil.rmtree(tmp_path / 'repos' / 'local' / 'log' / 'gone') + before = index_bytes(tmp_path) + + for ref in ('gone', uid): + r = reindex(cm, ref) + assert r['return'] == 1 and 'no folder with its UID' in r['error'], r + assert index_bytes(tmp_path) == before + + # delete, as the error says, removes the record of the missing folder + r = access(cm, category='log', command='delete', arg1='gone', force=True) + assert r['return'] == 0, r.get('error') + assert find(cm, 'gone')['return'] == 16 + + +def test_a_folder_replaced_by_another_artifact(cm, tmp_path): + # The folder of an artifact was removed and another artifact was put under the same name by hand + uid = make(cm, 'slot', yaml=True) + other = make(cm, 'other', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + shutil.rmtree(base / 'slot') + (base / 'slot').mkdir() + (base / 'slot' / '_cmeta.yaml').write_text('artifact: slot,aaaaaaaaaaaaaaaa\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'slot') + assert r['return'] == 0, r.get('error') + assert uids(r) == ['aaaaaaaaaaaaaaaa'] + # The record of the artifact that is not there any more is dropped, and said + assert len(r['dropped_records']) == 1 and uid in r['dropped_records'][0]['ref'] and 'aaaaaaaaaaaaaaaa' in r['dropped_records'][0]['now'] + assert uids(find(cm, 'slot')) == ['aaaaaaaaaaaaaaaa'] + assert find(cm, uid)['return'] == 16 + assert uid not in index_data(tmp_path)['uids'] + assert index_data(tmp_path)['lowercase_aliases']['slot'] == ['aaaaaaaaaaaaaaaa'] + assert uids(find(cm, 'other')) == [other] + # The new artifact's folder is untouched + assert (base / 'slot' / '_cmeta.yaml').is_file() + + +def test_a_folder_moved_and_its_old_name_taken(cm, tmp_path): + # The artifact was renamed by hand and another artifact took its old name + uid1 = make(cm, 'taken', yaml=True, payload='one') + base = tmp_path / 'repos' / 'local' / 'log' + (base / 'taken').rename(base / 'moved-away') + (base / 'taken').mkdir() + (base / 'taken' / '_cmeta.yaml').write_text('artifact: taken,bbbbbbbbbbbbbbbb\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'taken') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([uid1, 'bbbbbbbbbbbbbbbb']) + assert r['dropped_records'] == [] + assert r['dropped_aliases'] == {uid1: ['taken']} + assert uids(find(cm, 'taken')) == ['bbbbbbbbbbbbbbbb'] + assert uids(find(cm, 'moved-away')) == [uid1] + assert record(cm, uid1)['cmeta']['payload'] == 'one' + + +def test_an_unknown_artifact_is_not_found(cm, tmp_path): + make(cm, 'exists', yaml=True) + r = reindex(cm, 'does-not-exist') + assert r['return'] == 16 and 'not found' in r['error'] + r = reindex(cm, 'ffffffffffffffff') + assert r['return'] == 16 + r = reindex(cm, 'no-such-repo:exists') + assert r['return'] == 16 + + +def test_a_hand_made_folder_without_a_valid_uid_is_not_indexed(cm, tmp_path): + make(cm, 'exists', yaml=True) + before = index_bytes(tmp_path) + bad = tmp_path / 'repos' / 'local' / 'log' / 'bad-uid' + bad.mkdir() + (bad / '_cmeta.yaml').write_text('artifact: bad-uid\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'bad-uid') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + no_key = tmp_path / 'repos' / 'local' / 'log' / 'no-key' + no_key.mkdir() + (no_key / '_cmeta.yaml').write_text('tags:\n- x\n', encoding='utf-8') + r = reindex(cm, 'no-key') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + +def test_a_folder_of_another_category_is_not_taken(cm, tmp_path): + # A folder copied into the wrong category folder: its meta names another category + make(cm, 'exists', yaml=True) + before = index_bytes(tmp_path) + wrong = tmp_path / 'repos' / 'local' / 'log' / 'an-asset' + wrong.mkdir() + (wrong / '_cmeta.yaml').write_text('artifact: an-asset,bbbbbbbbbbbbbbbb\ncategory: asset,e0982f19227743c8\n', encoding='utf-8') + r = reindex(cm, 'an-asset') + assert r['return'] == 16, r + assert index_bytes(tmp_path) == before + + +def test_other_records_are_untouched(cm, tmp_path): + uid_a = make(cm, 'art-a', yaml=True, tags=['a']) + uid_b = make(cm, 'art-b', yaml=True, tags=['b']) + uid_c = make(cm, 'art-c', tags=['c']) + data = index_data(tmp_path) + + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'art-b', 'owner: me\n') + r = reindex(cm, 'art-b') + assert r['return'] == 0, r.get('error') + + after = index_data(tmp_path) + for uid in (uid_a, uid_c): + assert after['uids'][uid] == data['uids'][uid] + assert after['uids'][uid_b]['cmeta']['owner'] == 'me' + assert after['lowercase_aliases'] == data['lowercase_aliases'] + + +########################################################################################## +# Selections: wildcards, tags, a repository, everything + +def test_wildcards_tags_and_a_repository(cm, tmp_path): + repo = make_repo(tmp_path, 'second', '2222222222222222') + plug(cm, repo) + base_local = tmp_path / 'repos' / 'local' / 'log' + base_second = repo / 'log' + + a1 = make(cm, 'wild-1', yaml=True, tags=['odd']) + a2 = make(cm, 'wild-2', yaml=True, tags=['even']) + s1 = make(cm, 'wild-1', repo='second', yaml=True, tags=['odd', 'second']) + other = make(cm, 'other', yaml=True) + for folder in (base_local / 'wild-1', base_local / 'wild-2', base_second / 'wild-1', base_local / 'other'): + hand_edit(folder, 'owner: me\n') + + r = reindex(cm, 'wild-*') + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([a1, a2, s1]) + assert 'owner' not in record(cm, other)['cmeta'] + + r = reindex(cm, 'other', tags='nope') + assert r['return'] == 16 + + hand_edit(base_local / 'wild-1', 'owner2: me\n') + hand_edit(base_second / 'wild-1', 'owner2: me\n') + r = reindex(cm, 'second:wild-1') # the one in the second repository only + assert r['return'] == 0, r.get('error') + assert uids(r) == [s1] + assert 'owner2' in record(cm, s1)['cmeta'] and 'owner2' not in record(cm, a1)['cmeta'] + # The same alias in two repositories: both UIDs stay under the alias + assert sorted(index_data(tmp_path)['lowercase_aliases']['wild-1']) == sorted([a1, s1]) + + r = reindex(cm, 'wild-*', tags='odd,-second') + assert r['return'] == 0, r.get('error') + assert uids(r) == [a1] + + r = reindex(cm, 'local:') # everything of the category in one repository + assert r['return'] == 0, r.get('error') + assert uids(r) == sorted([a1, a2, other]) + + +def test_everything_of_the_category_takes_the_folders_made_by_hand(cm, tmp_path): + make(cm, 'made', yaml=True) + base = tmp_path / 'repos' / 'local' / 'log' + for i in range(3): + folder = base / f'hand-{i}' + folder.mkdir() + (folder / '_cmeta.yaml').write_text(f'artifact: hand-{i},{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}{i}\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm) + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['hand-0', 'hand-1', 'hand-2', 'made'] + assert uids(find(cm, 'hand-1')) == ['1111111111111111'] + + +def test_a_folder_named_by_its_uid(cm, tmp_path): + base = tmp_path / 'repos' / 'local' / 'log' + make(cm, 'seed', yaml=True) # the index exists + folder = base / 'cccccccccccccccc' + folder.mkdir() + (folder / '_cmeta.yaml').write_text('artifact: cccccccccccccccc\ncategory: log,487a7639093a4685\n', encoding='utf-8') + + r = reindex(cm, 'cccccccccccccccc') + assert r['return'] == 0, r.get('error') + parts = record(cm, 'cccccccccccccccc')['cmeta_ref_parts'] + assert parts['artifact_uid'] == 'cccccccccccccccc' + assert parts.get('artifact_alias') == 'cccccccccccccccc' # the folder's name, as the full reindex records it + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, 'cccccccccccccccc')['cmeta_ref_parts'] == parts + + +########################################################################################## +# Categories with special layouts + +def make_category_repo(tmp_path, name, repo_uid, category_meta_extra): + """A repository with a category of its own (`shard` or `noidx`) and nothing else.""" + repo = tmp_path / 'extra' / name + category = repo / 'category' / name + (category / 'api').mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + (category / '_cmeta.yaml').write_text(f'artifact: {name},{CATEGORY_UID}\ncategory: category,dd9ea50e7f76467f\n' + 'last_api_version: 1\nbase_category_default_api_versions:\n "1": 1\n' + category_meta_extra, + encoding='utf-8') + (category / 'api' / 'v1.py').write_text('from cmeta.category import InitCategory\n\nclass Category(InitCategory):\n' + ' def __init__(self, *args, **kwargs):\n' + ' super().__init__(*args, module_file_path = __file__, **kwargs)\n', encoding='utf-8') + return repo + + +def test_a_sharded_category(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', 'sharding_slices:\n- 2\n') + plug(cm, repo) + + uid = make(cm, 'abcdef', category='shard', repo='shard', yaml=True) + r = find(cm, 'abcdef', category='shard') + folder = r['artifacts'][0]['path'] + assert r['artifacts'][0]['sharding_slices_num'] == 1 + assert os.path.basename(os.path.dirname(folder)) == 'ab' # the shard + + hand_edit(pytest.importorskip('pathlib').Path(folder), 'owner: me\n') + r = reindex(cm, 'abcdef', category='shard') + assert r['return'] == 0, r.get('error') + assert r['artifacts'][0]['sharding_slices_num'] == 1 + after_reindex = record(cm, uid, category='shard') + assert after_reindex['cmeta']['owner'] == 'me' and after_reindex['sharding_slices_num'] == 1 + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, uid, category='shard') == after_reindex + + +def test_a_no_index_category_is_refused(cm, tmp_path): + repo = make_category_repo(tmp_path, 'noidx', '4444444444444444', 'no_index: true\n') + plug(cm, repo) + scanned = make(cm, 'scanned', category='noidx', repo='noidx', yaml=True) + make(cm, 'other', category='noidx', repo='noidx', yaml=True) + r = reindex(cm, 'scanned', category='noidx') + assert r['return'] == 1 and 'no_index' in r['error'] + + # The scan that finds the artifacts of such a category honours a wildcard (it used to return every folder) + assert uids(find(cm, 'sc*', category='noidx')) == [scanned] + assert uids(find(cm, 'SCAN?ED', category='noidx')) == [scanned] + assert find(cm, 'nothing-*', category='noidx')['return'] == 16 + assert len(find(cm, '*', category='noidx')['artifacts']) == 2 + + +def test_the_record_of_a_category_itself(cm, tmp_path): + repo = make_category_repo(tmp_path, 'shard', '3333333333333333', '') + plug(cm, repo) + assert 'extra' not in record(cm, 'shard', category='category')['cmeta'] + + f = repo / 'category' / 'shard' / '_cmeta.yaml' + f.write_text(f.read_text(encoding='utf-8') + 'extra: 1\n', encoding='utf-8') + before = file_state(f) + + r = reindex(cm, 'shard', category='category') + assert r['return'] == 0, r.get('error') + assert record(cm, 'shard', category='category')['cmeta']['extra'] == 1 + assert file_state(f) == before + after_reindex = record(cm, 'shard', category='category') + + r = access(cm, reindex=True) + assert r['return'] == 0, r.get('error') + assert record(cm, 'shard', category='category') == after_reindex + + +def test_a_migration_stub_is_reindexed_as_itself(cm, tmp_path): + uid = make(cm, 'old-name', yaml=True) + r = access(cm, category='log', command='migrate', arg1='old-name', arg2='new-name') + assert r['return'] == 0, r.get('error') + stub_uid = r['stub_meta']['artifact'] + + r = reindex(cm, 'old-name') + assert r['return'] == 0, r.get('error') + assert uids(r) == [stub_uid] # the stub, not the artifact it points to + assert record(cm, stub_uid)['cmeta']['migrated_to'] == f'new-name,{uid}' + assert uids(find(cm, 'old-name')) == [uid] # the old alias alone still follows the stub + + +########################################################################################## +# The commands that exist keep their answers + +def test_index_and_update_are_unchanged(cm, tmp_path): + uid = make(cm, 'kept', yaml=True) + # "index" still refuses an artifact that is indexed + r = access(cm, category='log', command='index', arg1='local:kept') + assert r['return'] == 8 + # an empty "update" still refreshes the record and rewrites the file (its timestamp) + f = meta_file(tmp_path / 'repos' / 'local' / 'log' / 'kept') + before = file_state(f) + r = access(cm, category='log', command='update', arg1='kept') + assert r['return'] == 0, r.get('error') + assert file_state(f) != before + assert 'last_update_timestamp' in record(cm, uid)['cmeta'] + + +def test_the_cli_route(cm, tmp_path): + uid = make(cm, 'via-cli', yaml=True) + hand_edit(tmp_path / 'repos' / 'local' / 'log' / 'via-cli', 'owner: me\n') + + env = {k: v for k, v in os.environ.items() if not k.startswith('CMETA_') and k not in ('VIRTUAL_ENV', 'CONDA_PREFIX')} + env['CMETA_HOME'] = str(tmp_path) + env['PYTHONIOENCODING'] = 'utf-8' + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'via-cli'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) + assert r.returncode == 0, r.stdout + r.stderr + assert f'Reindexed log "via-cli" ({uid})' in r.stdout + assert record(cm, uid)['cmeta']['owner'] == 'me' + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', 'nothing-here'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) + assert r.returncode == 16, r.stdout + r.stderr + + r = subprocess.run([sys.executable, '-m', 'cmeta', 'log', 'reindex', '--help'], stdin=subprocess.DEVNULL, capture_output=True, text=True, env=env) + assert r.returncode == 0 and 'ignore_errors' in r.stdout, r.stdout + r.stderr diff --git a/tests/internal_repo_tests/test_integration_repo_reindex.py b/tests/internal_repo_tests/test_integration_repo_reindex.py new file mode 100644 index 0000000..cd31dd7 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_reindex.py @@ -0,0 +1,242 @@ +""" +Integration tests for `cx repo reindex `: one registered repository is indexed again +(its `_cmr.yaml` re-read, its artifacts rescanned) while the records of the other repositories +stay as they are; a repository whose folder is not there is refused. Also the incremental index +that pull, plug and unzip use: the records of a category of which the repository holds no +artifact any more are dropped. + +Each test runs against a fresh in tmp_path so it does not touch the user's real +cMeta state. + +cMeta author and developer: (C) 2025-2026 Grigori Fursin + +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import pickle +import shutil + +import pytest + +from cmeta import CMeta + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid): + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def make(cm, alias, repo, category='log', **meta): + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=True) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def hand_made(repo, alias, uid, category='log', category_ref='log,487a7639093a4685'): + folder = repo / category / alias + folder.mkdir(parents=True) + (folder / '_cmeta.yaml').write_text(f'artifact: {alias},{uid}\ncategory: {category_ref}\n', encoding='utf-8') + return folder + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def repo_record(cm, ref): + r = access(cm, category='repo', command='find', arg1=ref) + assert r['return'] == 0, r.get('error') + assert len(r['artifacts']) == 1 + return r['artifacts'][0] + + +def index_data(tmp_path, category): + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def repos_json(tmp_path): + return json.loads((tmp_path / 'repos.json').read_text(encoding='utf-8')) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def test_repo_reindex_rereads_the_descriptor_and_rescans_the_artifacts(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + local_uid = make(cm, 'in-local', 'local', tags=['local']) + kept = make(cm, 'kept', 'mine') + removed = make(cm, 'removed', 'mine') + local_before = index_data(tmp_path, 'log')['uids'][local_uid] + + # Changes made behind cMeta's back: the descriptor, a new folder, a folder removed + f = repo / '_cmr.yaml' + f.write_text(f.read_text(encoding='utf-8') + 'owner: me\n', encoding='utf-8') + hand_made(repo, 'added', 'aaaaaaaaaaaaaaaa') + shutil.rmtree(repo / 'log' / 'removed') + + assert 'owner' not in repo_record(cm, 'mine')['cmeta'] + assert find(cm, 'added')['return'] == 16 + assert uids(find(cm, 'removed')) == [removed] + + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + assert [a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']] == ['mine'] + assert [same(p) for p in r['paths']] == [same(repo)] + + assert repo_record(cm, 'mine')['cmeta']['owner'] == 'me' + assert uids(find(cm, 'added')) == ['aaaaaaaaaaaaaaaa'] + assert uids(find(cm, 'kept')) == [kept] + assert find(cm, 'removed')['return'] == 16 + assert 'removed' not in index_data(tmp_path, 'log')['lowercase_aliases'] + # The other repository's records are as they were + assert index_data(tmp_path, 'log')['uids'][local_uid] == local_before + assert repos_json(tmp_path) == repos_json(tmp_path) # still readable + assert same(repo) in [same(p) for p in repos_json(tmp_path)] + + +def test_repo_reindex_by_uid_path_and_dot(cm, tmp_path, monkeypatch): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + make(cm, 'seed', 'mine') + + hand_made(repo, 'by-uid', 'aaaaaaaaaaaaaaaa') + r = access(cm, category='repo', command='reindex', arg1='1111111111111111') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-uid')) == ['aaaaaaaaaaaaaaaa'] + + hand_made(repo, 'by-path', 'bbbbbbbbbbbbbbbb') + r = access(cm, category='repo', command='reindex', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-path')) == ['bbbbbbbbbbbbbbbb'] + + hand_made(repo, 'by-sub-path', 'cccccccccccccccc') + r = access(cm, category='repo', command='reindex', arg1=str(repo / 'log')) # a folder inside the repository + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-sub-path')) == ['cccccccccccccccc'] + + hand_made(repo, 'by-dot', 'dddddddddddddddd') + monkeypatch.chdir(repo / 'log' / 'by-dot') + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'by-dot')) == ['dddddddddddddddd'] + + monkeypatch.chdir(tmp_path) + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 16 and 'plug' in r['error'] + + +def test_repo_reindex_refuses_what_it_cannot_do_safely(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'seed', 'mine') + before = index_data(tmp_path, 'log') + repos_before = repos_json(tmp_path) + + r = access(cm, category='repo', command='reindex') + assert r['return'] == 1 and 'cx --reindex' in r['error'] + + r = access(cm, category='repo', command='reindex', arg1='no-such-repo') + assert r['return'] == 16 + + # The repository's folder is not there (a detached drive): refused, nothing changed, the + # registration and the records stay + moved = tmp_path / 'extra' / 'mine-away' + (tmp_path / 'extra' / 'mine').rename(moved) + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 1 and 'not there' in r['error'], r + assert index_data(tmp_path, 'log') == before + assert repos_json(tmp_path) == repos_before + assert uids(find(cm, 'seed')) == [uid] + + moved.rename(tmp_path / 'extra' / 'mine') + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'seed')) == [uid] + + +def test_repo_reindex_after_the_descriptor_renamed_the_repository(cm, tmp_path): + repo = make_repo(tmp_path, 'old-repo', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'art', 'old-repo') + + (repo / '_cmr.yaml').write_text('artifact: new-repo,1111111111111111\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + + r = access(cm, category='repo', command='reindex', arg1='1111111111111111') + assert r['return'] == 0, r.get('error') + assert repo_record(cm, 'new-repo')['cmeta_ref_parts']['artifact_uid'] == '1111111111111111' + assert access(cm, category='repo', command='find', arg1='old-repo')['return'] == 16 + assert find(cm, 'art')['artifacts'][0]['cmeta_ref_parts']['repo_alias'] == 'new-repo' + assert uids(find(cm, 'new-repo:art')) == [uid] + + +def test_repo_reindex_with_a_wildcard_and_several_repositories(cm, tmp_path): + repo_a = make_repo(tmp_path, 'mine-a', '1111111111111111') + repo_b = make_repo(tmp_path, 'mine-b', '2222222222222222') + other = make_repo(tmp_path, 'other', '3333333333333333') + for repo in (repo_a, repo_b, other): + plug(cm, repo) + hand_made(repo_a, 'in-a', 'aaaaaaaaaaaaaaaa') + hand_made(repo_b, 'in-b', 'bbbbbbbbbbbbbbbb') + hand_made(other, 'in-other', 'cccccccccccccccc') + + r = access(cm, category='repo', command='reindex', arg1='mine-*') + assert r['return'] == 0, r.get('error') + assert sorted(a['cmeta_ref_parts']['artifact_alias'] for a in r['artifacts']) == ['mine-a', 'mine-b'] + assert uids(find(cm, 'in-a')) == ['aaaaaaaaaaaaaaaa'] and uids(find(cm, 'in-b')) == ['bbbbbbbbbbbbbbbb'] + assert find(cm, 'in-other')['return'] == 16 # not asked for, not scanned + + +def test_the_incremental_index_drops_the_records_of_an_emptied_category(cm, tmp_path): + # The index that pull, plug and reindex share: a repository whose last artifact of a category + # was removed by hand loses that record too + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + local_uid = make(cm, 'in-local', 'local') + only = make(cm, 'only-one', 'mine') + asset = make(cm, 'an-asset', 'mine', category='asset') + + shutil.rmtree(repo / 'log' / 'only-one') + r = access(cm, category='repo', command='reindex', arg1='mine') + assert r['return'] == 0, r.get('error') + + assert find(cm, 'only-one')['return'] == 16 + assert only not in index_data(tmp_path, 'log')['uids'] + assert uids(find(cm, 'in-local')) == [local_uid] + assert uids(find(cm, 'an-asset', category='asset')) == [asset] + + # The same through plug of a second copy: plugging indexes incrementally as well + other = make_repo(tmp_path, 'other', '2222222222222222') + hand_made(other, 'in-other', 'eeeeeeeeeeeeeeee') + plug(cm, other) + assert uids(find(cm, 'in-other')) == ['eeeeeeeeeeeeeeee'] + assert uids(find(cm, 'in-local')) == [local_uid] diff --git a/uv.lock b/uv.lock index 567a818..95531da 100644 --- a/uv.lock +++ b/uv.lock @@ -164,7 +164,6 @@ wheels = [ name = "cmeta" source = { editable = "." } dependencies = [ - { name = "filelock" }, { name = "packaging" }, { name = "psutil" }, { name = "pyyaml" }, @@ -206,7 +205,6 @@ requires-dist = [ { name = "cmeta", extras = ["async"], marker = "extra == 'server'" }, { name = "cmeta", extras = ["async", "dev", "server"], marker = "extra == 'all'" }, { name = "fastapi", marker = "extra == 'server'" }, - { name = "filelock" }, { name = "flake8", marker = "extra == 'dev'" }, { name = "httpx", marker = "extra == 'dev'" }, { name = "itsdangerous", marker = "extra == 'server'" }, @@ -378,15 +376,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3e/57/aa70121b5008f44031be645a61a7c4abc24e0e888ad3fc8fda916f4d188e/fastapi-0.124.4-py3-none-any.whl", hash = "sha256:6d1e703698443ccb89e50abe4893f3c84d9d6689c0cf1ca4fad6d3c15cf69f15", size = 113281, upload-time = "2025-12-12T15:00:42.44Z" }, ] -[[package]] -name = "filelock" -version = "3.19.1" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/40/bb/0ab3e58d22305b6f5440629d20683af28959bf793d98d11950e305c1c326/filelock-3.19.1.tar.gz", hash = "sha256:66eda1888b0171c998b35be2bcc0f6d75c388a7ce20c3f3f37aa8e96c2dddf58", size = 17687, upload-time = "2025-08-14T16:56:03.016Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/42/14/42b2651a2f46b022ccd948bca9f2d5af0fd8929c4eec235b8d6d844fbe67/filelock-3.19.1-py3-none-any.whl", hash = "sha256:d38e30481def20772f5baf097c122c3babc4fcdb7e14e57049eb9d88c6dc017d", size = 15988, upload-time = "2025-08-14T16:56:01.633Z" }, -] - [[package]] name = "flake8" version = "7.3.0"