20261009 - cMeta v0.34.0: the engine's own file lock, per-artifact and per-repository reindex

.claude/skills/add-cserver-plugin/SKILL.md

@@ -47,12 +47,14 @@ that one artifact instead of rebuilding the whole home:
 
 ```bash
 cx category index  <repo>:cserver.<name>   # index a hand-made folder
-cx category update <repo>:cserver.<name>   # after editing its _cmeta.* meta
+cx category reindex cserver.<name>         # after editing its _cmeta.* meta (nothing written)
+cx category update <repo>:cserver.<name>   # the same, but it rewrites the file too
+cx repo reindex <repo>                     # the whole repository, when several things changed in it
 ```
 
-`cx --reindex` walks every category and is slow — keep it for moves, bulk
-`git pull`, or a wiped `CMETA_HOME`. **Editing `api/`, `files/` or `src/` needs
-no reindex at all**; a server restart (or `--reload`) picks those up.
+`cx --reindex` walks every category and is slow — keep it for a wiped
+`CMETA_HOME` or a bulk change across repositories. **Editing `api/`, `files/` or
+`src/` needs no reindex at all**; a server restart (or `--reload`) picks those up.
 
 Layout:
 

.claude/skills/add-plugin/SKILL.md

@@ -340,13 +340,15 @@ narrowest command that covers it:
 
 ```bash
 cx <category> index  <repo>:<artifact>   # you created the folder by hand (mkdir + _cmeta.*)
-cx <category> update <repo>:<artifact>   # you edited an existing artifact's _cmeta.* meta
+cx <category> reindex <artifact>         # you edited, renamed or copied in an existing artifact's folder (nothing is written)
+cx repo reindex <repo>                   # one repository: its _cmr.yaml and its artifacts
+cx <category> update <repo>:<artifact>   # you edited an existing artifact's _cmeta.* meta (rewrites the file too)
 cx --reindex                             # rebuilds ALL of <CMETA_HOME>/index/*.pkl — slow
 ```
 
-Prefer the per-artifact forms: `--reindex` walks every category and is markedly
-slower on a large home. Save it for renames/moves, a bulk `git pull`, or a wiped
-`CMETA_HOME`. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no
+Prefer the narrow forms: `--reindex` walks every category and is markedly
+slower on a large home. Save it for a wiped `CMETA_HOME` or a bulk change across
+repositories. Editing only `api/`, `files/`, `src/` or `_desc.yaml` needs no
 reindex — the index tracks meta, not content. Category-level content
 caches (task results, downloaded blobs) live under the `cache` category:
 
@@ -382,7 +384,8 @@ Before reporting the plugin as done:
   `TypeError: foo() takes 2 positional arguments but ...`. Rename to `foo_`.
 - **Command name clashes with a Python builtin** (`list`, `type`, ...). Name
   the method `list__` — the CLI name is still `list`.
-- **New category isn't found.** The artifact index is cached; run `cx --reindex`.
+- **New category isn't found.** The artifact index is cached; run
+  `cx repo reindex <repo>` (or `cx --reindex`).
 - **Base command not exposed.** The category's `_cmeta.yaml` sets
   `skip_base_category_commands: true` (e.g. `utils`). Remove that flag or
   reimplement the command.

.claude/skills/add-repo/SKILL.md

@@ -147,6 +147,7 @@ cx repo find <alias-or-uid>        # find a repo
 cx repo status <alias>             # git status + remote URL for git-backed repos
 cx repo pull <alias>               # git pull; also refreshes the index
 cx repo update <alias>             # equivalent to pull
+cx repo reindex <alias>            # re-read its _cmr.yaml and rescan its artifacts (files copied in by other means)
 cx repo checkout <alias> <ref>     # git checkout branch/tag/commit
 cx repo space <alias>              # disk usage
 cx repo zip <alias>                # dump to cmr-<alias>-YYYYMMDD-HHMMSS.zip
@@ -172,10 +173,11 @@ After adding a repo, sanity-check:
 - [ ] `<CMETA_HOME>/repos.json` contains the repo's path.
 
 If a repo is registered but its artifacts don't show up in `find`/`ls`, the
-fast index is stale — force a rebuild:
+fast index is stale — rescan that repository (or, as a last resort, everything):
 
 ```bash
-cx --reindex
+cx repo reindex <alias>            # its _cmr.yaml and its artifacts; other repos untouched
+cx --reindex                       # everything
 ```
 
 ---
@@ -198,8 +200,9 @@ cx --reindex
   does.
 - **Renaming a repo alias.** Not supported. Reference repos by `alias,UID` in
   cross-repo references so alias renames wouldn't matter anyway.
-- **`_cmr.yaml` edited by hand and things look off.** `cx --reindex` to
-  rebuild `<CMETA_HOME>/index/*.pkl`.
+- **`_cmr.yaml` edited by hand and things look off.** `cx repo reindex <alias>`
+  re-reads it and rescans the repo's artifacts (or `cx --reindex` to rebuild
+  `<CMETA_HOME>/index/*.pkl`).
 - **`permanent: true` in `_cmr.yaml`.** `cx repo delete` refuses. Remove the
   flag first if you truly want to delete.
 - **`subdir:` in `_cmr.yaml`.** Only that sub-tree is scanned for artifacts —

.claude/skills/use-cmeta-cli/SKILL.md

@@ -349,20 +349,30 @@ cx <cat> <cmd> ... --dump                        # capture full ctx
 ### 10.7 Reindex when things look stale — narrowest command first
 
 `cx --reindex` rebuilds *every* category pickle and gets slow as a home grows.
-Reach for the per-artifact commands first; they touch one index entry:
+Reach for the narrow commands first; they touch one index entry or one repository:
 
 ```bash
 cx <category> index  <repo>:<artifact>   # register ONE hand-made (mkdir-ed) artifact
-cx <category> update <repo>:<artifact>   # refresh entry after editing its _cmeta.* meta
+cx <category> reindex <artifact>         # refresh ONE record from its folder: a meta edited by hand, a folder
+                                         #   renamed/moved/copied in; nothing is written on disk
+cx <category> reindex                    # every artifact of the category, hand-made folders included
+cx repo reindex <repo>                   # ONE repository: its _cmr.yaml and its artifacts; other repos untouched
+cx <category> update <repo>:<artifact>   # refresh entry after editing its _cmeta.* meta (rewrites the file too)
 cx --reindex                             # whole-home rebuild — last resort
 ```
 
 - `cx <category> add <repo>:<artifact>` already indexes as it creates — nothing
-  extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`).
+  extra needed. `index` is for folders you made by hand (`mkdir` + `_cmeta.*`);
+  `reindex` takes those too, and an artifact the index knows already.
+- `reindex` never touches a folder; the only record it removes is that of an
+  artifact whose folder now holds another artifact, and it says so. A folder it
+  cannot find, a meta it cannot read, or one UID in two folders is an error that
+  changes nothing. After a rename by hand the new name, the UID and the old name
+  all find the folder, and the old alias is dropped.
 - Editing only an artifact's **payload** (`api/`, `files/`, `src/`, `_desc.yaml`)
   needs no reindex at all — the index tracks meta, not content.
-- Keep `--reindex` for the broad cases: bulk `git pull`, moved/renamed folders,
-  `CMETA_HOME` wipe, or an index that looks generally wrong. Safe, idempotent.
+- Keep `--reindex` for the broad cases: a wiped `CMETA_HOME`, a bulk change
+  across repositories, or an index that looks generally wrong. Safe, idempotent.
 
 ## 11. Scripting the CLI
 

.claude/skills/use-cmeta-python/SKILL.md

@@ -289,7 +289,11 @@ loaded = r['artifact']['loaded_files']
 ### 6.5 Reindex on-demand
 
 ```python
-cm.repos.reindex(con=False)             # or: cm.access({'reindex': True})
+cm.repos.reindex(con=False)             # or: cm.access({'reindex': True})   - everything, slow
+
+# One artifact's record from its folder (nothing written), one repository
+r = cm.access({'category': 'note', 'command': 'reindex', 'arg1': 'my-note'})   # r['artifacts'], r['dropped_aliases']
+r = cm.access({'category': 'repo', 'command': 'reindex', 'arg1': 'myrepo'})    # its _cmr.yaml and its artifacts
 ```
 
 ## 6.6 Use `cm.utils.*` from category code — **recommended**

AGENTS.md

@@ -57,7 +57,8 @@ The commands above are cross-platform truth. Any Windows `.bat` wrappers around
 them are the author's local workflow and are not tracked in this repository.
 
 Runtime deps are intentionally minimal: pyyaml, requests, setuptools, wheel,
-tabulate, tqdm, filelock, packaging, psutil.
+tabulate, tqdm, packaging, psutil. (File locks are the engine's own
+`cmeta.utils.files.PathLock` since 0.34.0; `filelock` is no longer used.)
 
 ### Useful environment variables
 `CMETA_HOME`, `CMETA_DEBUG=1`, `CMETA_LOG=DEBUG|INFO`, `CMETA_LOG_FILE=<path>`,
@@ -75,7 +76,7 @@ tabulate, tqdm, filelock, packaging, psutil.
 | `core.py` | `CMeta` class + `access()` — the one dispatch every request funnels through. Resolves the category artifact, loads its API module, calls the matching command function. |
 | `core_async.py` | Async variant of `access()`. |
 | `category.py` | `InitCategory` base class every category API subclasses. Provides `_prepare_input_from_params/ctx` helpers used to re-enter `cm.access()`. |
-| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`info`/`tags`/`get`/`set`/`index`/`test`. |
+| `category_api_v1.py` | Standard **base** category — implements the shared CRUD commands every category inherits: `find`/`list`/`read`/`update`/`create`/`delete`/`move`/`copy`/`migrate`/`info`/`tags`/`get`/`set`/`index`/`reindex`/`test`. |
 | `repos.py` | `Repos` — index and resolve content repositories + artifacts (by alias / UID / tags / wildcards). Owns the per-category `.pkl` index files. |
 | `packages.py` | `Packages` — detect installed tools/versions and auto-install Python packages. |
 | `config.py` | Global `cfg` dict, param descriptors, env-var handling, command aliases (`add→create`, `rm→delete`, `ls→list`, `search→find`, `mv→move`, ...). |
@@ -141,10 +142,17 @@ tabulate, tqdm, filelock, packaging, psutil.
   refreshes automatically on `create`/`update`/`delete` and on `cx repo`
   changes. If artifacts were touched outside cMeta, reindex with the narrowest
   command: `cx <category> index <repo>:<artifact>` registers a single folder you
-  created by hand (`mkdir` + `_cmeta.*`); `cx <category> update <repo>:<artifact>`
-  refreshes the entry after you edit an existing artifact's `_cmeta.*` meta;
-  `cx --reindex` rebuilds every pickle and is slow — keep it for manual moves,
-  bulk `git pull`, or a cleared `CMETA_HOME`. Payload-only edits (`api/`,
+  created by hand (`mkdir` + `_cmeta.*`); `cx <category> reindex <artifact>`
+  rewrites the record of one artifact (or of several, with wildcards) from the
+  meta file in its folder without writing anything — a meta edited by hand, a
+  folder renamed, moved or copied in (its stale aliases are dropped, and the
+  record of a folder that now holds another artifact; a folder it cannot find or
+  one UID in two folders is an error that changes nothing);
+  `cx <category> update <repo>:<artifact>` refreshes the entry too but rewrites
+  the meta file (a new timestamp); `cx repo reindex <repo>` rescans one
+  repository (its `_cmr.yaml` and its artifacts) and leaves the others alone;
+  `cx --reindex` rebuilds every pickle and is slow — keep it for a cleared
+  `CMETA_HOME` or a bulk change across repositories. Payload-only edits (`api/`,
   `files/`, `src/`, `_desc.yaml`) need no reindex. Categories with
   `no_index: true` in `_cmeta.yaml` are always found by filesystem scan.
 - **Content-addressed task caching / reproducibility.** Workflow caches keyed

CHANGELOG.md

@@ -3,6 +3,86 @@
 All notable changes to cMeta are documented here, newest first.
 
 
+## 0.34.0
+- **The engine's own path lock replaces the `filelock` library: two processes can no longer hold one
+  lock at once, and lock files are still removed after use.** The previous lock removed its file after
+  releasing it; with `flock` on Linux and macOS a process waiting on the removed file and a newcomer
+  that created a new one then both acquired, and one of two concurrent index writes was lost (seen with
+  six `cx log reindex` at once on WSL2 and in a container). `cmeta.utils.files.PathLock` removes the
+  file BEFORE releasing (after closing it on Windows, where an open file cannot be removed) and checks
+  after acquiring that the file it holds is still the file at the path, trying again otherwise. It takes
+  a byte-range lock (`msvcrt.locking`) on Windows, `fcntl.flock` on POSIX, POSIX record locks
+  (`fcntl.lockf`) where the file system refuses `flock` (NFS and some network mounts), and a soft lock
+  file (`pid host time`, created exclusively; a stale one - its process dead on this host, or older than
+  ten minutes when left by another host - is removed by the waiter that finds it) where the file system
+  refuses both (some FUSE, 9p and SMB mounts). The threads of one process are serialized on the path
+  first, so the lock is safe for threads as well as processes in every mode. A busy lock is polled every
+  1 to 25 ms instead of every 50 ms. The timeouts, the lock file names and the functions that use them
+  (`safe_read_file(lock=True)`, `safe_write_file`, `safe_delete_directory`, `lock_path` / `unlock_path`)
+  are unchanged; `filelock` is no longer a dependency. Tests: `tests/core_tests/test_utils_files_path_lock.py`
+  (one holder at a time, the file removed on release, eight processes and sixteen threads through one
+  counter, the engine's read-modify-write cycle under contention, a killed holder, the removed-file race
+  replayed deterministically, a lock file that is not its own left alone, the soft lock and its stale
+  files, the POSIX record-lock fallback, timeouts, `safe_delete_directory` and `safe_write_file`);
+  `test_several_processes_reindex_at_once` passes on every platform (it was expected to fail on Linux).
+- **`tests/benchmarks/benchmark_engine.py`: a benchmark of the engine's hot paths** - the index reads
+  behind `find`, the locked writes behind `create`, `update`, `delete` and `reindex`, the full reindex, the
+  reload after a change, the primitives (stat, lock, record copy, pickle, atomic write), concurrent creates
+  and the `cx` command line - on a home of its own with N hand-made artifacts; `--json` writes the numbers
+  and `--compare <json>` prints the ratio against an earlier run. A smoke test runs it in the suite.
+- **`cx <category> reindex [<artifact>]`: the index record of one artifact (or of several) is refreshed
+  from its folder, and no file is changed.** The meta file is read from the artifact's folder on disk
+  and the record in the category's index is rewritten from it: the alias is the folder's name, every
+  other alias that still pointed at the artifact's UID is dropped, and the path, the meta and the
+  sharding depth are what a full `cx --reindex` records. Use it after a meta was edited by hand, after
+  a folder was renamed, moved or copied in from another machine (bisync, a USB disk), or when a lookup
+  looks stale - instead of the whole `cx --reindex`, and instead of an empty `cx <category> update`,
+  which refreshes the record but also rewrites the meta file (a new timestamp and a generator record,
+  which a bisynced folder then carries back). The artifact is named by its alias, UID, alias,UID or
+  repo:alias, with wildcards; `--tags` prunes; nothing names every artifact of the category. A folder
+  the index does not know yet (made or renamed by hand) is found on disk in the repositories; an
+  artifact whose indexed folder is gone is looked for by its UID in its repository, so the new name,
+  the UID or the old name all work after a rename by hand. No folder is touched, and the only record
+  `reindex` ever removes is that of an artifact whose folder, still there, holds another artifact now
+  (replaced by hand) - a record a `delete` would act on by removing the other artifact's folder - and
+  it says so. A folder that cannot be found, a meta that cannot be read or has no valid `artifact`
+  UID, a meta of another category, or one UID found in two existing folders is an error that leaves
+  the index as it was (`--ignore_errors` reindexes the other artifacts and reports these). `index`,
+  `update`, `move` and `migrate` keep their behaviour. Tests: `tests/internal_repo_tests/test_integration_reindex_artifact.py`
+  (a hand-edited meta, YAML and JSON, the record of a full reindex, renames by hand found by the new
+  name, the UID or the old name, a rename twice, a case-only rename, two spellings on a case-sensitive
+  file system, a folder replaced by another artifact, a folder moved and its old name taken, one UID
+  in two folders, a missing folder, an unreadable meta, an invalid UID, an upper-case UID, another
+  category's folder, a non-ASCII alias, wildcards, tags, a repository, a repository with `subdir`,
+  everything, a UID-named folder, a sharded category and a rename inside it, a `no_index` category, a
+  category without an index file yet, the record of a category itself, the shipped internal repository,
+  a migration stub, the CLI, several processes at once).
+- **`cx repo reindex <repository>`: one registered repository is indexed again.** Its `_cmr.yaml` is
+  re-read and its artifacts are rescanned with the incremental index that `pull`, `plug` and `unzip`
+  use, and the records of the other repositories stay as they are. The repository is named by its
+  alias, UID or alias,UID (wildcards allowed), by the path of its folder (or of a folder inside it),
+  or by `.` for the repository of the current directory. A repository whose folder or `_cmr.yaml` is
+  not there (a detached drive) is refused and nothing is changed. `cx --reindex` stays the full
+  rebuild of every repository. Tests: `tests/internal_repo_tests/test_integration_repo_reindex.py`.
+- **The incremental index drops the records of a category of which a repository holds no artifact
+  any more.** When a repository was indexed again (`pull`, `plug`, `unzip`, now `repo reindex`), the
+  records of its deleted artifacts were removed only from the index files of the categories in which
+  the repository still had an artifact: when its last artifact of a category was gone, that record
+  stayed and the lookup kept answering with a folder that did not exist. Every existing category
+  index now takes part in the removal. The full `cx --reindex` is unchanged.
+- **A wildcard search in a `no_index` category returns only the folders the pattern names.** The scan
+  that finds the artifacts of such a category (and that `reindex` uses to find folders on disk)
+  applied the pattern only in sharded categories; in a plain one, `cx <category> find "name-*"`
+  returned every artifact. The pattern is now matched against the folder names, case-insensitively,
+  as the sharded scan does; and an exact alias now finds every folder that spells it (two spellings
+  can coexist on a case-sensitive file system), not only the first. The same scan no longer stops
+  with a `KeyError` on a meta file that has no `category` key: the folder is skipped with a warning,
+  as a folder without an `artifact` key is.
+- `Repos.add_to_index` takes `clean_aliases=True` to drop, in the same locked write, every other
+  alias of the alias index that still lists the UID being written; its result carries them as
+  `dropped_aliases`. The default is unchanged.
+
+
 ## 0.33.1
 - **A command's output is decoded on any machine in any language, and never fails.** `utils.sys.run`
   captured output in the text mode of `subprocess`, which decodes with the locale's encoding and

CITATION.cff

@@ -13,8 +13,8 @@ authors:
     website: "https://cTuning.ai/@gfursin"
     affiliation: "cTuning Labs"
     orcid: "https://orcid.org/0000-0001-7719-1624"
-version: "0.33.1"
-date-released: "2026-10-07"
+version: "0.34.0"
+date-released: "2026-10-09"
 license: Apache-2.0
 repository-code: "https://github.com/cTuningLabs/cmeta"
 url: "https://cTuning.ai"

CLAUDE.md

@@ -48,7 +48,8 @@ separate content repos (e.g. `cmeta-aops`).
 ## Conventions worth repeating
 
 - Runtime deps are intentionally minimal (pyyaml, requests, setuptools, wheel,
-  tabulate, tqdm, filelock, packaging, psutil). Justify any new one.
+  tabulate, tqdm, packaging, psutil). Justify any new one. File locks are the
+  engine's own `cmeta.utils.files.PathLock` (no `filelock`).
 - Preserve module docstring/copyright headers — never strip or rewrite them.
   New files get the standard header; new artifacts/categories record `authors`
   and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and