cMeta v0.35.0: the soft lock across systems, a registered repository that stays registered, the engine's modules imported once

.github/PULL_REQUEST_TEMPLATE.md

@@ -29,7 +29,7 @@ a small fix.
 - [ ] No local paths, credentials, API keys or personal scratch files included.
 
 <!--
-Python 3.9–3.14 is supported, so please avoid newer-only syntax.
+Python 3.9–3.15 is supported, so please avoid newer-only syntax.
 Full details: CONTRIBUTING.md. Architecture and conventions: AGENTS.md.
 Security issue or a leaked secret? Do NOT open a PR or issue — email gfursin@gmail.com.
 -->

.github/workflows/test-core.yml

@@ -28,7 +28,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, windows-latest, macos-latest]
-        python-version: ["3.9", "3.14"]
+        python-version: ["3.9", "3.14", "3.15"]
 
     runs-on: ${{ matrix.os }}
 

AGENTS.md

@@ -16,7 +16,7 @@ uniform interface:
 - Python: `cm.access({'category': ..., 'command': ..., ...})`
 - CLI: `cx <category> <command> [args] [--flags]`
 
-**License:** Apache-2.0. **Python:** 3.9–3.14.
+**License:** Apache-2.0. **Python:** 3.9–3.15.
 
 This repo ships the framework and a built-in content repository
 (`cmeta/internal-repo/`) with foundational categories. All other artifacts live
@@ -244,7 +244,7 @@ Notable shipped artifacts:
 - **Cross-category calls.** Inside `api/v1.py`, use `self.cm.access(...)`. Refer
   to other categories via `self.cmeta['uses_categories']['<name>']` declared in
   your `_cmeta.yaml`, not by hard-coded alias.
-- Python 3.9–3.14 supported; avoid newer-only syntax.
+- Python 3.9–3.15 supported; avoid newer-only syntax.
 - Tests: `tests/`, files `test_*.py`, pytest config in `pyproject.toml`.
 - Public Apache-2.0 framework — keep product vision/strategy prose out; public
   docs describe *functionality* (`README.md`, `docs/installation.md`,

CHANGELOG.md

@@ -3,6 +3,89 @@
 All notable changes to cMeta are documented here, newest first.
 
 
+## 0.35.0
+- **A registered repository that cannot be reached stays registered.** When the index was built - a full
+  `cx --reindex`, or the incremental index of `pull`, `plug`, `unplug` and `cx repo reindex` - and the folder
+  of a registered repository was not there (a detached USB drive, a network share that was down) or its
+  `_cmr.yaml` could not be read, the repository was dropped from `repos.json` without a word and had to be
+  plugged again; one transient miss was enough. Now its line stays in `repos.json`, in its place, marked
+  `unreachable: true`; the repository is left out of the index (in an incremental run its records are
+  removed, so that a find does not hand out artifacts whose folder is away); a notice names it and the two
+  commands (on stderr with the console, always in the log). When the folder is there again, the next index
+  run of any kind indexes it as if it had just been plugged and clears the mark - also `cx repo reindex
+  <path>`, which now accepts the folder of a registered repository that is not in the index. `cx repo unplug
+  <path>` forgets a repository by the path of its folder, reachable or not (before, `unplug` knew aliases
+  only). Fixed on the way: the stored descriptor of a repository plugged with `--keep` was left out of
+  `repos.json` whenever the file was rewritten while the repository was reachable, so the next time its
+  drive was away it was dropped like any other; a `_cmr.yaml` without an `artifact` name ended the index with
+  a `KeyError` (now: unreachable, with the reason); the pause "Fix it or press Enter to continue!" of an
+  ambiguous category alias ended the index with `EOFError` when nobody could answer (now: it goes on).
+  Tests: `tests/internal_repo_tests/test_integration_repo_unreachable.py`.
+- **The soft lock: no clock of the drive, a heartbeat, the system in its line, a stale one taken over by
+  one waiter in one step, and a way to ask for it on a shared folder.** The soft lock is the fallback of
+  `PathLock` where a file system refuses OS locks (some FUSE, 9p and SMB mounts); nothing changes where OS
+  locks work. (1) How long ago a soft lock was taken is now counted from the time its holder wrote into the
+  file; before, it was the time the file system gave the file, compared with this machine's clock - on a
+  drive that keeps another clock (a Windows drive inside WSL2 is seconds off, a network share can be more)
+  a lock file made a moment ago looked old, so the grace of a holder that has just released and exited did
+  not hold there. An empty lock file, which nothing dates, is a leftover once a waiter has seen it empty for
+  2 seconds (or when the drive dates it more than 10 minutes back). (2) **The holder beats.** It keeps its
+  lock file open and rewrites the time in its line every 2 seconds (a daemon thread; the line is rewritten
+  in place, so the file is never empty or shorter); a waiter that has seen the same line for 20 seconds on
+  its own monotonic clock knows the holder is gone - without comparing two machines' clocks, however long
+  the lock has been held (the old rule took a lock of another host for abandoned after 10 minutes, a build
+  of an hour included), and without a pid it cannot check (so a holder that crashed on ANOTHER system is
+  seen as running - by `cx cache classify` too - for those 20 seconds; one of this system is known at once by
+  its pid). (3) **The line names the system** (`pid host
+  time system`: a short hash of the kernel's boot id and the pid namespace on Linux, of the machine id on
+  Windows and macOS), so a waiter checks the pid only when it is a pid of its own system: Windows and WSL2
+  share the host name and have different pids, a container shares the kernel of its host and has its own
+  pids; a dead pid of the same system is still known at once, a live one keeps its lock. A lock file of
+  cMeta before 0.35.0 (three words, no heartbeat) is judged as before. (4) A stale lock file was renamed
+  away, checked and - when it was not the judged one - put back; between the two a third process could
+  take the free path, which under load could end with two holders, a leftover `<lock file>.stale-*`, or a
+  lock file carrying the line of a live process that did not hold it any more - a lock nobody could take
+  for as long as that process lived. Now the waiter makes the file that replaces the stale one exclusively,
+  under a name of that stale file (`<lock file>.take-<checksum>`: one waiter only), checks that the file at
+  the path is still the one it judged, and puts its own in place with one rename - the path is never free
+  in between, and a waiter that comes a step later finds another file there and leaves it. The file is
+  closed before the rename on every platform (Windows refuses to rename a file the process has open; a
+  Windows drive seen through 9p - WSL2's `/mnt/d`, a container's bind mount - shows nothing at the new path
+  while the descriptor is open) and opened again at its new path to be held. (5) A single try
+  (`timeout 0`: the probe behind `cx cache classify`) takes the lock of a dead process in that try; before,
+  it cleared the file and still reported the lock held once. (6) A holder checks, after it took the lock,
+  that the file at the path is its own (the open descriptor, or its line), and removes the file at release
+  only when it still is; on Windows a held lock file cannot be replaced at all, so a live holder there is
+  never taken over by mistake. (7) The empty file that a refused OS-lock attempt leaves is taken over the
+  same way instead of being removed outright. (8) A forked child of a holder never releases its parent's
+  lock, and starts with thread locks of its own (the ones it inherits may have been held by a thread of the
+  parent that does not exist in the child). (9) **Asking for it.** A home or repository that several systems write at once through a shared
+  drive - a Windows tree plugged from WSL2 or from a container, a network share - has no lock between those
+  systems with the OS locks (each system's lock is invisible to the others; the exclusive create of the
+  soft lock file is honoured by all). `CMETA_SOFT_LOCK_PATHS=<folder>[<os.pathsep><folder>...]` (`*` for
+  everywhere) makes every lock file under those folders use the soft lock in that process, and `lock: soft`
+  in a repository's `_cmr.yaml` does it for that repository on every system that plugs it (registered when
+  it is plugged and at the start of every process, from the index of repositories; the API is
+  `utils.files.use_soft_lock_under`). Everything else keeps the OS locks: a lock cycle costs what it did.
+  (Each system still keeps its own index of such a repository in its own home: what another system created or
+  changed there is seen after `cx repo reindex <alias>`; the files are always current and no update is lost.)
+- **The engine's own category API is imported once.** `category_api_v1.py` - the base commands of every
+  category - was loaded by its path under a name made from its folders (`cmeta.cmeta.category_api_v1`),
+  and its relative imports made a second copy of the engine's modules in the process (`cmeta.cmeta.utils.
+  files` next to `cmeta.utils.files`), each with a state of its own: the registries of the path lock among
+  them, so the soft lock asked for a folder was unknown to the copy that wrote the artifacts, and the
+  threads of one process were serialized on a path by two registries instead of one (the OS locks kept
+  them apart all the same). The loader now imports a module of the engine's own package by its real name
+  (`cmeta.category_api_v1`); category APIs of repositories are loaded as before.
+- **A lock with a note leaves no lock file behind on exFAT under macOS.** There an empty file reports a
+  placeholder inode that becomes a real one at its first write: the lock noted the identity of its (empty)
+  file when it took it, the note of the holder (the index lock of a reindex, a cache entry being built)
+  changed it, and at release "the file at the path is not ours" left `<path>.lock` behind. The identity is
+  now asked from the open file at release.
+  Tests of both: `tests/core_tests/test_utils_files_path_lock.py`.
+- **Python 3.15.** The suite runs on Python 3.15 in CI, next to 3.9 and 3.14 (Linux, Windows, macOS); nothing
+  in the engine needed a change. `pyproject.toml` carries the classifier.
+
 ## 0.34.2
 - **The cache category knows the state of an entry, and a delete holds the folder's lock across its two
   steps.** `cx cache classify` (the API `classify`) tells for every entry whether it is `ok`, `running` (the

CITATION.cff

@@ -13,8 +13,8 @@ authors:
     website: "https://cTuning.ai/@gfursin"
     affiliation: "cTuning Labs"
     orcid: "https://orcid.org/0000-0001-7719-1624"
-version: "0.34.2"
-date-released: "2026-10-10"
+version: "0.35.0"
+date-released: "2026-10-11"
 license: Apache-2.0
 repository-code: "https://github.com/cTuningLabs/cmeta"
 url: "https://cTuning.ai"

CLAUDE.md

@@ -54,7 +54,7 @@ separate content repos (e.g. `cmeta-aops`).
   New files get the standard header; new artifacts/categories record `authors`
   and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and
   citation: `AGENTS.md` §5.1 and `NOTICE`.
-- Python 3.9–3.14 supported; don't use newer-only syntax.
+- Python 3.9–3.15 supported; don't use newer-only syntax.
 - Tests live in `tests/` as `test_*.py` (pytest config in `pyproject.toml`).
 - This is the public Apache-2.0 framework — keep product vision/strategy prose
   out; public docs describe *functionality*.

CONTRIBUTING.md

@@ -105,7 +105,7 @@ flake8 cmeta                       # lint
 python -m build                    # build wheel + sdist
 ```
 
-cMeta supports **Python 3.9–3.14**; please avoid newer-only syntax. Runtime
+cMeta supports **Python 3.9–3.15**; please avoid newer-only syntax. Runtime
 dependencies are intentionally minimal — justify any new one in your pull
 request. See [`AGENTS.md`](AGENTS.md) for architecture and conventions.
 

cmeta/internal-repo/_cmr.yaml

@@ -3,5 +3,5 @@ authors: "[Grigori Fursin](https://cTuning.ai/@gfursin)"
 category: repo,f4f792ab40c7498f
 copyright: "2025-2026 Grigori Fursin and cTuning Labs. See the cMeta COPYRIGHT and LICENSE files for details."
 permanent: True
-version: '0.34.2'
-version_date: '20261010'
+version: '0.35.0'
+version_date: '20261011'

cmeta/internal-repo/category/repo/api/v1.py

@@ -874,7 +874,15 @@ def delete(
             p['arg1'] = params['arg1']
 
         result = self.cm.access(p)
-        if result['return']>0: return result
+        if result['return']>0:
+            if unplug and result['return'] == 16:
+                # Not in the index by that name: a registered FOLDER - a repository that is out of reach (a
+                # detached drive: registered, not indexed) or simply named by its path - is forgotten by its path
+                path = self._registered_path(params.get('arg1'))
+                if path is not None:
+                    return self._forget_registered_path(path, con, verbose)
+
+            return result
 
         if unplug:
             result['deleted_artifacts'] = result.pop('artifacts', [])
@@ -922,6 +930,80 @@ def delete(
         return result
 
 
+    ###############################################################################################
+    def _registered_path(
+        self,
+        name,  # A folder: the path of a registered repository (or, with inside=True, a folder inside one).
+        inside = False,  # If True, a folder inside a registered repository names that repository too (the deepest one).
+    ):
+        """
+            The path, as repos.json has it, of the registered repository that `name` is (or, with
+            inside=True, lies in) - whether that repository is in the index or not. None when `name` is no
+            such folder.
+        """
+
+        if name is None or str(name).strip() == '':
+            return None
+
+        wanted = os.path.abspath(os.path.normpath(str(name).strip()))
+
+        r = utils.files.safe_read_file(self.cm.repos_config_path, lock=False, fail_on_error=False, logger=self.logger)
+        if r['return']>0 or not isinstance(r.get('data'), dict):
+            return None
+
+        best = None
+        best_full = None
+
+        for path in r['data']:
+            full = os.path.abspath(os.path.normpath(path))
+
+            if os.path.normcase(full) == os.path.normcase(wanted) or (inside and utils.files.is_path_within(full, wanted)):
+                if best is None or len(full) > len(best_full):
+                    best = path
+                    best_full = full
+
+        return best
+
+
+    ###############################################################################################
+    def _forget_registered_path(
+        self,
+        path,  # The path of the repository as repos.json has it.
+        con = False,  # If True, print output to console.
+        verbose = False,  # If True, enable verbose output.
+    ):
+        """
+            Unregister a repository by its path: its line leaves repos.json and its records leave the
+            index; its folder, reachable or not, is left as it is.
+        """
+
+        repos_config_path = self.cm.repos_config_path
+
+        r = utils.files.safe_read_file(repos_config_path, lock=True, keep_locked=True, fail_on_error=self.fail_on_error, logger=self.logger)
+        if r['return']>0: return r
+
+        repos_paths = r['data']
+        repos_paths_file_lock = r['file_lock']
+
+        repos_paths.pop(path, None)
+
+        # Do not sort keys - preserve order
+        r = utils.files.safe_write_file(repos_config_path, repos_paths, file_lock=repos_paths_file_lock, atomic=True,
+                                        fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False)
+        if r['return']>0: return r
+
+        if con:
+            print('')
+
+        r = self.cm.repos.index(clean=False, con=con, verbose=verbose, delete_repo_paths = [path])
+        if r['return']>0: return r
+
+        if con:
+            print (f'Unregistered the repository in "{path}" (its folder is left as it is)')
+
+        return {'return':0, 'deleted_artifacts':[], 'unregistered_paths':[path]}
+
+
     ###############################################################################################
     def move(
         self,
@@ -1498,6 +1580,7 @@ def reindex_(
         arg1 = str(arg1).strip()
 
         repo_artifacts = []
+        unindexed_path = None
 
         if arg1 == '.' or os.path.isdir(arg1):
             # A folder: the registered repository that holds it (the deepest one, for nested repositories)
@@ -1518,9 +1601,15 @@ def reindex_(
                         best_path = repo_path
 
             if best is None:
-                return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug <path>"'}
+                # Registered but not in the index: a repository that was out of reach when the index was last
+                # built (it stays registered) and is there again
+                unindexed_path = self._registered_path(path, inside=True)
 
-            repo_artifacts = [best]
+                if unindexed_path is None:
+                    return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug <path>"'}
+
+            else:
+                repo_artifacts = [best]
 
         else:
             p = {'category': ctx['category'],
@@ -1561,6 +1650,12 @@ def reindex_(
 
             paths.append(path)
 
+        if unindexed_path is not None:
+            if not os.path.isdir(unindexed_path) or not os.path.isfile(os.path.join(unindexed_path, self.cm.cfg['repo_meta_desc'])):
+                return {'return':1, 'error':f'the folder of the repository registered as "{unindexed_path}" or its {self.cm.cfg["repo_meta_desc"]} is not there (a detached drive?) - nothing was reindexed'}
+
+            paths.append(unindexed_path)
+
         if con:
             print ('')
 
@@ -1575,6 +1670,9 @@ def reindex_(
 
                 print (f'Reindexed repository "{repo_alias}" ({repo_uid}): {path}')
 
+            if unindexed_path is not None:
+                print (f'Reindexed the repository in "{unindexed_path}" (it was registered and not in the index)')
+
         return {'return':0, 'artifacts': repo_artifacts, 'paths': paths}
 
     ###############################################################################################