From b0bc1191d152d940840f0cd21e2632b204e58d4e Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 14:09:20 +0200 Subject: [PATCH 1/6] docs: the API pages regenerated by the 0.34.2 docs build (utils.common.ask, utils.files.acquire_with_notice, the resume record suffix of utils.net) Co-Authored-By: Claude Opus 5.5 Signed-off-by: Grigori Fursin --- docs/en/api/cmeta_utils_common.rst | 2 ++ docs/en/api/cmeta_utils_files.rst | 2 ++ docs/en/api/cmeta_utils_net.rst | 5 +++++ 3 files changed, 9 insertions(+) diff --git a/docs/en/api/cmeta_utils_common.rst b/docs/en/api/cmeta_utils_common.rst index 6422b33..eece534 100644 --- a/docs/en/api/cmeta_utils_common.rst +++ b/docs/en/api/cmeta_utils_common.rst @@ -10,6 +10,8 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. Functions --------- +.. autofunction:: cmeta.utils.common.ask + .. autofunction:: cmeta.utils.common.build_sort_key .. autofunction:: cmeta.utils.common.check_params diff --git a/docs/en/api/cmeta_utils_files.rst b/docs/en/api/cmeta_utils_files.rst index e89107e..6666e4f 100644 --- a/docs/en/api/cmeta_utils_files.rst +++ b/docs/en/api/cmeta_utils_files.rst @@ -18,6 +18,8 @@ Classes Functions --------- +.. autofunction:: cmeta.utils.files.acquire_with_notice + .. autofunction:: cmeta.utils.files.apply_sharding_to_path .. autofunction:: cmeta.utils.files.ask_to_delete diff --git a/docs/en/api/cmeta_utils_net.rst b/docs/en/api/cmeta_utils_net.rst index 230bd4b..f6c42d7 100644 --- a/docs/en/api/cmeta_utils_net.rst +++ b/docs/en/api/cmeta_utils_net.rst @@ -15,3 +15,8 @@ Functions .. autofunction:: cmeta.utils.net.download .. autofunction:: cmeta.utils.net.unify_request + +Constants +--------- + +.. autodata:: cmeta.utils.net.RESUME_RECORD_SUFFIX From 45927131e26c631a84ee65bd0bd291a5a4929c09 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 16:53:28 +0200 Subject: [PATCH 2/6] CI: Python 3.15 in the test matrix Python 3.15.0 is out: the core tests run on Linux, Windows and macOS with 3.9, 3.14 and 3.15. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Grigori Fursin --- .github/workflows/test-core.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/test-core.yml b/.github/workflows/test-core.yml index 34d0d8e..342c35a 100644 --- a/.github/workflows/test-core.yml +++ b/.github/workflows/test-core.yml @@ -28,7 +28,7 @@ jobs: strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] - python-version: ["3.9", "3.14"] + python-version: ["3.9", "3.14", "3.15"] runs-on: ${{ matrix.os }} From d1a7863c1634fa13bf50db5b9d26e12c4dd17c77 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 17:07:00 +0200 Subject: [PATCH 3/6] Python 3.15 is supported: the places that name the supported versions Both CI matrices passed on Python 3.15.0 (Linux, Windows, macOS). AGENTS.md, CLAUDE.md, CONTRIBUTING.md and the pull request template say 3.9-3.15, docs/releasing.md names the three versions of the CI, pyproject.toml gets the classifier. Co-Authored-By: Claude Opus 5.5 Signed-off-by: Grigori Fursin --- .github/PULL_REQUEST_TEMPLATE.md | 2 +- AGENTS.md | 4 ++-- CLAUDE.md | 2 +- CONTRIBUTING.md | 2 +- docs/releasing.md | 2 +- pyproject.toml | 1 + 6 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index f67ccfa..6452a84 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -29,7 +29,7 @@ a small fix. - [ ] No local paths, credentials, API keys or personal scratch files included. diff --git a/AGENTS.md b/AGENTS.md index 8d087c5..c6eb434 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ uniform interface: - Python: `cm.access({'category': ..., 'command': ..., ...})` - CLI: `cx [args] [--flags]` -**License:** Apache-2.0. **Python:** 3.9–3.14. +**License:** Apache-2.0. **Python:** 3.9–3.15. This repo ships the framework and a built-in content repository (`cmeta/internal-repo/`) with foundational categories. All other artifacts live @@ -244,7 +244,7 @@ Notable shipped artifacts: - **Cross-category calls.** Inside `api/v1.py`, use `self.cm.access(...)`. Refer to other categories via `self.cmeta['uses_categories']['']` declared in your `_cmeta.yaml`, not by hard-coded alias. -- Python 3.9–3.14 supported; avoid newer-only syntax. +- Python 3.9–3.15 supported; avoid newer-only syntax. - Tests: `tests/`, files `test_*.py`, pytest config in `pyproject.toml`. - Public Apache-2.0 framework — keep product vision/strategy prose out; public docs describe *functionality* (`README.md`, `docs/installation.md`, diff --git a/CLAUDE.md b/CLAUDE.md index 3d78007..5c3c246 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,7 +54,7 @@ separate content repos (e.g. `cmeta-aops`). New files get the standard header; new artifacts/categories record `authors` and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and citation: `AGENTS.md` §5.1 and `NOTICE`. -- Python 3.9–3.14 supported; don't use newer-only syntax. +- Python 3.9–3.15 supported; don't use newer-only syntax. - Tests live in `tests/` as `test_*.py` (pytest config in `pyproject.toml`). - This is the public Apache-2.0 framework — keep product vision/strategy prose out; public docs describe *functionality*. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index adba71d..e243b76 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -105,7 +105,7 @@ flake8 cmeta # lint python -m build # build wheel + sdist ``` -cMeta supports **Python 3.9–3.14**; please avoid newer-only syntax. Runtime +cMeta supports **Python 3.9–3.15**; please avoid newer-only syntax. Runtime dependencies are intentionally minimal — justify any new one in your pull request. See [`AGENTS.md`](AGENTS.md) for architecture and conventions. diff --git a/docs/releasing.md b/docs/releasing.md index a51a958..f8ef769 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -32,7 +32,7 @@ gh workflow run test-core.yml --repo cTuningLabs/cmeta --ref dev gh run list --repo cTuningLabs/cmeta --workflow test-core.yml --limit 1 # then: gh run watch --repo cTuningLabs/cmeta ``` -The CI workflow runs only on request; it covers Linux, Windows and macOS with Python 3.9 and 3.14. +The CI workflow runs only on request; it covers Linux, Windows and macOS with Python 3.9, 3.14 and 3.15. ## 2. Bump the version diff --git a/pyproject.toml b/pyproject.toml index aafefed..d4aaf4b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,6 +34,7 @@ classifiers = [ "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", + "Programming Language :: Python :: 3.15", ] dependencies = [ "pyyaml", From 2371c3048f2e7ebcd6f64c146575af55336e57dc Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 21:43:57 +0200 Subject: [PATCH 4/6] A registered repository that cannot be reached stays registered When the index was built - cx --reindex, or the incremental index of pull, plug, unplug and cx repo reindex - and the folder of a registered repository was not there (a detached drive, a share that was down) or its _cmr.yaml could not be read, the repository was dropped from repos.json without a word and had to be plugged again; one transient miss was enough. Now its line stays in repos.json, marked "unreachable: true"; the repository is left out of the index (an incremental run removes its records, so that a find does not hand out artifacts whose folder is away); a notice names it and the two commands. When the folder is there again, the next index run of any kind indexes it as if it had just been plugged and clears the mark - also cx repo reindex , which accepts the folder of a registered repository that is not in the index. cx repo unplug forgets a repository by the path of its folder. Fixed on the way: the stored descriptor of a repository plugged with --keep was left out of repos.json whenever the file was rewritten while the repository was reachable; a _cmr.yaml without an artifact name ended the index with a KeyError; the pause of an ambiguous category alias ended the index with EOFError when nobody could answer. Tests: tests/internal_repo_tests/test_integration_repo_unreachable.py (8). Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- CHANGELOG.md | 21 ++ cmeta/internal-repo/category/repo/api/v1.py | 104 +++++- cmeta/repos.py | 122 ++++++- cmeta/version.py | 2 +- docs/using-cmeta.md | 16 +- .../test_integration_repo_unreachable.py | 305 ++++++++++++++++++ 6 files changed, 557 insertions(+), 13 deletions(-) create mode 100644 tests/internal_repo_tests/test_integration_repo_unreachable.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fdc7f9..3663d8b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,27 @@ All notable changes to cMeta are documented here, newest first. +## 0.34.2.1 (in development) +- **A registered repository that cannot be reached stays registered.** When the index was built - a full + `cx --reindex`, or the incremental index of `pull`, `plug`, `unplug` and `cx repo reindex` - and the folder + of a registered repository was not there (a detached USB drive, a network share that was down) or its + `_cmr.yaml` could not be read, the repository was dropped from `repos.json` without a word and had to be + plugged again; one transient miss was enough. Now its line stays in `repos.json`, in its place, marked + `unreachable: true`; the repository is left out of the index (in an incremental run its records are + removed, so that a find does not hand out artifacts whose folder is away); a notice names it and the two + commands (on stderr with the console, always in the log). When the folder is there again, the next index + run of any kind indexes it as if it had just been plugged and clears the mark - also `cx repo reindex + `, which now accepts the folder of a registered repository that is not in the index. `cx repo unplug + ` forgets a repository by the path of its folder, reachable or not (before, `unplug` knew aliases + only). Fixed on the way: the stored descriptor of a repository plugged with `--keep` was left out of + `repos.json` whenever the file was rewritten while the repository was reachable, so the next time its + drive was away it was dropped like any other; a `_cmr.yaml` without an `artifact` name ended the index with + a `KeyError` (now: unreachable, with the reason); the pause "Fix it or press Enter to continue!" of an + ambiguous category alias ended the index with `EOFError` when nobody could answer (now: it goes on). + Tests: `tests/internal_repo_tests/test_integration_repo_unreachable.py`. +- **Python 3.15.** The suite runs on Python 3.15 in CI, next to 3.9 and 3.14 (Linux, Windows, macOS); nothing + in the engine needed a change. `pyproject.toml` carries the classifier. + ## 0.34.2 - **The cache category knows the state of an entry, and a delete holds the folder's lock across its two steps.** `cx cache classify` (the API `classify`) tells for every entry whether it is `ok`, `running` (the diff --git a/cmeta/internal-repo/category/repo/api/v1.py b/cmeta/internal-repo/category/repo/api/v1.py index cb3fb1e..a77653c 100644 --- a/cmeta/internal-repo/category/repo/api/v1.py +++ b/cmeta/internal-repo/category/repo/api/v1.py @@ -874,7 +874,15 @@ def delete( p['arg1'] = params['arg1'] result = self.cm.access(p) - if result['return']>0: return result + if result['return']>0: + if unplug and result['return'] == 16: + # Not in the index by that name: a registered FOLDER - a repository that is out of reach (a + # detached drive: registered, not indexed) or simply named by its path - is forgotten by its path + path = self._registered_path(params.get('arg1')) + if path is not None: + return self._forget_registered_path(path, con, verbose) + + return result if unplug: result['deleted_artifacts'] = result.pop('artifacts', []) @@ -922,6 +930,80 @@ def delete( return result + ############################################################################################### + def _registered_path( + self, + name, # A folder: the path of a registered repository (or, with inside=True, a folder inside one). + inside = False, # If True, a folder inside a registered repository names that repository too (the deepest one). + ): + """ + The path, as repos.json has it, of the registered repository that `name` is (or, with + inside=True, lies in) - whether that repository is in the index or not. None when `name` is no + such folder. + """ + + if name is None or str(name).strip() == '': + return None + + wanted = os.path.abspath(os.path.normpath(str(name).strip())) + + r = utils.files.safe_read_file(self.cm.repos_config_path, lock=False, fail_on_error=False, logger=self.logger) + if r['return']>0 or not isinstance(r.get('data'), dict): + return None + + best = None + best_full = None + + for path in r['data']: + full = os.path.abspath(os.path.normpath(path)) + + if os.path.normcase(full) == os.path.normcase(wanted) or (inside and utils.files.is_path_within(full, wanted)): + if best is None or len(full) > len(best_full): + best = path + best_full = full + + return best + + + ############################################################################################### + def _forget_registered_path( + self, + path, # The path of the repository as repos.json has it. + con = False, # If True, print output to console. + verbose = False, # If True, enable verbose output. + ): + """ + Unregister a repository by its path: its line leaves repos.json and its records leave the + index; its folder, reachable or not, is left as it is. + """ + + repos_config_path = self.cm.repos_config_path + + r = utils.files.safe_read_file(repos_config_path, lock=True, keep_locked=True, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + repos_paths = r['data'] + repos_paths_file_lock = r['file_lock'] + + repos_paths.pop(path, None) + + # Do not sort keys - preserve order + r = utils.files.safe_write_file(repos_config_path, repos_paths, file_lock=repos_paths_file_lock, atomic=True, + fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) + if r['return']>0: return r + + if con: + print('') + + r = self.cm.repos.index(clean=False, con=con, verbose=verbose, delete_repo_paths = [path]) + if r['return']>0: return r + + if con: + print (f'Unregistered the repository in "{path}" (its folder is left as it is)') + + return {'return':0, 'deleted_artifacts':[], 'unregistered_paths':[path]} + + ############################################################################################### def move( self, @@ -1498,6 +1580,7 @@ def reindex_( arg1 = str(arg1).strip() repo_artifacts = [] + unindexed_path = None if arg1 == '.' or os.path.isdir(arg1): # A folder: the registered repository that holds it (the deepest one, for nested repositories) @@ -1518,9 +1601,15 @@ def reindex_( best_path = repo_path if best is None: - return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + # Registered but not in the index: a repository that was out of reach when the index was last + # built (it stays registered) and is there again + unindexed_path = self._registered_path(path, inside=True) - repo_artifacts = [best] + if unindexed_path is None: + return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + + else: + repo_artifacts = [best] else: p = {'category': ctx['category'], @@ -1561,6 +1650,12 @@ def reindex_( paths.append(path) + if unindexed_path is not None: + if not os.path.isdir(unindexed_path) or not os.path.isfile(os.path.join(unindexed_path, self.cm.cfg['repo_meta_desc'])): + return {'return':1, 'error':f'the folder of the repository registered as "{unindexed_path}" or its {self.cm.cfg["repo_meta_desc"]} is not there (a detached drive?) - nothing was reindexed'} + + paths.append(unindexed_path) + if con: print ('') @@ -1575,6 +1670,9 @@ def reindex_( print (f'Reindexed repository "{repo_alias}" ({repo_uid}): {path}') + if unindexed_path is not None: + print (f'Reindexed the repository in "{unindexed_path}" (it was registered and not in the index)') + return {'return':0, 'artifacts': repo_artifacts, 'paths': paths} ############################################################################################### diff --git a/cmeta/repos.py b/cmeta/repos.py index 7483f95..c75f161 100644 --- a/cmeta/repos.py +++ b/cmeta/repos.py @@ -86,6 +86,12 @@ def __init__( self.KEY_INDEX_UIDS = 'uids' self.KEY_INDEX_LOWERCASE_ALIASES = 'lowercase_aliases' + # In repos.json: a registered repository whose folder could not be reached when the index was last built + # (a detached drive, a share that is down). It stays registered and is indexed again when it is back. + self.KEY_REPO_UNREACHABLE = 'unreachable' + + self._soft_lock_registered = False # the repositories that ask for the soft lock were read once (init) + ################################################################################################### def init( self, @@ -165,6 +171,46 @@ def init( finally: index_lock.release() + # The repositories that ask for the soft lock (`lock: soft` in their descriptor) are known to the path + # lock of this process from the start (once per instance: init runs with every access) + if not self._soft_lock_registered: + r = self._register_soft_lock_repos() + if r['return']>0: return r + self._soft_lock_registered = True + + return {'return':0} + + ################################################################################################### + def _register_soft_lock( + self, + path, # The registered folder of the repository. + repo_meta, # Its descriptor (_cmr.yaml, with the extra meta of the registration). + ): + """ + A repository whose descriptor says `lock: soft` - one that several systems write at once through + a shared drive (a Windows tree plugged from WSL2 or a container, a network share), where the OS + locks of the systems do not see each other - asks the path lock for the soft lock under its folder + (`utils.files.use_soft_lock_under`), for this process. + """ + if path and str(repo_meta.get('lock', '')).strip().lower() == 'soft': + utils.files.use_soft_lock_under(path) + self.logger.debug(f"repository {path} asks for the soft lock (lock: soft)") + + def _register_soft_lock_repos(self): + """ + At the start of a process: the repositories of the index that ask for the soft lock (see + `_register_soft_lock`). The index of repositories is small and read through the cache. + """ + index_repo_file = os.path.join(self.index_path, 'repo' + self.index_extension) + if not os.path.isfile(index_repo_file): + return {'return':0} + + r = utils.files.safe_read_file_via_cache(index_repo_file, self.file_cache, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + for entry in (r['data'].get(self.KEY_INDEX_UIDS) or {}).values(): + self._register_soft_lock(entry.get('path'), entry.get('_cmr') or entry.get('cmeta') or {}) + return {'return':0} ################################################################################################### @@ -1129,7 +1175,12 @@ def _index( repos_config_lock = r['file_lock'] - paths_to_repos = {} + paths_to_repos = {} # the repositories indexed by this run + repos_to_write = {} # what repos.json lists after it: those, and - in their places - the registered + # repositories that cannot be reached now + unreachable = {} # path -> why + newly_unreachable = [] # those of them that were reachable at the run before + back = [] # paths that could not be reached at an earlier run and can now original_paths_to_repos = r['data'] to_update = False @@ -1145,6 +1196,8 @@ def _index( path_to_repo_desc = os.path.join(path, self.cfg['repo_meta_desc']) + why_not = '' + if not os.path.isfile(path_to_repo_desc): if extra_meta.get('keep') and 'keep_repo_meta' in local_meta: repo_meta = local_meta['keep_repo_meta'] @@ -1153,13 +1206,15 @@ def _index( paths_to_repos[path]={'meta': extra_meta, 'keep_repo_meta': repo_meta} - - else: - to_update = True + repos_to_write[path] = paths_to_repos[path] + + continue + + why_not = 'its folder or its ' + self.cfg['repo_meta_desc'] + ' is not there' else: r = utils.files.safe_read_file(path_to_repo_desc, retry_if_not_found=3, fail_on_error=self.fail_on_error, logger=self.logger) - if r['return']==0: + if r['return']==0 and isinstance(r.get('data'), dict) and r['data'].get('artifact'): repo_meta = r['data'] if len(extra_meta)>0: @@ -1170,15 +1225,59 @@ def _index( paths_to_repos[path] = {} if len(extra_meta)>0: paths_to_repos[path]['meta'] = extra_meta + if 'keep_repo_meta' in local_meta: + # the stored copy of a kept repository stays with its registration (it was left out + # of the file whenever the file was rewritten while the repository was reachable) + paths_to_repos[path]['keep_repo_meta'] = local_meta['keep_repo_meta'] + repos_to_write[path] = paths_to_repos[path] + + if local_meta.get(self.KEY_REPO_UNREACHABLE): + back.append(path) + to_update = True + + continue + + if r['return']==0: + why_not = 'its ' + self.cfg['repo_meta_desc'] + ' names no artifact' + else: + why_not = 'its ' + self.cfg['repo_meta_desc'] + ' cannot be read' + (f" ({r.get('error')})" if r.get('error') else '') + + # A registered repository that cannot be reached - a detached drive, a share that is down, a + # descriptor that is being edited - stays registered (it was dropped from repos.json without a + # word, and had to be plugged again) and is left out of this index + unreachable[path] = why_not + + entry = dict(local_meta) + if not entry.get(self.KEY_REPO_UNREACHABLE): + entry[self.KEY_REPO_UNREACHABLE] = True + to_update = True + newly_unreachable.append(path) + repos_to_write[path] = entry if to_update: # Do not sort keys - preserve order! - r = utils.files.safe_write_file(repos_config_path, paths_to_repos, file_lock=repos_config_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) + r = utils.files.safe_write_file(repos_config_path, repos_to_write, file_lock=repos_config_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r else: r = utils.files.unlock_path(repos_config_path, repos_config_lock, fail_on_error=self.fail_on_error, logger=self.logger) if r['return']>0: return r + for path in unreachable: + message = (f'repository "{path}" cannot be reached ({unreachable[path]}): it stays registered and is left out of the index - ' + f'"cx repo reindex {path}" or "cx --reindex" brings it back when it is there again, "cx repo unplug {path}" forgets it') + self.logger.warning(message) + if con: + print (f'cMeta notice: {message}', file=sys.stderr) + + if len(back) > 0: + # Reachable again: indexed with this run, as if it had just been plugged + add_repo_paths = list(add_repo_paths) + [path for path in back if path not in add_repo_paths] + + if len(newly_unreachable) > 0: + # Out of reach since the run before: its records leave the index with this run (an incremental run + # keeps the records of the repositories it does not scan; a full rebuild has none of them anyway) + delete_repo_paths = list(delete_repo_paths) + [path for path in newly_unreachable if path not in delete_repo_paths] + if len(paths_to_repos) == 0: return {'return':1, 'error':f'could not find any repository in {repos_config_path}'} @@ -1259,7 +1358,12 @@ def _index( # Use atomic write to avoid corrupting large index files r = utils.files.safe_write_file(index_repo_file, index_repos, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r - + + # A repository that asks for the soft lock is known to the path lock of this process at once (the + # processes that start later read it from this index: see init) + for path in paths_to_repos.keys(): + self._register_soft_lock(path, repos_meta[path]) + ###################################################################################################################### # Indexing categories index_categories = {self.KEY_INDEX_UIDS:{}, self.KEY_INDEX_LOWERCASE_ALIASES:{}} @@ -1383,7 +1487,9 @@ def _index( print (' * ' + index_categories[self.KEY_INDEX_UIDS][uid]['path']) print (' * ' + category_path) if con: - input (' Fix it or press Enter to continue!') + # A pause for the person at the terminal; with nobody there (a detached + # job, a pipe at its end) the index goes on instead of ending in EOFError + utils.common.ask(' Fix it or press Enter to continue!', optional=True) cmeta_ref_parts['artifact_alias'] = alias if alias != alias.lower(): diff --git a/cmeta/version.py b/cmeta/version.py index 17dbf6f..a15c2c5 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.34.2" +__version__ = "0.34.2.1" __release_date__ = "2026-10-10" diff --git a/docs/using-cmeta.md b/docs/using-cmeta.md index 5c25bb8..5f6a0cd 100644 --- a/docs/using-cmeta.md +++ b/docs/using-cmeta.md @@ -820,12 +820,25 @@ cx repo plug --path= # Detach without deleting the files: cx repo unplug # unregister a repo +cx repo unplug # ... by the path of its folder (also when the folder cannot be reached) cx repo unplug # unregister repo detected in current directory ``` `plug` is `get_` with `--local --path=...`. `unplug` is `delete` with the `--unplug` flag, which unregisters without touching the folder. +**A repository that cannot be reached stays registered.** When the index is built +and the folder of a registered repository is not there - a detached USB drive, a +network share that is down - or its `_cmr.yaml` cannot be read, the repository is +left out of the index, its line in `repos.json` is kept and marked +`unreachable: true`, and a notice names it. Nothing has to be plugged again: the +next index run of any kind (`cx --reindex`, a `pull` or `plug` of another +repository, `cx repo reindex `) finds it back and indexes it. `cx repo unplug +` forgets it for good. (Before 0.35.0 such a repository was dropped from +`repos.json` without a word.) `cx repo plug --keep` goes further for a removable +drive: it stores the descriptor, so the repository stays in the repository index +while the drive is away. + ### 7.10 Everyday repo commands ```bash @@ -1571,7 +1584,8 @@ stay as it is, for instance in a folder that another machine syncs. inside it — does what `cx repo pull` and `cx repo plug` do once a repository is in place: its `_cmr.yaml` is re-read and its artifacts are indexed again, and the records of the other repositories stay as they are. A repository whose folder is -not there (a detached drive) is refused and nothing changes. +not there (a detached drive) is refused and nothing changes; once the folder is +back, `cx repo reindex ` - or any other index run - indexes it again (§7.9). `cx --reindex` rebuilds every index file from scratch — after a `CMETA_HOME` wipe, a bulk change across repositories, or when in doubt. It's safe, idempotent diff --git a/tests/internal_repo_tests/test_integration_repo_unreachable.py b/tests/internal_repo_tests/test_integration_repo_unreachable.py new file mode 100644 index 0000000..4f582c4 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_unreachable.py @@ -0,0 +1,305 @@ +""" +Integration tests for a registered repository whose folder cannot be reached when the index is built - a detached +drive, a share that is down, a descriptor that cannot be read: it stays registered (repos.json keeps its line, +marked "unreachable"), is left out of the index with a notice, comes back by itself with the next index run once +its folder is there again, and can be forgotten by its path with `cx repo unplug `. Before, the repository +was dropped from repos.json without a word and had to be plugged again. + +Each test runs against a fresh in tmp_path so it does not touch the user's real cMeta state. + +Licensed under the Apache License, Version 2.0. +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import pickle + +import pytest + +from cmeta import CMeta + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid, where='extra'): + repo = tmp_path / where / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo, **params): + r = access(cm, category='repo', command='plug', arg1=str(repo), **params) + assert r['return'] == 0, r.get('error') + + +def make(cm, alias, repo, category='log', **meta): + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=True) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def repos_json(tmp_path): + return json.loads((tmp_path / 'repos.json').read_text(encoding='utf-8')) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def registered(tmp_path, repo): + """The entry of a repository in repos.json (None when it is not listed).""" + for path, entry in repos_json(tmp_path).items(): + if same(path) == same(repo): + return entry + return None + + +def marked(tmp_path, repo): + """True / False: the repository is listed, with / without the "unreachable" mark. None: it is not listed.""" + entry = registered(tmp_path, repo) + return None if entry is None else bool(entry.get('unreachable')) + + +def index_data(tmp_path, category): + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def reindex_all(cm, **params): + r = cm.repos.index(clean=True, **params) + assert r['return'] == 0, r.get('error') + + +def detach(repo): + away = repo.with_name(repo.name + '-detached') + repo.rename(away) + return away + + +def test_an_unreachable_repository_stays_registered_and_comes_back(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, repo) + plug(cm, other) + uid = make(cm, 'on-the-drive', 'mine') + other_uid = make(cm, 'elsewhere', 'other') + order = [same(p) for p in repos_json(tmp_path)] + + away = detach(repo) + + # A full reindex while the drive is away: the line stays where it was, marked; a notice says what to do + reindex_all(cm, con=True) + err = capsys.readouterr().err + assert 'cMeta notice' in err and 'cannot be reached' in err and 'cx repo unplug' in err and str(repo) in err + assert [same(p) for p in repos_json(tmp_path)] == order + assert marked(tmp_path, repo) is True + assert marked(tmp_path, other) is False + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 16 + assert find(cm, 'on-the-drive')['return'] == 16 + assert uids(find(cm, 'elsewhere')) == [other_uid] + + # ... and again: nothing changes, the file is not rewritten for it + before = (tmp_path / 'repos.json').read_bytes() + reindex_all(cm) + assert (tmp_path / 'repos.json').read_bytes() == before + assert capsys.readouterr().err == '' # without the console, the notice goes to the log only + + # The drive is back: the next index run - of any kind - indexes it again and clears the mark + away.rename(repo) + reindex_all(cm) + assert marked(tmp_path, repo) is False + assert [same(p) for p in repos_json(tmp_path)] == order + assert uids(find(cm, 'on-the-drive')) == [uid] + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 0 + + +def test_an_incremental_index_drops_its_records_and_brings_them_back(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + local_uid = make(cm, 'in-local', 'local') + + away = detach(repo) + + # The incremental index of another operation (here: a third repository is plugged) notices it: registered, + # marked, and its records are out of the index - a find must not hand out artifacts whose folder is away + third = make_repo(tmp_path, 'third', '3333333333333333') + plug(cm, third) + assert marked(tmp_path, repo) is True + assert find(cm, 'on-the-drive')['return'] == 16 + assert uid not in index_data(tmp_path, 'log')['uids'] + assert uids(find(cm, 'in-local')) == [local_uid] + + # Back, and another incremental operation runs (a fourth repository is plugged): the repository is indexed + # with it, although nobody named it + away.rename(repo) + fourth = make_repo(tmp_path, 'fourth', '4444444444444444') + plug(cm, fourth) + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'on-the-drive')) == [uid] + assert uids(find(cm, 'in-local')) == [local_uid] + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 0 + + +def test_repo_reindex_by_its_path_brings_an_unindexed_repository_back(cm, tmp_path, monkeypatch): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + away = detach(repo) + reindex_all(cm) + assert marked(tmp_path, repo) is True + + # Still away: refused, nothing changes + r = access(cm, category='repo', command='reindex', arg1=str(repo)) + assert r['return'] > 0 + assert marked(tmp_path, repo) is True + + away.rename(repo) + r = access(cm, category='repo', command='reindex', arg1=str(repo)) # not by its alias: the index does not know it + assert r['return'] == 0, r.get('error') + assert [same(p) for p in r['paths']] == [same(repo)] and r['artifacts'] == [] + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'on-the-drive')) == [uid] + + # ... also from inside it + detached_again = detach(repo) + reindex_all(cm) + detached_again.rename(repo) + monkeypatch.chdir(repo / 'log') + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'on-the-drive')) == [uid] + + +def test_unplug_by_path_forgets_an_unreachable_repository(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, repo) + plug(cm, other) + make(cm, 'on-the-drive', 'mine') + other_uid = make(cm, 'elsewhere', 'other') + away = detach(repo) + reindex_all(cm) + + # By its alias it is not known any more; by its path it is forgotten + assert access(cm, category='repo', command='unplug', arg1='mine')['return'] == 16 + assert marked(tmp_path, repo) is True + r = access(cm, category='repo', command='unplug', arg1=str(repo), con=True) + assert r['return'] == 0, r.get('error') + assert [same(p) for p in r['unregistered_paths']] == [same(repo)] and r['deleted_artifacts'] == [] + assert 'Unregistered' in capsys.readouterr().out + assert registered(tmp_path, repo) is None + assert marked(tmp_path, other) is False and uids(find(cm, 'elsewhere')) == [other_uid] + assert away.is_dir() and (away / 'log' / 'on-the-drive').is_dir() # the folder is nobody's business + + # A folder that is no registered repository is still "not found", and a folder inside one does not unplug it + assert access(cm, category='repo', command='unplug', arg1=str(tmp_path / 'extra'))['return'] == 16 + assert access(cm, category='repo', command='unplug', arg1=str(other / 'log'))['return'] == 16 + assert marked(tmp_path, other) is False + + # A reachable, indexed repository can be unplugged by its path too + r = access(cm, category='repo', command='unplug', arg1=str(other)) + assert r['return'] == 0, r.get('error') + assert registered(tmp_path, other) is None and find(cm, 'elsewhere')['return'] == 16 + assert other.is_dir() + + +def test_plugging_it_again_is_not_needed_but_works(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + away = detach(repo) + reindex_all(cm) + away.rename(repo) + + plug(cm, repo) # the old habit after a drive came back + assert marked(tmp_path, repo) is False + assert len([p for p in repos_json(tmp_path) if same(p) == same(repo)]) == 1 + assert uids(find(cm, 'on-the-drive')) == [uid] + + +def test_a_descriptor_that_cannot_be_read_does_not_unregister_the_repository(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'art', 'mine') + descriptor = repo / '_cmr.yaml' + good = descriptor.read_text(encoding='utf-8') + + for broken, why in (('artifact: [unclosed\n', 'cannot be read'), ('category: repo,f4f792ab40c7498f\n', 'names no artifact')): + descriptor.write_text(broken, encoding='utf-8') # a slip while editing it by hand + reindex_all(cm, con=True) + assert why in capsys.readouterr().err + assert marked(tmp_path, repo) is True + assert find(cm, 'art')['return'] == 16 + + descriptor.write_text(good, encoding='utf-8') + reindex_all(cm) + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'art')) == [uid] + + +def test_a_kept_repository_keeps_its_stored_descriptor_through_rewrites(cm, tmp_path): + """`cx repo plug --keep` stores the descriptor of a repository on a removable drive, so that it stays in the + repository index while the drive is away. The stored copy must survive a rewrite of repos.json made while the + repository is reachable (it was left out then, and the next time the drive was away the repository was dropped).""" + kept = make_repo(tmp_path, 'on-usb', '1111111111111111') + plug(cm, kept, keep=True) + assert registered(tmp_path, kept)['keep_repo_meta']['artifact'] == 'on-usb,1111111111111111' + + # another repository goes away and comes back: repos.json is rewritten twice while the kept one is reachable + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, other) + away = detach(other) + reindex_all(cm) + away.rename(other) + reindex_all(cm) + assert registered(tmp_path, kept)['keep_repo_meta']['artifact'] == 'on-usb,1111111111111111' + + # the drive of the kept repository is away: still registered, still in the repository index, never "unreachable" + kept_away = detach(kept) + reindex_all(cm) + entry = registered(tmp_path, kept) + assert entry['meta']['keep'] is True and 'unreachable' not in entry + assert access(cm, category='repo', command='find', arg1='on-usb')['return'] == 0 + kept_away.rename(kept) + + +def test_the_lines_of_repos_json_keep_their_order_and_only_the_mark_is_added(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + before = repos_json(tmp_path) + assert len(before) >= 2 # the internal repository, local, mine + + away = detach(repo) + reindex_all(cm) + after = repos_json(tmp_path) + assert list(after) == list(before) # the same lines in the same order + for path in before: + expected = dict(before[path], unreachable=True) if same(path) == same(repo) else before[path] + assert after[path] == expected, path + + away.rename(repo) + reindex_all(cm) + assert repos_json(tmp_path) == before and list(repos_json(tmp_path)) == list(before) # and back to what it was From 612a5e965cc88ccfcf5d517b759f49733c23b46f Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 21:43:57 +0200 Subject: [PATCH 5/6] The soft lock: a heartbeat, the system in its line, a take-over in one step, and a way to ask for it on a shared folder The soft lock is the fallback of PathLock where a file system refuses OS locks; nothing changes where OS locks work. - How long ago a soft lock was taken is counted from the time its holder wrote into the file, never from the time the file system gave the file (a drive can keep another clock: a Windows drive inside WSL2, a network share). An empty lock file is a leftover once a waiter has seen it empty for 2 seconds. - The holder keeps its lock file open and one thread per process rewrites the time in the line of every held soft lock every 2 seconds; a waiter that has seen the same line for 20 seconds on its own clock knows the holder is gone - no comparison of two machines' clocks, no limit on how long a lock is held, no pid it cannot check. - The line names the system (pid host time system: the kernel's boot id and pid namespace on Linux, the machine id on Windows and macOS), so the pid is checked only on the holder's own system: Windows and WSL2 share the host name and have different pids; a container shares the kernel and has its own pids. A lock file of an earlier cMeta (three words) is judged as before. - A stale lock file is replaced in one rename by one waiter (.take- made exclusively, checked, renamed over the path), never renamed away and put back: under load the old way gave two holders, leftover .stale-* files and a lock file carrying the line of a live process that did not hold it any more. - A single try takes the lock of a dead process in that try; a holder confirms the file at the path is its own and removes it at release only when it still is; the empty file a refused OS-lock attempt leaves is taken over the same way; a forked child never releases its parent's lock and starts with thread locks of its own. - A home or repository that several systems write at once through a shared drive asks for the soft lock: CMETA_SOFT_LOCK_PATHS (folders, or *) for a process, lock: soft in a repository's _cmr.yaml for every system that plugs it (registered when it is plugged and at the first access of every process); utils.files.use_soft_lock_under is the API. Each system keeps its own index of such a repository: another system's change is seen after cx repo reindex. - A lock with a note leaves no lock file behind on exFAT under macOS, where an empty file reports a placeholder inode that changes at the first write: the identity is asked from the open file at release. - The engine's own category API (category_api_v1.py) is imported once by its real name: loaded by its path under a name made from its folders, its relative imports made a second copy of the engine's modules in the process, each with a state of its own. Tests: tests/core_tests/test_utils_files_path_lock.py (21 -> 31), tests/internal_repo_tests/test_integration_repo_soft_lock.py (3); a test of the writes skips where a file system has no symbolic links. Docs: docs/async-and-concurrency.md, docs/using-cmeta.md. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- CHANGELOG.md | 62 ++ cmeta/utils/files.py | 678 +++++++++++++++--- cmeta/utils/sys.py | 61 +- docs/async-and-concurrency.md | 59 +- docs/using-cmeta.md | 6 +- .../core_tests/test_utils_files_path_lock.py | 579 ++++++++++++++- tests/core_tests/test_utils_files_writes.py | 5 +- .../test_integration_repo_soft_lock.py | 130 ++++ 8 files changed, 1428 insertions(+), 152 deletions(-) create mode 100644 tests/internal_repo_tests/test_integration_repo_soft_lock.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 3663d8b..858251b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,68 @@ All notable changes to cMeta are documented here, newest first. a `KeyError` (now: unreachable, with the reason); the pause "Fix it or press Enter to continue!" of an ambiguous category alias ended the index with `EOFError` when nobody could answer (now: it goes on). Tests: `tests/internal_repo_tests/test_integration_repo_unreachable.py`. +- **The soft lock: no clock of the drive, a heartbeat, the system in its line, a stale one taken over by + one waiter in one step, and a way to ask for it on a shared folder.** The soft lock is the fallback of + `PathLock` where a file system refuses OS locks (some FUSE, 9p and SMB mounts); nothing changes where OS + locks work. (1) How long ago a soft lock was taken is now counted from the time its holder wrote into the + file; before, it was the time the file system gave the file, compared with this machine's clock - on a + drive that keeps another clock (a Windows drive inside WSL2 is seconds off, a network share can be more) + a lock file made a moment ago looked old, so the grace of a holder that has just released and exited did + not hold there. An empty lock file, which nothing dates, is a leftover once a waiter has seen it empty for + 2 seconds (or when the drive dates it more than 10 minutes back). (2) **The holder beats.** It keeps its + lock file open and rewrites the time in its line every 2 seconds (a daemon thread; the line is rewritten + in place, so the file is never empty or shorter); a waiter that has seen the same line for 20 seconds on + its own monotonic clock knows the holder is gone - without comparing two machines' clocks, however long + the lock has been held (the old rule took a lock of another host for abandoned after 10 minutes, a build + of an hour included), and without a pid it cannot check (so a holder that crashed on ANOTHER system is + seen as running - by `cx cache classify` too - for those 20 seconds; one of this system is known at once by + its pid). (3) **The line names the system** (`pid host + time system`: a short hash of the kernel's boot id and the pid namespace on Linux, of the machine id on + Windows and macOS), so a waiter checks the pid only when it is a pid of its own system: Windows and WSL2 + share the host name and have different pids, a container shares the kernel of its host and has its own + pids; a dead pid of the same system is still known at once, a live one keeps its lock. A lock file of + cMeta before 0.35.0 (three words, no heartbeat) is judged as before. (4) A stale lock file was renamed + away, checked and - when it was not the judged one - put back; between the two a third process could + take the free path, which under load could end with two holders, a leftover `.stale-*`, or a + lock file carrying the line of a live process that did not hold it any more - a lock nobody could take + for as long as that process lived. Now the waiter makes the file that replaces the stale one exclusively, + under a name of that stale file (`.take-`: one waiter only), checks that the file at + the path is still the one it judged, and puts its own in place with one rename - the path is never free + in between, and a waiter that comes a step later finds another file there and leaves it. The file is + closed before the rename on every platform (Windows refuses to rename a file the process has open; a + Windows drive seen through 9p - WSL2's `/mnt/d`, a container's bind mount - shows nothing at the new path + while the descriptor is open) and opened again at its new path to be held. (5) A single try + (`timeout 0`: the probe behind `cx cache classify`) takes the lock of a dead process in that try; before, + it cleared the file and still reported the lock held once. (6) A holder checks, after it took the lock, + that the file at the path is its own (the open descriptor, or its line), and removes the file at release + only when it still is; on Windows a held lock file cannot be replaced at all, so a live holder there is + never taken over by mistake. (7) The empty file that a refused OS-lock attempt leaves is taken over the + same way instead of being removed outright. (8) A forked child of a holder never releases its parent's + lock, and starts with thread locks of its own (the ones it inherits may have been held by a thread of the + parent that does not exist in the child). (9) **Asking for it.** A home or repository that several systems write at once through a shared + drive - a Windows tree plugged from WSL2 or from a container, a network share - has no lock between those + systems with the OS locks (each system's lock is invisible to the others; the exclusive create of the + soft lock file is honoured by all). `CMETA_SOFT_LOCK_PATHS=[...]` (`*` for + everywhere) makes every lock file under those folders use the soft lock in that process, and `lock: soft` + in a repository's `_cmr.yaml` does it for that repository on every system that plugs it (registered when + it is plugged and at the start of every process, from the index of repositories; the API is + `utils.files.use_soft_lock_under`). Everything else keeps the OS locks: a lock cycle costs what it did. + (Each system still keeps its own index of such a repository in its own home: what another system created or + changed there is seen after `cx repo reindex `; the files are always current and no update is lost.) +- **The engine's own category API is imported once.** `category_api_v1.py` - the base commands of every + category - was loaded by its path under a name made from its folders (`cmeta.cmeta.category_api_v1`), + and its relative imports made a second copy of the engine's modules in the process (`cmeta.cmeta.utils. + files` next to `cmeta.utils.files`), each with a state of its own: the registries of the path lock among + them, so the soft lock asked for a folder was unknown to the copy that wrote the artifacts, and the + threads of one process were serialized on a path by two registries instead of one (the OS locks kept + them apart all the same). The loader now imports a module of the engine's own package by its real name + (`cmeta.category_api_v1`); category APIs of repositories are loaded as before. +- **A lock with a note leaves no lock file behind on exFAT under macOS.** There an empty file reports a + placeholder inode that becomes a real one at its first write: the lock noted the identity of its (empty) + file when it took it, the note of the holder (the index lock of a reindex, a cache entry being built) + changed it, and at release "the file at the path is not ours" left `.lock` behind. The identity is + now asked from the open file at release. + Tests of both: `tests/core_tests/test_utils_files_path_lock.py`. - **Python 3.15.** The suite runs on Python 3.15 in CI, next to 3.9 and 3.14 (Linux, Windows, macOS); nothing in the engine needed a change. `pyproject.toml` carries the classifier. diff --git a/cmeta/utils/files.py b/cmeta/utils/files.py index 91db7d7..63d369b 100644 --- a/cmeta/utils/files.py +++ b/cmeta/utils/files.py @@ -23,6 +23,7 @@ import logging import random import threading +import binascii from .common import _error, ask as _ask @@ -44,11 +45,20 @@ LOCK_SUFFIX = ".lock" LOCK_POLL_START = 0.001 # the first wait when a lock is busy; it doubles up to LOCK_POLL_MAX LOCK_POLL_MAX = 0.025 -LOCK_SOFT_STALE_SECONDS = 600 # a soft lock file older than this, whose holder is on another host, is stale -LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file older than this is a leftover (a holder writes its content at once) -LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock file of a dead pid younger than this is a release in flight, not a crash +# The age of a soft lock is counted from the time its holder WROTE INTO IT, read on this machine's clock, and an +# empty file by how long the waiter itself has seen it - never by the time the file system gave the file: a drive +# can keep another clock (a network share, a Windows drive inside WSL2) or keep times to the whole second. +LOCK_SOFT_STALE_SECONDS = 600 # a soft lock taken longer ago than this by a holder on another host is stale +LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file that a waiter has seen empty for this long is a leftover (a holder writes its content at once) +LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock of a dead pid taken less than this ago is a release in flight, not a crash LOCK_SOFT_CHECK_SECONDS = 0.25 # how often a waiter on a soft lock reads the file to check for a stale holder LOCK_SOFT_REMOVE_SECONDS = 2.0 # how long a releasing soft lock retries the removal of its file (Windows: a reader may hold it) +LOCK_SOFT_TAKE_TRIES = 25 # tries of the rename that takes a stale soft lock over (Windows: a reader may hold a file for an instant) +LOCK_SOFT_TAKE_REMOVE_SECONDS = 0.2 # how long a waiter retries the removal of the take file it did not use +LOCK_SOFT_READ_MAX = 1024 # bytes read from a soft lock file (its one line: pid host time system) +LOCK_SOFT_HEARTBEAT_SECONDS = 2.0 # a soft holder rewrites the time in its line this often (the heartbeat; soft mode only) +LOCK_SOFT_SILENT_SECONDS = 20.0 # a soft lock whose line a waiter has seen unchanged for this long is the leftover of a holder that is gone +LOCK_SOFT_PATHS_ENV = 'CMETA_SOFT_LOCK_PATHS' # folders whose lock files use the soft lock (a home or repository written by several systems at once) LOCK_IDENTITY_RETRIES = 50 # consecutive identity mismatches after which a path is locked with the soft lock LOCK_NOTE_MAX = 512 # bytes of the note a holder may leave in its lock file for the message of a waiter LOCK_TIMEOUT_ENV = 'CMETA_LOCK_TIMEOUT' @@ -155,10 +165,19 @@ def is_path_within( # POSIX: fcntl.flock (per open file description, so threads of one process exclude each other too), # then fcntl.lockf (POSIX record locks: NFS and some network mounts refuse flock), each with # the identity check (fstat of the descriptor vs stat of the path, inode and device); -# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time") where the file -# system refuses both OS locks (some FUSE, 9p and SMB mounts) or the identity check keeps -# failing - a stale soft lock (its pid dead on this host, or older than LOCK_SOFT_STALE_SECONDS -# when left by another host) is renamed away and removed by the waiter that finds it. +# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time system") where the +# file system refuses both OS locks (some FUSE, 9p and SMB mounts), where the identity check +# keeps failing, or where it is ASKED FOR - a home or repository that several systems write at +# once through a shared drive (a Windows tree plugged from WSL2 or a container, a network +# share), whose OS locks do not see each other: CMETA_SOFT_LOCK_PATHS names such folders, +# `lock: soft` in a repository's _cmr.yaml names the repository (use_soft_lock_under). The +# holder keeps its file open and rewrites the time in its line every LOCK_SOFT_HEARTBEAT_SECONDS; +# a stale soft lock (its pid dead on this system, or its line unchanged for +# LOCK_SOFT_SILENT_SECONDS on the waiter's own clock - the holder is on another system or host) +# is taken over by ONE of the waiters that find it: it replaces the file by its own in one +# rename (PathLock._take_stale_soft_lock). A lock file of cMeta before 0.35.0 (no system, no +# heartbeat) is judged as before: the pid when the host name is this one, else its age against +# LOCK_SOFT_STALE_SECONDS. # In every mode the threads of one process are serialized on the path first (a threading.Lock per path), # which POSIX record locks and soft locks need (they are per process), and the busy wait polls with a # delay that doubles from LOCK_POLL_START to LOCK_POLL_MAX until the timeout. The engine's own writers @@ -171,8 +190,48 @@ def is_path_within( _lock_registry = {} # normalized lock file path -> threading.Lock (the threads of this process) _lock_registry_guard = threading.Lock() _lock_soft_paths = set() # lock files locked with the soft lock from now on (the OS lock was refused) +_lock_soft_dirs = [] # folders whose lock files use the soft lock because it was asked for (use_soft_lock_under) _lock_no_flock_dirs = set() # folders where flock was refused (POSIX record locks are tried first there) _lock_host = None +_lock_system = None +_beat_lock = threading.Lock() # guards _beat_held, and every write of a beat +_beat_held = {} # id(lock) -> the soft locks this process holds and beats (PathLock._start_heartbeat) +_beat_thread = None # the one heartbeat thread of the process, started with the first soft lock held +_beat_wake = threading.Event() # the thread sleeps on it (never set: a plain sleep that reads the interval each time) + + +def _after_fork_in_child(): + """A forked child starts with locks of its own: the thread locks of the parent (the registry of the paths, + the guard of the registry, the lock of the heartbeat) may have been held by a thread that does not exist in + the child - the parent's heartbeat thread, another thread of the parent - and would be held for ever; the + soft locks the parent beats are not the child's to beat (its copy of a lock never releases it either: + PathLock.release).""" + global _lock_registry_guard, _beat_lock, _beat_thread + _lock_registry_guard = threading.Lock() + _lock_registry.clear() + _beat_lock = threading.Lock() + _beat_held.clear() + _beat_thread = None + + +if hasattr(os, 'register_at_fork'): + os.register_at_fork(after_in_child=_after_fork_in_child) + + +def _beat_forever(): + """The heartbeat thread: every LOCK_SOFT_HEARTBEAT_SECONDS, the line of every soft lock this process holds + is rewritten (PathLock._write_beat). The locks a forked child inherited are its parent's: dropped.""" + while True: + _beat_wake.wait(LOCK_SOFT_HEARTBEAT_SECONDS) + try: + with _beat_lock: + for key, lock in list(_beat_held.items()): + if lock._owner_pid != os.getpid(): + _beat_held.pop(key, None) + continue + lock._write_beat() + except Exception: + pass def _lock_thread_lock(lock_file): @@ -204,6 +263,92 @@ def _lock_pid_alive(pid): return True # unknown: never treat it as dead +def _lock_system_id(): + """ + A short name of the system this process runs on, written into a soft lock file so that a waiter knows + whether the pid in the file is one it can check: the same name means the same kernel and the same pid + space. Windows and WSL2 share the host name and have different pids; a container shares the kernel of + its host (one boot) and has its own pid space; a cloned machine keeps its host name. + Linux: the boot id of the kernel and the pid namespace of this process (/proc); + Windows: the machine's installation id (the registry); + macOS: the machine's id (sysctl kern.uuid); + else, or where those cannot be read: the host name and the platform. + Twelve hex digits of a hash of it, computed once per process. + """ + global _lock_system + if _lock_system is None: + parts = [sys.platform] + try: + if sys.platform.startswith('linux'): + with open('/proc/sys/kernel/random/boot_id') as f: + parts.append(f.read().strip()) + parts.append(str(os.stat('/proc/self/ns/pid').st_ino)) + elif sys.platform == 'win32': + import winreg + with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Microsoft\Cryptography') as key: + parts.append(str(winreg.QueryValueEx(key, 'MachineGuid')[0])) + elif sys.platform == 'darwin': + import ctypes + libc = ctypes.CDLL(None) + size = ctypes.c_size_t(0) + if libc.sysctlbyname(b'kern.uuid', None, ctypes.byref(size), None, 0) == 0 and 0 < size.value < 256: + buf = ctypes.create_string_buffer(size.value) + if libc.sysctlbyname(b'kern.uuid', buf, ctypes.byref(size), None, 0) == 0: + parts.append(buf.value.decode('ascii', 'replace')) + except Exception: + pass + if len(parts) < 2: + parts.append(_lock_host_name()) + _lock_system = hashlib.sha1('\n'.join(parts).encode('utf-8', 'replace')).hexdigest()[:12] + return _lock_system + + +def _soft_lock_paths_from_env(): + """The folders named by CMETA_SOFT_LOCK_PATHS (os.pathsep between them; '*' = everywhere), normalized.""" + value = os.environ.get(LOCK_SOFT_PATHS_ENV, '') + folders = [] + for folder in value.split(os.pathsep): + folder = folder.strip() + if folder == '': + continue + folders.append('*' if folder == '*' else os.path.normcase(os.path.abspath(folder))) + return folders + + +def use_soft_lock_under( + folder: str, # The folder (a home, a repository) whose lock files use the soft lock from now on in this process. +): + """ + Ask for the soft lock for every lock file under `folder` (see the notes on the path lock: a home or + repository that several systems write at once through a shared drive, whose OS locks do not see + each other). The engine calls it for a repository whose descriptor says `lock: soft`; the environment + variable CMETA_SOFT_LOCK_PATHS names folders at the start of a process. '*' means everywhere. + + Returns: + dict: {'return': 0} + """ + folder = '*' if folder == '*' else os.path.normcase(os.path.abspath(folder)) + if folder not in _lock_soft_dirs: + _lock_soft_dirs.append(folder) + return {'return': 0} + + +def _soft_lock_asked_for(lock_file): + """Whether `lock_file` lies under a folder for which the soft lock was asked (use_soft_lock_under).""" + if not _lock_soft_dirs: + return False + if '*' in _lock_soft_dirs: + return True + path = os.path.normcase(os.path.abspath(lock_file)) + for folder in _lock_soft_dirs: + if path == folder or path.startswith(folder.rstrip(os.sep) + os.sep): + return True + return False + + +_lock_soft_dirs.extend(_soft_lock_paths_from_env()) # CMETA_SOFT_LOCK_PATHS, read once at the start of the process + + class _LockUnsupported(Exception): pass @@ -232,6 +377,13 @@ def __init__( self._mismatches = 0 self._last_error = None self._next_stale_check = 0.0 + self._empty_seen = {} # path -> (identity, monotonic time): the empty soft lock files this waiter watches + self._own_empty = None # (identity,) of the empty file this object's refused OS-lock attempt left + self._noted = False # a note was written into the held lock file (see _own_identity) + self._line = None # the line of the soft lock this object holds (the heartbeat rewrites it) + self._wrote = None # the line of a soft lock file this object wrote and could not confirm + self._beat_seen = {} # path -> (identity, content, monotonic time): the lines this waiter watches for a heartbeat + self._owner_pid = None # the process that took the lock (a forked child never releases its parent's lock) def __enter__(self): self.acquire() @@ -258,6 +410,7 @@ def acquire( if not thread_lock.acquire(timeout=timeout): raise TimeoutError(f"the lock {self.lock_file} is held by another thread of this process (waited {timeout} s)") + self._owner_pid = os.getpid() # before the lock is taken: the heartbeat thread reads it try: deadline = time.monotonic() + timeout delay = LOCK_POLL_START @@ -279,6 +432,8 @@ def acquire( delay = min(delay * 2, LOCK_POLL_MAX) except BaseException: + self._owner_pid = None + self._drop_unconfirmed() thread_lock.release() raise @@ -287,17 +442,27 @@ def release(self): if not self.is_locked: return + if self._owner_pid is not None and self._owner_pid != os.getpid(): + # A forked child of the holder: the lock is its parent's. The child's copy forgets it and touches + # neither the file nor the descriptor (a POSIX record lock is not even the child's to release) + self._fd = self._thread_lock = None + self._ident = self._line = None + self._noted = self.is_locked = False + self.mode = None + return + try: if self.mode == 'windows': try: msvcrt.locking(self._fd, msvcrt.LK_UNLCK, 1) except OSError: pass + ident = self._own_identity() os.close(self._fd) - self._unlink_own_file() + self._unlink_own_file(ident) elif self.mode in ('flock', 'lockf'): - self._unlink_own_file() + self._unlink_own_file(self._own_identity()) try: self._posix_unlock(self._fd, self.mode) except OSError: @@ -305,11 +470,15 @@ def release(self): os.close(self._fd) elif self.mode == 'soft': + self._stop_heartbeat() self._remove_soft_file() finally: self._fd = None self._ident = None + self._line = None + self._noted = False + self._owner_pid = None self.is_locked = False self.mode = None thread_lock = self._thread_lock @@ -325,6 +494,7 @@ def write_note(self, note): if not self.is_locked or self._fd is None or self.mode == 'soft': return data = b'\n' + str(note).encode('utf-8', 'replace')[:LOCK_NOTE_MAX] + b'\n' + self._noted = True try: os.ftruncate(self._fd, 0) os.lseek(self._fd, 0, os.SEEK_SET) @@ -362,7 +532,7 @@ def read_note(lock_file): # ---- one attempt ---------------------------------------------------------------------- def _try(self): - if self.lock_file in _lock_soft_paths: + if self.lock_file in _lock_soft_paths or _soft_lock_asked_for(self.lock_file): return self._try_soft() if msvcrt is not None: return self._try_windows() @@ -495,151 +665,435 @@ def _posix_unlock(fd, kind): else: fcntl.lockf(fd, fcntl.LOCK_UN) + @staticmethod + def _soft_line(): + """The line of a soft lock this process holds: `pid host time system`. The time is the moment of the + last write (the heartbeat rewrites the line in place, so its length never changes); the system (see + _lock_system_id) tells a waiter whether it can check the pid.""" + return f"{os.getpid()} {_lock_host_name()} {time.time():.3f} {_lock_system_id()}\n".encode('utf-8', 'replace') + def _try_soft(self): + line = self._soft_line() try: fd = os.open(self.lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) except FileExistsError as e: self._last_error = e now = time.monotonic() - if now >= self._next_stale_check: - self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS - self._clear_stale_soft_lock() - return False + if now < self._next_stale_check: + return False + self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS + # The file of a holder that is gone is taken over in the same attempt: a single try (timeout 0, the + # probe of a cache entry) finds the lock of a dead process free + return self._take_stale_soft_lock(line) except OSError as e: if self._open_failed_for_the_moment(e): return False raise try: - os.write(fd, (f"{os.getpid()} {_lock_host_name()} {time.time():.3f}\n").encode('utf-8', 'replace')) - finally: + os.write(fd, line) + except BaseException: os.close(fd) + raise + return self._soft_lock_taken(line, fd) - try: - self._ident = self._identity(os.stat(self.lock_file)) - except OSError: - self._ident = None + def _soft_lock_taken(self, line, fd=None): + """ + The last step of taking the soft lock: the file at the path must be the file this object wrote. It is + not when that file was taken over in the instant it was still empty (this process was stopped between + creating it and writing into it for as long as an empty file is watched) - the lock is then not held, + and nothing at the path is touched. + + fd: the open descriptor of the written file, when there is one. It stays open while the lock is held: + an open file keeps its identity (no new file gets its inode number), so "the file at the path is the + one this descriptor holds" is an exact answer, here and at release, for the price of two calls that + open nothing. (And on Windows a file that is open cannot be replaced: the lock of a live holder + cannot be taken over there at all.) Where the identity does not say so - no descriptor (a file + renamed into place by a take-over, a file found later), a file system without inode numbers or one + that reports them differently by descriptor and by path - the line in the file decides. + """ + mine = None + if fd is not None: + try: + mine = self._identity(os.fstat(fd)) + if mine is not None and self._identity(os.stat(self.lock_file)) == mine: + return self._hold_soft(line, fd, mine) + except OSError: + pass + + status, held = 'unreadable', None + for _ in range(3): + status, held = self._open_if_mine(line) + if status in ('mine', 'other'): + break + time.sleep(0.002) # not readable for an instant (Windows: a scanner holds the new file), or not shown yet + + if status == 'mine': + if fd is not None: + try: + os.close(fd) + except OSError: + pass + try: + mine = self._identity(os.fstat(held)) + except OSError: + mine = None + return self._hold_soft(line, held, mine) + if status == 'unreadable' and fd is not None: + return self._hold_soft(line, fd, mine) # it cannot be asked: the file this object made is its lock, as before + + # Not held. Should the file with this line be at the path later after all (a file system slow to show + # a new file), it is this object's lock (_take_stale_soft_lock), and an acquire that gives up removes + # it (_drop_unconfirmed) + if fd is not None: + os.close(fd) + self._wrote = line + return False + + def _hold_soft(self, line, fd, ident): + self._fd = fd + self._ident = ident + self._line = line + self._wrote = None + self._own_empty = None + self._empty_seen = {} + self._beat_seen = {} self.mode = 'soft' + self._start_heartbeat() return True - def _clear_stale_soft_lock(self): - """A soft lock file whose holder is dead (this host; the file at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS - old, so that a holder that just released and exited is not mistaken for a crash), that was marked - released by a holder that could not remove it, that is empty and old (a leftover) or too old (another - host) is renamed away and removed - by one waiter only (the rename is atomic), and only when the file - at the path is still the file that was judged (a fresh lock file of a live holder moved by mistake in - the instant between the check and the rename is put back).""" + # ---- the heartbeat of a soft holder ------------------------------------------------------ + # + # A soft holder rewrites the time in its line every LOCK_SOFT_HEARTBEAT_SECONDS (one daemon thread per + # process for all the soft locks it holds; the line is rewritten in place through the held descriptor: + # same length, never empty). A waiter that sees the same line for LOCK_SOFT_SILENT_SECONDS on its own + # monotonic clock knows the holder is gone - whatever the other system's clock says, however long the + # holder has held the lock, and without a pid it could not check (a holder on another host, or on + # another system of this host: Windows and WSL2 share the host name and have different pids). A lock + # file without a system (3 words: written by cMeta before 0.35.0, which did not beat) is judged as + # before: the pid when the host name is this one, else the age of its time against LOCK_SOFT_STALE_SECONDS. + + def _start_heartbeat(self): + if self._fd is None: + return # no descriptor to write through (the line decides for the waiters: see the rules) + global _beat_thread + with _beat_lock: + _beat_held[id(self)] = self + if _beat_thread is None or not _beat_thread.is_alive(): + _beat_thread = threading.Thread(target=_beat_forever, name='cmeta-soft-lock-heartbeat', daemon=True) + _beat_thread.start() # its first beat comes after one interval: a lock held for less never beats + + def _stop_heartbeat(self): + with _beat_lock: # a beat of this lock in flight ends before the lock goes on + _beat_held.pop(id(self), None) + + def _write_beat(self): + fd = self._fd + if fd is None: + return + line = self._soft_line() + try: + if hasattr(os, 'pwrite'): + os.pwrite(fd, line, 0) + else: + os.lseek(fd, 0, os.SEEK_SET) + os.write(fd, line) + except (OSError, ValueError): + return # the drive refuses for the moment: the line stays as it is + self._line = line + + def _open_if_mine(self, line): + """Open the file at the lock path and keep it open when it carries `line`: ('mine', descriptor); + ('other', None) when it carries something else, ('gone', None) when it is not there, ('unreadable', + None) when it cannot be read at the moment. Open for writing too: the heartbeat writes through it.""" try: - fd = os.open(self.lock_file, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + fd = os.open(self.lock_file, os.O_RDWR | getattr(os, 'O_BINARY', 0)) + except FileNotFoundError: + return 'gone', None except OSError: + return 'unreadable', None + try: + data = os.read(fd, LOCK_SOFT_READ_MAX) + except OSError: + os.close(fd) + return 'unreadable', None + if data != line: + os.close(fd) + return 'other', None + return 'mine', fd + + def _drop_unconfirmed(self): + """An acquire gives up: a file at the path that carries a line this object wrote and could not confirm + (see _soft_lock_taken) would keep everybody out for as long as this process lives - it goes.""" + wrote, self._wrote = self._wrote, None + if wrote is None: return + seen = self._read_lock_file(self.lock_file) + if seen and seen[1] == wrote: + self._remove_file(self.lock_file, LOCK_SOFT_TAKE_REMOVE_SECONDS) + + @classmethod + def _read_lock_file(cls, path): + """(identity, content, time of the file) of a soft lock file; False when it is not there, None when it + cannot be read at the moment.""" + try: + fd = os.open(path, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + except FileNotFoundError: + return False + except OSError: + return None try: st = os.fstat(fd) - words = os.read(fd, 256).decode('utf-8', 'replace').split() if st.st_size > 0 else [] + data = os.read(fd, LOCK_SOFT_READ_MAX) if st.st_size > 0 else b'' except OSError: - return + return None finally: os.close(fd) - - ident = self._identity(st) - age = time.time() - st.st_mtime - pid = int(words[0]) if len(words) > 0 and words[0].isdigit() else None - host = words[1] if len(words) > 1 else None + return cls._identity(st), data, st.st_mtime + + def _lock_file_is(self, path, seen): + """Whether the file at `path` is still the one read as `seen`: the same content and, where the file + system reports one, the same identity.""" + now = self._read_lock_file(path) + return bool(now) and now[1] == seen[1] and (now[0] is None or seen[0] is None or now[0] == seen[0]) + + def _soft_file_is_stale(self, path, seen): + """Whether the soft lock file read as `seen` (or the take file of a waiter, which carries the same + line) is the leftover of a holder that is gone: + - marked released by a holder that could not remove it; + - a line of four words (`pid host time system`: a holder that beats, cMeta 0.35.0 and later): its pid + is dead on THIS system (the system is this one; taken at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS ago, so + that a holder that has just released and exited is not a crash), or this waiter has seen the same + line for LOCK_SOFT_SILENT_SECONDS on its own monotonic clock (the holder beats no more: dead on + another system or host, or stopped); + - a line of three words (a holder of cMeta before 0.35.0, which does not beat): its pid is dead on + this host (the host name is this one), or it was taken longer ago than LOCK_SOFT_STALE_SECONDS; + - it is empty and this waiter has seen it empty for LOCK_SOFT_EMPTY_STALE_SECONDS (a holder writes + its line right after it creates the file), or it is the empty file this object's own OS-lock + attempt has just left. + How long ago a lock was taken is read from the time its holder wrote into it, on this machine's clock. + The time the file system gave the file is not used - on a drive that keeps another clock (a Windows + drive inside WSL2 is seconds off, a network share can be more) a fresh lock file looked old - except + where nothing else dates the file: a line without a time (not written by this code), and an empty file + that nobody watches (older than LOCK_SOFT_STALE_SECONDS it is a leftover whatever the clock of the drive).""" + ident, data, mtime = seen + words = data.decode('utf-8', 'replace').split() if not words: - stale = age > LOCK_SOFT_EMPTY_STALE_SECONDS - elif words[0] == 'released': - stale = True - elif pid is not None and host == _lock_host_name(): - stale = age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) - else: - stale = age > LOCK_SOFT_STALE_SECONDS + if path == self.lock_file and self._own_empty is not None and self._own_empty[0] == ident: + return True + now = time.monotonic() + watched = self._empty_seen.get(path) + if watched is None or watched[0] != ident: + watched = self._empty_seen[path] = (ident, now) + return now - watched[1] > LOCK_SOFT_EMPTY_STALE_SECONDS or time.time() - mtime > LOCK_SOFT_STALE_SECONDS - if not stale: - return + self._empty_seen.pop(path, None) + if words[0] == 'released': + return True try: - if ident is not None and self._identity(os.stat(self.lock_file)) != ident: - return # the file changed under us: judge it again next time - except OSError: - return + age = time.time() - float(words[2]) + except (IndexError, ValueError): + age = None + if age is None or age != age or abs(age) == float('inf'): + age = time.time() - mtime + + pid = int(words[0]) if words[0].isascii() and words[0].isdigit() else None + + if len(words) >= 4: + # A holder that beats (cMeta 0.35.0 and later: the fourth word names its system). Its pid is checked + # when it is a pid of THIS system; whatever the system, a line that has not changed for + # LOCK_SOFT_SILENT_SECONDS on this waiter's own clock is the leftover of a holder that is gone. + if pid is not None and words[3] == _lock_system_id() and age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid): + return True + now = time.monotonic() + watched = self._beat_seen.get(path) + if watched is None or watched[0] != ident or watched[1] != data: + self._beat_seen[path] = (ident, data, now) + return False + return now - watched[2] > LOCK_SOFT_SILENT_SECONDS + + # A holder that does not beat (cMeta before 0.35.0): its pid when the host name is this one, else how + # long ago it was taken + if pid is not None and len(words) > 1 and words[1] == _lock_host_name(): + return age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) + return age > LOCK_SOFT_STALE_SECONDS + + def _take_stale_soft_lock(self, line): + """ + Take the soft lock over from a holder that is gone (see _soft_file_is_stale). Returns True when this + object holds the lock. + + The stale file is never removed: the waiter REPLACES it by a file that carries its own line (one + atomic rename), so the path is never free in between and nobody else can slip in. The file to put + there is made first, exclusively, under a name that says which stale file it is for + (`.take-`): of the waiters that judged the + same stale file only one gets to make it, and it then checks that the file at the path is still the + one it judged before it replaces it. A waiter that comes later finds another file at the path and + gives up. (Before, the stale file was renamed away, checked and - when it was not the judged one - + put back: between the two a third process could take the free path, which under load could end + with two holders and a leftover `.stale-*` file.) + """ + seen = self._read_lock_file(self.lock_file) + if not seen: + return False + if self._wrote is not None and seen[1] == self._wrote: + return self._soft_lock_taken(self._wrote) # the file this object wrote a moment ago + if not self._soft_file_is_stale(self.lock_file, seen): + return False - moved = f"{self.lock_file}.stale-{os.getpid()}-{random.getrandbits(32):08x}" + data = seen[1] + take = self._take_file(seen) try: - os.rename(self.lock_file, moved) # atomic: only one waiter gets to remove it + fd = os.open(take, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) + except FileExistsError: + # Another waiter is taking this very file over - or died doing so, and its take file is a leftover + theirs = self._read_lock_file(take) + if theirs and self._soft_file_is_stale(take, theirs) and self._lock_file_is(take, theirs): + self._remove_file(take) + return False except OSError: - return + return False - if ident is not None: - try: - moved_ident = self._identity(os.stat(moved)) - except OSError: - return - if moved_ident != ident: - # Not the file judged stale but a fresh one made in between: back where it was, without - # clobbering (a hard link fails where a file exists already), then the extra name goes + taken = False + try: + os.write(fd, line) + # Closed before the rename, everywhere: Windows refuses to rename a file this process has open, + # and a Windows drive seen through 9p (WSL2's /mnt/d, a container's bind mount) renames it but + # shows nothing at the new path until the descriptor is closed (the lock looked "gone" to the + # taker itself); the file is opened again at its new path to hold it (_soft_lock_taken) + os.close(fd) + fd = None + for _ in range(LOCK_SOFT_TAKE_TRIES): + if not self._lock_file_is(self.lock_file, seen): + break # not the file that was judged any more: judged again at the next check try: - os.link(moved, self.lock_file) + os.replace(take, self.lock_file) + except PermissionError: + # Windows: a waiter that reads one of the two files holds it for an instant - and a holder + # that is alive after all holds its file open for as long as it holds the lock + time.sleep(0.002) + continue except OSError: - try: - if not os.path.exists(self.lock_file): - os.rename(moved, self.lock_file) - except OSError: - pass - return + break + taken = True + break + finally: + if fd is not None: try: - os.unlink(moved) + os.close(fd) # the write failed before the close above except OSError: pass - return + if not taken: + self._wrote = line # see _soft_lock_taken: a file with this line that reaches the path is this object's + self._remove_file(take, LOCK_SOFT_TAKE_REMOVE_SECONDS) - try: - os.unlink(moved) - except OSError: - pass - self.logger.warning(f"utils.files.PathLock - removed the stale lock file {self.lock_file} (left by pid {pid} on {host})") + if not taken: + return False + + words = data.decode('utf-8', 'replace').split() + left_by = "an empty file" if not words else ("marked released" if words[0] == 'released' else ' '.join(words[:2])) + self.logger.warning(f"utils.files.PathLock - took over the stale lock file {self.lock_file} (left by {left_by})") + return self._soft_lock_taken(line) + + def _take_file(self, seen): + """The name of the file a waiter makes to replace the stale lock file read as `seen`: one name per + stale file (its content and identity), so that one waiter only can make it.""" + return f"{self.lock_file}.take-{binascii.crc32(seen[1] + repr(seen[0]).encode('ascii', 'replace')) & 0xffffffff:08x}" + + @staticmethod + def _remove_file(path, seconds=0.0): + """Remove a file, retrying for `seconds` while it is held (Windows: a reader has it open); best effort.""" + deadline = time.monotonic() + seconds + while True: + try: + os.unlink(path) + return + except PermissionError: + if time.monotonic() >= deadline: + return + time.sleep(0.002) + except OSError: + return def _fall_back_to_soft(self, why): if self.lock_file not in _lock_soft_paths: _lock_soft_paths.add(self.lock_file) self.logger.warning(f"utils.files.PathLock - the file system refuses OS locks for {self.lock_file} ({why}): a soft lock file is used instead (best effort between hosts)") - # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its - # content at once): out of the way, or the soft lock could never be taken + # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its line + # at once): this object takes it over at once, without the time an empty file is otherwise watched try: - if os.path.getsize(self.lock_file) == 0: - os.unlink(self.lock_file) + st = os.stat(self.lock_file) + self._own_empty = (self._identity(st),) if st.st_size == 0 else None except OSError: - pass + self._own_empty = None def _remove_soft_file(self): - """Remove the soft lock file; on Windows a waiter reading it holds it open for a moment, so the removal - is retried, and a file that cannot be removed in time is marked released for the waiters to clear.""" - deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS - while True: - if self._ident is not None: + """Remove the soft lock file - when it is still the file of this lock (a file taken over by a waiter + that judged this holder gone is theirs): the file at the path is the one this lock's descriptor holds + or, where the identity does not say so, it carries this lock's line. On Windows a waiter reading the + file holds it open for a moment, so the removal is retried, and a file that cannot be removed in time + is marked released for the waiters to take over.""" + fd, self._fd = self._fd, None + mine = self._ident + try: + if fd is not None: try: - if self._identity(os.stat(self.lock_file)) != self._ident: - return # not ours any more + mine = self._identity(os.fstat(fd)) except OSError: + pass + # Closed before the removal, everywhere: Windows cannot remove a file this process has open, + # and a Windows drive seen through 9p keeps the path of an open file in a removed-but-there + # state until the descriptor is closed. Nothing is lost by it: the soft lock is the line in + # the file, not the descriptor (a waiter judges the holder by its pid or its heartbeat) + try: + os.close(fd) + except OSError: + pass + fd = None + + deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS + while True: + ours = False + if mine is not None: + try: + ours = self._identity(os.stat(self.lock_file)) == mine + except FileNotFoundError: + return + except OSError: + ours = False + if not ours: + seen = self._read_lock_file(self.lock_file) + if seen is False: + return + if seen is not None and self._line is not None and seen[1] != self._line: + return # not ours any more + try: + os.unlink(self.lock_file) return - try: - os.unlink(self.lock_file) - return - except FileNotFoundError: - return - except PermissionError: - if time.monotonic() >= deadline: + except FileNotFoundError: + return + except PermissionError: + if time.monotonic() >= deadline: + break + time.sleep(0.002) + except OSError: break - time.sleep(0.002) + + try: + with open(self.lock_file, 'w') as f: + f.write(f"released {os.getpid()}\n") except OSError: - break + pass + self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") - try: - with open(self.lock_file, 'w') as f: - f.write(f"released {os.getpid()}\n") - except OSError: - pass - self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") + finally: + if fd is not None: + try: + os.close(fd) + except OSError: + pass @staticmethod def _identity(st): @@ -648,12 +1102,26 @@ def _identity(st): return None return (st.st_ino, st.st_dev) - def _unlink_own_file(self): - """Remove the lock file, unless the file at the path is no longer the one this lock took (someone - removed it under us and another process created a new one: that one is theirs).""" - if self._ident is not None: + def _own_identity(self): + """The identity of the file this lock's descriptor holds. It is the one seen when the lock was taken - + unless a note was written into the file since: a file system without real inodes can give a file + another number at its first write (exFAT on macOS: an empty file has a placeholder until `write_note` + fills it), and the old number would say "not ours" for our own file - the lock file stayed behind + after every lock with a note. Then the descriptor is asked again, now.""" + if not self._noted: + return self._ident + try: + return self._identity(os.fstat(self._fd)) + except (OSError, TypeError, ValueError): + return self._ident + + def _unlink_own_file(self, ident): + """Remove the lock file, unless the file at the path is no longer the one this lock holds (someone + removed it under us and another process created a new one: that one is theirs). ident: the identity + of our file (`_own_identity`, taken while the descriptor is open).""" + if ident is not None: try: - if self._identity(os.stat(self.lock_file)) != self._ident: + if self._identity(os.stat(self.lock_file)) != ident: return except OSError: return diff --git a/cmeta/utils/sys.py b/cmeta/utils/sys.py index 23f4d69..c652bc1 100644 --- a/cmeta/utils/sys.py +++ b/cmeta/utils/sys.py @@ -160,6 +160,17 @@ def sanitize( full_package_name = f"{cat_name}.{pkg_name}" full_module_name = f"{full_package_name}.{module_name}" + # A module of the engine's own package (category_api_v1.py: the base commands of every category) is + # imported by its real name, once. Loaded by its path under a name made from its folders ("cmeta.cmeta. + # category_api_v1"), its relative imports made a SECOND copy of the engine's modules in the process - + # with a state of their own: the path lock's registries among them, so a soft lock asked for a folder + # was unknown to the copy that wrote the artifacts + engine_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + engine_module = os.path.normcase(module_dir) == os.path.normcase(engine_dir) + if engine_module: + full_package_name = __name__.rsplit('.', 2)[0] + full_module_name = f"{full_package_name}.{module_name}" + timestamp = os.path.getmtime(module_path) if module_path in module_cache: @@ -168,29 +179,33 @@ def sanitize( return {"return": 0, "cache": cached} try: - # Ensure category package exists - if cat_name not in sys.modules: - spec = importlib.machinery.ModuleSpec(cat_name, loader=None, is_package=True) - pkg = importlib.util.module_from_spec(spec) - pkg.__path__ = [category_dir] - sys.modules[cat_name] = pkg - - # Ensure api subpackage exists - if full_package_name not in sys.modules: - spec = importlib.machinery.ModuleSpec(full_package_name, loader=None, is_package=True) - pkg = importlib.util.module_from_spec(spec) - pkg.__path__ = [module_dir] - sys.modules[full_package_name] = pkg - - # Load the plugin module - spec = importlib.util.spec_from_file_location(full_module_name, module_path) - module = importlib.util.module_from_spec(spec) - - module.__package__ = full_package_name - module.__file__ = module_path - - sys.modules[full_module_name] = module - spec.loader.exec_module(module) + if engine_module: + module = importlib.import_module(full_module_name) + + else: + # Ensure category package exists + if cat_name not in sys.modules: + spec = importlib.machinery.ModuleSpec(cat_name, loader=None, is_package=True) + pkg = importlib.util.module_from_spec(spec) + pkg.__path__ = [category_dir] + sys.modules[cat_name] = pkg + + # Ensure api subpackage exists + if full_package_name not in sys.modules: + spec = importlib.machinery.ModuleSpec(full_package_name, loader=None, is_package=True) + pkg = importlib.util.module_from_spec(spec) + pkg.__path__ = [module_dir] + sys.modules[full_package_name] = pkg + + # Load the plugin module + spec = importlib.util.spec_from_file_location(full_module_name, module_path) + module = importlib.util.module_from_spec(spec) + + module.__package__ = full_package_name + module.__file__ = module_path + + sys.modules[full_module_name] = module + spec.loader.exec_module(module) cache_data = { "python_module": module, diff --git a/docs/async-and-concurrency.md b/docs/async-and-concurrency.md index 2675fc8..fefa167 100644 --- a/docs/async-and-concurrency.md +++ b/docs/async-and-concurrency.md @@ -249,13 +249,53 @@ takes `fcntl.flock` (per open file description, so threads exclude each other too), or POSIX record locks (`fcntl.lockf`) where the file system refuses `flock` (NFS and some network mounts), each with the identity check. Where the file system refuses both (some FUSE, 9p and SMB mounts) it falls back to a -**soft lock**: the file is created exclusively and holds `pid host time`; a -stale one (its process dead on this host, or older than 10 minutes when left -by another host) is removed by the waiter that finds it. In every mode the -threads of one process are serialized on the path first. A busy lock is polled -with a delay that doubles from 1 ms to 25 ms until the timeout. A folder used -by two operating systems at once (a Windows tree plugged from WSL) is best -effort: each side locks with its own mechanism. +**soft lock**: the file is created exclusively and holds `pid host time +system`. The holder keeps the file open and rewrites the time in its line +every 2 seconds (the heartbeat: a daemon thread, the line rewritten in place +so the file is never empty). A stale lock file - its pid dead on this system, +or its line unchanged for 20 seconds on the waiter's own monotonic clock (the +holder is on another system or host, or stopped beating) - is taken over by one +of the waiters that find it. Nothing is read from the clock of the drive or of +another machine (a network share, a Windows drive inside WSL2 keep other +clocks): the `system` word (a short hash of the kernel's boot id and the pid +namespace on Linux, of the machine id on Windows and macOS) tells the waiter +whether the pid is one it can check - Windows and WSL2 share the host name and +have different pids, a container shares the kernel of its host and has its own +pids. An empty lock file, which nothing dates, is a leftover once a waiter has +seen it empty for 2 seconds. The waiter does not remove the stale file: it +makes its own file exclusively under a name that belongs to that stale file +(`.take-`, so one waiter only gets to make it), checks +that the file at the path is still the one it judged, and puts its own in its +place with one rename - the path is never free in between. A holder checks +that the file at the path is its own after it took the lock, and removes it at +release only when it still is; a forked child never releases its parent's lock. +A lock file written by cMeta before 0.35.0 (three words, no heartbeat) is +judged as it was then: the pid when the host name is this one, else its age +against 10 minutes. In every mode the threads of one process are serialized on +the path first. A busy lock is polled with a delay that doubles from 1 ms to +25 ms until the timeout. + +**A folder written by several systems at once** - a Windows tree plugged from +WSL2 or from a container through a bind mount, a network share mounted on two +machines - has no lock between those systems with the OS locks: each system's +lock is invisible to the others (measured on one folder of a Windows drive seen +by Windows, WSL2 and a container: no exclusion in any direction, while an +exclusive create is honoured by all). Such a home or repository asks for the +soft lock: `CMETA_SOFT_LOCK_PATHS=[...]` (`*`: +everywhere) names folders for a process, `lock: soft` in a repository's +`_cmr.yaml` names the repository for every system that plugs it (the engine +registers it when the repository is plugged and at the start of every process, +from the index of repositories; `utils.files.use_soft_lock_under` is the API). +With it, the systems exclude each other through the exclusive create, a holder +that dies on one system is replaced from another after the 20 seconds of +silence, and a live holder is never taken over (on Windows its open file cannot +even be replaced). Everything else keeps the OS locks at the same cost. One +thing the lock does not change: every system keeps its own index of the +repository in its own home, and a find answers from that index - what another +system created or changed in the shared repository is seen here after +`cx repo reindex ` (or `cx reindex `); the files +in the repository are always the current ones, and every update reads and +rewrites the artifact's file under the lock, so no update is lost. ### The index lock @@ -296,7 +336,10 @@ updates, and no stale `.lock` files left behind. The lock's own tests (`tests/core_tests/test_utils_files_path_lock.py`) run eight processes and sixteen threads through one counter, replay the removed-file race deterministically, kill a holder, and exercise the soft lock and its stale -files; `tests/benchmarks/benchmark_engine.py` measures the cost. +files - with a drive whose clock is off in both directions, and with six +processes that keep taking over the lock files of "crashed" holders side by +side without ever being two inside the lock; `tests/benchmarks/benchmark_engine.py` +measures the cost. ### What is *not* concurrency-safe diff --git a/docs/using-cmeta.md b/docs/using-cmeta.md index 5f6a0cd..50fe758 100644 --- a/docs/using-cmeta.md +++ b/docs/using-cmeta.md @@ -37,7 +37,11 @@ Environment variables worth knowing: `CMETA_FAIL_ON_ERROR=yes`, `CMETA_INDEX_LOCK_TIMEOUT=` (how long a write to the index waits for a running `cx --reindex`; 600), `CMETA_LOCK_TIMEOUT=` (how long a writer waits for the lock of a file -held by another process; 30, with a notice after 3). +held by another process; 30, with a notice after 3), +`CMETA_SOFT_LOCK_PATHS=[...]` (the folders - a home, +a repository - that several systems write at once through a shared drive, whose +lock files use the soft lock; `*` for everywhere; a repository can ask for it in +its own `_cmr.yaml` with `lock: soft` - see the concurrency guide). --- diff --git a/tests/core_tests/test_utils_files_path_lock.py b/tests/core_tests/test_utils_files_path_lock.py index bb1e5d7..7659786 100644 --- a/tests/core_tests/test_utils_files_path_lock.py +++ b/tests/core_tests/test_utils_files_path_lock.py @@ -74,6 +74,72 @@ print('RELEASED', flush=True) ''' +STALE_RACE_WORKER = r''' +import logging, os, random, subprocess, sys, time +from cmeta.utils import files +logging.disable(logging.CRITICAL) +folder, rounds, crash = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) +lock_file, marker = os.path.join(folder, 'x.lock'), os.path.join(folder, 'in-the-lock') +files._lock_soft_paths.add(lock_file) +files.LOCK_SOFT_CHECK_SECONDS = 0.0 # every failed attempt judges the file: the waiters meet as often as they can +gone = 2147483644 # no process has this pid (a pid that was used could be given to a new process meanwhile) +if files._lock_pid_alive(gone): + child = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL) + child.wait() + gone = child.pid +rnd = random.Random(os.getpid()) +together = 0 +for i in range(rounds): + lock = files.PathLock(lock_file).acquire(timeout=120) + try: + fd = os.open(marker, os.O_WRONLY | os.O_CREAT | os.O_EXCL) + except FileExistsError: + together += 1 # another process is inside the lock + except OSError: + pass # Windows: the marker of the previous holder is still being removed + else: + os.close(fd) + time.sleep(0.0003) + try: + os.unlink(marker) + except OSError: + pass + if rnd.randrange(crash) == 0: + # the crash of a holder: its file stays, with the line of a process that is gone; it beats no more and + # has no file open any more + lock._stop_heartbeat() + with open(lock_file, 'w') as f: + f.write('%d %s %.3f %s\n' % (gone, files._lock_host_name(), time.time() - 5, files._lock_system_id())) + os.close(lock._fd) + lock._fd = None + lock._remove_soft_file = lambda: None + lock.release() +print('OK together', together) +''' + +SOFT_ASKED = r''' +import os, sys +from cmeta.utils import files +asked, other = sys.argv[1], sys.argv[2] +a = files.PathLock(os.path.join(asked, 'a.lock')).acquire(timeout=5) +b = files.PathLock(os.path.join(other, 'b.lock')).acquire(timeout=5) +print(a.mode, b.mode, files._lock_system_id()) +a.release() +b.release() +''' + +SOFT_HOLDER = r''' +import sys, time +from cmeta.utils import files +files._lock_soft_paths.add(sys.argv[1]) +lock = files.PathLock(sys.argv[1]).acquire(timeout=60) +print('LOCKED', lock.mode, flush=True) +sys.stdin.readline() # holds the lock until told to release it (or killed) +lock.release() +print('RELEASED', flush=True) +time.sleep(float(sys.argv[2])) +''' + HOOKED_WAITER = r''' import os, sys, time from cmeta.utils import files @@ -356,8 +422,8 @@ def test_soft_lock_holds_pid_and_host_and_is_removed_on_release(soft, caplog): lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' with open(soft) as f: - pid, host, when = f.read().split() - assert int(pid) == os.getpid() and host == files._lock_host_name() + pid, host, when, system = f.read().split() + assert int(pid) == os.getpid() and host == files._lock_host_name() and system == files._lock_system_id() with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) @@ -387,14 +453,21 @@ def test_a_stale_soft_lock_of_a_dead_process_is_removed(soft, caplog, monkeypatc f.write(f'{gone.pid} {files._lock_host_name()} {time.time():.3f}\n') caplog.set_level('WARNING', logger='cmeta.utils.files') - # Fresh, the file of a dead pid is a release in flight: waited for + # Fresh, the file of a dead pid is a release in flight: waited for - whatever time the drive gave the file + # (an hour old here: a drive whose clock is behind this machine's) + hour_ago = time.time() - 3600 + os.utime(soft, (hour_ago, hour_ago)) with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) - old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS - os.utime(soft, (old, old)) - lock = files.PathLock(soft).acquire(timeout=5) + # Taken long enough ago - by the time its holder wrote into it - it is the leftover of a crash: one try + # takes the lock (the probe of a cache entry is such a try), whatever time the drive gave the file + with open(soft, 'w') as f: + f.write(f'{gone.pid} {files._lock_host_name()} {time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS:.3f}\n') + hour_ahead = time.time() + 3600 + os.utime(soft, (hour_ahead, hour_ahead)) + lock = files.PathLock(soft).acquire(timeout=0) assert lock.mode == 'soft' with open(soft) as f: assert int(f.read().split()[0]) == os.getpid() @@ -415,23 +488,42 @@ def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkey monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) with open(soft, 'w') as f: f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + old = time.time() - 3600 + os.utime(soft, (old, old)) # the drive's time for the file says nothing with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) - old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS - os.utime(soft, (old, old)) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS:.3f}\n') lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) - # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its content at once): - # fresh, it is waited for; old, it is cleared + # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its line at once). A waiter + # dates it by its own watch: seen empty for less than the time, it is waited for - a drive five minutes + # behind this machine changes nothing; seen empty for longer, it is taken over + five_minutes = time.time() - 300 open(soft, 'w').close() + os.utime(soft, (five_minutes, five_minutes)) monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 30.0) with pytest.raises(TimeoutError): - files.PathLock(soft).acquire(timeout=0.3) - monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + files.PathLock(soft).acquire(timeout=0.6) + assert os.path.isfile(soft) + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.4) + started = time.monotonic() lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.4 <= time.monotonic() - started < 3 + lock.release() + assert not os.path.exists(soft) + # a single try has not watched the file: it leaves it ... + open(soft, 'w').close() + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert os.path.isfile(soft) + # ... unless the drive dates it beyond any difference of clocks (the limit of a lock left by another host) + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=0) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) @@ -440,17 +532,291 @@ def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkey def test_a_soft_lock_of_another_host_is_stale_only_after_the_age(soft, monkeypatch): with open(soft, 'w') as f: f.write(f'12345 some-other-host {time.time():.3f}\n') + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) # a drive twenty minutes behind: the lock was taken a moment ago with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) - old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS - os.utime(soft, (old, old)) + # Taken longer ago than the limit, by the time its holder wrote into it + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {old:.3f}\n') + ahead = time.time() + 3600 + os.utime(soft, (ahead, ahead)) # ... and a drive an hour ahead changes nothing lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) + # A holder whose clock is ahead of this machine's (its time lies in our future) is not stale for it + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time() + 3600:.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + os.unlink(soft) + + +def test_a_soft_lock_line_without_a_time_is_dated_by_its_file(soft): + # A lock file that this code did not write (no time in its line, a time that is no number, digits that are + # no number): the time of the file is all there is, and no content makes the lock fail + for content in ('12345 some-other-host\n', '12345 some-other-host nan\n', '²³ x y\n', 'pid 1 reindexes\n'): + with open(soft, 'w', encoding='utf-8') as f: + f.write(content) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def stale_line(seconds_ago=5.0, system=True): + """The line of a soft lock whose holder is gone: a pid that no process has, on this system, taken a while + ago (`system=False`: the line of a holder of cMeta before 0.35.0, without the system).""" + gone = 2147483644 + if files._lock_pid_alive(gone): + child = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + child.wait() + gone = child.pid + tail = f' {files._lock_system_id()}' if system else '' + return f'{gone} {files._lock_host_name()} {time.time() - seconds_ago:.3f}{tail}\n' + + +def test_a_stale_soft_lock_is_replaced_in_one_step_and_never_moved_from_the_path(soft, monkeypatch): + """The waiter replaces the stale file by its own in one rename: the path is never free in between, and no + file is renamed AWAY from the path (the previous way - away, checked, put back when it was not the judged + one - left the path free for a third process meanwhile).""" + moves = [] + real_rename, real_replace = os.rename, os.replace + + def rename(src, dst, *args, **kwargs): + moves.append((src, dst)) + return real_rename(src, dst, *args, **kwargs) + + def replace(src, dst, *args, **kwargs): + moves.append((src, dst)) + return real_replace(src, dst, *args, **kwargs) + monkeypatch.setattr(os, 'rename', rename) + monkeypatch.setattr(os, 'replace', replace) + + with open(soft, 'w') as f: + f.write(stale_line()) + lock = files.PathLock(soft).acquire(timeout=0) + assert lock.mode == 'soft' + assert len(moves) == 1 and moves[0][0].startswith(soft + '.take-') and moves[0][1] == soft + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_a_waiter_a_step_behind_leaves_the_file_of_the_one_that_took_the_stale_lock(soft, monkeypatch): + """Two waiters judge the same stale file and one takes the lock over. The other, a step behind, finds that + the file at the path is not the one it judged, and leaves it where it is.""" + with open(soft, 'w') as f: + f.write(stale_line()) + # the line of the other waiter: a live process (this one), taken half a second ago - never the very line the + # waiter under test writes in the same millisecond (a line tells its writer by pid, host and time) + fresh = f'{os.getpid()} {files._lock_host_name()} {time.time() - 0.5:.3f}\n' + real = files.PathLock._soft_file_is_stale + seen = {} + + def judged_then_overtaken(self, path, read): + stale = real(self, path, read) + if stale and path == soft and not seen: + with open(soft + '.other', 'w') as f: # the other waiter's take-over, right after the judgement + f.write(fresh) + os.replace(soft + '.other', soft) + seen['ino'] = os.stat(soft).st_ino + return stale + monkeypatch.setattr(files.PathLock, '_soft_file_is_stale', judged_then_overtaken) + + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert seen and os.stat(soft).st_ino == seen['ino'] + with open(soft) as f: + assert f.read() == fresh + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + os.unlink(soft) + + +def test_one_waiter_at_a_time_takes_a_stale_soft_lock_over(soft): + """The file that replaces a stale lock file is made exclusively, under a name of that stale file: a second + waiter finds it and waits. The take file of a waiter that died meanwhile is a leftover, and goes.""" + stale = stale_line() + with open(soft, 'w') as f: + f.write(stale) + probe = files.PathLock(soft) + take = probe._take_file(probe._read_lock_file(soft)) + assert take.startswith(soft + '.take-') + with open(take, 'w') as f: # another waiter (alive: this process) is taking it over right now + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.6) + with open(soft) as f: + assert f.read() == stale + assert os.path.isfile(take) + + with open(take, 'w') as f: # that waiter died before it replaced the file + f.write(stale_line()) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_a_soft_lock_taken_over_is_not_removed_by_its_former_holder(soft): + lock = files.PathLock(soft).acquire(timeout=5) + theirs = f'12345 some-other-host {time.time():.3f}\n' + with open(soft + '.new', 'w') as f: + f.write(theirs) + if POSIX: + os.replace(soft + '.new', soft) # this holder was judged gone (it was not) and its lock taken over + lock.release() + with open(soft) as f: + assert f.read() == theirs + os.unlink(soft) + else: + # Windows: a holder keeps its lock file open, and an open file cannot be replaced - the lock of a holder + # that is alive cannot be taken over at all + with pytest.raises(PermissionError): + os.replace(soft + '.new', soft) + os.unlink(soft + '.new') + lock.release() + assert not os.path.exists(soft) + + +def test_a_live_soft_holder_judged_gone_by_mistake(tmp_path, monkeypatch): + """A waiter that takes a live holder for a dead one (it cannot see its pid: another system on a shared + folder). On Windows it cannot take the lock over - the holder has its file open - and nothing is left behind; + elsewhere it can, and the former holder then leaves the lock of the other alone when it releases.""" + lock_file = str(tmp_path / 'live.lock') + holder = subprocess.Popen([sys.executable, '-c', SOFT_HOLDER, lock_file, '60'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + line = holder.stdout.readline().decode('utf-8', 'replace') + assert line.startswith('LOCKED soft'), line + try: + monkeypatch.setattr(files, '_lock_soft_paths', {lock_file}) + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) + monkeypatch.setattr(files, 'LOCK_SOFT_DEAD_MIN_AGE_SECONDS', 0.0) + if POSIX: + lock = files.PathLock(lock_file).acquire(timeout=5) + with open(lock_file) as f: + assert int(f.read().split()[0]) == os.getpid() + holder.stdin.write(b'release\n') + holder.stdin.flush() + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('RELEASED') + with open(lock_file) as f: # the former holder left it alone + assert int(f.read().split()[0]) == os.getpid() + lock.release() + else: + with pytest.raises(TimeoutError): + files.PathLock(lock_file).acquire(timeout=1) + with open(lock_file) as f: + assert int(f.read().split()[0]) == holder.pid + assert os.listdir(str(tmp_path)) == [os.path.basename(lock_file)] + holder.stdin.write(b'release\n') + holder.stdin.flush() + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('RELEASED') + assert os.listdir(str(tmp_path)) == [] + finally: + holder.kill() + holder.wait(30) + + +@pytest.mark.skipif(not POSIX, reason='on Windows a file cannot be replaced while its maker has it open') +def test_a_soft_lock_file_taken_over_while_it_is_made_is_not_held(soft, monkeypatch): + """A process stopped between creating its soft lock file and writing into it finds, when it goes on, that a + waiter has taken the empty file over: it does not hold the lock, and leaves the file of the other.""" + theirs = f'12345 some-other-host {time.time():.3f}\n' + real_write = os.write + state = {'n': 0} + + def write_after_a_take_over(fd, data): + state['n'] += 1 + if state['n'] == 1: + with open(soft + '.new', 'w') as f: + f.write(theirs) + os.replace(soft + '.new', soft) + return real_write(fd, data) + monkeypatch.setattr(os, 'write', write_after_a_take_over) + + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + monkeypatch.setattr(os, 'write', real_write) + with open(soft) as f: + assert f.read() == theirs + os.unlink(soft) + + +def test_a_soft_lock_file_that_the_file_system_shows_late_is_still_this_lock(soft, monkeypatch): + """A file system that does not show a new file at once: the holder cannot confirm its lock file in its first + attempt. The file with its line, found at the path in the next attempt, is its lock - and when the acquire + gives up before that (a single try), the file goes, or it would keep everybody out while the process lives.""" + real = files.PathLock._open_if_mine + state = {'n': 0} + + def not_there_at_first(self, line): + state['n'] += 1 + return ('gone', None) if state['n'] <= 3 else real(self, line) # the three looks of the first attempt + monkeypatch.setattr(files.PathLock, '_open_if_mine', not_there_at_first) + monkeypatch.setattr(files.PathLock, '_identity', staticmethod(lambda st: None)) # ... and it reports no inode numbers + + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and state['n'] > 3 + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + lock.release() + assert not os.path.exists(soft) + + state['n'] = 0 + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert state['n'] == 3 and not os.path.exists(soft) + lock = files.PathLock(soft).acquire(timeout=0) # ... and the lock is free for the next one + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_waiters_on_the_soft_lock_of_crashed_holders_never_hold_it_together(tmp_path): + """Six processes take one soft lock sixty times each, and one holder in four "crashes" (its lock file stays, + with the line of a process that is gone): the waiters judge and take over stale files side by side, and + never two of them are inside the lock at once. No take file is left.""" + folder = tmp_path / 'race' + folder.mkdir() + outputs = run_workers(STALE_RACE_WORKER, [(str(folder), 60, 4)] * 6) + for rc, out in outputs: + assert rc == 0 and out.strip().endswith('OK together 0'), out + assert os.listdir(str(folder)) in ([], ['x.lock']) # the last crash may have left its lock file + + +def test_the_lock_file_goes_where_a_file_gets_its_identity_at_the_first_write(tmp_path, monkeypatch): + """exFAT on macOS: an empty file reports a placeholder inode that becomes a real one at the first write. The + lock noted the identity of its (empty) file when it took it, the note changed it, and at release "the file + at the path is not ours" left the lock file behind after every noted lock.""" + real = files.PathLock._identity + + def placeholder_until_written(st): + ident = real(st) + if ident is None: + return None + return (2 ** 64 - 1, ident[1]) if st.st_size == 0 else ident + monkeypatch.setattr(files.PathLock, '_identity', staticmethod(placeholder_until_written)) + + lock_file = str(tmp_path / 'noted.lock') + lock = files.PathLock(lock_file).acquire(timeout=5, note='pid 1 reindexes') + assert lock.mode != 'soft' and files.PathLock.read_note(lock_file) == 'pid 1 reindexes' + lock.release() + assert not os.path.exists(lock_file) + + lock = files.PathLock(lock_file).acquire(timeout=5) # without a note the file stays empty + lock.release() + assert not os.path.exists(lock_file) + def test_soft_lock_is_used_where_the_os_lock_is_refused(tmp_path, monkeypatch, caplog): lock_file = str(tmp_path / 'refused.lock') @@ -468,16 +834,21 @@ def refuse(*args, **kwargs): monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) caplog.set_level('WARNING', logger='cmeta.utils.files') + started = time.monotonic() lock = files.PathLock(lock_file).acquire(timeout=5) assert lock.mode == 'soft' + # the empty file the refused attempt left is taken over at once, not watched like an empty file of unknown origin + assert time.monotonic() - started < files.LOCK_SOFT_EMPTY_STALE_SECONDS / 2 assert lock_file in files._lock_soft_paths assert any('soft lock' in rec.getMessage() for rec in caplog.records) + assert os.listdir(str(tmp_path)) == [os.path.basename(lock_file)] lock.release() assert not os.path.exists(lock_file) again = files.PathLock(lock_file).acquire(timeout=5) # remembered: soft at once assert again.mode == 'soft' again.release() + assert os.listdir(str(tmp_path)) == [] @pytest.mark.skipif(not POSIX, reason='POSIX record locks are the POSIX fallback') @@ -496,6 +867,186 @@ def refuse(*args, **kwargs): assert not os.path.exists(lock_file) +########################################################################################## +# The heartbeat of a soft holder, the system named in its line, and the soft lock asked for + +def test_a_soft_lock_line_names_the_system_and_the_heartbeat_rewrites_its_time(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_HEARTBEAT_SECONDS', 0.05) + lock = files.PathLock(soft).acquire(timeout=5) + with open(soft) as f: + first = f.read() + words = first.split() + assert len(words) == 4 and int(words[0]) == os.getpid() and words[1] == files._lock_host_name() + assert words[3] == files._lock_system_id() and len(words[3]) == 12 + assert id(lock) in files._beat_held and files._beat_thread.is_alive() + + deadline = time.monotonic() + 5 + data = first + while data == first and time.monotonic() < deadline: + time.sleep(0.02) + with open(soft) as f: + data = f.read() + beat = data.split() + assert float(beat[2]) > float(words[2]) and beat[0] == words[0] and beat[1] == words[1] and beat[3] == words[3] + assert len(data) == len(first) and data.endswith('\n') # rewritten in place: the same length, never empty + assert lock._line != first.encode() # the holder knows the line it wrote last + + # the processes of one system agree on its name + out = subprocess.run([sys.executable, '-c', 'from cmeta.utils import files; print(files._lock_system_id())'], + stdin=subprocess.DEVNULL, capture_output=True) + assert out.stdout.decode('utf-8', 'replace').split() == [files._lock_system_id()] + + lock.release() + assert id(lock) not in files._beat_held # the one thread of the process beats on for the other soft locks + assert not os.path.exists(soft) + + +def test_a_silent_holder_of_another_system_is_taken_over_after_the_silence_and_a_beating_one_never(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 0.6) + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) # that pid is dead HERE - but it is a pid of another system + foreign = 'ffffffffffff' + with open(soft, 'w') as f: # taken a moment ago, by its own time: no age rule can take it + f.write(f'12345 some-other-host {time.time():.3f} {foreign}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + started = time.monotonic() + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.6 <= time.monotonic() - started < 4 + lock.release() + assert not os.path.exists(soft) + + # A holder that beats is never taken over - whatever its clock reads (an hour behind here: the time in its + # line is not compared with this machine's clock) + stop = threading.Event() + + def beat(): + n = 0 + while not stop.wait(0.1): + n += 1 + with open(soft, 'r+') as f: + f.write(f'12345 some-other-host {time.time() - 3600 + n / 1000:.3f} {foreign}\n') + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time() - 3600:.3f} {foreign}\n') + beater = threading.Thread(target=beat, daemon=True) + beater.start() + try: + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=1.5) + finally: + stop.set() + beater.join(5) + os.unlink(soft) + + +def test_a_dead_holder_of_this_system_is_known_by_its_pid_before_the_silence(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 3600.0) + with open(soft, 'w') as f: + f.write(stale_line()) # four words: a pid of this system that is gone, taken 5 s ago + lock = files.PathLock(soft).acquire(timeout=0) + assert lock.mode == 'soft' + lock.release() + + # ... while a pid of this system that is alive keeps its lock, silent or not + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 5:.3f} {files._lock_system_id()}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + + # ... and the same pid of ANOTHER system is not checked here: that holder is judged by its silence alone + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 0.4) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 5:.3f} ffffffffffff\n') + started = time.monotonic() + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.4 <= time.monotonic() - started < 4 + lock.release() + assert not os.path.exists(soft) + + +@pytest.mark.skipif(not hasattr(os, 'fork'), reason='fork is a POSIX matter') +def test_a_forked_child_never_releases_the_lock_of_its_parent(tmp_path, monkeypatch): + import warnings + soft_file, own = str(tmp_path / 'soft.lock'), str(tmp_path / 'own.lock') + monkeypatch.setattr(files, '_lock_soft_paths', {soft_file, own}) + for lock_file in (str(tmp_path / 'os.lock'), soft_file): + lock = files.PathLock(lock_file).acquire(timeout=5) + with warnings.catch_warnings(): + warnings.simplefilter('ignore', DeprecationWarning) # the heartbeat thread of the soft holder is running + pid = os.fork() + if pid == 0: # the child: a copy of the holder that is not the holder + try: + lock.release() + code = 0 if os.path.exists(lock_file) and not lock.is_locked else 1 + # ... and a process of its own: the parent's lock is held by another process for it, and it + # takes and beats soft locks of its own (the module's thread locks were made anew for it) + try: + files.PathLock(lock_file).acquire(timeout=0.3) + code = 3 + except TimeoutError as e: + if 'another process' not in str(e): + code = 4 + mine = files.PathLock(own).acquire(timeout=5) + if not mine.is_locked or id(mine) not in files._beat_held or files._beat_thread is None: + code = 5 + mine.release() + if os.path.exists(own): + code = 6 + except BaseException: + code = 2 + os._exit(code) + _, status = os.waitpid(pid, 0) + assert os.WEXITSTATUS(status) == 0, lock_file + assert lock.is_locked and os.path.exists(lock_file) + with pytest.raises(TimeoutError): # still held + files.PathLock(lock_file).acquire(timeout=0.2) + lock.release() + assert not os.path.exists(lock_file) + + +def test_the_soft_lock_is_used_under_the_folders_the_environment_names(tmp_path): + asked, other = tmp_path / 'shared', tmp_path / 'own' + asked.mkdir() + other.mkdir() + + def modes(value): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + env.pop('CMETA_SOFT_LOCK_PATHS', None) + if value is not None: + env['CMETA_SOFT_LOCK_PATHS'] = value + out = subprocess.run([sys.executable, '-c', SOFT_ASKED, str(asked), str(other)], env=env, + stdin=subprocess.DEVNULL, capture_output=True) + assert out.returncode == 0, out.stderr.decode('utf-8', 'replace') + a, b, system = out.stdout.decode('utf-8', 'replace').split() + assert system == files._lock_system_id() + return a, b + + os_mode = 'windows' if not POSIX else 'flock' + assert modes(None) == (os_mode, os_mode) + assert modes(str(asked)) == ('soft', os_mode) + assert modes(os.pathsep.join([str(other), str(asked)])) == ('soft', 'soft') + assert modes('*') == ('soft', 'soft') + assert sorted(os.listdir(str(asked))) == [] and sorted(os.listdir(str(other))) == [] + + +def test_use_soft_lock_under_a_folder_and_not_next_to_it(tmp_path, monkeypatch): + monkeypatch.setattr(files, '_lock_soft_dirs', []) + folder = tmp_path / 'repo' + (folder / 'sub').mkdir(parents=True) + (tmp_path / 'repo2').mkdir() + assert files.use_soft_lock_under(str(folder)) == {'return': 0} + files.use_soft_lock_under(str(folder)) # once is enough + assert len(files._lock_soft_dirs) == 1 + + inside = files.PathLock(str(folder / 'sub' / 'x.lock')).acquire(timeout=5) + beside = files.PathLock(str(tmp_path / 'repo2' / 'x.lock')).acquire(timeout=5) # the same letters, another folder + outside = files.PathLock(str(tmp_path / 'x.lock')).acquire(timeout=5) + assert inside.mode == 'soft' and beside.mode != 'soft' and outside.mode != 'soft' + for lock in (inside, beside, outside): + lock.release() + assert not any(name.endswith('.lock') for root, dirs, names in os.walk(str(tmp_path)) for name in names) + + ########################################################################################## # The engine's functions diff --git a/tests/core_tests/test_utils_files_writes.py b/tests/core_tests/test_utils_files_writes.py index d6ed062..8d35e0c 100644 --- a/tests/core_tests/test_utils_files_writes.py +++ b/tests/core_tests/test_utils_files_writes.py @@ -148,7 +148,10 @@ def test_a_symbolic_link_is_written_through(tmp_path): real = str(tmp_path / 'real.json') link = str(tmp_path / 'link.json') assert files.safe_write_file(real, {'a': 1})['return'] == 0 - os.symlink(real, link) + try: + os.symlink(real, link) + except OSError as e: + pytest.skip(f'the file system has no symbolic links ({e.__class__.__name__}: FAT, exFAT)') assert files.safe_write_file(link, {'a': 2})['return'] == 0 assert os.path.islink(link) assert json.load(open(real)) == {'a': 2} diff --git a/tests/internal_repo_tests/test_integration_repo_soft_lock.py b/tests/internal_repo_tests/test_integration_repo_soft_lock.py new file mode 100644 index 0000000..9e36f64 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_soft_lock.py @@ -0,0 +1,130 @@ +""" +Integration tests of the soft lock a repository asks for: a `_cmr.yaml` with `lock: soft` - a repository that +several systems write at once through a shared drive (a Windows tree plugged from WSL2 or from a container, a +network share), where the OS locks of the systems do not see each other - makes every lock file under that +repository use the soft lock: in the process that plugs it, and in every process that starts after it (the +engine registers it from the index of repositories at its start). Other repositories and the home keep the OS lock. + +Each test runs against a fresh in tmp_path so it does not touch the user's real cMeta state. + +Licensed under the Apache License, Version 2.0. +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import logging +import os +import subprocess +import sys + +import pytest + +from cmeta import CMeta +from cmeta.utils import files + +POSIX = sys.platform != 'win32' +OS_MODE = 'flock' if POSIX else 'windows' + +LATER_PROCESS = r''' +import os, sys +from cmeta import CMeta +from cmeta.utils import files +home, shared, own = sys.argv[1:4] +cm = CMeta(home=home) +r = cm.access({'category': 'repo', 'command': 'find', 'arg1': 'shared', 'con': False}) # any first access of the engine +assert r['return'] == 0, r +a = files.PathLock(os.path.join(shared, 'z.lock')).acquire(timeout=5) +b = files.PathLock(os.path.join(own, 'z.lock')).acquire(timeout=5) +c = files.PathLock(os.path.join(home, 'z.lock')).acquire(timeout=5) +print(a.mode, b.mode, c.mode) +a.release() +b.release() +c.release() +''' + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR', 'CMETA_SOFT_LOCK_PATHS'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + monkeypatch.setattr(files, '_lock_soft_dirs', []) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid, lock=None): + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + text = f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n' + if lock is not None: + text += f'lock: {lock}\n' + (repo / '_cmr.yaml').write_text(text, encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def mode_under(folder): + lock = files.PathLock(os.path.join(str(folder), 'probe.lock')).acquire(timeout=5) + mode = lock.mode + lock.release() + assert not os.path.exists(os.path.join(str(folder), 'probe.lock')) + return mode + + +def test_a_repository_that_asks_for_the_soft_lock_gets_it_and_the_others_keep_the_os_lock(cm, tmp_path): + shared = make_repo(tmp_path, 'shared', 'a1b2c3d4e5f60718', lock='soft') + own = make_repo(tmp_path, 'own', 'b1b2c3d4e5f60718') + assert mode_under(shared) == OS_MODE # not plugged yet: nothing asked + plug(cm, shared) + plug(cm, own) + + assert mode_under(shared) == 'soft' # the process that plugged it + (shared / 'sub').mkdir() + assert mode_under(shared / 'sub') == 'soft' + assert mode_under(own) == OS_MODE + assert mode_under(tmp_path) == OS_MODE # the home keeps the OS lock + + # a process that starts later knows it from the index of repositories, with its first access of the engine + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + out = subprocess.run([sys.executable, '-c', LATER_PROCESS, str(tmp_path), str(shared), str(own)], env=env, + stdin=subprocess.DEVNULL, capture_output=True) + assert out.returncode == 0, out.stderr.decode('utf-8', 'replace') + assert out.stdout.decode('utf-8', 'replace').split() == ['soft', OS_MODE, OS_MODE] + + +def test_the_writes_of_the_engine_in_such_a_repository_take_the_soft_lock(cm, tmp_path, caplog): + shared = make_repo(tmp_path, 'shared', 'a1b2c3d4e5f60718', lock='soft') + own = make_repo(tmp_path, 'own', 'b1b2c3d4e5f60718') + plug(cm, shared) + plug(cm, own) + caplog.set_level(logging.DEBUG) + + for repo, mode in ((shared, 'soft'), (own, OS_MODE)): + caplog.clear() + r = access(cm, category='log', command='create', arg1=f'{repo.name}:note-{repo.name}', meta={'x': 1}, yaml=True) + assert r['return'] == 0, r.get('error') + r = access(cm, category='log', command='update', arg1=f'{repo.name}:note-{repo.name}', meta={'y': 2}) + assert r['return'] == 0, r.get('error') + taken = [rec.getMessage() for rec in caplog.records if 'PathLock - acquired' in rec.getMessage()] + inside = [m for m in taken if os.path.normcase(str(repo)) in os.path.normcase(m)] + assert inside, taken + assert all(m.endswith(f'({mode})') for m in inside), inside + + assert not any(name.endswith('.lock') for root, dirs, names in os.walk(str(tmp_path)) for name in names) + + +def test_a_repository_that_asks_for_nothing_else_is_untouched(cm, tmp_path): + other = make_repo(tmp_path, 'other', 'c1b2c3d4e5f60718', lock='os') # any other value: the default + plug(cm, other) + assert mode_under(other) == OS_MODE + assert files._lock_soft_dirs == [] From b9ae980c88cba053913df8291380a960d6deced3 Mon Sep 17 00:00:00 2001 From: Grigori Fursin Date: Sun, 11 Oct 2026 21:43:57 +0200 Subject: [PATCH 6/6] cMeta v0.35.0 The version in every file that carries it (docs/releasing.md, step 2); release date 2026-10-11. Co-Authored-By: Claude Fable 5.1 Signed-off-by: Grigori Fursin --- CHANGELOG.md | 2 +- CITATION.cff | 4 ++-- cmeta/internal-repo/_cmr.yaml | 4 ++-- cmeta/version.py | 4 ++-- cmeta/versions.json | 4 ++++ docs/en/api/index.rst | 4 ++-- docs/en/index.rst | 4 ++-- docs/history.md | 2 +- docs/installation.md | 4 ++-- 9 files changed, 18 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 858251b..6afd3ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,7 +3,7 @@ All notable changes to cMeta are documented here, newest first. -## 0.34.2.1 (in development) +## 0.35.0 - **A registered repository that cannot be reached stays registered.** When the index was built - a full `cx --reindex`, or the incremental index of `pull`, `plug`, `unplug` and `cx repo reindex` - and the folder of a registered repository was not there (a detached USB drive, a network share that was down) or its diff --git a/CITATION.cff b/CITATION.cff index f68e094..1b2c382 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -13,8 +13,8 @@ authors: website: "https://cTuning.ai/@gfursin" affiliation: "cTuning Labs" orcid: "https://orcid.org/0000-0001-7719-1624" -version: "0.34.2" -date-released: "2026-10-10" +version: "0.35.0" +date-released: "2026-10-11" license: Apache-2.0 repository-code: "https://github.com/cTuningLabs/cmeta" url: "https://cTuning.ai" diff --git a/cmeta/internal-repo/_cmr.yaml b/cmeta/internal-repo/_cmr.yaml index b3f24d0..0d7e58f 100644 --- a/cmeta/internal-repo/_cmr.yaml +++ b/cmeta/internal-repo/_cmr.yaml @@ -3,5 +3,5 @@ authors: "[Grigori Fursin](https://cTuning.ai/@gfursin)" category: repo,f4f792ab40c7498f copyright: "2025-2026 Grigori Fursin and cTuning Labs. See the cMeta COPYRIGHT and LICENSE files for details." permanent: True -version: '0.34.2' -version_date: '20261010' +version: '0.35.0' +version_date: '20261011' diff --git a/cmeta/version.py b/cmeta/version.py index a15c2c5..a92c65d 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.34.2.1" -__release_date__ = "2026-10-10" +__version__ = "0.35.0" +__release_date__ = "2026-10-11" diff --git a/cmeta/versions.json b/cmeta/versions.json index 40263a9..669d0a3 100644 --- a/cmeta/versions.json +++ b/cmeta/versions.json @@ -101,6 +101,10 @@ { "version": "0.34.2", "version_date": "20261010" + }, + { + "version": "0.35.0", + "version_date": "20261011" } ] } diff --git a/docs/en/api/index.rst b/docs/en/api/index.rst index 7bff7a2..f8424fb 100644 --- a/docs/en/api/index.rst +++ b/docs/en/api/index.rst @@ -1,7 +1,7 @@ -API Reference (v0.34.2) +API Reference (v0.35.0) ======================= -This section contains the complete API reference for cMeta v0.34.2. +This section contains the complete API reference for cMeta v0.35.0. .. toctree:: :maxdepth: 2 diff --git a/docs/en/index.rst b/docs/en/index.rst index 7f3af71..42fe80e 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -1,11 +1,11 @@ cMeta Documentation =================== -Welcome to cMeta v0.34.2 documentation! +Welcome to cMeta v0.35.0 documentation! .. toctree:: :maxdepth: 2 - :caption: cMeta v0.34.2 Contents: + :caption: cMeta v0.35.0 Contents: home diff --git a/docs/history.md b/docs/history.md index 07d4a8b..355c9e6 100644 --- a/docs/history.md +++ b/docs/history.md @@ -82,7 +82,7 @@ produces APA and BibTeX automatically. The BibTeX entry: author = {Fursin, Grigori}, title = {{cMeta (Common Meta Framework)}}, year = {2026}, - version = {0.34.2}, + version = {0.35.0}, license = {Apache-2.0}, url = {https://github.com/cTuningLabs/cmeta}, note = {cTuning Labs} diff --git a/docs/installation.md b/docs/installation.md index e030f47..596ca7c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -157,8 +157,8 @@ repositories) into these commands. **A specific version**, to pin or to go back: ```bash -uv tool install --force "cmeta[server]==0.34.2" -pip install "cmeta==0.34.2" +uv tool install --force "cmeta[server]==0.35.0" +pip install "cmeta==0.35.0" ``` **Moving an install from git to PyPI** (or back) is a reinstall from the other