diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index f67ccfa..6452a84 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -29,7 +29,7 @@ a small fix. - [ ] No local paths, credentials, API keys or personal scratch files included. diff --git a/.github/workflows/test-core.yml b/.github/workflows/test-core.yml index 34d0d8e..342c35a 100644 --- a/.github/workflows/test-core.yml +++ b/.github/workflows/test-core.yml @@ -28,7 +28,7 @@ jobs: strategy: matrix: os: [ubuntu-latest, windows-latest, macos-latest] - python-version: ["3.9", "3.14"] + python-version: ["3.9", "3.14", "3.15"] runs-on: ${{ matrix.os }} diff --git a/AGENTS.md b/AGENTS.md index 8d087c5..c6eb434 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,7 @@ uniform interface: - Python: `cm.access({'category': ..., 'command': ..., ...})` - CLI: `cx [args] [--flags]` -**License:** Apache-2.0. **Python:** 3.9–3.14. +**License:** Apache-2.0. **Python:** 3.9–3.15. This repo ships the framework and a built-in content repository (`cmeta/internal-repo/`) with foundational categories. All other artifacts live @@ -244,7 +244,7 @@ Notable shipped artifacts: - **Cross-category calls.** Inside `api/v1.py`, use `self.cm.access(...)`. Refer to other categories via `self.cmeta['uses_categories']['']` declared in your `_cmeta.yaml`, not by hard-coded alias. -- Python 3.9–3.14 supported; avoid newer-only syntax. +- Python 3.9–3.15 supported; avoid newer-only syntax. - Tests: `tests/`, files `test_*.py`, pytest config in `pyproject.toml`. - Public Apache-2.0 framework — keep product vision/strategy prose out; public docs describe *functionality* (`README.md`, `docs/installation.md`, diff --git a/CHANGELOG.md b/CHANGELOG.md index 4fdc7f9..6afd3ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,89 @@ All notable changes to cMeta are documented here, newest first. +## 0.35.0 +- **A registered repository that cannot be reached stays registered.** When the index was built - a full + `cx --reindex`, or the incremental index of `pull`, `plug`, `unplug` and `cx repo reindex` - and the folder + of a registered repository was not there (a detached USB drive, a network share that was down) or its + `_cmr.yaml` could not be read, the repository was dropped from `repos.json` without a word and had to be + plugged again; one transient miss was enough. Now its line stays in `repos.json`, in its place, marked + `unreachable: true`; the repository is left out of the index (in an incremental run its records are + removed, so that a find does not hand out artifacts whose folder is away); a notice names it and the two + commands (on stderr with the console, always in the log). When the folder is there again, the next index + run of any kind indexes it as if it had just been plugged and clears the mark - also `cx repo reindex + `, which now accepts the folder of a registered repository that is not in the index. `cx repo unplug + ` forgets a repository by the path of its folder, reachable or not (before, `unplug` knew aliases + only). Fixed on the way: the stored descriptor of a repository plugged with `--keep` was left out of + `repos.json` whenever the file was rewritten while the repository was reachable, so the next time its + drive was away it was dropped like any other; a `_cmr.yaml` without an `artifact` name ended the index with + a `KeyError` (now: unreachable, with the reason); the pause "Fix it or press Enter to continue!" of an + ambiguous category alias ended the index with `EOFError` when nobody could answer (now: it goes on). + Tests: `tests/internal_repo_tests/test_integration_repo_unreachable.py`. +- **The soft lock: no clock of the drive, a heartbeat, the system in its line, a stale one taken over by + one waiter in one step, and a way to ask for it on a shared folder.** The soft lock is the fallback of + `PathLock` where a file system refuses OS locks (some FUSE, 9p and SMB mounts); nothing changes where OS + locks work. (1) How long ago a soft lock was taken is now counted from the time its holder wrote into the + file; before, it was the time the file system gave the file, compared with this machine's clock - on a + drive that keeps another clock (a Windows drive inside WSL2 is seconds off, a network share can be more) + a lock file made a moment ago looked old, so the grace of a holder that has just released and exited did + not hold there. An empty lock file, which nothing dates, is a leftover once a waiter has seen it empty for + 2 seconds (or when the drive dates it more than 10 minutes back). (2) **The holder beats.** It keeps its + lock file open and rewrites the time in its line every 2 seconds (a daemon thread; the line is rewritten + in place, so the file is never empty or shorter); a waiter that has seen the same line for 20 seconds on + its own monotonic clock knows the holder is gone - without comparing two machines' clocks, however long + the lock has been held (the old rule took a lock of another host for abandoned after 10 minutes, a build + of an hour included), and without a pid it cannot check (so a holder that crashed on ANOTHER system is + seen as running - by `cx cache classify` too - for those 20 seconds; one of this system is known at once by + its pid). (3) **The line names the system** (`pid host + time system`: a short hash of the kernel's boot id and the pid namespace on Linux, of the machine id on + Windows and macOS), so a waiter checks the pid only when it is a pid of its own system: Windows and WSL2 + share the host name and have different pids, a container shares the kernel of its host and has its own + pids; a dead pid of the same system is still known at once, a live one keeps its lock. A lock file of + cMeta before 0.35.0 (three words, no heartbeat) is judged as before. (4) A stale lock file was renamed + away, checked and - when it was not the judged one - put back; between the two a third process could + take the free path, which under load could end with two holders, a leftover `.stale-*`, or a + lock file carrying the line of a live process that did not hold it any more - a lock nobody could take + for as long as that process lived. Now the waiter makes the file that replaces the stale one exclusively, + under a name of that stale file (`.take-`: one waiter only), checks that the file at + the path is still the one it judged, and puts its own in place with one rename - the path is never free + in between, and a waiter that comes a step later finds another file there and leaves it. The file is + closed before the rename on every platform (Windows refuses to rename a file the process has open; a + Windows drive seen through 9p - WSL2's `/mnt/d`, a container's bind mount - shows nothing at the new path + while the descriptor is open) and opened again at its new path to be held. (5) A single try + (`timeout 0`: the probe behind `cx cache classify`) takes the lock of a dead process in that try; before, + it cleared the file and still reported the lock held once. (6) A holder checks, after it took the lock, + that the file at the path is its own (the open descriptor, or its line), and removes the file at release + only when it still is; on Windows a held lock file cannot be replaced at all, so a live holder there is + never taken over by mistake. (7) The empty file that a refused OS-lock attempt leaves is taken over the + same way instead of being removed outright. (8) A forked child of a holder never releases its parent's + lock, and starts with thread locks of its own (the ones it inherits may have been held by a thread of the + parent that does not exist in the child). (9) **Asking for it.** A home or repository that several systems write at once through a shared + drive - a Windows tree plugged from WSL2 or from a container, a network share - has no lock between those + systems with the OS locks (each system's lock is invisible to the others; the exclusive create of the + soft lock file is honoured by all). `CMETA_SOFT_LOCK_PATHS=[...]` (`*` for + everywhere) makes every lock file under those folders use the soft lock in that process, and `lock: soft` + in a repository's `_cmr.yaml` does it for that repository on every system that plugs it (registered when + it is plugged and at the start of every process, from the index of repositories; the API is + `utils.files.use_soft_lock_under`). Everything else keeps the OS locks: a lock cycle costs what it did. + (Each system still keeps its own index of such a repository in its own home: what another system created or + changed there is seen after `cx repo reindex `; the files are always current and no update is lost.) +- **The engine's own category API is imported once.** `category_api_v1.py` - the base commands of every + category - was loaded by its path under a name made from its folders (`cmeta.cmeta.category_api_v1`), + and its relative imports made a second copy of the engine's modules in the process (`cmeta.cmeta.utils. + files` next to `cmeta.utils.files`), each with a state of its own: the registries of the path lock among + them, so the soft lock asked for a folder was unknown to the copy that wrote the artifacts, and the + threads of one process were serialized on a path by two registries instead of one (the OS locks kept + them apart all the same). The loader now imports a module of the engine's own package by its real name + (`cmeta.category_api_v1`); category APIs of repositories are loaded as before. +- **A lock with a note leaves no lock file behind on exFAT under macOS.** There an empty file reports a + placeholder inode that becomes a real one at its first write: the lock noted the identity of its (empty) + file when it took it, the note of the holder (the index lock of a reindex, a cache entry being built) + changed it, and at release "the file at the path is not ours" left `.lock` behind. The identity is + now asked from the open file at release. + Tests of both: `tests/core_tests/test_utils_files_path_lock.py`. +- **Python 3.15.** The suite runs on Python 3.15 in CI, next to 3.9 and 3.14 (Linux, Windows, macOS); nothing + in the engine needed a change. `pyproject.toml` carries the classifier. + ## 0.34.2 - **The cache category knows the state of an entry, and a delete holds the folder's lock across its two steps.** `cx cache classify` (the API `classify`) tells for every entry whether it is `ok`, `running` (the diff --git a/CITATION.cff b/CITATION.cff index f68e094..1b2c382 100644 --- a/CITATION.cff +++ b/CITATION.cff @@ -13,8 +13,8 @@ authors: website: "https://cTuning.ai/@gfursin" affiliation: "cTuning Labs" orcid: "https://orcid.org/0000-0001-7719-1624" -version: "0.34.2" -date-released: "2026-10-10" +version: "0.35.0" +date-released: "2026-10-11" license: Apache-2.0 repository-code: "https://github.com/cTuningLabs/cmeta" url: "https://cTuning.ai" diff --git a/CLAUDE.md b/CLAUDE.md index 3d78007..5c3c246 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -54,7 +54,7 @@ separate content repos (e.g. `cmeta-aops`). New files get the standard header; new artifacts/categories record `authors` and `copyright` in their `_cmeta.yaml`. Full rule, including reuse and citation: `AGENTS.md` §5.1 and `NOTICE`. -- Python 3.9–3.14 supported; don't use newer-only syntax. +- Python 3.9–3.15 supported; don't use newer-only syntax. - Tests live in `tests/` as `test_*.py` (pytest config in `pyproject.toml`). - This is the public Apache-2.0 framework — keep product vision/strategy prose out; public docs describe *functionality*. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index adba71d..e243b76 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -105,7 +105,7 @@ flake8 cmeta # lint python -m build # build wheel + sdist ``` -cMeta supports **Python 3.9–3.14**; please avoid newer-only syntax. Runtime +cMeta supports **Python 3.9–3.15**; please avoid newer-only syntax. Runtime dependencies are intentionally minimal — justify any new one in your pull request. See [`AGENTS.md`](AGENTS.md) for architecture and conventions. diff --git a/cmeta/internal-repo/_cmr.yaml b/cmeta/internal-repo/_cmr.yaml index b3f24d0..0d7e58f 100644 --- a/cmeta/internal-repo/_cmr.yaml +++ b/cmeta/internal-repo/_cmr.yaml @@ -3,5 +3,5 @@ authors: "[Grigori Fursin](https://cTuning.ai/@gfursin)" category: repo,f4f792ab40c7498f copyright: "2025-2026 Grigori Fursin and cTuning Labs. See the cMeta COPYRIGHT and LICENSE files for details." permanent: True -version: '0.34.2' -version_date: '20261010' +version: '0.35.0' +version_date: '20261011' diff --git a/cmeta/internal-repo/category/repo/api/v1.py b/cmeta/internal-repo/category/repo/api/v1.py index cb3fb1e..a77653c 100644 --- a/cmeta/internal-repo/category/repo/api/v1.py +++ b/cmeta/internal-repo/category/repo/api/v1.py @@ -874,7 +874,15 @@ def delete( p['arg1'] = params['arg1'] result = self.cm.access(p) - if result['return']>0: return result + if result['return']>0: + if unplug and result['return'] == 16: + # Not in the index by that name: a registered FOLDER - a repository that is out of reach (a + # detached drive: registered, not indexed) or simply named by its path - is forgotten by its path + path = self._registered_path(params.get('arg1')) + if path is not None: + return self._forget_registered_path(path, con, verbose) + + return result if unplug: result['deleted_artifacts'] = result.pop('artifacts', []) @@ -922,6 +930,80 @@ def delete( return result + ############################################################################################### + def _registered_path( + self, + name, # A folder: the path of a registered repository (or, with inside=True, a folder inside one). + inside = False, # If True, a folder inside a registered repository names that repository too (the deepest one). + ): + """ + The path, as repos.json has it, of the registered repository that `name` is (or, with + inside=True, lies in) - whether that repository is in the index or not. None when `name` is no + such folder. + """ + + if name is None or str(name).strip() == '': + return None + + wanted = os.path.abspath(os.path.normpath(str(name).strip())) + + r = utils.files.safe_read_file(self.cm.repos_config_path, lock=False, fail_on_error=False, logger=self.logger) + if r['return']>0 or not isinstance(r.get('data'), dict): + return None + + best = None + best_full = None + + for path in r['data']: + full = os.path.abspath(os.path.normpath(path)) + + if os.path.normcase(full) == os.path.normcase(wanted) or (inside and utils.files.is_path_within(full, wanted)): + if best is None or len(full) > len(best_full): + best = path + best_full = full + + return best + + + ############################################################################################### + def _forget_registered_path( + self, + path, # The path of the repository as repos.json has it. + con = False, # If True, print output to console. + verbose = False, # If True, enable verbose output. + ): + """ + Unregister a repository by its path: its line leaves repos.json and its records leave the + index; its folder, reachable or not, is left as it is. + """ + + repos_config_path = self.cm.repos_config_path + + r = utils.files.safe_read_file(repos_config_path, lock=True, keep_locked=True, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + repos_paths = r['data'] + repos_paths_file_lock = r['file_lock'] + + repos_paths.pop(path, None) + + # Do not sort keys - preserve order + r = utils.files.safe_write_file(repos_config_path, repos_paths, file_lock=repos_paths_file_lock, atomic=True, + fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) + if r['return']>0: return r + + if con: + print('') + + r = self.cm.repos.index(clean=False, con=con, verbose=verbose, delete_repo_paths = [path]) + if r['return']>0: return r + + if con: + print (f'Unregistered the repository in "{path}" (its folder is left as it is)') + + return {'return':0, 'deleted_artifacts':[], 'unregistered_paths':[path]} + + ############################################################################################### def move( self, @@ -1498,6 +1580,7 @@ def reindex_( arg1 = str(arg1).strip() repo_artifacts = [] + unindexed_path = None if arg1 == '.' or os.path.isdir(arg1): # A folder: the registered repository that holds it (the deepest one, for nested repositories) @@ -1518,9 +1601,15 @@ def reindex_( best_path = repo_path if best is None: - return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + # Registered but not in the index: a repository that was out of reach when the index was last + # built (it stays registered) and is there again + unindexed_path = self._registered_path(path, inside=True) - repo_artifacts = [best] + if unindexed_path is None: + return {'return':16, 'error':f'no registered repository holds "{path}" - register it first with "cx repo plug "'} + + else: + repo_artifacts = [best] else: p = {'category': ctx['category'], @@ -1561,6 +1650,12 @@ def reindex_( paths.append(path) + if unindexed_path is not None: + if not os.path.isdir(unindexed_path) or not os.path.isfile(os.path.join(unindexed_path, self.cm.cfg['repo_meta_desc'])): + return {'return':1, 'error':f'the folder of the repository registered as "{unindexed_path}" or its {self.cm.cfg["repo_meta_desc"]} is not there (a detached drive?) - nothing was reindexed'} + + paths.append(unindexed_path) + if con: print ('') @@ -1575,6 +1670,9 @@ def reindex_( print (f'Reindexed repository "{repo_alias}" ({repo_uid}): {path}') + if unindexed_path is not None: + print (f'Reindexed the repository in "{unindexed_path}" (it was registered and not in the index)') + return {'return':0, 'artifacts': repo_artifacts, 'paths': paths} ############################################################################################### diff --git a/cmeta/repos.py b/cmeta/repos.py index 7483f95..c75f161 100644 --- a/cmeta/repos.py +++ b/cmeta/repos.py @@ -86,6 +86,12 @@ def __init__( self.KEY_INDEX_UIDS = 'uids' self.KEY_INDEX_LOWERCASE_ALIASES = 'lowercase_aliases' + # In repos.json: a registered repository whose folder could not be reached when the index was last built + # (a detached drive, a share that is down). It stays registered and is indexed again when it is back. + self.KEY_REPO_UNREACHABLE = 'unreachable' + + self._soft_lock_registered = False # the repositories that ask for the soft lock were read once (init) + ################################################################################################### def init( self, @@ -165,6 +171,46 @@ def init( finally: index_lock.release() + # The repositories that ask for the soft lock (`lock: soft` in their descriptor) are known to the path + # lock of this process from the start (once per instance: init runs with every access) + if not self._soft_lock_registered: + r = self._register_soft_lock_repos() + if r['return']>0: return r + self._soft_lock_registered = True + + return {'return':0} + + ################################################################################################### + def _register_soft_lock( + self, + path, # The registered folder of the repository. + repo_meta, # Its descriptor (_cmr.yaml, with the extra meta of the registration). + ): + """ + A repository whose descriptor says `lock: soft` - one that several systems write at once through + a shared drive (a Windows tree plugged from WSL2 or a container, a network share), where the OS + locks of the systems do not see each other - asks the path lock for the soft lock under its folder + (`utils.files.use_soft_lock_under`), for this process. + """ + if path and str(repo_meta.get('lock', '')).strip().lower() == 'soft': + utils.files.use_soft_lock_under(path) + self.logger.debug(f"repository {path} asks for the soft lock (lock: soft)") + + def _register_soft_lock_repos(self): + """ + At the start of a process: the repositories of the index that ask for the soft lock (see + `_register_soft_lock`). The index of repositories is small and read through the cache. + """ + index_repo_file = os.path.join(self.index_path, 'repo' + self.index_extension) + if not os.path.isfile(index_repo_file): + return {'return':0} + + r = utils.files.safe_read_file_via_cache(index_repo_file, self.file_cache, fail_on_error=self.fail_on_error, logger=self.logger) + if r['return']>0: return r + + for entry in (r['data'].get(self.KEY_INDEX_UIDS) or {}).values(): + self._register_soft_lock(entry.get('path'), entry.get('_cmr') or entry.get('cmeta') or {}) + return {'return':0} ################################################################################################### @@ -1129,7 +1175,12 @@ def _index( repos_config_lock = r['file_lock'] - paths_to_repos = {} + paths_to_repos = {} # the repositories indexed by this run + repos_to_write = {} # what repos.json lists after it: those, and - in their places - the registered + # repositories that cannot be reached now + unreachable = {} # path -> why + newly_unreachable = [] # those of them that were reachable at the run before + back = [] # paths that could not be reached at an earlier run and can now original_paths_to_repos = r['data'] to_update = False @@ -1145,6 +1196,8 @@ def _index( path_to_repo_desc = os.path.join(path, self.cfg['repo_meta_desc']) + why_not = '' + if not os.path.isfile(path_to_repo_desc): if extra_meta.get('keep') and 'keep_repo_meta' in local_meta: repo_meta = local_meta['keep_repo_meta'] @@ -1153,13 +1206,15 @@ def _index( paths_to_repos[path]={'meta': extra_meta, 'keep_repo_meta': repo_meta} - - else: - to_update = True + repos_to_write[path] = paths_to_repos[path] + + continue + + why_not = 'its folder or its ' + self.cfg['repo_meta_desc'] + ' is not there' else: r = utils.files.safe_read_file(path_to_repo_desc, retry_if_not_found=3, fail_on_error=self.fail_on_error, logger=self.logger) - if r['return']==0: + if r['return']==0 and isinstance(r.get('data'), dict) and r['data'].get('artifact'): repo_meta = r['data'] if len(extra_meta)>0: @@ -1170,15 +1225,59 @@ def _index( paths_to_repos[path] = {} if len(extra_meta)>0: paths_to_repos[path]['meta'] = extra_meta + if 'keep_repo_meta' in local_meta: + # the stored copy of a kept repository stays with its registration (it was left out + # of the file whenever the file was rewritten while the repository was reachable) + paths_to_repos[path]['keep_repo_meta'] = local_meta['keep_repo_meta'] + repos_to_write[path] = paths_to_repos[path] + + if local_meta.get(self.KEY_REPO_UNREACHABLE): + back.append(path) + to_update = True + + continue + + if r['return']==0: + why_not = 'its ' + self.cfg['repo_meta_desc'] + ' names no artifact' + else: + why_not = 'its ' + self.cfg['repo_meta_desc'] + ' cannot be read' + (f" ({r.get('error')})" if r.get('error') else '') + + # A registered repository that cannot be reached - a detached drive, a share that is down, a + # descriptor that is being edited - stays registered (it was dropped from repos.json without a + # word, and had to be plugged again) and is left out of this index + unreachable[path] = why_not + + entry = dict(local_meta) + if not entry.get(self.KEY_REPO_UNREACHABLE): + entry[self.KEY_REPO_UNREACHABLE] = True + to_update = True + newly_unreachable.append(path) + repos_to_write[path] = entry if to_update: # Do not sort keys - preserve order! - r = utils.files.safe_write_file(repos_config_path, paths_to_repos, file_lock=repos_config_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) + r = utils.files.safe_write_file(repos_config_path, repos_to_write, file_lock=repos_config_lock, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r else: r = utils.files.unlock_path(repos_config_path, repos_config_lock, fail_on_error=self.fail_on_error, logger=self.logger) if r['return']>0: return r + for path in unreachable: + message = (f'repository "{path}" cannot be reached ({unreachable[path]}): it stays registered and is left out of the index - ' + f'"cx repo reindex {path}" or "cx --reindex" brings it back when it is there again, "cx repo unplug {path}" forgets it') + self.logger.warning(message) + if con: + print (f'cMeta notice: {message}', file=sys.stderr) + + if len(back) > 0: + # Reachable again: indexed with this run, as if it had just been plugged + add_repo_paths = list(add_repo_paths) + [path for path in back if path not in add_repo_paths] + + if len(newly_unreachable) > 0: + # Out of reach since the run before: its records leave the index with this run (an incremental run + # keeps the records of the repositories it does not scan; a full rebuild has none of them anyway) + delete_repo_paths = list(delete_repo_paths) + [path for path in newly_unreachable if path not in delete_repo_paths] + if len(paths_to_repos) == 0: return {'return':1, 'error':f'could not find any repository in {repos_config_path}'} @@ -1259,7 +1358,12 @@ def _index( # Use atomic write to avoid corrupting large index files r = utils.files.safe_write_file(index_repo_file, index_repos, atomic=True, fail_on_error=self.fail_on_error, logger=self.logger, sort_keys=False) if r['return']>0: return r - + + # A repository that asks for the soft lock is known to the path lock of this process at once (the + # processes that start later read it from this index: see init) + for path in paths_to_repos.keys(): + self._register_soft_lock(path, repos_meta[path]) + ###################################################################################################################### # Indexing categories index_categories = {self.KEY_INDEX_UIDS:{}, self.KEY_INDEX_LOWERCASE_ALIASES:{}} @@ -1383,7 +1487,9 @@ def _index( print (' * ' + index_categories[self.KEY_INDEX_UIDS][uid]['path']) print (' * ' + category_path) if con: - input (' Fix it or press Enter to continue!') + # A pause for the person at the terminal; with nobody there (a detached + # job, a pipe at its end) the index goes on instead of ending in EOFError + utils.common.ask(' Fix it or press Enter to continue!', optional=True) cmeta_ref_parts['artifact_alias'] = alias if alias != alias.lower(): diff --git a/cmeta/utils/files.py b/cmeta/utils/files.py index 91db7d7..63d369b 100644 --- a/cmeta/utils/files.py +++ b/cmeta/utils/files.py @@ -23,6 +23,7 @@ import logging import random import threading +import binascii from .common import _error, ask as _ask @@ -44,11 +45,20 @@ LOCK_SUFFIX = ".lock" LOCK_POLL_START = 0.001 # the first wait when a lock is busy; it doubles up to LOCK_POLL_MAX LOCK_POLL_MAX = 0.025 -LOCK_SOFT_STALE_SECONDS = 600 # a soft lock file older than this, whose holder is on another host, is stale -LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file older than this is a leftover (a holder writes its content at once) -LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock file of a dead pid younger than this is a release in flight, not a crash +# The age of a soft lock is counted from the time its holder WROTE INTO IT, read on this machine's clock, and an +# empty file by how long the waiter itself has seen it - never by the time the file system gave the file: a drive +# can keep another clock (a network share, a Windows drive inside WSL2) or keep times to the whole second. +LOCK_SOFT_STALE_SECONDS = 600 # a soft lock taken longer ago than this by a holder on another host is stale +LOCK_SOFT_EMPTY_STALE_SECONDS = 2.0 # an EMPTY soft lock file that a waiter has seen empty for this long is a leftover (a holder writes its content at once) +LOCK_SOFT_DEAD_MIN_AGE_SECONDS = 1.0 # a soft lock of a dead pid taken less than this ago is a release in flight, not a crash LOCK_SOFT_CHECK_SECONDS = 0.25 # how often a waiter on a soft lock reads the file to check for a stale holder LOCK_SOFT_REMOVE_SECONDS = 2.0 # how long a releasing soft lock retries the removal of its file (Windows: a reader may hold it) +LOCK_SOFT_TAKE_TRIES = 25 # tries of the rename that takes a stale soft lock over (Windows: a reader may hold a file for an instant) +LOCK_SOFT_TAKE_REMOVE_SECONDS = 0.2 # how long a waiter retries the removal of the take file it did not use +LOCK_SOFT_READ_MAX = 1024 # bytes read from a soft lock file (its one line: pid host time system) +LOCK_SOFT_HEARTBEAT_SECONDS = 2.0 # a soft holder rewrites the time in its line this often (the heartbeat; soft mode only) +LOCK_SOFT_SILENT_SECONDS = 20.0 # a soft lock whose line a waiter has seen unchanged for this long is the leftover of a holder that is gone +LOCK_SOFT_PATHS_ENV = 'CMETA_SOFT_LOCK_PATHS' # folders whose lock files use the soft lock (a home or repository written by several systems at once) LOCK_IDENTITY_RETRIES = 50 # consecutive identity mismatches after which a path is locked with the soft lock LOCK_NOTE_MAX = 512 # bytes of the note a holder may leave in its lock file for the message of a waiter LOCK_TIMEOUT_ENV = 'CMETA_LOCK_TIMEOUT' @@ -155,10 +165,19 @@ def is_path_within( # POSIX: fcntl.flock (per open file description, so threads of one process exclude each other too), # then fcntl.lockf (POSIX record locks: NFS and some network mounts refuse flock), each with # the identity check (fstat of the descriptor vs stat of the path, inode and device); -# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time") where the file -# system refuses both OS locks (some FUSE, 9p and SMB mounts) or the identity check keeps -# failing - a stale soft lock (its pid dead on this host, or older than LOCK_SOFT_STALE_SECONDS -# when left by another host) is renamed away and removed by the waiter that finds it. +# anywhere: the soft lock (the file created with O_EXCL, holding "pid host time system") where the +# file system refuses both OS locks (some FUSE, 9p and SMB mounts), where the identity check +# keeps failing, or where it is ASKED FOR - a home or repository that several systems write at +# once through a shared drive (a Windows tree plugged from WSL2 or a container, a network +# share), whose OS locks do not see each other: CMETA_SOFT_LOCK_PATHS names such folders, +# `lock: soft` in a repository's _cmr.yaml names the repository (use_soft_lock_under). The +# holder keeps its file open and rewrites the time in its line every LOCK_SOFT_HEARTBEAT_SECONDS; +# a stale soft lock (its pid dead on this system, or its line unchanged for +# LOCK_SOFT_SILENT_SECONDS on the waiter's own clock - the holder is on another system or host) +# is taken over by ONE of the waiters that find it: it replaces the file by its own in one +# rename (PathLock._take_stale_soft_lock). A lock file of cMeta before 0.35.0 (no system, no +# heartbeat) is judged as before: the pid when the host name is this one, else its age against +# LOCK_SOFT_STALE_SECONDS. # In every mode the threads of one process are serialized on the path first (a threading.Lock per path), # which POSIX record locks and soft locks need (they are per process), and the busy wait polls with a # delay that doubles from LOCK_POLL_START to LOCK_POLL_MAX until the timeout. The engine's own writers @@ -171,8 +190,48 @@ def is_path_within( _lock_registry = {} # normalized lock file path -> threading.Lock (the threads of this process) _lock_registry_guard = threading.Lock() _lock_soft_paths = set() # lock files locked with the soft lock from now on (the OS lock was refused) +_lock_soft_dirs = [] # folders whose lock files use the soft lock because it was asked for (use_soft_lock_under) _lock_no_flock_dirs = set() # folders where flock was refused (POSIX record locks are tried first there) _lock_host = None +_lock_system = None +_beat_lock = threading.Lock() # guards _beat_held, and every write of a beat +_beat_held = {} # id(lock) -> the soft locks this process holds and beats (PathLock._start_heartbeat) +_beat_thread = None # the one heartbeat thread of the process, started with the first soft lock held +_beat_wake = threading.Event() # the thread sleeps on it (never set: a plain sleep that reads the interval each time) + + +def _after_fork_in_child(): + """A forked child starts with locks of its own: the thread locks of the parent (the registry of the paths, + the guard of the registry, the lock of the heartbeat) may have been held by a thread that does not exist in + the child - the parent's heartbeat thread, another thread of the parent - and would be held for ever; the + soft locks the parent beats are not the child's to beat (its copy of a lock never releases it either: + PathLock.release).""" + global _lock_registry_guard, _beat_lock, _beat_thread + _lock_registry_guard = threading.Lock() + _lock_registry.clear() + _beat_lock = threading.Lock() + _beat_held.clear() + _beat_thread = None + + +if hasattr(os, 'register_at_fork'): + os.register_at_fork(after_in_child=_after_fork_in_child) + + +def _beat_forever(): + """The heartbeat thread: every LOCK_SOFT_HEARTBEAT_SECONDS, the line of every soft lock this process holds + is rewritten (PathLock._write_beat). The locks a forked child inherited are its parent's: dropped.""" + while True: + _beat_wake.wait(LOCK_SOFT_HEARTBEAT_SECONDS) + try: + with _beat_lock: + for key, lock in list(_beat_held.items()): + if lock._owner_pid != os.getpid(): + _beat_held.pop(key, None) + continue + lock._write_beat() + except Exception: + pass def _lock_thread_lock(lock_file): @@ -204,6 +263,92 @@ def _lock_pid_alive(pid): return True # unknown: never treat it as dead +def _lock_system_id(): + """ + A short name of the system this process runs on, written into a soft lock file so that a waiter knows + whether the pid in the file is one it can check: the same name means the same kernel and the same pid + space. Windows and WSL2 share the host name and have different pids; a container shares the kernel of + its host (one boot) and has its own pid space; a cloned machine keeps its host name. + Linux: the boot id of the kernel and the pid namespace of this process (/proc); + Windows: the machine's installation id (the registry); + macOS: the machine's id (sysctl kern.uuid); + else, or where those cannot be read: the host name and the platform. + Twelve hex digits of a hash of it, computed once per process. + """ + global _lock_system + if _lock_system is None: + parts = [sys.platform] + try: + if sys.platform.startswith('linux'): + with open('/proc/sys/kernel/random/boot_id') as f: + parts.append(f.read().strip()) + parts.append(str(os.stat('/proc/self/ns/pid').st_ino)) + elif sys.platform == 'win32': + import winreg + with winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r'SOFTWARE\Microsoft\Cryptography') as key: + parts.append(str(winreg.QueryValueEx(key, 'MachineGuid')[0])) + elif sys.platform == 'darwin': + import ctypes + libc = ctypes.CDLL(None) + size = ctypes.c_size_t(0) + if libc.sysctlbyname(b'kern.uuid', None, ctypes.byref(size), None, 0) == 0 and 0 < size.value < 256: + buf = ctypes.create_string_buffer(size.value) + if libc.sysctlbyname(b'kern.uuid', buf, ctypes.byref(size), None, 0) == 0: + parts.append(buf.value.decode('ascii', 'replace')) + except Exception: + pass + if len(parts) < 2: + parts.append(_lock_host_name()) + _lock_system = hashlib.sha1('\n'.join(parts).encode('utf-8', 'replace')).hexdigest()[:12] + return _lock_system + + +def _soft_lock_paths_from_env(): + """The folders named by CMETA_SOFT_LOCK_PATHS (os.pathsep between them; '*' = everywhere), normalized.""" + value = os.environ.get(LOCK_SOFT_PATHS_ENV, '') + folders = [] + for folder in value.split(os.pathsep): + folder = folder.strip() + if folder == '': + continue + folders.append('*' if folder == '*' else os.path.normcase(os.path.abspath(folder))) + return folders + + +def use_soft_lock_under( + folder: str, # The folder (a home, a repository) whose lock files use the soft lock from now on in this process. +): + """ + Ask for the soft lock for every lock file under `folder` (see the notes on the path lock: a home or + repository that several systems write at once through a shared drive, whose OS locks do not see + each other). The engine calls it for a repository whose descriptor says `lock: soft`; the environment + variable CMETA_SOFT_LOCK_PATHS names folders at the start of a process. '*' means everywhere. + + Returns: + dict: {'return': 0} + """ + folder = '*' if folder == '*' else os.path.normcase(os.path.abspath(folder)) + if folder not in _lock_soft_dirs: + _lock_soft_dirs.append(folder) + return {'return': 0} + + +def _soft_lock_asked_for(lock_file): + """Whether `lock_file` lies under a folder for which the soft lock was asked (use_soft_lock_under).""" + if not _lock_soft_dirs: + return False + if '*' in _lock_soft_dirs: + return True + path = os.path.normcase(os.path.abspath(lock_file)) + for folder in _lock_soft_dirs: + if path == folder or path.startswith(folder.rstrip(os.sep) + os.sep): + return True + return False + + +_lock_soft_dirs.extend(_soft_lock_paths_from_env()) # CMETA_SOFT_LOCK_PATHS, read once at the start of the process + + class _LockUnsupported(Exception): pass @@ -232,6 +377,13 @@ def __init__( self._mismatches = 0 self._last_error = None self._next_stale_check = 0.0 + self._empty_seen = {} # path -> (identity, monotonic time): the empty soft lock files this waiter watches + self._own_empty = None # (identity,) of the empty file this object's refused OS-lock attempt left + self._noted = False # a note was written into the held lock file (see _own_identity) + self._line = None # the line of the soft lock this object holds (the heartbeat rewrites it) + self._wrote = None # the line of a soft lock file this object wrote and could not confirm + self._beat_seen = {} # path -> (identity, content, monotonic time): the lines this waiter watches for a heartbeat + self._owner_pid = None # the process that took the lock (a forked child never releases its parent's lock) def __enter__(self): self.acquire() @@ -258,6 +410,7 @@ def acquire( if not thread_lock.acquire(timeout=timeout): raise TimeoutError(f"the lock {self.lock_file} is held by another thread of this process (waited {timeout} s)") + self._owner_pid = os.getpid() # before the lock is taken: the heartbeat thread reads it try: deadline = time.monotonic() + timeout delay = LOCK_POLL_START @@ -279,6 +432,8 @@ def acquire( delay = min(delay * 2, LOCK_POLL_MAX) except BaseException: + self._owner_pid = None + self._drop_unconfirmed() thread_lock.release() raise @@ -287,17 +442,27 @@ def release(self): if not self.is_locked: return + if self._owner_pid is not None and self._owner_pid != os.getpid(): + # A forked child of the holder: the lock is its parent's. The child's copy forgets it and touches + # neither the file nor the descriptor (a POSIX record lock is not even the child's to release) + self._fd = self._thread_lock = None + self._ident = self._line = None + self._noted = self.is_locked = False + self.mode = None + return + try: if self.mode == 'windows': try: msvcrt.locking(self._fd, msvcrt.LK_UNLCK, 1) except OSError: pass + ident = self._own_identity() os.close(self._fd) - self._unlink_own_file() + self._unlink_own_file(ident) elif self.mode in ('flock', 'lockf'): - self._unlink_own_file() + self._unlink_own_file(self._own_identity()) try: self._posix_unlock(self._fd, self.mode) except OSError: @@ -305,11 +470,15 @@ def release(self): os.close(self._fd) elif self.mode == 'soft': + self._stop_heartbeat() self._remove_soft_file() finally: self._fd = None self._ident = None + self._line = None + self._noted = False + self._owner_pid = None self.is_locked = False self.mode = None thread_lock = self._thread_lock @@ -325,6 +494,7 @@ def write_note(self, note): if not self.is_locked or self._fd is None or self.mode == 'soft': return data = b'\n' + str(note).encode('utf-8', 'replace')[:LOCK_NOTE_MAX] + b'\n' + self._noted = True try: os.ftruncate(self._fd, 0) os.lseek(self._fd, 0, os.SEEK_SET) @@ -362,7 +532,7 @@ def read_note(lock_file): # ---- one attempt ---------------------------------------------------------------------- def _try(self): - if self.lock_file in _lock_soft_paths: + if self.lock_file in _lock_soft_paths or _soft_lock_asked_for(self.lock_file): return self._try_soft() if msvcrt is not None: return self._try_windows() @@ -495,151 +665,435 @@ def _posix_unlock(fd, kind): else: fcntl.lockf(fd, fcntl.LOCK_UN) + @staticmethod + def _soft_line(): + """The line of a soft lock this process holds: `pid host time system`. The time is the moment of the + last write (the heartbeat rewrites the line in place, so its length never changes); the system (see + _lock_system_id) tells a waiter whether it can check the pid.""" + return f"{os.getpid()} {_lock_host_name()} {time.time():.3f} {_lock_system_id()}\n".encode('utf-8', 'replace') + def _try_soft(self): + line = self._soft_line() try: fd = os.open(self.lock_file, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) except FileExistsError as e: self._last_error = e now = time.monotonic() - if now >= self._next_stale_check: - self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS - self._clear_stale_soft_lock() - return False + if now < self._next_stale_check: + return False + self._next_stale_check = now + LOCK_SOFT_CHECK_SECONDS + # The file of a holder that is gone is taken over in the same attempt: a single try (timeout 0, the + # probe of a cache entry) finds the lock of a dead process free + return self._take_stale_soft_lock(line) except OSError as e: if self._open_failed_for_the_moment(e): return False raise try: - os.write(fd, (f"{os.getpid()} {_lock_host_name()} {time.time():.3f}\n").encode('utf-8', 'replace')) - finally: + os.write(fd, line) + except BaseException: os.close(fd) + raise + return self._soft_lock_taken(line, fd) - try: - self._ident = self._identity(os.stat(self.lock_file)) - except OSError: - self._ident = None + def _soft_lock_taken(self, line, fd=None): + """ + The last step of taking the soft lock: the file at the path must be the file this object wrote. It is + not when that file was taken over in the instant it was still empty (this process was stopped between + creating it and writing into it for as long as an empty file is watched) - the lock is then not held, + and nothing at the path is touched. + + fd: the open descriptor of the written file, when there is one. It stays open while the lock is held: + an open file keeps its identity (no new file gets its inode number), so "the file at the path is the + one this descriptor holds" is an exact answer, here and at release, for the price of two calls that + open nothing. (And on Windows a file that is open cannot be replaced: the lock of a live holder + cannot be taken over there at all.) Where the identity does not say so - no descriptor (a file + renamed into place by a take-over, a file found later), a file system without inode numbers or one + that reports them differently by descriptor and by path - the line in the file decides. + """ + mine = None + if fd is not None: + try: + mine = self._identity(os.fstat(fd)) + if mine is not None and self._identity(os.stat(self.lock_file)) == mine: + return self._hold_soft(line, fd, mine) + except OSError: + pass + + status, held = 'unreadable', None + for _ in range(3): + status, held = self._open_if_mine(line) + if status in ('mine', 'other'): + break + time.sleep(0.002) # not readable for an instant (Windows: a scanner holds the new file), or not shown yet + + if status == 'mine': + if fd is not None: + try: + os.close(fd) + except OSError: + pass + try: + mine = self._identity(os.fstat(held)) + except OSError: + mine = None + return self._hold_soft(line, held, mine) + if status == 'unreadable' and fd is not None: + return self._hold_soft(line, fd, mine) # it cannot be asked: the file this object made is its lock, as before + + # Not held. Should the file with this line be at the path later after all (a file system slow to show + # a new file), it is this object's lock (_take_stale_soft_lock), and an acquire that gives up removes + # it (_drop_unconfirmed) + if fd is not None: + os.close(fd) + self._wrote = line + return False + + def _hold_soft(self, line, fd, ident): + self._fd = fd + self._ident = ident + self._line = line + self._wrote = None + self._own_empty = None + self._empty_seen = {} + self._beat_seen = {} self.mode = 'soft' + self._start_heartbeat() return True - def _clear_stale_soft_lock(self): - """A soft lock file whose holder is dead (this host; the file at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS - old, so that a holder that just released and exited is not mistaken for a crash), that was marked - released by a holder that could not remove it, that is empty and old (a leftover) or too old (another - host) is renamed away and removed - by one waiter only (the rename is atomic), and only when the file - at the path is still the file that was judged (a fresh lock file of a live holder moved by mistake in - the instant between the check and the rename is put back).""" + # ---- the heartbeat of a soft holder ------------------------------------------------------ + # + # A soft holder rewrites the time in its line every LOCK_SOFT_HEARTBEAT_SECONDS (one daemon thread per + # process for all the soft locks it holds; the line is rewritten in place through the held descriptor: + # same length, never empty). A waiter that sees the same line for LOCK_SOFT_SILENT_SECONDS on its own + # monotonic clock knows the holder is gone - whatever the other system's clock says, however long the + # holder has held the lock, and without a pid it could not check (a holder on another host, or on + # another system of this host: Windows and WSL2 share the host name and have different pids). A lock + # file without a system (3 words: written by cMeta before 0.35.0, which did not beat) is judged as + # before: the pid when the host name is this one, else the age of its time against LOCK_SOFT_STALE_SECONDS. + + def _start_heartbeat(self): + if self._fd is None: + return # no descriptor to write through (the line decides for the waiters: see the rules) + global _beat_thread + with _beat_lock: + _beat_held[id(self)] = self + if _beat_thread is None or not _beat_thread.is_alive(): + _beat_thread = threading.Thread(target=_beat_forever, name='cmeta-soft-lock-heartbeat', daemon=True) + _beat_thread.start() # its first beat comes after one interval: a lock held for less never beats + + def _stop_heartbeat(self): + with _beat_lock: # a beat of this lock in flight ends before the lock goes on + _beat_held.pop(id(self), None) + + def _write_beat(self): + fd = self._fd + if fd is None: + return + line = self._soft_line() + try: + if hasattr(os, 'pwrite'): + os.pwrite(fd, line, 0) + else: + os.lseek(fd, 0, os.SEEK_SET) + os.write(fd, line) + except (OSError, ValueError): + return # the drive refuses for the moment: the line stays as it is + self._line = line + + def _open_if_mine(self, line): + """Open the file at the lock path and keep it open when it carries `line`: ('mine', descriptor); + ('other', None) when it carries something else, ('gone', None) when it is not there, ('unreadable', + None) when it cannot be read at the moment. Open for writing too: the heartbeat writes through it.""" try: - fd = os.open(self.lock_file, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + fd = os.open(self.lock_file, os.O_RDWR | getattr(os, 'O_BINARY', 0)) + except FileNotFoundError: + return 'gone', None except OSError: + return 'unreadable', None + try: + data = os.read(fd, LOCK_SOFT_READ_MAX) + except OSError: + os.close(fd) + return 'unreadable', None + if data != line: + os.close(fd) + return 'other', None + return 'mine', fd + + def _drop_unconfirmed(self): + """An acquire gives up: a file at the path that carries a line this object wrote and could not confirm + (see _soft_lock_taken) would keep everybody out for as long as this process lives - it goes.""" + wrote, self._wrote = self._wrote, None + if wrote is None: return + seen = self._read_lock_file(self.lock_file) + if seen and seen[1] == wrote: + self._remove_file(self.lock_file, LOCK_SOFT_TAKE_REMOVE_SECONDS) + + @classmethod + def _read_lock_file(cls, path): + """(identity, content, time of the file) of a soft lock file; False when it is not there, None when it + cannot be read at the moment.""" + try: + fd = os.open(path, os.O_RDONLY | getattr(os, 'O_BINARY', 0)) + except FileNotFoundError: + return False + except OSError: + return None try: st = os.fstat(fd) - words = os.read(fd, 256).decode('utf-8', 'replace').split() if st.st_size > 0 else [] + data = os.read(fd, LOCK_SOFT_READ_MAX) if st.st_size > 0 else b'' except OSError: - return + return None finally: os.close(fd) - - ident = self._identity(st) - age = time.time() - st.st_mtime - pid = int(words[0]) if len(words) > 0 and words[0].isdigit() else None - host = words[1] if len(words) > 1 else None + return cls._identity(st), data, st.st_mtime + + def _lock_file_is(self, path, seen): + """Whether the file at `path` is still the one read as `seen`: the same content and, where the file + system reports one, the same identity.""" + now = self._read_lock_file(path) + return bool(now) and now[1] == seen[1] and (now[0] is None or seen[0] is None or now[0] == seen[0]) + + def _soft_file_is_stale(self, path, seen): + """Whether the soft lock file read as `seen` (or the take file of a waiter, which carries the same + line) is the leftover of a holder that is gone: + - marked released by a holder that could not remove it; + - a line of four words (`pid host time system`: a holder that beats, cMeta 0.35.0 and later): its pid + is dead on THIS system (the system is this one; taken at least LOCK_SOFT_DEAD_MIN_AGE_SECONDS ago, so + that a holder that has just released and exited is not a crash), or this waiter has seen the same + line for LOCK_SOFT_SILENT_SECONDS on its own monotonic clock (the holder beats no more: dead on + another system or host, or stopped); + - a line of three words (a holder of cMeta before 0.35.0, which does not beat): its pid is dead on + this host (the host name is this one), or it was taken longer ago than LOCK_SOFT_STALE_SECONDS; + - it is empty and this waiter has seen it empty for LOCK_SOFT_EMPTY_STALE_SECONDS (a holder writes + its line right after it creates the file), or it is the empty file this object's own OS-lock + attempt has just left. + How long ago a lock was taken is read from the time its holder wrote into it, on this machine's clock. + The time the file system gave the file is not used - on a drive that keeps another clock (a Windows + drive inside WSL2 is seconds off, a network share can be more) a fresh lock file looked old - except + where nothing else dates the file: a line without a time (not written by this code), and an empty file + that nobody watches (older than LOCK_SOFT_STALE_SECONDS it is a leftover whatever the clock of the drive).""" + ident, data, mtime = seen + words = data.decode('utf-8', 'replace').split() if not words: - stale = age > LOCK_SOFT_EMPTY_STALE_SECONDS - elif words[0] == 'released': - stale = True - elif pid is not None and host == _lock_host_name(): - stale = age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) - else: - stale = age > LOCK_SOFT_STALE_SECONDS + if path == self.lock_file and self._own_empty is not None and self._own_empty[0] == ident: + return True + now = time.monotonic() + watched = self._empty_seen.get(path) + if watched is None or watched[0] != ident: + watched = self._empty_seen[path] = (ident, now) + return now - watched[1] > LOCK_SOFT_EMPTY_STALE_SECONDS or time.time() - mtime > LOCK_SOFT_STALE_SECONDS - if not stale: - return + self._empty_seen.pop(path, None) + if words[0] == 'released': + return True try: - if ident is not None and self._identity(os.stat(self.lock_file)) != ident: - return # the file changed under us: judge it again next time - except OSError: - return + age = time.time() - float(words[2]) + except (IndexError, ValueError): + age = None + if age is None or age != age or abs(age) == float('inf'): + age = time.time() - mtime + + pid = int(words[0]) if words[0].isascii() and words[0].isdigit() else None + + if len(words) >= 4: + # A holder that beats (cMeta 0.35.0 and later: the fourth word names its system). Its pid is checked + # when it is a pid of THIS system; whatever the system, a line that has not changed for + # LOCK_SOFT_SILENT_SECONDS on this waiter's own clock is the leftover of a holder that is gone. + if pid is not None and words[3] == _lock_system_id() and age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid): + return True + now = time.monotonic() + watched = self._beat_seen.get(path) + if watched is None or watched[0] != ident or watched[1] != data: + self._beat_seen[path] = (ident, data, now) + return False + return now - watched[2] > LOCK_SOFT_SILENT_SECONDS + + # A holder that does not beat (cMeta before 0.35.0): its pid when the host name is this one, else how + # long ago it was taken + if pid is not None and len(words) > 1 and words[1] == _lock_host_name(): + return age > LOCK_SOFT_DEAD_MIN_AGE_SECONDS and not _lock_pid_alive(pid) + return age > LOCK_SOFT_STALE_SECONDS + + def _take_stale_soft_lock(self, line): + """ + Take the soft lock over from a holder that is gone (see _soft_file_is_stale). Returns True when this + object holds the lock. + + The stale file is never removed: the waiter REPLACES it by a file that carries its own line (one + atomic rename), so the path is never free in between and nobody else can slip in. The file to put + there is made first, exclusively, under a name that says which stale file it is for + (`.take-`): of the waiters that judged the + same stale file only one gets to make it, and it then checks that the file at the path is still the + one it judged before it replaces it. A waiter that comes later finds another file at the path and + gives up. (Before, the stale file was renamed away, checked and - when it was not the judged one - + put back: between the two a third process could take the free path, which under load could end + with two holders and a leftover `.stale-*` file.) + """ + seen = self._read_lock_file(self.lock_file) + if not seen: + return False + if self._wrote is not None and seen[1] == self._wrote: + return self._soft_lock_taken(self._wrote) # the file this object wrote a moment ago + if not self._soft_file_is_stale(self.lock_file, seen): + return False - moved = f"{self.lock_file}.stale-{os.getpid()}-{random.getrandbits(32):08x}" + data = seen[1] + take = self._take_file(seen) try: - os.rename(self.lock_file, moved) # atomic: only one waiter gets to remove it + fd = os.open(take, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, 'O_BINARY', 0), 0o644) + except FileExistsError: + # Another waiter is taking this very file over - or died doing so, and its take file is a leftover + theirs = self._read_lock_file(take) + if theirs and self._soft_file_is_stale(take, theirs) and self._lock_file_is(take, theirs): + self._remove_file(take) + return False except OSError: - return + return False - if ident is not None: - try: - moved_ident = self._identity(os.stat(moved)) - except OSError: - return - if moved_ident != ident: - # Not the file judged stale but a fresh one made in between: back where it was, without - # clobbering (a hard link fails where a file exists already), then the extra name goes + taken = False + try: + os.write(fd, line) + # Closed before the rename, everywhere: Windows refuses to rename a file this process has open, + # and a Windows drive seen through 9p (WSL2's /mnt/d, a container's bind mount) renames it but + # shows nothing at the new path until the descriptor is closed (the lock looked "gone" to the + # taker itself); the file is opened again at its new path to hold it (_soft_lock_taken) + os.close(fd) + fd = None + for _ in range(LOCK_SOFT_TAKE_TRIES): + if not self._lock_file_is(self.lock_file, seen): + break # not the file that was judged any more: judged again at the next check try: - os.link(moved, self.lock_file) + os.replace(take, self.lock_file) + except PermissionError: + # Windows: a waiter that reads one of the two files holds it for an instant - and a holder + # that is alive after all holds its file open for as long as it holds the lock + time.sleep(0.002) + continue except OSError: - try: - if not os.path.exists(self.lock_file): - os.rename(moved, self.lock_file) - except OSError: - pass - return + break + taken = True + break + finally: + if fd is not None: try: - os.unlink(moved) + os.close(fd) # the write failed before the close above except OSError: pass - return + if not taken: + self._wrote = line # see _soft_lock_taken: a file with this line that reaches the path is this object's + self._remove_file(take, LOCK_SOFT_TAKE_REMOVE_SECONDS) - try: - os.unlink(moved) - except OSError: - pass - self.logger.warning(f"utils.files.PathLock - removed the stale lock file {self.lock_file} (left by pid {pid} on {host})") + if not taken: + return False + + words = data.decode('utf-8', 'replace').split() + left_by = "an empty file" if not words else ("marked released" if words[0] == 'released' else ' '.join(words[:2])) + self.logger.warning(f"utils.files.PathLock - took over the stale lock file {self.lock_file} (left by {left_by})") + return self._soft_lock_taken(line) + + def _take_file(self, seen): + """The name of the file a waiter makes to replace the stale lock file read as `seen`: one name per + stale file (its content and identity), so that one waiter only can make it.""" + return f"{self.lock_file}.take-{binascii.crc32(seen[1] + repr(seen[0]).encode('ascii', 'replace')) & 0xffffffff:08x}" + + @staticmethod + def _remove_file(path, seconds=0.0): + """Remove a file, retrying for `seconds` while it is held (Windows: a reader has it open); best effort.""" + deadline = time.monotonic() + seconds + while True: + try: + os.unlink(path) + return + except PermissionError: + if time.monotonic() >= deadline: + return + time.sleep(0.002) + except OSError: + return def _fall_back_to_soft(self, why): if self.lock_file not in _lock_soft_paths: _lock_soft_paths.add(self.lock_file) self.logger.warning(f"utils.files.PathLock - the file system refuses OS locks for {self.lock_file} ({why}): a soft lock file is used instead (best effort between hosts)") - # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its - # content at once): out of the way, or the soft lock could never be taken + # The empty file the OS-lock attempt created or found is not a soft lock (a soft holder writes its line + # at once): this object takes it over at once, without the time an empty file is otherwise watched try: - if os.path.getsize(self.lock_file) == 0: - os.unlink(self.lock_file) + st = os.stat(self.lock_file) + self._own_empty = (self._identity(st),) if st.st_size == 0 else None except OSError: - pass + self._own_empty = None def _remove_soft_file(self): - """Remove the soft lock file; on Windows a waiter reading it holds it open for a moment, so the removal - is retried, and a file that cannot be removed in time is marked released for the waiters to clear.""" - deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS - while True: - if self._ident is not None: + """Remove the soft lock file - when it is still the file of this lock (a file taken over by a waiter + that judged this holder gone is theirs): the file at the path is the one this lock's descriptor holds + or, where the identity does not say so, it carries this lock's line. On Windows a waiter reading the + file holds it open for a moment, so the removal is retried, and a file that cannot be removed in time + is marked released for the waiters to take over.""" + fd, self._fd = self._fd, None + mine = self._ident + try: + if fd is not None: try: - if self._identity(os.stat(self.lock_file)) != self._ident: - return # not ours any more + mine = self._identity(os.fstat(fd)) except OSError: + pass + # Closed before the removal, everywhere: Windows cannot remove a file this process has open, + # and a Windows drive seen through 9p keeps the path of an open file in a removed-but-there + # state until the descriptor is closed. Nothing is lost by it: the soft lock is the line in + # the file, not the descriptor (a waiter judges the holder by its pid or its heartbeat) + try: + os.close(fd) + except OSError: + pass + fd = None + + deadline = time.monotonic() + LOCK_SOFT_REMOVE_SECONDS + while True: + ours = False + if mine is not None: + try: + ours = self._identity(os.stat(self.lock_file)) == mine + except FileNotFoundError: + return + except OSError: + ours = False + if not ours: + seen = self._read_lock_file(self.lock_file) + if seen is False: + return + if seen is not None and self._line is not None and seen[1] != self._line: + return # not ours any more + try: + os.unlink(self.lock_file) return - try: - os.unlink(self.lock_file) - return - except FileNotFoundError: - return - except PermissionError: - if time.monotonic() >= deadline: + except FileNotFoundError: + return + except PermissionError: + if time.monotonic() >= deadline: + break + time.sleep(0.002) + except OSError: break - time.sleep(0.002) + + try: + with open(self.lock_file, 'w') as f: + f.write(f"released {os.getpid()}\n") except OSError: - break + pass + self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") - try: - with open(self.lock_file, 'w') as f: - f.write(f"released {os.getpid()}\n") - except OSError: - pass - self.logger.warning(f"utils.files.PathLock - the soft lock file {self.lock_file} could not be removed and was marked released") + finally: + if fd is not None: + try: + os.close(fd) + except OSError: + pass @staticmethod def _identity(st): @@ -648,12 +1102,26 @@ def _identity(st): return None return (st.st_ino, st.st_dev) - def _unlink_own_file(self): - """Remove the lock file, unless the file at the path is no longer the one this lock took (someone - removed it under us and another process created a new one: that one is theirs).""" - if self._ident is not None: + def _own_identity(self): + """The identity of the file this lock's descriptor holds. It is the one seen when the lock was taken - + unless a note was written into the file since: a file system without real inodes can give a file + another number at its first write (exFAT on macOS: an empty file has a placeholder until `write_note` + fills it), and the old number would say "not ours" for our own file - the lock file stayed behind + after every lock with a note. Then the descriptor is asked again, now.""" + if not self._noted: + return self._ident + try: + return self._identity(os.fstat(self._fd)) + except (OSError, TypeError, ValueError): + return self._ident + + def _unlink_own_file(self, ident): + """Remove the lock file, unless the file at the path is no longer the one this lock holds (someone + removed it under us and another process created a new one: that one is theirs). ident: the identity + of our file (`_own_identity`, taken while the descriptor is open).""" + if ident is not None: try: - if self._identity(os.stat(self.lock_file)) != self._ident: + if self._identity(os.stat(self.lock_file)) != ident: return except OSError: return diff --git a/cmeta/utils/sys.py b/cmeta/utils/sys.py index 23f4d69..c652bc1 100644 --- a/cmeta/utils/sys.py +++ b/cmeta/utils/sys.py @@ -160,6 +160,17 @@ def sanitize( full_package_name = f"{cat_name}.{pkg_name}" full_module_name = f"{full_package_name}.{module_name}" + # A module of the engine's own package (category_api_v1.py: the base commands of every category) is + # imported by its real name, once. Loaded by its path under a name made from its folders ("cmeta.cmeta. + # category_api_v1"), its relative imports made a SECOND copy of the engine's modules in the process - + # with a state of their own: the path lock's registries among them, so a soft lock asked for a folder + # was unknown to the copy that wrote the artifacts + engine_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + engine_module = os.path.normcase(module_dir) == os.path.normcase(engine_dir) + if engine_module: + full_package_name = __name__.rsplit('.', 2)[0] + full_module_name = f"{full_package_name}.{module_name}" + timestamp = os.path.getmtime(module_path) if module_path in module_cache: @@ -168,29 +179,33 @@ def sanitize( return {"return": 0, "cache": cached} try: - # Ensure category package exists - if cat_name not in sys.modules: - spec = importlib.machinery.ModuleSpec(cat_name, loader=None, is_package=True) - pkg = importlib.util.module_from_spec(spec) - pkg.__path__ = [category_dir] - sys.modules[cat_name] = pkg - - # Ensure api subpackage exists - if full_package_name not in sys.modules: - spec = importlib.machinery.ModuleSpec(full_package_name, loader=None, is_package=True) - pkg = importlib.util.module_from_spec(spec) - pkg.__path__ = [module_dir] - sys.modules[full_package_name] = pkg - - # Load the plugin module - spec = importlib.util.spec_from_file_location(full_module_name, module_path) - module = importlib.util.module_from_spec(spec) - - module.__package__ = full_package_name - module.__file__ = module_path - - sys.modules[full_module_name] = module - spec.loader.exec_module(module) + if engine_module: + module = importlib.import_module(full_module_name) + + else: + # Ensure category package exists + if cat_name not in sys.modules: + spec = importlib.machinery.ModuleSpec(cat_name, loader=None, is_package=True) + pkg = importlib.util.module_from_spec(spec) + pkg.__path__ = [category_dir] + sys.modules[cat_name] = pkg + + # Ensure api subpackage exists + if full_package_name not in sys.modules: + spec = importlib.machinery.ModuleSpec(full_package_name, loader=None, is_package=True) + pkg = importlib.util.module_from_spec(spec) + pkg.__path__ = [module_dir] + sys.modules[full_package_name] = pkg + + # Load the plugin module + spec = importlib.util.spec_from_file_location(full_module_name, module_path) + module = importlib.util.module_from_spec(spec) + + module.__package__ = full_package_name + module.__file__ = module_path + + sys.modules[full_module_name] = module + spec.loader.exec_module(module) cache_data = { "python_module": module, diff --git a/cmeta/version.py b/cmeta/version.py index 17dbf6f..a92c65d 100644 --- a/cmeta/version.py +++ b/cmeta/version.py @@ -6,5 +6,5 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. """ -__version__ = "0.34.2" -__release_date__ = "2026-10-10" +__version__ = "0.35.0" +__release_date__ = "2026-10-11" diff --git a/cmeta/versions.json b/cmeta/versions.json index 40263a9..669d0a3 100644 --- a/cmeta/versions.json +++ b/cmeta/versions.json @@ -101,6 +101,10 @@ { "version": "0.34.2", "version_date": "20261010" + }, + { + "version": "0.35.0", + "version_date": "20261011" } ] } diff --git a/docs/async-and-concurrency.md b/docs/async-and-concurrency.md index 2675fc8..fefa167 100644 --- a/docs/async-and-concurrency.md +++ b/docs/async-and-concurrency.md @@ -249,13 +249,53 @@ takes `fcntl.flock` (per open file description, so threads exclude each other too), or POSIX record locks (`fcntl.lockf`) where the file system refuses `flock` (NFS and some network mounts), each with the identity check. Where the file system refuses both (some FUSE, 9p and SMB mounts) it falls back to a -**soft lock**: the file is created exclusively and holds `pid host time`; a -stale one (its process dead on this host, or older than 10 minutes when left -by another host) is removed by the waiter that finds it. In every mode the -threads of one process are serialized on the path first. A busy lock is polled -with a delay that doubles from 1 ms to 25 ms until the timeout. A folder used -by two operating systems at once (a Windows tree plugged from WSL) is best -effort: each side locks with its own mechanism. +**soft lock**: the file is created exclusively and holds `pid host time +system`. The holder keeps the file open and rewrites the time in its line +every 2 seconds (the heartbeat: a daemon thread, the line rewritten in place +so the file is never empty). A stale lock file - its pid dead on this system, +or its line unchanged for 20 seconds on the waiter's own monotonic clock (the +holder is on another system or host, or stopped beating) - is taken over by one +of the waiters that find it. Nothing is read from the clock of the drive or of +another machine (a network share, a Windows drive inside WSL2 keep other +clocks): the `system` word (a short hash of the kernel's boot id and the pid +namespace on Linux, of the machine id on Windows and macOS) tells the waiter +whether the pid is one it can check - Windows and WSL2 share the host name and +have different pids, a container shares the kernel of its host and has its own +pids. An empty lock file, which nothing dates, is a leftover once a waiter has +seen it empty for 2 seconds. The waiter does not remove the stale file: it +makes its own file exclusively under a name that belongs to that stale file +(`.take-`, so one waiter only gets to make it), checks +that the file at the path is still the one it judged, and puts its own in its +place with one rename - the path is never free in between. A holder checks +that the file at the path is its own after it took the lock, and removes it at +release only when it still is; a forked child never releases its parent's lock. +A lock file written by cMeta before 0.35.0 (three words, no heartbeat) is +judged as it was then: the pid when the host name is this one, else its age +against 10 minutes. In every mode the threads of one process are serialized on +the path first. A busy lock is polled with a delay that doubles from 1 ms to +25 ms until the timeout. + +**A folder written by several systems at once** - a Windows tree plugged from +WSL2 or from a container through a bind mount, a network share mounted on two +machines - has no lock between those systems with the OS locks: each system's +lock is invisible to the others (measured on one folder of a Windows drive seen +by Windows, WSL2 and a container: no exclusion in any direction, while an +exclusive create is honoured by all). Such a home or repository asks for the +soft lock: `CMETA_SOFT_LOCK_PATHS=[...]` (`*`: +everywhere) names folders for a process, `lock: soft` in a repository's +`_cmr.yaml` names the repository for every system that plugs it (the engine +registers it when the repository is plugged and at the start of every process, +from the index of repositories; `utils.files.use_soft_lock_under` is the API). +With it, the systems exclude each other through the exclusive create, a holder +that dies on one system is replaced from another after the 20 seconds of +silence, and a live holder is never taken over (on Windows its open file cannot +even be replaced). Everything else keeps the OS locks at the same cost. One +thing the lock does not change: every system keeps its own index of the +repository in its own home, and a find answers from that index - what another +system created or changed in the shared repository is seen here after +`cx repo reindex ` (or `cx reindex `); the files +in the repository are always the current ones, and every update reads and +rewrites the artifact's file under the lock, so no update is lost. ### The index lock @@ -296,7 +336,10 @@ updates, and no stale `.lock` files left behind. The lock's own tests (`tests/core_tests/test_utils_files_path_lock.py`) run eight processes and sixteen threads through one counter, replay the removed-file race deterministically, kill a holder, and exercise the soft lock and its stale -files; `tests/benchmarks/benchmark_engine.py` measures the cost. +files - with a drive whose clock is off in both directions, and with six +processes that keep taking over the lock files of "crashed" holders side by +side without ever being two inside the lock; `tests/benchmarks/benchmark_engine.py` +measures the cost. ### What is *not* concurrency-safe diff --git a/docs/en/api/cmeta_utils_common.rst b/docs/en/api/cmeta_utils_common.rst index 6422b33..eece534 100644 --- a/docs/en/api/cmeta_utils_common.rst +++ b/docs/en/api/cmeta_utils_common.rst @@ -10,6 +10,8 @@ See the cMeta COPYRIGHT and LICENSE files in the project root for details. Functions --------- +.. autofunction:: cmeta.utils.common.ask + .. autofunction:: cmeta.utils.common.build_sort_key .. autofunction:: cmeta.utils.common.check_params diff --git a/docs/en/api/cmeta_utils_files.rst b/docs/en/api/cmeta_utils_files.rst index e89107e..6666e4f 100644 --- a/docs/en/api/cmeta_utils_files.rst +++ b/docs/en/api/cmeta_utils_files.rst @@ -18,6 +18,8 @@ Classes Functions --------- +.. autofunction:: cmeta.utils.files.acquire_with_notice + .. autofunction:: cmeta.utils.files.apply_sharding_to_path .. autofunction:: cmeta.utils.files.ask_to_delete diff --git a/docs/en/api/cmeta_utils_net.rst b/docs/en/api/cmeta_utils_net.rst index 230bd4b..f6c42d7 100644 --- a/docs/en/api/cmeta_utils_net.rst +++ b/docs/en/api/cmeta_utils_net.rst @@ -15,3 +15,8 @@ Functions .. autofunction:: cmeta.utils.net.download .. autofunction:: cmeta.utils.net.unify_request + +Constants +--------- + +.. autodata:: cmeta.utils.net.RESUME_RECORD_SUFFIX diff --git a/docs/en/api/index.rst b/docs/en/api/index.rst index 7bff7a2..f8424fb 100644 --- a/docs/en/api/index.rst +++ b/docs/en/api/index.rst @@ -1,7 +1,7 @@ -API Reference (v0.34.2) +API Reference (v0.35.0) ======================= -This section contains the complete API reference for cMeta v0.34.2. +This section contains the complete API reference for cMeta v0.35.0. .. toctree:: :maxdepth: 2 diff --git a/docs/en/index.rst b/docs/en/index.rst index 7f3af71..42fe80e 100644 --- a/docs/en/index.rst +++ b/docs/en/index.rst @@ -1,11 +1,11 @@ cMeta Documentation =================== -Welcome to cMeta v0.34.2 documentation! +Welcome to cMeta v0.35.0 documentation! .. toctree:: :maxdepth: 2 - :caption: cMeta v0.34.2 Contents: + :caption: cMeta v0.35.0 Contents: home diff --git a/docs/history.md b/docs/history.md index 07d4a8b..355c9e6 100644 --- a/docs/history.md +++ b/docs/history.md @@ -82,7 +82,7 @@ produces APA and BibTeX automatically. The BibTeX entry: author = {Fursin, Grigori}, title = {{cMeta (Common Meta Framework)}}, year = {2026}, - version = {0.34.2}, + version = {0.35.0}, license = {Apache-2.0}, url = {https://github.com/cTuningLabs/cmeta}, note = {cTuning Labs} diff --git a/docs/installation.md b/docs/installation.md index e030f47..596ca7c 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -157,8 +157,8 @@ repositories) into these commands. **A specific version**, to pin or to go back: ```bash -uv tool install --force "cmeta[server]==0.34.2" -pip install "cmeta==0.34.2" +uv tool install --force "cmeta[server]==0.35.0" +pip install "cmeta==0.35.0" ``` **Moving an install from git to PyPI** (or back) is a reinstall from the other diff --git a/docs/releasing.md b/docs/releasing.md index a51a958..f8ef769 100644 --- a/docs/releasing.md +++ b/docs/releasing.md @@ -32,7 +32,7 @@ gh workflow run test-core.yml --repo cTuningLabs/cmeta --ref dev gh run list --repo cTuningLabs/cmeta --workflow test-core.yml --limit 1 # then: gh run watch --repo cTuningLabs/cmeta ``` -The CI workflow runs only on request; it covers Linux, Windows and macOS with Python 3.9 and 3.14. +The CI workflow runs only on request; it covers Linux, Windows and macOS with Python 3.9, 3.14 and 3.15. ## 2. Bump the version diff --git a/docs/using-cmeta.md b/docs/using-cmeta.md index 5c25bb8..50fe758 100644 --- a/docs/using-cmeta.md +++ b/docs/using-cmeta.md @@ -37,7 +37,11 @@ Environment variables worth knowing: `CMETA_FAIL_ON_ERROR=yes`, `CMETA_INDEX_LOCK_TIMEOUT=` (how long a write to the index waits for a running `cx --reindex`; 600), `CMETA_LOCK_TIMEOUT=` (how long a writer waits for the lock of a file -held by another process; 30, with a notice after 3). +held by another process; 30, with a notice after 3), +`CMETA_SOFT_LOCK_PATHS=[...]` (the folders - a home, +a repository - that several systems write at once through a shared drive, whose +lock files use the soft lock; `*` for everywhere; a repository can ask for it in +its own `_cmr.yaml` with `lock: soft` - see the concurrency guide). --- @@ -820,12 +824,25 @@ cx repo plug --path= # Detach without deleting the files: cx repo unplug # unregister a repo +cx repo unplug # ... by the path of its folder (also when the folder cannot be reached) cx repo unplug # unregister repo detected in current directory ``` `plug` is `get_` with `--local --path=...`. `unplug` is `delete` with the `--unplug` flag, which unregisters without touching the folder. +**A repository that cannot be reached stays registered.** When the index is built +and the folder of a registered repository is not there - a detached USB drive, a +network share that is down - or its `_cmr.yaml` cannot be read, the repository is +left out of the index, its line in `repos.json` is kept and marked +`unreachable: true`, and a notice names it. Nothing has to be plugged again: the +next index run of any kind (`cx --reindex`, a `pull` or `plug` of another +repository, `cx repo reindex `) finds it back and indexes it. `cx repo unplug +` forgets it for good. (Before 0.35.0 such a repository was dropped from +`repos.json` without a word.) `cx repo plug --keep` goes further for a removable +drive: it stores the descriptor, so the repository stays in the repository index +while the drive is away. + ### 7.10 Everyday repo commands ```bash @@ -1571,7 +1588,8 @@ stay as it is, for instance in a folder that another machine syncs. inside it — does what `cx repo pull` and `cx repo plug` do once a repository is in place: its `_cmr.yaml` is re-read and its artifacts are indexed again, and the records of the other repositories stay as they are. A repository whose folder is -not there (a detached drive) is refused and nothing changes. +not there (a detached drive) is refused and nothing changes; once the folder is +back, `cx repo reindex ` - or any other index run - indexes it again (§7.9). `cx --reindex` rebuilds every index file from scratch — after a `CMETA_HOME` wipe, a bulk change across repositories, or when in doubt. It's safe, idempotent diff --git a/pyproject.toml b/pyproject.toml index aafefed..d4aaf4b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,6 +34,7 @@ classifiers = [ "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", + "Programming Language :: Python :: 3.15", ] dependencies = [ "pyyaml", diff --git a/tests/core_tests/test_utils_files_path_lock.py b/tests/core_tests/test_utils_files_path_lock.py index bb1e5d7..7659786 100644 --- a/tests/core_tests/test_utils_files_path_lock.py +++ b/tests/core_tests/test_utils_files_path_lock.py @@ -74,6 +74,72 @@ print('RELEASED', flush=True) ''' +STALE_RACE_WORKER = r''' +import logging, os, random, subprocess, sys, time +from cmeta.utils import files +logging.disable(logging.CRITICAL) +folder, rounds, crash = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) +lock_file, marker = os.path.join(folder, 'x.lock'), os.path.join(folder, 'in-the-lock') +files._lock_soft_paths.add(lock_file) +files.LOCK_SOFT_CHECK_SECONDS = 0.0 # every failed attempt judges the file: the waiters meet as often as they can +gone = 2147483644 # no process has this pid (a pid that was used could be given to a new process meanwhile) +if files._lock_pid_alive(gone): + child = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL) + child.wait() + gone = child.pid +rnd = random.Random(os.getpid()) +together = 0 +for i in range(rounds): + lock = files.PathLock(lock_file).acquire(timeout=120) + try: + fd = os.open(marker, os.O_WRONLY | os.O_CREAT | os.O_EXCL) + except FileExistsError: + together += 1 # another process is inside the lock + except OSError: + pass # Windows: the marker of the previous holder is still being removed + else: + os.close(fd) + time.sleep(0.0003) + try: + os.unlink(marker) + except OSError: + pass + if rnd.randrange(crash) == 0: + # the crash of a holder: its file stays, with the line of a process that is gone; it beats no more and + # has no file open any more + lock._stop_heartbeat() + with open(lock_file, 'w') as f: + f.write('%d %s %.3f %s\n' % (gone, files._lock_host_name(), time.time() - 5, files._lock_system_id())) + os.close(lock._fd) + lock._fd = None + lock._remove_soft_file = lambda: None + lock.release() +print('OK together', together) +''' + +SOFT_ASKED = r''' +import os, sys +from cmeta.utils import files +asked, other = sys.argv[1], sys.argv[2] +a = files.PathLock(os.path.join(asked, 'a.lock')).acquire(timeout=5) +b = files.PathLock(os.path.join(other, 'b.lock')).acquire(timeout=5) +print(a.mode, b.mode, files._lock_system_id()) +a.release() +b.release() +''' + +SOFT_HOLDER = r''' +import sys, time +from cmeta.utils import files +files._lock_soft_paths.add(sys.argv[1]) +lock = files.PathLock(sys.argv[1]).acquire(timeout=60) +print('LOCKED', lock.mode, flush=True) +sys.stdin.readline() # holds the lock until told to release it (or killed) +lock.release() +print('RELEASED', flush=True) +time.sleep(float(sys.argv[2])) +''' + HOOKED_WAITER = r''' import os, sys, time from cmeta.utils import files @@ -356,8 +422,8 @@ def test_soft_lock_holds_pid_and_host_and_is_removed_on_release(soft, caplog): lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' with open(soft) as f: - pid, host, when = f.read().split() - assert int(pid) == os.getpid() and host == files._lock_host_name() + pid, host, when, system = f.read().split() + assert int(pid) == os.getpid() and host == files._lock_host_name() and system == files._lock_system_id() with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) @@ -387,14 +453,21 @@ def test_a_stale_soft_lock_of_a_dead_process_is_removed(soft, caplog, monkeypatc f.write(f'{gone.pid} {files._lock_host_name()} {time.time():.3f}\n') caplog.set_level('WARNING', logger='cmeta.utils.files') - # Fresh, the file of a dead pid is a release in flight: waited for + # Fresh, the file of a dead pid is a release in flight: waited for - whatever time the drive gave the file + # (an hour old here: a drive whose clock is behind this machine's) + hour_ago = time.time() - 3600 + os.utime(soft, (hour_ago, hour_ago)) with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) - old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS - os.utime(soft, (old, old)) - lock = files.PathLock(soft).acquire(timeout=5) + # Taken long enough ago - by the time its holder wrote into it - it is the leftover of a crash: one try + # takes the lock (the probe of a cache entry is such a try), whatever time the drive gave the file + with open(soft, 'w') as f: + f.write(f'{gone.pid} {files._lock_host_name()} {time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS:.3f}\n') + hour_ahead = time.time() + 3600 + os.utime(soft, (hour_ahead, hour_ahead)) + lock = files.PathLock(soft).acquire(timeout=0) assert lock.mode == 'soft' with open(soft) as f: assert int(f.read().split()[0]) == os.getpid() @@ -415,23 +488,42 @@ def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkey monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) with open(soft, 'w') as f: f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + old = time.time() - 3600 + os.utime(soft, (old, old)) # the drive's time for the file says nothing with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) - old = time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS - os.utime(soft, (old, old)) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 2 * files.LOCK_SOFT_DEAD_MIN_AGE_SECONDS:.3f}\n') lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) - # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its content at once): - # fresh, it is waited for; old, it is cleared + # An empty lock file is a leftover of an OS-lock attempt (a soft holder writes its line at once). A waiter + # dates it by its own watch: seen empty for less than the time, it is waited for - a drive five minutes + # behind this machine changes nothing; seen empty for longer, it is taken over + five_minutes = time.time() - 300 open(soft, 'w').close() + os.utime(soft, (five_minutes, five_minutes)) monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 30.0) with pytest.raises(TimeoutError): - files.PathLock(soft).acquire(timeout=0.3) - monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + files.PathLock(soft).acquire(timeout=0.6) + assert os.path.isfile(soft) + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.4) + started = time.monotonic() lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.4 <= time.monotonic() - started < 3 + lock.release() + assert not os.path.exists(soft) + # a single try has not watched the file: it leaves it ... + open(soft, 'w').close() + monkeypatch.setattr(files, 'LOCK_SOFT_EMPTY_STALE_SECONDS', 0.0) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert os.path.isfile(soft) + # ... unless the drive dates it beyond any difference of clocks (the limit of a lock left by another host) + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=0) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) @@ -440,17 +532,291 @@ def test_a_soft_lock_marked_released_or_an_old_empty_one_is_cleared(soft, monkey def test_a_soft_lock_of_another_host_is_stale_only_after_the_age(soft, monkeypatch): with open(soft, 'w') as f: f.write(f'12345 some-other-host {time.time():.3f}\n') + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) # a drive twenty minutes behind: the lock was taken a moment ago with pytest.raises(TimeoutError): files.PathLock(soft).acquire(timeout=0.3) assert os.path.isfile(soft) - old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS - os.utime(soft, (old, old)) + # Taken longer ago than the limit, by the time its holder wrote into it + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {old:.3f}\n') + ahead = time.time() + 3600 + os.utime(soft, (ahead, ahead)) # ... and a drive an hour ahead changes nothing lock = files.PathLock(soft).acquire(timeout=5) assert lock.mode == 'soft' lock.release() assert not os.path.exists(soft) + # A holder whose clock is ahead of this machine's (its time lies in our future) is not stale for it + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time() + 3600:.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + os.unlink(soft) + + +def test_a_soft_lock_line_without_a_time_is_dated_by_its_file(soft): + # A lock file that this code did not write (no time in its line, a time that is no number, digits that are + # no number): the time of the file is all there is, and no content makes the lock fail + for content in ('12345 some-other-host\n', '12345 some-other-host nan\n', '²³ x y\n', 'pid 1 reindexes\n'): + with open(soft, 'w', encoding='utf-8') as f: + f.write(content) + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + old = time.time() - 2 * files.LOCK_SOFT_STALE_SECONDS + os.utime(soft, (old, old)) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + lock.release() + assert not os.path.exists(soft) + + +def stale_line(seconds_ago=5.0, system=True): + """The line of a soft lock whose holder is gone: a pid that no process has, on this system, taken a while + ago (`system=False`: the line of a holder of cMeta before 0.35.0, without the system).""" + gone = 2147483644 + if files._lock_pid_alive(gone): + child = subprocess.Popen([sys.executable, '-c', 'pass'], stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + child.wait() + gone = child.pid + tail = f' {files._lock_system_id()}' if system else '' + return f'{gone} {files._lock_host_name()} {time.time() - seconds_ago:.3f}{tail}\n' + + +def test_a_stale_soft_lock_is_replaced_in_one_step_and_never_moved_from_the_path(soft, monkeypatch): + """The waiter replaces the stale file by its own in one rename: the path is never free in between, and no + file is renamed AWAY from the path (the previous way - away, checked, put back when it was not the judged + one - left the path free for a third process meanwhile).""" + moves = [] + real_rename, real_replace = os.rename, os.replace + + def rename(src, dst, *args, **kwargs): + moves.append((src, dst)) + return real_rename(src, dst, *args, **kwargs) + + def replace(src, dst, *args, **kwargs): + moves.append((src, dst)) + return real_replace(src, dst, *args, **kwargs) + monkeypatch.setattr(os, 'rename', rename) + monkeypatch.setattr(os, 'replace', replace) + + with open(soft, 'w') as f: + f.write(stale_line()) + lock = files.PathLock(soft).acquire(timeout=0) + assert lock.mode == 'soft' + assert len(moves) == 1 and moves[0][0].startswith(soft + '.take-') and moves[0][1] == soft + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_a_waiter_a_step_behind_leaves_the_file_of_the_one_that_took_the_stale_lock(soft, monkeypatch): + """Two waiters judge the same stale file and one takes the lock over. The other, a step behind, finds that + the file at the path is not the one it judged, and leaves it where it is.""" + with open(soft, 'w') as f: + f.write(stale_line()) + # the line of the other waiter: a live process (this one), taken half a second ago - never the very line the + # waiter under test writes in the same millisecond (a line tells its writer by pid, host and time) + fresh = f'{os.getpid()} {files._lock_host_name()} {time.time() - 0.5:.3f}\n' + real = files.PathLock._soft_file_is_stale + seen = {} + + def judged_then_overtaken(self, path, read): + stale = real(self, path, read) + if stale and path == soft and not seen: + with open(soft + '.other', 'w') as f: # the other waiter's take-over, right after the judgement + f.write(fresh) + os.replace(soft + '.other', soft) + seen['ino'] = os.stat(soft).st_ino + return stale + monkeypatch.setattr(files.PathLock, '_soft_file_is_stale', judged_then_overtaken) + + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert seen and os.stat(soft).st_ino == seen['ino'] + with open(soft) as f: + assert f.read() == fresh + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + os.unlink(soft) + + +def test_one_waiter_at_a_time_takes_a_stale_soft_lock_over(soft): + """The file that replaces a stale lock file is made exclusively, under a name of that stale file: a second + waiter finds it and waits. The take file of a waiter that died meanwhile is a leftover, and goes.""" + stale = stale_line() + with open(soft, 'w') as f: + f.write(stale) + probe = files.PathLock(soft) + take = probe._take_file(probe._read_lock_file(soft)) + assert take.startswith(soft + '.take-') + with open(take, 'w') as f: # another waiter (alive: this process) is taking it over right now + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time():.3f}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.6) + with open(soft) as f: + assert f.read() == stale + assert os.path.isfile(take) + + with open(take, 'w') as f: # that waiter died before it replaced the file + f.write(stale_line()) + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' + assert os.listdir(os.path.dirname(soft)) == [os.path.basename(soft)] + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_a_soft_lock_taken_over_is_not_removed_by_its_former_holder(soft): + lock = files.PathLock(soft).acquire(timeout=5) + theirs = f'12345 some-other-host {time.time():.3f}\n' + with open(soft + '.new', 'w') as f: + f.write(theirs) + if POSIX: + os.replace(soft + '.new', soft) # this holder was judged gone (it was not) and its lock taken over + lock.release() + with open(soft) as f: + assert f.read() == theirs + os.unlink(soft) + else: + # Windows: a holder keeps its lock file open, and an open file cannot be replaced - the lock of a holder + # that is alive cannot be taken over at all + with pytest.raises(PermissionError): + os.replace(soft + '.new', soft) + os.unlink(soft + '.new') + lock.release() + assert not os.path.exists(soft) + + +def test_a_live_soft_holder_judged_gone_by_mistake(tmp_path, monkeypatch): + """A waiter that takes a live holder for a dead one (it cannot see its pid: another system on a shared + folder). On Windows it cannot take the lock over - the holder has its file open - and nothing is left behind; + elsewhere it can, and the former holder then leaves the lock of the other alone when it releases.""" + lock_file = str(tmp_path / 'live.lock') + holder = subprocess.Popen([sys.executable, '-c', SOFT_HOLDER, lock_file, '60'], + stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + line = holder.stdout.readline().decode('utf-8', 'replace') + assert line.startswith('LOCKED soft'), line + try: + monkeypatch.setattr(files, '_lock_soft_paths', {lock_file}) + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) + monkeypatch.setattr(files, 'LOCK_SOFT_DEAD_MIN_AGE_SECONDS', 0.0) + if POSIX: + lock = files.PathLock(lock_file).acquire(timeout=5) + with open(lock_file) as f: + assert int(f.read().split()[0]) == os.getpid() + holder.stdin.write(b'release\n') + holder.stdin.flush() + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('RELEASED') + with open(lock_file) as f: # the former holder left it alone + assert int(f.read().split()[0]) == os.getpid() + lock.release() + else: + with pytest.raises(TimeoutError): + files.PathLock(lock_file).acquire(timeout=1) + with open(lock_file) as f: + assert int(f.read().split()[0]) == holder.pid + assert os.listdir(str(tmp_path)) == [os.path.basename(lock_file)] + holder.stdin.write(b'release\n') + holder.stdin.flush() + assert holder.stdout.readline().decode('utf-8', 'replace').startswith('RELEASED') + assert os.listdir(str(tmp_path)) == [] + finally: + holder.kill() + holder.wait(30) + + +@pytest.mark.skipif(not POSIX, reason='on Windows a file cannot be replaced while its maker has it open') +def test_a_soft_lock_file_taken_over_while_it_is_made_is_not_held(soft, monkeypatch): + """A process stopped between creating its soft lock file and writing into it finds, when it goes on, that a + waiter has taken the empty file over: it does not hold the lock, and leaves the file of the other.""" + theirs = f'12345 some-other-host {time.time():.3f}\n' + real_write = os.write + state = {'n': 0} + + def write_after_a_take_over(fd, data): + state['n'] += 1 + if state['n'] == 1: + with open(soft + '.new', 'w') as f: + f.write(theirs) + os.replace(soft + '.new', soft) + return real_write(fd, data) + monkeypatch.setattr(os, 'write', write_after_a_take_over) + + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + monkeypatch.setattr(os, 'write', real_write) + with open(soft) as f: + assert f.read() == theirs + os.unlink(soft) + + +def test_a_soft_lock_file_that_the_file_system_shows_late_is_still_this_lock(soft, monkeypatch): + """A file system that does not show a new file at once: the holder cannot confirm its lock file in its first + attempt. The file with its line, found at the path in the next attempt, is its lock - and when the acquire + gives up before that (a single try), the file goes, or it would keep everybody out while the process lives.""" + real = files.PathLock._open_if_mine + state = {'n': 0} + + def not_there_at_first(self, line): + state['n'] += 1 + return ('gone', None) if state['n'] <= 3 else real(self, line) # the three looks of the first attempt + monkeypatch.setattr(files.PathLock, '_open_if_mine', not_there_at_first) + monkeypatch.setattr(files.PathLock, '_identity', staticmethod(lambda st: None)) # ... and it reports no inode numbers + + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and state['n'] > 3 + with open(soft) as f: + assert int(f.read().split()[0]) == os.getpid() + lock.release() + assert not os.path.exists(soft) + + state['n'] = 0 + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0) + assert state['n'] == 3 and not os.path.exists(soft) + lock = files.PathLock(soft).acquire(timeout=0) # ... and the lock is free for the next one + lock.release() + assert os.listdir(os.path.dirname(soft)) == [] + + +def test_waiters_on_the_soft_lock_of_crashed_holders_never_hold_it_together(tmp_path): + """Six processes take one soft lock sixty times each, and one holder in four "crashes" (its lock file stays, + with the line of a process that is gone): the waiters judge and take over stale files side by side, and + never two of them are inside the lock at once. No take file is left.""" + folder = tmp_path / 'race' + folder.mkdir() + outputs = run_workers(STALE_RACE_WORKER, [(str(folder), 60, 4)] * 6) + for rc, out in outputs: + assert rc == 0 and out.strip().endswith('OK together 0'), out + assert os.listdir(str(folder)) in ([], ['x.lock']) # the last crash may have left its lock file + + +def test_the_lock_file_goes_where_a_file_gets_its_identity_at_the_first_write(tmp_path, monkeypatch): + """exFAT on macOS: an empty file reports a placeholder inode that becomes a real one at the first write. The + lock noted the identity of its (empty) file when it took it, the note changed it, and at release "the file + at the path is not ours" left the lock file behind after every noted lock.""" + real = files.PathLock._identity + + def placeholder_until_written(st): + ident = real(st) + if ident is None: + return None + return (2 ** 64 - 1, ident[1]) if st.st_size == 0 else ident + monkeypatch.setattr(files.PathLock, '_identity', staticmethod(placeholder_until_written)) + + lock_file = str(tmp_path / 'noted.lock') + lock = files.PathLock(lock_file).acquire(timeout=5, note='pid 1 reindexes') + assert lock.mode != 'soft' and files.PathLock.read_note(lock_file) == 'pid 1 reindexes' + lock.release() + assert not os.path.exists(lock_file) + + lock = files.PathLock(lock_file).acquire(timeout=5) # without a note the file stays empty + lock.release() + assert not os.path.exists(lock_file) + def test_soft_lock_is_used_where_the_os_lock_is_refused(tmp_path, monkeypatch, caplog): lock_file = str(tmp_path / 'refused.lock') @@ -468,16 +834,21 @@ def refuse(*args, **kwargs): monkeypatch.setattr(files, '_lock_no_flock_dirs', set()) caplog.set_level('WARNING', logger='cmeta.utils.files') + started = time.monotonic() lock = files.PathLock(lock_file).acquire(timeout=5) assert lock.mode == 'soft' + # the empty file the refused attempt left is taken over at once, not watched like an empty file of unknown origin + assert time.monotonic() - started < files.LOCK_SOFT_EMPTY_STALE_SECONDS / 2 assert lock_file in files._lock_soft_paths assert any('soft lock' in rec.getMessage() for rec in caplog.records) + assert os.listdir(str(tmp_path)) == [os.path.basename(lock_file)] lock.release() assert not os.path.exists(lock_file) again = files.PathLock(lock_file).acquire(timeout=5) # remembered: soft at once assert again.mode == 'soft' again.release() + assert os.listdir(str(tmp_path)) == [] @pytest.mark.skipif(not POSIX, reason='POSIX record locks are the POSIX fallback') @@ -496,6 +867,186 @@ def refuse(*args, **kwargs): assert not os.path.exists(lock_file) +########################################################################################## +# The heartbeat of a soft holder, the system named in its line, and the soft lock asked for + +def test_a_soft_lock_line_names_the_system_and_the_heartbeat_rewrites_its_time(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_HEARTBEAT_SECONDS', 0.05) + lock = files.PathLock(soft).acquire(timeout=5) + with open(soft) as f: + first = f.read() + words = first.split() + assert len(words) == 4 and int(words[0]) == os.getpid() and words[1] == files._lock_host_name() + assert words[3] == files._lock_system_id() and len(words[3]) == 12 + assert id(lock) in files._beat_held and files._beat_thread.is_alive() + + deadline = time.monotonic() + 5 + data = first + while data == first and time.monotonic() < deadline: + time.sleep(0.02) + with open(soft) as f: + data = f.read() + beat = data.split() + assert float(beat[2]) > float(words[2]) and beat[0] == words[0] and beat[1] == words[1] and beat[3] == words[3] + assert len(data) == len(first) and data.endswith('\n') # rewritten in place: the same length, never empty + assert lock._line != first.encode() # the holder knows the line it wrote last + + # the processes of one system agree on its name + out = subprocess.run([sys.executable, '-c', 'from cmeta.utils import files; print(files._lock_system_id())'], + stdin=subprocess.DEVNULL, capture_output=True) + assert out.stdout.decode('utf-8', 'replace').split() == [files._lock_system_id()] + + lock.release() + assert id(lock) not in files._beat_held # the one thread of the process beats on for the other soft locks + assert not os.path.exists(soft) + + +def test_a_silent_holder_of_another_system_is_taken_over_after_the_silence_and_a_beating_one_never(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 0.6) + monkeypatch.setattr(files, '_lock_pid_alive', lambda pid: False) # that pid is dead HERE - but it is a pid of another system + foreign = 'ffffffffffff' + with open(soft, 'w') as f: # taken a moment ago, by its own time: no age rule can take it + f.write(f'12345 some-other-host {time.time():.3f} {foreign}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + started = time.monotonic() + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.6 <= time.monotonic() - started < 4 + lock.release() + assert not os.path.exists(soft) + + # A holder that beats is never taken over - whatever its clock reads (an hour behind here: the time in its + # line is not compared with this machine's clock) + stop = threading.Event() + + def beat(): + n = 0 + while not stop.wait(0.1): + n += 1 + with open(soft, 'r+') as f: + f.write(f'12345 some-other-host {time.time() - 3600 + n / 1000:.3f} {foreign}\n') + with open(soft, 'w') as f: + f.write(f'12345 some-other-host {time.time() - 3600:.3f} {foreign}\n') + beater = threading.Thread(target=beat, daemon=True) + beater.start() + try: + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=1.5) + finally: + stop.set() + beater.join(5) + os.unlink(soft) + + +def test_a_dead_holder_of_this_system_is_known_by_its_pid_before_the_silence(soft, monkeypatch): + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 3600.0) + with open(soft, 'w') as f: + f.write(stale_line()) # four words: a pid of this system that is gone, taken 5 s ago + lock = files.PathLock(soft).acquire(timeout=0) + assert lock.mode == 'soft' + lock.release() + + # ... while a pid of this system that is alive keeps its lock, silent or not + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 5:.3f} {files._lock_system_id()}\n') + with pytest.raises(TimeoutError): + files.PathLock(soft).acquire(timeout=0.3) + + # ... and the same pid of ANOTHER system is not checked here: that holder is judged by its silence alone + monkeypatch.setattr(files, 'LOCK_SOFT_SILENT_SECONDS', 0.4) + with open(soft, 'w') as f: + f.write(f'{os.getpid()} {files._lock_host_name()} {time.time() - 5:.3f} ffffffffffff\n') + started = time.monotonic() + lock = files.PathLock(soft).acquire(timeout=5) + assert lock.mode == 'soft' and 0.4 <= time.monotonic() - started < 4 + lock.release() + assert not os.path.exists(soft) + + +@pytest.mark.skipif(not hasattr(os, 'fork'), reason='fork is a POSIX matter') +def test_a_forked_child_never_releases_the_lock_of_its_parent(tmp_path, monkeypatch): + import warnings + soft_file, own = str(tmp_path / 'soft.lock'), str(tmp_path / 'own.lock') + monkeypatch.setattr(files, '_lock_soft_paths', {soft_file, own}) + for lock_file in (str(tmp_path / 'os.lock'), soft_file): + lock = files.PathLock(lock_file).acquire(timeout=5) + with warnings.catch_warnings(): + warnings.simplefilter('ignore', DeprecationWarning) # the heartbeat thread of the soft holder is running + pid = os.fork() + if pid == 0: # the child: a copy of the holder that is not the holder + try: + lock.release() + code = 0 if os.path.exists(lock_file) and not lock.is_locked else 1 + # ... and a process of its own: the parent's lock is held by another process for it, and it + # takes and beats soft locks of its own (the module's thread locks were made anew for it) + try: + files.PathLock(lock_file).acquire(timeout=0.3) + code = 3 + except TimeoutError as e: + if 'another process' not in str(e): + code = 4 + mine = files.PathLock(own).acquire(timeout=5) + if not mine.is_locked or id(mine) not in files._beat_held or files._beat_thread is None: + code = 5 + mine.release() + if os.path.exists(own): + code = 6 + except BaseException: + code = 2 + os._exit(code) + _, status = os.waitpid(pid, 0) + assert os.WEXITSTATUS(status) == 0, lock_file + assert lock.is_locked and os.path.exists(lock_file) + with pytest.raises(TimeoutError): # still held + files.PathLock(lock_file).acquire(timeout=0.2) + lock.release() + assert not os.path.exists(lock_file) + + +def test_the_soft_lock_is_used_under_the_folders_the_environment_names(tmp_path): + asked, other = tmp_path / 'shared', tmp_path / 'own' + asked.mkdir() + other.mkdir() + + def modes(value): + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + env.pop('CMETA_SOFT_LOCK_PATHS', None) + if value is not None: + env['CMETA_SOFT_LOCK_PATHS'] = value + out = subprocess.run([sys.executable, '-c', SOFT_ASKED, str(asked), str(other)], env=env, + stdin=subprocess.DEVNULL, capture_output=True) + assert out.returncode == 0, out.stderr.decode('utf-8', 'replace') + a, b, system = out.stdout.decode('utf-8', 'replace').split() + assert system == files._lock_system_id() + return a, b + + os_mode = 'windows' if not POSIX else 'flock' + assert modes(None) == (os_mode, os_mode) + assert modes(str(asked)) == ('soft', os_mode) + assert modes(os.pathsep.join([str(other), str(asked)])) == ('soft', 'soft') + assert modes('*') == ('soft', 'soft') + assert sorted(os.listdir(str(asked))) == [] and sorted(os.listdir(str(other))) == [] + + +def test_use_soft_lock_under_a_folder_and_not_next_to_it(tmp_path, monkeypatch): + monkeypatch.setattr(files, '_lock_soft_dirs', []) + folder = tmp_path / 'repo' + (folder / 'sub').mkdir(parents=True) + (tmp_path / 'repo2').mkdir() + assert files.use_soft_lock_under(str(folder)) == {'return': 0} + files.use_soft_lock_under(str(folder)) # once is enough + assert len(files._lock_soft_dirs) == 1 + + inside = files.PathLock(str(folder / 'sub' / 'x.lock')).acquire(timeout=5) + beside = files.PathLock(str(tmp_path / 'repo2' / 'x.lock')).acquire(timeout=5) # the same letters, another folder + outside = files.PathLock(str(tmp_path / 'x.lock')).acquire(timeout=5) + assert inside.mode == 'soft' and beside.mode != 'soft' and outside.mode != 'soft' + for lock in (inside, beside, outside): + lock.release() + assert not any(name.endswith('.lock') for root, dirs, names in os.walk(str(tmp_path)) for name in names) + + ########################################################################################## # The engine's functions diff --git a/tests/core_tests/test_utils_files_writes.py b/tests/core_tests/test_utils_files_writes.py index d6ed062..8d35e0c 100644 --- a/tests/core_tests/test_utils_files_writes.py +++ b/tests/core_tests/test_utils_files_writes.py @@ -148,7 +148,10 @@ def test_a_symbolic_link_is_written_through(tmp_path): real = str(tmp_path / 'real.json') link = str(tmp_path / 'link.json') assert files.safe_write_file(real, {'a': 1})['return'] == 0 - os.symlink(real, link) + try: + os.symlink(real, link) + except OSError as e: + pytest.skip(f'the file system has no symbolic links ({e.__class__.__name__}: FAT, exFAT)') assert files.safe_write_file(link, {'a': 2})['return'] == 0 assert os.path.islink(link) assert json.load(open(real)) == {'a': 2} diff --git a/tests/internal_repo_tests/test_integration_repo_soft_lock.py b/tests/internal_repo_tests/test_integration_repo_soft_lock.py new file mode 100644 index 0000000..9e36f64 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_soft_lock.py @@ -0,0 +1,130 @@ +""" +Integration tests of the soft lock a repository asks for: a `_cmr.yaml` with `lock: soft` - a repository that +several systems write at once through a shared drive (a Windows tree plugged from WSL2 or from a container, a +network share), where the OS locks of the systems do not see each other - makes every lock file under that +repository use the soft lock: in the process that plugs it, and in every process that starts after it (the +engine registers it from the index of repositories at its start). Other repositories and the home keep the OS lock. + +Each test runs against a fresh in tmp_path so it does not touch the user's real cMeta state. + +Licensed under the Apache License, Version 2.0. +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import logging +import os +import subprocess +import sys + +import pytest + +from cmeta import CMeta +from cmeta.utils import files + +POSIX = sys.platform != 'win32' +OS_MODE = 'flock' if POSIX else 'windows' + +LATER_PROCESS = r''' +import os, sys +from cmeta import CMeta +from cmeta.utils import files +home, shared, own = sys.argv[1:4] +cm = CMeta(home=home) +r = cm.access({'category': 'repo', 'command': 'find', 'arg1': 'shared', 'con': False}) # any first access of the engine +assert r['return'] == 0, r +a = files.PathLock(os.path.join(shared, 'z.lock')).acquire(timeout=5) +b = files.PathLock(os.path.join(own, 'z.lock')).acquire(timeout=5) +c = files.PathLock(os.path.join(home, 'z.lock')).acquire(timeout=5) +print(a.mode, b.mode, c.mode) +a.release() +b.release() +c.release() +''' + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR', 'CMETA_SOFT_LOCK_PATHS'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + monkeypatch.setattr(files, '_lock_soft_dirs', []) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid, lock=None): + repo = tmp_path / 'extra' / name + repo.mkdir(parents=True) + text = f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n' + if lock is not None: + text += f'lock: {lock}\n' + (repo / '_cmr.yaml').write_text(text, encoding='utf-8') + return repo + + +def plug(cm, repo): + r = access(cm, category='repo', command='plug', arg1=str(repo)) + assert r['return'] == 0, r.get('error') + + +def mode_under(folder): + lock = files.PathLock(os.path.join(str(folder), 'probe.lock')).acquire(timeout=5) + mode = lock.mode + lock.release() + assert not os.path.exists(os.path.join(str(folder), 'probe.lock')) + return mode + + +def test_a_repository_that_asks_for_the_soft_lock_gets_it_and_the_others_keep_the_os_lock(cm, tmp_path): + shared = make_repo(tmp_path, 'shared', 'a1b2c3d4e5f60718', lock='soft') + own = make_repo(tmp_path, 'own', 'b1b2c3d4e5f60718') + assert mode_under(shared) == OS_MODE # not plugged yet: nothing asked + plug(cm, shared) + plug(cm, own) + + assert mode_under(shared) == 'soft' # the process that plugged it + (shared / 'sub').mkdir() + assert mode_under(shared / 'sub') == 'soft' + assert mode_under(own) == OS_MODE + assert mode_under(tmp_path) == OS_MODE # the home keeps the OS lock + + # a process that starts later knows it from the index of repositories, with its first access of the engine + env = dict(os.environ) + env['PYTHONIOENCODING'] = 'utf-8' + out = subprocess.run([sys.executable, '-c', LATER_PROCESS, str(tmp_path), str(shared), str(own)], env=env, + stdin=subprocess.DEVNULL, capture_output=True) + assert out.returncode == 0, out.stderr.decode('utf-8', 'replace') + assert out.stdout.decode('utf-8', 'replace').split() == ['soft', OS_MODE, OS_MODE] + + +def test_the_writes_of_the_engine_in_such_a_repository_take_the_soft_lock(cm, tmp_path, caplog): + shared = make_repo(tmp_path, 'shared', 'a1b2c3d4e5f60718', lock='soft') + own = make_repo(tmp_path, 'own', 'b1b2c3d4e5f60718') + plug(cm, shared) + plug(cm, own) + caplog.set_level(logging.DEBUG) + + for repo, mode in ((shared, 'soft'), (own, OS_MODE)): + caplog.clear() + r = access(cm, category='log', command='create', arg1=f'{repo.name}:note-{repo.name}', meta={'x': 1}, yaml=True) + assert r['return'] == 0, r.get('error') + r = access(cm, category='log', command='update', arg1=f'{repo.name}:note-{repo.name}', meta={'y': 2}) + assert r['return'] == 0, r.get('error') + taken = [rec.getMessage() for rec in caplog.records if 'PathLock - acquired' in rec.getMessage()] + inside = [m for m in taken if os.path.normcase(str(repo)) in os.path.normcase(m)] + assert inside, taken + assert all(m.endswith(f'({mode})') for m in inside), inside + + assert not any(name.endswith('.lock') for root, dirs, names in os.walk(str(tmp_path)) for name in names) + + +def test_a_repository_that_asks_for_nothing_else_is_untouched(cm, tmp_path): + other = make_repo(tmp_path, 'other', 'c1b2c3d4e5f60718', lock='os') # any other value: the default + plug(cm, other) + assert mode_under(other) == OS_MODE + assert files._lock_soft_dirs == [] diff --git a/tests/internal_repo_tests/test_integration_repo_unreachable.py b/tests/internal_repo_tests/test_integration_repo_unreachable.py new file mode 100644 index 0000000..4f582c4 --- /dev/null +++ b/tests/internal_repo_tests/test_integration_repo_unreachable.py @@ -0,0 +1,305 @@ +""" +Integration tests for a registered repository whose folder cannot be reached when the index is built - a detached +drive, a share that is down, a descriptor that cannot be read: it stays registered (repos.json keeps its line, +marked "unreachable"), is left out of the index with a notice, comes back by itself with the next index run once +its folder is there again, and can be forgotten by its path with `cx repo unplug `. Before, the repository +was dropped from repos.json without a word and had to be plugged again. + +Each test runs against a fresh in tmp_path so it does not touch the user's real cMeta state. + +Licensed under the Apache License, Version 2.0. +See the cMeta COPYRIGHT and LICENSE files in the project root for details. +""" + +import json +import os +import pickle + +import pytest + +from cmeta import CMeta + + +@pytest.fixture() +def cm(tmp_path, monkeypatch): + for var in ('CMETA_HOME', 'CMETA_HOME2', 'VIRTUAL_ENV', 'CONDA_PREFIX', + 'CMETA_DEBUG', 'CMETA_VERBOSE', 'CMETA_FAIL_ON_ERROR'): + monkeypatch.delenv(var, raising=False) + monkeypatch.setenv('CMETA_HOME', str(tmp_path)) + return CMeta(home=str(tmp_path)) + + +def access(cm, **params): + params.setdefault('con', False) + return cm.access(params) + + +def make_repo(tmp_path, name, repo_uid, where='extra'): + repo = tmp_path / where / name + repo.mkdir(parents=True) + (repo / '_cmr.yaml').write_text(f'artifact: {name},{repo_uid}\ncategory: repo,f4f792ab40c7498f\n', encoding='utf-8') + return repo + + +def plug(cm, repo, **params): + r = access(cm, category='repo', command='plug', arg1=str(repo), **params) + assert r['return'] == 0, r.get('error') + + +def make(cm, alias, repo, category='log', **meta): + r = access(cm, category=category, command='create', arg1=f'{repo}:{alias}', meta=meta, yaml=True) + assert r['return'] == 0, r.get('error') + return r['meta']['artifact'] + + +def find(cm, ref=None, category='log', **params): + return access(cm, category=category, command='find', arg1=ref, **params) + + +def uids(r): + return sorted(a['cmeta_ref_parts']['artifact_uid'] for a in r.get('artifacts', [])) + + +def repos_json(tmp_path): + return json.loads((tmp_path / 'repos.json').read_text(encoding='utf-8')) + + +def same(path): + return os.path.normcase(os.path.normpath(str(path))) + + +def registered(tmp_path, repo): + """The entry of a repository in repos.json (None when it is not listed).""" + for path, entry in repos_json(tmp_path).items(): + if same(path) == same(repo): + return entry + return None + + +def marked(tmp_path, repo): + """True / False: the repository is listed, with / without the "unreachable" mark. None: it is not listed.""" + entry = registered(tmp_path, repo) + return None if entry is None else bool(entry.get('unreachable')) + + +def index_data(tmp_path, category): + with open(tmp_path / 'index' / f'{category}.pkl', 'rb') as f: + return pickle.load(f) + + +def reindex_all(cm, **params): + r = cm.repos.index(clean=True, **params) + assert r['return'] == 0, r.get('error') + + +def detach(repo): + away = repo.with_name(repo.name + '-detached') + repo.rename(away) + return away + + +def test_an_unreachable_repository_stays_registered_and_comes_back(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, repo) + plug(cm, other) + uid = make(cm, 'on-the-drive', 'mine') + other_uid = make(cm, 'elsewhere', 'other') + order = [same(p) for p in repos_json(tmp_path)] + + away = detach(repo) + + # A full reindex while the drive is away: the line stays where it was, marked; a notice says what to do + reindex_all(cm, con=True) + err = capsys.readouterr().err + assert 'cMeta notice' in err and 'cannot be reached' in err and 'cx repo unplug' in err and str(repo) in err + assert [same(p) for p in repos_json(tmp_path)] == order + assert marked(tmp_path, repo) is True + assert marked(tmp_path, other) is False + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 16 + assert find(cm, 'on-the-drive')['return'] == 16 + assert uids(find(cm, 'elsewhere')) == [other_uid] + + # ... and again: nothing changes, the file is not rewritten for it + before = (tmp_path / 'repos.json').read_bytes() + reindex_all(cm) + assert (tmp_path / 'repos.json').read_bytes() == before + assert capsys.readouterr().err == '' # without the console, the notice goes to the log only + + # The drive is back: the next index run - of any kind - indexes it again and clears the mark + away.rename(repo) + reindex_all(cm) + assert marked(tmp_path, repo) is False + assert [same(p) for p in repos_json(tmp_path)] == order + assert uids(find(cm, 'on-the-drive')) == [uid] + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 0 + + +def test_an_incremental_index_drops_its_records_and_brings_them_back(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + local_uid = make(cm, 'in-local', 'local') + + away = detach(repo) + + # The incremental index of another operation (here: a third repository is plugged) notices it: registered, + # marked, and its records are out of the index - a find must not hand out artifacts whose folder is away + third = make_repo(tmp_path, 'third', '3333333333333333') + plug(cm, third) + assert marked(tmp_path, repo) is True + assert find(cm, 'on-the-drive')['return'] == 16 + assert uid not in index_data(tmp_path, 'log')['uids'] + assert uids(find(cm, 'in-local')) == [local_uid] + + # Back, and another incremental operation runs (a fourth repository is plugged): the repository is indexed + # with it, although nobody named it + away.rename(repo) + fourth = make_repo(tmp_path, 'fourth', '4444444444444444') + plug(cm, fourth) + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'on-the-drive')) == [uid] + assert uids(find(cm, 'in-local')) == [local_uid] + assert access(cm, category='repo', command='find', arg1='mine')['return'] == 0 + + +def test_repo_reindex_by_its_path_brings_an_unindexed_repository_back(cm, tmp_path, monkeypatch): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + away = detach(repo) + reindex_all(cm) + assert marked(tmp_path, repo) is True + + # Still away: refused, nothing changes + r = access(cm, category='repo', command='reindex', arg1=str(repo)) + assert r['return'] > 0 + assert marked(tmp_path, repo) is True + + away.rename(repo) + r = access(cm, category='repo', command='reindex', arg1=str(repo)) # not by its alias: the index does not know it + assert r['return'] == 0, r.get('error') + assert [same(p) for p in r['paths']] == [same(repo)] and r['artifacts'] == [] + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'on-the-drive')) == [uid] + + # ... also from inside it + detached_again = detach(repo) + reindex_all(cm) + detached_again.rename(repo) + monkeypatch.chdir(repo / 'log') + r = access(cm, category='repo', command='reindex', arg1='.') + assert r['return'] == 0, r.get('error') + assert uids(find(cm, 'on-the-drive')) == [uid] + + +def test_unplug_by_path_forgets_an_unreachable_repository(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, repo) + plug(cm, other) + make(cm, 'on-the-drive', 'mine') + other_uid = make(cm, 'elsewhere', 'other') + away = detach(repo) + reindex_all(cm) + + # By its alias it is not known any more; by its path it is forgotten + assert access(cm, category='repo', command='unplug', arg1='mine')['return'] == 16 + assert marked(tmp_path, repo) is True + r = access(cm, category='repo', command='unplug', arg1=str(repo), con=True) + assert r['return'] == 0, r.get('error') + assert [same(p) for p in r['unregistered_paths']] == [same(repo)] and r['deleted_artifacts'] == [] + assert 'Unregistered' in capsys.readouterr().out + assert registered(tmp_path, repo) is None + assert marked(tmp_path, other) is False and uids(find(cm, 'elsewhere')) == [other_uid] + assert away.is_dir() and (away / 'log' / 'on-the-drive').is_dir() # the folder is nobody's business + + # A folder that is no registered repository is still "not found", and a folder inside one does not unplug it + assert access(cm, category='repo', command='unplug', arg1=str(tmp_path / 'extra'))['return'] == 16 + assert access(cm, category='repo', command='unplug', arg1=str(other / 'log'))['return'] == 16 + assert marked(tmp_path, other) is False + + # A reachable, indexed repository can be unplugged by its path too + r = access(cm, category='repo', command='unplug', arg1=str(other)) + assert r['return'] == 0, r.get('error') + assert registered(tmp_path, other) is None and find(cm, 'elsewhere')['return'] == 16 + assert other.is_dir() + + +def test_plugging_it_again_is_not_needed_but_works(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'on-the-drive', 'mine') + away = detach(repo) + reindex_all(cm) + away.rename(repo) + + plug(cm, repo) # the old habit after a drive came back + assert marked(tmp_path, repo) is False + assert len([p for p in repos_json(tmp_path) if same(p) == same(repo)]) == 1 + assert uids(find(cm, 'on-the-drive')) == [uid] + + +def test_a_descriptor_that_cannot_be_read_does_not_unregister_the_repository(cm, tmp_path, capsys): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + uid = make(cm, 'art', 'mine') + descriptor = repo / '_cmr.yaml' + good = descriptor.read_text(encoding='utf-8') + + for broken, why in (('artifact: [unclosed\n', 'cannot be read'), ('category: repo,f4f792ab40c7498f\n', 'names no artifact')): + descriptor.write_text(broken, encoding='utf-8') # a slip while editing it by hand + reindex_all(cm, con=True) + assert why in capsys.readouterr().err + assert marked(tmp_path, repo) is True + assert find(cm, 'art')['return'] == 16 + + descriptor.write_text(good, encoding='utf-8') + reindex_all(cm) + assert marked(tmp_path, repo) is False + assert uids(find(cm, 'art')) == [uid] + + +def test_a_kept_repository_keeps_its_stored_descriptor_through_rewrites(cm, tmp_path): + """`cx repo plug --keep` stores the descriptor of a repository on a removable drive, so that it stays in the + repository index while the drive is away. The stored copy must survive a rewrite of repos.json made while the + repository is reachable (it was left out then, and the next time the drive was away the repository was dropped).""" + kept = make_repo(tmp_path, 'on-usb', '1111111111111111') + plug(cm, kept, keep=True) + assert registered(tmp_path, kept)['keep_repo_meta']['artifact'] == 'on-usb,1111111111111111' + + # another repository goes away and comes back: repos.json is rewritten twice while the kept one is reachable + other = make_repo(tmp_path, 'other', '2222222222222222') + plug(cm, other) + away = detach(other) + reindex_all(cm) + away.rename(other) + reindex_all(cm) + assert registered(tmp_path, kept)['keep_repo_meta']['artifact'] == 'on-usb,1111111111111111' + + # the drive of the kept repository is away: still registered, still in the repository index, never "unreachable" + kept_away = detach(kept) + reindex_all(cm) + entry = registered(tmp_path, kept) + assert entry['meta']['keep'] is True and 'unreachable' not in entry + assert access(cm, category='repo', command='find', arg1='on-usb')['return'] == 0 + kept_away.rename(kept) + + +def test_the_lines_of_repos_json_keep_their_order_and_only_the_mark_is_added(cm, tmp_path): + repo = make_repo(tmp_path, 'mine', '1111111111111111') + plug(cm, repo) + before = repos_json(tmp_path) + assert len(before) >= 2 # the internal repository, local, mine + + away = detach(repo) + reindex_all(cm) + after = repos_json(tmp_path) + assert list(after) == list(before) # the same lines in the same order + for path in before: + expected = dict(before[path], unreachable=True) if same(path) == same(repo) else before[path] + assert after[path] == expected, path + + away.rename(repo) + reindex_all(cm) + assert repos_json(tmp_path) == before and list(repos_json(tmp_path)) == list(before) # and back to what it was