Repository navigation
Conversation
…ock constants, the lock timeout, the notice helper, the install and replace helpers) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
…; downloads check their size and can be continued; questions without a terminal (0.34.1.1)
The cache category (cmeta/internal-repo/category/cache/api/v1.py) derives the state of an entry from its
tags, its files and the lock of its folder - ok, running (the lock of the folder is held: an attempt of the
task engine), crashed (the `tmp` tag of an attempt whose process is gone), failed (the `failed` tag, or a
result with return > 0), broken (a result file that cannot be read, a result missing from an entry the
task engine made - its request record or the ctx file of a finished attempt is there - or the recorded
tool_path / git_path gone; a cache artifact that never carries a result, such as a program's build
workspace, is ok) - and
exposes it: `cx cache classify` (the API the task engine of cmeta-aops uses to serve, resume, wait for or
replace an entry), `cx cache show` prints the state and filters with --state, `cx cache clean` removes by
state (crashed by default; --failed, --broken, --unfinished; --all needs --force; a running entry is never
removed). The files of an entry are the contract with the task engine: cmeta-task-cached-result.json and
cmeta-task-running.json (who runs the current attempt).
A delete (category_api_v1.py) takes the lock of the artifact's folder, removes the FOLDER under it and only
then the record: a folder locked by another process (an attempt in progress) fails the delete as a whole
(it waits CMETA_LOCK_TIMEOUT first) instead of leaving an orphan folder without a record - the first cause
of the "weird orphan cache entries". safe_delete_directory takes the caller's lock (file_lock=) for that.
utils/files: acquire_with_notice is the public name of the notice-then-wait helper (the task engine waits on
a cache entry with it); safe_write_file passes safe_dump to write_file (the ctx file of a cached run).
utils/net download(): what arrived is compared with the size the server announced. A connection that ended
early returned success with a short file (http.client's read(n) returns what it got and raises nothing), and
the caller unpacked a truncated archive; it is an error now ('incomplete', 'size', 'total_size'), the bytes
stay in the file. With the new resume=True (the last parameter, off by default: other callers unchanged) a
file left by an interrupted call is continued with an HTTP range request under If-Range - the strong ETag of
the file, else its Last-Modified date, kept with the URL and the size in "<file>.resume" - so the server
sends the rest only for the same file. The download starts over whenever that cannot be guaranteed (no
record, another URL, no validator or a weak ETag, a server without ranges, a file that changed, a range
answer that does not fit); a server that fails on a range request gets one request for the whole file; the
partial file is replaced only when that answer has arrived; a partial file that holds every byte needs no
request. The result has 'resumed_from'.
utils/common ask(): a question to the person who runs the command. Without a terminal input() raises
EOFError and a command that asked ended in a traceback; the engine's questions (the deletion of an artifact,
a selection among several, --ask) now end in an error that names the flag which answers in advance (-f, -q,
the argument), never in a made-up answer; the optional name of a dated artifact is the empty answer. A
terminal and a piped answer are as before.
Docs: docs/common-commands.md and docs/using-cmeta.md list the states and the clean flags, and the two
helpers.
Tests: tests/internal_repo_tests/test_cache_states.py (the states, show/clean by state, a locked entry is
running and neither cleaned nor deleted - the record stays with the folder, safe_delete_directory with the
caller's lock), tests/core_tests/test_utils_net_download.py (a local server that cuts a transfer, serves
ranges or not, changes its file, fails on ranges, redirects; every case that continues and every case that
starts over), tests/internal_repo_tests/test_no_terminal_questions.py (the helper, the deletion, the
selection, the command line with a closed standard input). The suite is green on Windows, macOS and Linux
(Python 3.9 to 3.14 and the free-threaded 3.14t), in seven Linux images (python 3.9-slim, 3.14-slim, ubuntu
22.04 and 24.04, debian bookworm, fedora 42, alpine 3.21) and under WSL2 on its own disk and on a Windows
drive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Grigori Fursin <gfursin@gmail.com>
New code in this repository carries no copyright line: "Licensed under the Apache License, Version 2.0." and the pointer to the COPYRIGHT and LICENSE files of the project. The eight test and benchmark files added in 0.34.0 and 0.34.1 had taken the header of their neighbours; the two test files of this release were written without it. Docstrings only, no code changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Grigori Fursin <gfursin@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cMeta v0.34.2: the engine's side of point 3 of the determinism work (the cache entry of a run), and what a resumed attempt needs from the engine.
Cache entries have states. The cache category derives the state of an entry from its tags, its files and the lock of its folder:
ok,running(the folder's lock is held by an attempt),crashed(thetmptag of an attempt whose process is gone),failed,broken(a result that cannot be read or is missing from an entry the task engine made, or the recorded tool path gone).cx cache classifyis the API the task engine of cmeta-aops 0.45.0 uses to serve, resume, wait for or replace an entry;cx cache showprints the state and takes--state;cx cache cleanremoves the crashed entries by default (--failed,--broken,--unfinished,--all --force) and never a running one.A delete removes the folder under its lock, then the record. A folder in use by another process (a cache entry being built) makes the delete wait and then fail as a whole, instead of removing the record and leaving an orphan folder.
A download cut by the network is an error, and can be continued.
utils.net.downloadcompares what arrived with the size the server announced (a connection that ended early used to return success with a short file). With the newresume=True(off by default) a partial file is continued with a range request underIf-Range- the strong ETag, else the Last-Modified date, kept in<file>.resume- and the download starts over whenever the server cannot guarantee that the file is the same.A question that nobody can answer is an error of the command. Without a terminal
input()raisedEOFErrorand a command ended in a traceback; the engine's questions go through the newutils.common.ask: an error that names the flag which answers in advance (-f,-q), never a made-up answer.Everything is backward compatible: no command, option or file format changes;
resumeand the new helpers are additions,cx cache cleanwithout flags is narrower (the crashed entries; before: everytmpentry, a running one included).The test and benchmark files added since 0.34.0 now carry the licence pointer only (no copyright line), like the new ones.
Tests:
test_cache_states.py,test_utils_net_download.py,test_no_terminal_questions.py(+ the suite); verified on Windows, macOS and Linux with Python 3.9 to 3.14 and the free-threaded 3.14t, in seven Linux images and under WSL2. See CHANGELOG.md.🤖 Generated with Claude Code