Repository navigation
See what runs and workers are doing: Overview, Runs, Incidents and Workers on server-side facts - #47
Merged
Merged
Conversation
added 30 commits
October 9, 2026 02:45
Keep incomplete listing metadata from vetoing admitted transitions, retain unknown historical schedule and start times, and allow typed called children to inherit test status.
- runs.list measures each visible page against the 8 MiB logical byte ceiling before it reads full runs, and withholds only policy-blocked items ahead of the authoritative read. - Chronological run pages read known times and the unknown-time tail as separate keyset scans, so the scoped time indexes serve every order without sorting the environment. - Step cursors carry the scheduled time and the full instance key only, so the longest valid key still fits. - Step outputs share the 8 MiB page budget, and outputs of agent, tool or yield instance keys stay withheld. - Steps of a run waiting for a platform that runs it answer WV-IR-UNSUPPORTED, as runs.read does. - The legacy step stream is unsorted, so its key cap also bounds the database work. - Token and current authority are each checked and audited once. - The keyword list adapter always pages by ID, and the SDK rejects a query combined with keyword filters. - Run pages select summary columns only, and a timeline reads its run metadata once.
New runs record how they started as `origin` on their `started` event, but history pages and exports treated that field as unclassified. They withheld it with a `classification_unavailable` omission, so replay stopped at the first event with WV-REPLAY-REDACTED and reported the history of every new run as incomplete. Disclose `origin` on `started` events when it is one of the run origins and keep withholding any other value, so replay verifies these runs again.
Incidents opened on a switch or parallel branch output or on a join come from control steps, which incident facts deliberately record without a step ID. Incident lists and reads treated that as an identity that could not be normalized and served these incidents as unavailable placeholders without an ETag, so they could not be shown or resolved. Serve them with the control-step ID stored on the incident and no instance key. Keep withholding every other disagreement, including facts that name a different step or instance than the incident's own key. Document that CLIs and SDKs must be upgraded with the server, which payloads keep their bytes, and that a resolve answer is the stored incident, so clients read it again.
…mbers Kubernetes member listings no longer print the pod status reason, so no value can carry a line break into the listing; evicted pods are already left out by their Failed phase. A Ready condition of Unknown now counts as not ready instead of leaving every member out. Compose checks each container state inside the guarded block, so a state that is not text leaves members out instead of failing the observation. Adds a boundary test at exactly 101 live pods.
…ort release usage workers.list reads each page with one query that groups active leases, filters by presence, draining, release, task type, target and a literal host prefix, lists current workers by default and orders by last contact or registration with keyset cursors bound to every filter. Worker reads and lists add placement, matched from the latest complete and unexpired observation of the targets both the token and the current principal may read, and the time a worker started draining. workers.retire_stale revokes at most 100 workers per command that hold no active lease and whose last contact is older than the threshold, oldest first, locking them with SKIP LOCKED and checking again before revoking; a dry run previews the same batch and both replay by Idempotency-Key. releases.usage counts a release's workers and pinning activations and pages the readable deployment components that run it. The SDK adds typed worker queries, retire_stale_workers and release_usage; the API reference, upgrade guide and changelog describe the new defaults, cursors and older-client effects.
Studio's Workers page asks for every worker, newest registered first, so revoked and long-offline workers stay listed, its revoked toggle works against the new default and rows no longer move between refreshes. The browser mock now answers like the server. The AI status check reads every worker of its release, so one unseen for hours reads as stale. Release usage no longer counts activations whose workflow version is retired, since runs cannot start from them. The changelog and upgrade guide say that earlier Studio hosts and desktop apps get the reduced worker list from an upgraded server, that older clients fail only on placed workers or workers drained since the upgrade, and how placement uniqueness depends on the targets a reader may see. Adds a test for the worker list argument guards.
…recent failures Loop health now ages successes with a monotonic clock and keeps the wall clock only for the times it shows, so a clock step neither hides a stalled loop nor degrades a healthy one. A cycle still running within its loop's cycle budget is on time: 5 seconds for the outbox and broker traversals, 30 seconds for recovery and schedules, and 60 seconds for provider and mailbox polling. A loop also reads degraded when 3 of its last 20 cycles failed, so a failure that alternates with successes, such as one run that keeps failing in a rotating traversal, is no longer hidden. An outbox cycle that finds no free delivery slot no longer counts as a success: like a cycle that waits for readiness, it is neither a success nor a failure, and the outbox reads degraded once no turn has started for more than three intervals.
…ttings operations_summary.read answers an environment's health, load, workers, targets, incidents and up to 20 items that need attention. It checks status.read on the token before opening one read transaction, reloads the caller's grants there, and gives each section a savepoint with two seconds of database time. A section the caller may not read is forbidden before any of its statements run; one that times out or meets a transient database failure is unavailable while the others are still served, and a lost connection, cancellation or any other error fails the request. Targets, jobs, runners and approvals count only the targets both principals may read, approvals also need deployment.approve on the same target, and run figures leave out test runs while task and worker figures measure shared capacity. Ready tasks count only once they are due. operations_telemetry.read answers whether telemetry export is on, the collector host names, the metrics interval and the dropped count, never endpoint paths, credentials or headers. Both reads are in the CLI as weave operations summary and weave operations telemetry, and Studio's host bridge forwards them together with releases.usage.
…ecent drops and pin the summary's access and figure rules
…just-started run first
added 27 commits
October 12, 2026 21:27
…Overview The identity answer that grants the Operate capabilities was registered after the page had loaded, so under load the first read could miss it and leave the Overview entry hidden.
Incident pages ordered by opening time sorted every incident of the environment. Known opening times now walk a scoped opening-time index in either direction, then incidents without a known time follow in ID order, as run pages already do; results and cursors are unchanged.
…vability # Conflicts: # CHANGELOG.md # src/firefly_weave/operations/exports.py # src/firefly_weave/operations/incidents.py # tests/integration/test_terminal_capacity.py
Journey step J5.1 reads the platform summary with the host's grants, waits for healthy background loops, then runs a one-step workflow on an external worker to success and to an incident, and checks the run summaries, steps, incident and worker the operational reads report. The new operations browser suite measures Overview's first meaningful paint and the size of every scheduled refresh of Overview, Runs, Workers and Incidents, checks that a viewer sees the sections they may not read as needing access, and that Runs lists the newest run first without test runs. The shared real-platform helpers move to their own module.
A deterministic dataset of 1,000,000 runs, 50,000 incidents, 500 workers, 10,000 ready tasks and 1,000 large deployment observations is seeded through the owner connection with usage accounting on, and each operational read's p95 is checked against its budget, for every single run summary filter too. A 200,000-run dataset checks that each chronological page walks its scoped index without reading payloads and that list statements do not grow with the page. The upgrade benchmark times the facts migration and compares usage counters with an independent recomputation, and a comparison test holds queue throughput within 5 percent of a baseline. The forbidden-section summary test only counts the records of its own read.
…vability # Conflicts: # CHANGELOG.md # docs/operations/upgrades.md # src/firefly_weave/persistence/migrations.py # studio/src/app/operations/records-view.ts # studio/tests/browser/real-platform.spec.ts # tests/integration/operations_support.py # tests/unit/test_acceptance_journeys.py
The facts migration now follows 0031_operations_key_collation, so a refused upgrade leaves that revision in place and the seeded prior schema is the one it produces.
…inear The facts backfill validates every run's state, activation and request, then classifies each distinct stored artifact once per scope and reads an artifact only the first time it is seen. It analyzes the new run facts after each batch, so later batches stop planning as if the table were empty. The output is unchanged: identical run and incident facts and usage counters on 10,000 and 50,000 runs, now in 17 and 32 seconds instead of 163 and 815. A run's reservation refresh now counts its task facts through a scoped run index, and updates of step, task and incident facts (and any run fact write) are charged to usage without refreshing the run's reservation, since they change no counted fact.
… it counts Task and incident facts are written and removed with their task or incident, whose own refresh already counts them, so of the fact writes only a step fact insert or delete refreshes the run's reservation. Every fact write is still charged to usage. The read benchmark creates its database from the migrated template.
…ich case the one-second target covers
…vability # Conflicts: # CHANGELOG.md # src/firefly_weave/operations/telemetry.py
…piled A run start imported and compiled its workflow artifact a second time to record its listing metadata, inside the transaction that holds the project's operations lock. The start now hands its imported artifact and compiled workflow to the fact projection, and admission accepts an artifact already imported. Measured on the parallel integration module, the start's lock hold is 97 ms at the median against 84 ms on main (it was 132 ms), with the same Python time as main.
… for Workers' Drain, Resume and Revoke and Runs' Cancel and Retry now note the workspace and person before their confirmation opens, and send nothing when either changed while it was open.
releases.usage adds deployments_hidden: true when the reader may read no deployment target, so an empty deployment list is not mistaken for a release nothing runs, and Studio's release detail names the access needed.
…vability # Conflicts: # CHANGELOG.md
The Runs page now lists runs through run summaries and shows a failed read in its own alert, so the test fails that read and checks the page's alert for the plain message, the support code and no raw payload.
The stage now runs more than 7,000 tests. It takes about 15 minutes locally, and shared CI runners have taken from 15 to more than 30 minutes for the same suite on main, so the 30-minute limit failed runs without any test failing.
A bulk Drain, Resume or Revoke compared the person it started for with the copy the page refreshes during change detection, and never waited for a sign-in check still out. After Try again on an unreachable platform, the next worker's command could go out before the check answered with another person, and the server ran it as them. The shell now tracks its identity checks. Before each send, the Workers and Runs bulk loops (and Retry before its POST) wait for any check still out, then compare the person read from the shell right then. A browser test holds the check open to prove nothing is sent meanwhile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Operators get accurate, scoped operational facts and usable Operate pages.
source rows and are charged to usage. They never decide execution.
0032_operations_factscreates them and backfills retained history in bounded batches, classifyingeach distinct stored artifact once. History it cannot establish stays unknown and is never invented.
never recorded as step instances.
run_summaries.listandruns.stepsserve metadata only. Each read checks both the token and the currentprincipal in one read transaction.
runs.listnow defaults to newest started first (order=idkeeps earlier cursors), accepts the summary filters,and leaves out test runs by default.
by status, code, workflow, step, time and test runs.
incidents.readserves a direct link with its revision.workers.listreads each page in one query, with presence, placement and lease counts.workers.retire_stalerevokes offline workers 100 at a time, with a dry run and replay by idempotency key.releases.usageshows where a release runs, and says (deployments_hidden) when the reader may read no target.operations_summary.readreports health, load, workers, targets, incidents and items that need attention.Each section needs its own permission, and a forbidden or unavailable section is never shown as zero.
bulk actions.
confirmation was open when the person changed.
Verification
Real services (local, colima on Apple silicon):
prprofile, on a fresh run, passed every stage, including a new platform-administration step: platformsummary, a workflow on an external worker to success and to an incident, and run, step, incident and worker
facts.
operate-real: Overview's first meaningful paint was 399 ms once Home had loaded (1.9 swhile Home was still reading, recorded, not gated). Every scheduled refresh of Overview, Runs, Workers and
Incidents was under 50,000 bytes (largest about 8 KB).
summaries 6 ms, steps 9 ms, summary 52 ms, workers 6 ms, incidents 7 ms. The budgets are 120, 120, 200, 100 and
100 ms. Query plans walk scoped indexes without sorting or reading payloads.
0032_operations_facts.integration module (6 runs each side):
as main. Before the start fix it was about 132 ms.
start (Kafka, Keycloak, release receipts, TLS, native image) and the load-sensitive tests listed below; the touched
modules were rerun after the final changes.
make check, the Studio gates and the touched browser and PostgreSQL suites passed.Mocked: Studio unit tests (2,007) and Playwright suites (239), with route fixtures.
CI only: Linux/macOS/Windows
Checks,Documentation,Desktop installersandAcceptance.Not verified:
Decisions
and their usage charges (required), 12.5 % more reservation refreshes and 11.8 % more counter updates.
the task or incident they are written with.
catalog.read, as specified: they are counts, never workeridentities. Deployment rows still need
deployment.readon their target.foreign keys hold their locks until it commits, so stop every writer first. Facts add to retained usage, so a
project near its ceiling reaches it a little sooner.
Follow-ups
test_parallelcases send two completions concurrently with no retry. They fail on main and somewhat moreoften here, because completions hold the project lock slightly longer. Workers retry these refusals in production.
Make the test retry them the way the SDK does.
--orderand filters for runs and incidents; loop-healthlast_failed_at; the Details popover on narrowscreens.
second).