Repository navigation
feat(native): add direct image pixel updates - #62
Open
monotykamary wants to merge 2 commits into
Open
monotykamary wants to merge 2 commits into
monotykamary wants to merge 2 commits into
Conversation
Stage owned BGRA overrides on existing img nodes using merged GPUI updates. Preserve image identity and bound upload memory and failed-version retries. Add lifecycle, atlas, pixel, React ordering and animated example regressions.
monotykamary
marked this pull request as draft
September 9, 2026 05:14
Decode file URLs for Windows subprocess imports. Sample decoded screenshots inside actual image bounds and cover larger windows with offset images.
monotykamary
marked this pull request as ready for review
September 9, 2026 05:24
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…hole as a unit, and a tick can say whether it could be drawn by building the units alone (remorses#62, first slice of the patch frames; no patcher yet, so a frame is drawn as before apart from the wrapping): at `Retain::enter` the previous record decides (`Retain::begin_unit`: the element's own motion, transitions, animations or motion values all paint only, `Record::paints_only` noted by `runs_paint_only`, the classification `Dormancy` uses; eligible, not lifted, not dormant, built in a plain context, at most `UNIT_MAX_ELEMENTS` = 256 elements), nothing inside it is skipped (`Kept::InUnit`), the build confirms it (`Retain::end_unit`: no exclusion, no scroller, nothing lifted out of it or positioned against what is outside, not absent), and the outermost such element on a path is wrapped as `gpui::retained(id, key of this frame, prebuilt).patchable()` with the `Taped` and `on_replay_translated` wiring of `retained_root` (`unit_root`, `replay_hook`); it keeps its `FrameEnv` in `Retain::units()`. A layout-only pass, the reference pass of a verified frame and a deferred build make none. `BuildCtx` is made by one `GpuixView::build_ctx` from a `BuildStart`, which `FrameEnv` holds. `GpuixView::build_unit(id, now)` builds one unit from its kept env at an instant with the retention closed (nothing entered, recorded or tapped, `BUILD_EFFECTS` truncated, events drained, `built_at` set, no frame asked for) inside a tape that captures: `Tape::capturing` notes the writes of a prepaint and paint without making them (`tape::is_capturing`, which `Measurements::record` honours for its own bookkeeping), and `Tapes::same_as_recorded` compares it with the tape of the recording. `Retain::find_deltas` is `compute_deltas` (pure) and `commit_deltas`, with peeks of what a frame takes (`Measurements::peek_changed`, `links::peek_changed`, `peek_commanded`), and `GpuixView::preview_tick` / `Retain::preview` answers whether, since the last frame built whole, only time-driven ids changed (dynamic, running motion, motion that ended, not yet built) and each is in a unit or dormant, the root revision is the same, no layout-only pass ran, no scroll animation or scrollbar fade is active, and that frame built no element that follows the clock (`Exclusion::varies_with_time`), and names the units to build. Counters `unitsBuilt` and `unitsOversized` in `getBuildCounters`. Test: unit tests of the unit choice (outermost, paint-only, size cap, exclusion, scroll, lift, absolute, nothing skipped inside), the preview (each decline, dormant, no effect on the next frame), the capturing tape and a page with a colour animation (unit built alone, records untouched, verified frame sound); cargo test --lib passes; hover-frame-work, live-invalidation, retained-frame-sources, retained-spine, retained-fuzz, retained-parity, retained-observers, offscreen-animation-work, css-animations, css-transitions, web-animations, svg-keyframes, inline-motion, span-motion and the motion-* files pass, also with GPUIX_RETAIN_VERIFY=1 (0 mismatches)
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…and painting them again in the frame shown (remorses#62, second slice of the patch frames): `retain/patch.rs` installs a gpui frame patcher (`GpuixView::install_patcher`, once per window, from `render`) that, for a tick (`Window::request_tick`), takes `now` from the clock, asks `preview_tick` (pure) which units changed or declines, steps what a frame steps before it builds (`sync_motion_states`, `sync_motion_values`, the clocks of the dormant animations, their events emitted), builds each unit with `build_unit` and gives them to `Window::repaint_in_place`, which accepts only if every scratch tape is `same_as_recorded`; on success it takes the active flags and asks for the next tick, on any decline it returns false and gpui draws the tick whole in the same draw (the clocks are sampled at an instant, so the frame sees the state it would have and each event was emitted once). The frame that ends a build asks for the next one with `request_next_frame`: `on_next_frame(request_tick)` when units were made (`Retain::has_units`) and patching is on, `request_animation_frame` as before otherwise or while a scroll animation runs. The reference pass of a verified frame wraps the units of the frame it stands for (`begin_unit` no longer refuses it, `add_unit` leaves the plan alone), because after a patch that pass is the frame shown and the next tick patches against it. `GPUIX_PATCH_OFF=1` and test renderer `setPatchEnabled` turn patching off; test renderer `getPatchCounters` (gpui's patchedFrames, patchedUnits and declines by reason, gpuix's unpatchable ticks by kind) and `clockAdvance` (moves the clock without notifying the view, as a live window's clock moves: `clockFastForward` notifies, so the frame after it is never a tick). Limits seen: an `<svg>` is a custom element with `Exclusion::Svg`, so a spinner's shape animation is built with its spine and the roots replayed each tick as before; an element with `opacity` below 1 is a stacking context that gpui lifts out of the recording, so a `motion` opacity loop is no unit; a `width` loop is no unit. Test: animation-frame-work (a keyframes colour, a native motion colour loop, a colour transition at 500 and 5000 nodes: equal elements built, prepainted, painted, primitives, retained and layout counters per tick, patchedFrames == ticks, also with a screen reader on; the picture after 21 ticks equals the frame built whole at the same instant; hover and React commit draw a full frame and patching resumes; animationiteration once per iteration and currentTime per tick across 300 patched ticks; live clock; width, opacity and svg are not patched), also with GPUIX_RETAIN_VERIFY; unit tests of a patched tick (verified) and of a tick declined after its clocks stepped (one animationiteration); cargo test --lib retain passes; offscreen-animation-work, hover-frame-work, live-invalidation, retained-*, css-animations, css-transitions, web-animations, svg-keyframes, inline-motion, span-motion and the motion-* files pass
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…tick is drawn by building the svg alone and rasterising it again in the frame shown (remorses#62, svg slice of the patch frames): the shapes of an `<svg>` are never built, so the svg host notes what runs on them (`shapes_running`: a CSS animation of a shape that `shows_something_new`, `CssAnimations::shape_moves`, or a shape `motion` that is animating) as its own `Dynamic` bits, and `runs_paint_only` of the host also asks every shape below it (`svg_tree::shapes`; the attributes of a shape change the picture of the svg and never its box), so the svg is the outermost element whose running animations all paint only. Nothing else was keeping a small spinner out: `SvgElement::retain_state` already names it settled once it has been painted. What it now says is whether painting the document of this render into the box of the last paint draws the image shown or makes its own in that paint (`SvgPaint::paints_in_place`: same document and colour, or a document that is inline-sized, 4096 bytes and 128 x 128 px), and a big document that moves is excluded (`Exclusion::Svg`, `CustomElement::draws_changes_at_once`) whatever its raster state, since a patched tick would start a job and show the image of the tick before; so only a document that rasterises in paint is a unit, and a big one is a frame built whole as before. An `<svg>` blocks culling around it, so it is never dormant and the offscreen cost of its loop is unchanged (a limit, not changed here). Test: animation-frame-work (an svg stroke dash loop like the button spinner of the FF demo, at 500 and 5000 nodes: equal elements built, prepainted, painted, primitives, retained and layout counters per tick, patchedFrames == ticks, built 1, prepainted 3, painted 3; the picture after 21 ticks equals the frame built whole at the same instant; a 200 x 200 svg is never patched and shows what a frame built whole does once its rasters have landed), also with GPUIX_RETAIN_VERIFY; unit tests of a small svg unit patched under verify, a big one that is no unit, `paints_in_place` and `rasterises_in_paint`; the `retain` test of an svg shape motion gave its duration in ms where seconds are meant and rested only because planning was off there; cargo test --lib passes; svg-*, offscreen-animation-work, hover-frame-work, live-invalidation, retained-*, css-animations, css-transitions, web-animations, inline-motion, span-motion and the motion-* files pass, also with GPUIX_RETAIN_VERIFY=1
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…leton pulse or the colour loop of a positioned box is drawn by building it alone and painting it again in the frame shown (remorses#62, lifted slice of the patch frames): below 1 an opacity makes an effect group, which gpuix lifts at z 0 into the context around it, and a positioned element is lifted at its z-index, so `begin_unit` refused every element that was lifted or lifted something. What a unit may not do is lift something out of it: `Retain::end_unit` looks at the build and refuses an element that holds a lifted descendant unless the element forms a stacking context that takes it in (an effect group, a positioned element with a z-index), and hands the z the element is lifted at to `unit_root`, which wraps the unit with the same `.lift_z(z)` a retained root has, before `.patchable()`. `Record::lifts` is true of every lifted element, so `begin_unit` only refuses an element that lifts something out of it by the record of the build before. gpui patches such a unit when the new element is lifted as the recording was (zed 14c355f37f); the tick at which an opacity is exactly 1 (no context, so no lift) and the one after it, which records the lifted element again, are frames built whole, and the loop patches again from the third. An element with `position: absolute` is still no unit: gpuix hands it to gpui as absolute in an ancestor, which escapes the recording. Test: animation-frame-work (a native motion opacity loop between 0.3 and 0.8, also holding a positioned child with a z-index, a keyframes opacity loop, the colour loop of a relative box with and without a z-index: equal elements built, prepainted, painted, primitives, retained and layout counters per tick at 500 and 5000 nodes, patchedFrames == ticks, the picture after 21 ticks equal to the frame built whole; a native motion loop 1 -> 0.4 -> 1 and a keyframes skeleton pulse: the two ticks around opacity 1 of each loop are drawn whole and the 196 others patched, the picture equal to the frame built whole at every instant around opacity 1; an absolute badge in a relative card is drawn whole), unit test of what a lifted element is a unit for; cargo test --lib passes; animation-frame-work, offscreen-animation-work, hover-frame-work, stacking-context, retained-lifts, retained-fuzz, retained-frame-sources and motion-* pass, also with GPUIX_RETAIN_VERIFY=1
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…enderer (remorses#62): `GpuixRenderer.getPatchCounters()` returns the same `PatchCounters` (`patchedFrames`, `patchedUnits`, the declines gpui counted by reason, the ticks gpuix did not try to patch by kind), so a run in a real window with the display link can tell a patched tick from one drawn whole. `PatchCounters` and its conversion (`patch_counters_js`) move from `test_renderer.rs` to `renderer.rs`, where both renderers use them. Measured with it, r7 (one colour keyframes text above N/5 rows of bordered cells, `startFrameLoop`, 75 Hz display) at 3000, 12000 and 30000 nodes for 8 s each: every tick after the first full frame is patched (patchedFrames == ticks, no decline of either kind), so the live ticks that took 6 ms at 3000 nodes were patched ticks; those 6 ms did not depend on the size of the page and came and went within a run, and were not seen again in ~60 later runs (0.6-1.0 ms at every size).
fdarian
added a commit
to fdarian/gpuix
that referenced
this pull request
Oct 21, 2026
…lace would make units of (remorses#62, the stray "the frame built whole did not record the root" of the verified react suite): `Retain::preview` now looks at the record of each unit it names and says `Unpatchable::Ended` for one whose last build found nothing left that only paints (`Record::is_unit_candidate`, the test `begin_unit` applies to the record of the build before, now shared). The case is the end of an SVG shape animation that holds its end value (`animation: ... forwards`): the frame that finds it over is still built as the unit the frame before found, `CssAnimations::svg_shapes` goes on listing the shape as running, and the next tick previewed the unit and patched it, while the frame built whole after the patch (the verifier's reference) reads the record that frame left, builds the svg as no unit, and has no root to compare, which the verifier reported as a mismatch (svg-keyframes "paints the dash a keyframe moves", seen once in a parallel run: the third verified patch of the test, reproducible in 17 of 20 serial runs of the file with the diagnostic build, none in the earlier 15 runs under 4 busy loops; the tick is a repaint of the end value, so the picture was the same). Not a stale patch of a time-dependent effect: nothing was recorded or lost in the reference (no `NonReusable` reason counted, no redraw before present), the root was never built. Now that tick is a frame built whole, which makes the svg no unit, and the ticks after it have none to patch. Test: the unit test `a_unit_whose_animation_ended_in_the_last_frame_is_not_patched` (preview says `Ended(2)` with the source still listing the element, and the frame after builds no unit) and `the_tick_after_the_frame_that_found_a_shape_animation_over_is_built_whole` in the test renderer (no patch, `assert_sound` clean; with the check off the verifier reports the mismatch with the new message, 21 subtrees built and no reason); cargo test --lib passes (2788); svg-keyframes 0 mismatches in 20 serial and 20 loaded runs, 0 in the full react suite with GPUIX_RETAIN_VERIFY=1 (the one failure is events "display-link wakes" timing out its 5 s spawn on a loaded machine); animation-frame-work, offscreen-animation-work, hover-frame-work, retained-*, stacking-context, motion-*, svg-*, css-*, web-animations and live-invalidation pass with GPUIX_RETAIN_VERIFY=1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two desktop primitives for ordinary retained
<img>nodes:updateImage(elementId, width, height, bgra)stages copied, tightly packed BGRA pixels without encoding them into the mutation JSON or committing React per frame.clearImage(elementId)restores the normal source; source changes, resize, and destruction release the previous override/atlas identity.Same-sized updates preserve
RenderImage::idand use GPUI's in-place atlas update. Normal GPUI image rendering still owns object-fit, clipping, radii, and layout. A root-frame snapshot keeps deferred virtual-list rows consistent; direct changes invalidate native measurements when needed.Scope and dependency
Refs #40. Advance only the Zed submodule to merged remorses/zed#4 (
81c99f816b4a5f69d3c014774068034c24d1d7af), reachable onremorses/zed'sgpuixbranch. No new GPUI source patch or toolchain change.No terminal/PTY/VT code, CEF, new image element, shared-GPU-texture import, global image registry, public cached-frame presentation, or custom painter. Sampling stays linear: GPUI's ordinary
Imghas no nearest-sampling builder at this revision. That belongs in a separate GPUI change rather than a GPUix painting workaround.Ownership and failure contract
The API uses an explicit pixel layout and copy-on-ingress boundary, informed by wgpu texture uploads and Flutter's pixel-buffer lifetime contract, without importing their broader texture APIs.
Dimensions are positive integers, capped at 4096 per axis / 64 MiB before copying. Invalid input preserves the previous image. Storage and staged/uploaded/failed versions belong to each renderer. GPU upload failures fall back to
src(or the normal empty placeholder), do not block other images, and are not retried on unrelated paints; a newly submitted version can retry. Failed versions never reachImg's implicit upload path.README, generated declarations, optional renderer methods, a changeset, and a tiny animated color-field example document the contract. Producers and their scheduling remain application code.
Validation
On macOS Apple Silicon, against the freshly rebuilt release addon:
git diff --checkpassed. New/changed Rust regions are formatted. Fullcargo fmt --checkreports four verified unchanged baseline regions (accessibility.rs,custom_elements/img.rs,custom_elements/input.rs,lib.rs); unrelated formatting is deliberately excluded.The private real-GPU Rust tests use a dev-only, environment-gated subprocess adapter because libtest runs tests on worker threads while AppKit requires main-thread window creation. It is excluded from production builds. PNG decoding dependencies are test-only. Windows/Linux runtime behavior was not exercised locally.
Grok 4.6 reviewed the design, implementation, upload-safety fix, and test harness. Final production and test reviews: APPROVE.
Agent provenance
Harness: Pi coding agent with Fabric
Agent: Main, with isolated
dynamic-image-pr/embedding-primplementation agentsModel:
openai-codex/gpt-6-astraDesign and code review:
xai/grok-4.6User prompts driving these primitive PRs (verbatim)
Perfect. Recall what PRs we wanted to do once the zed PR we made was merged.
Does GPUIx need them baked in? Or leave it up to the user? Our colleague wants to make sure we expose primitives and create small scoped PRs.
Help me add (1) and (2) while considering carefully best practices of related primitives out in the wild (feel free to search the web and check). Make sure grok 4.6 reviews your work and code; it needs to be tight and small in scope and PR, follow code conventions and most importantly good code style. Create those PRs for me.
gh watch the CI and resolve the issues.
I've converted them to a draft. Open them back up once all fixed. Proceed.
CI follow-up
Commit
5375b6bfixes Windows subprocess file-URL conversion and samples pixels inside actual image bounds rather than relative to the entire native window. Additional larger-window/offset regressions retain the strict color checks. Targeted local tests: 76 passed; package build passed. Grok 4.6 approved the test-only changes. All platform builds, macOS/Windows test jobs, and the publish job passed in CI run 34314125957. Marked ready only after the exact updated head was green.