Skip to content

Commit 66d0eb3

Browse files
dkackmandkackman
committed
Merge docs/1062-close-out-regional-edit: #1062 design record
2 parents c6a5878 + 56d1819 commit 66d0eb3

2 files changed

Lines changed: 197 additions & 0 deletions

File tree

‎docs/proposals/complete/h3-regional-edit-complete.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
# H3 regional edit: source_video and a mask (#1062)
2+
3+
Written by model `claude-opus-5-5` via provider `anthropic` (close-out,
4+
2026-10-31). Plan v1 approved by Don with its defaults (Q1 per-step blend,
5+
Q2 build smaller, Q3 no paste-back, Q4 names as proposed). Plan v2, written
6+
after Don's ruling on #1077's M-F218 bounce, approved 2026-10-31 with its
7+
defaults (Q5 keep the feature, Q6 document the shifted-σ meaning rather
8+
than re-scale). Two stages, #1077 and #1078, both verified 2026-10-31.
9+
10+
Prior art: einhorn13/mmh3_media (MIT, ComfyUI): `av_edit.py`'s per-stream
11+
masks, `inpaint_prepare.py`, and `h3_tile_refine.py`. The concept was
12+
re-implemented on the diffusers modular pipeline. No code was taken.
13+
14+
## The question
15+
16+
Does the diffusers MiniMax-H3 modular pipeline let the noise or latents be
17+
masked, per stream, during the denoise? The audio half existed already
18+
(`hold_audio`); the video half was the unknown.
19+
20+
## The answer: feasible as dw blocks, no upstream needed
21+
22+
The modular pipeline has no mask input, but it has what a mask needs:
23+
- **per-row noise levels are native** (`row_timestep_plan`): conditioning
24+
and held audio rows already ride clean beside noisy rows;
25+
- **one block writes rows** (`MiniMaxH3LoopSchedulerStep`), so a dw loop
26+
sub-block after `update` can put the kept rows back after every step,
27+
which is ComfyUI's per-stream `noise_mask`;
28+
- **the audio-locked tile refine had already shipped** (#1043 / #1048,
29+
`refine_tiles`).
30+
31+
A `denoise_mask` input on `MiniMaxH3LoopSchedulerStep` would be a clean
32+
upstream PR later, and the dw block would shrink to a pass-through. Not
33+
pursued here.
34+
35+
## Verdict
36+
37+
**Build smaller.** A (engine) and B (template and skill). Audio intervals,
38+
masked tiled refine and masked chained steps were cut, each with a
39+
trigger (below).
40+
41+
## What was built
42+
43+
### Stage A, #1077: engine regional edit
44+
45+
Arguments on an H3 `t2va`, `fl2va` or `ref2va` step:
46+
- `source_video`: the take to edit. It must match the step's `width`,
47+
`height` and `num_frames`. It stands in for a refine's `latents`.
48+
- `mask_video` **or** `mask_image`: white regenerates, binarized at luma
49+
128 through Pillow's `L` conversion (the same rule as
50+
`remove_object.binarize_mask`), max-pooled onto the patch grid (16 px ×
51+
patch size; one latent frame, about 3.4 pixel frames, × patch time
52+
size), then dilated by `mask_grow` cells (a whole number, 0 or more,
53+
default 1). With no mask the whole source is re-denoised.
54+
- `refine_strength`: required, in (0, 1] with a `source_video` (1.0
55+
regenerates the region from noise). Without a source the refine's range
56+
stays (0, 1), with its message unchanged.
57+
- `hold_audio`: normally `source_video` itself; the source mp4's own track
58+
is held.
59+
60+
Code:
61+
- `dw/pipeline_processors/h3_mask.py`: `insert_mask_blocks`, and the
62+
mask-to-rows arithmetic (`binarized`, `regenerate_cells`, `keep_rows`).
63+
- `dw/pipeline_processors/h3_mask_steps.py`:
64+
- the source-latents block before `set_timesteps`, which VAE-encodes the
65+
source on the canvas;
66+
- the mask-blend block in the `denoise` loop after `update`. It writes
67+
each kept row back as the source noised to the next sigma, with the
68+
refine's own noise, so the kept rows end as the source latent;
69+
- `edit_summary`, the regional-edit `log` job event (regenerated/kept
70+
cells, `mask_grow`, `refine_strength`), added after Don's ruling.
71+
- `dw/pipeline_processors/h3_rules.py`: `mask_problems`, `media_problems`
72+
and `edits_region`, shared by validate and `_with_mask` in
73+
`pipeline.py`.
74+
- `dw/hold_audio.py`: `mask_errors`, Check `regional_edit`. It probes the
75+
source and mask through `ValidationContext.probe`, so every refusal
76+
fires at `validate_workflow` and nothing is queued.
77+
- Docs: `WORKFLOW_GUIDE.md` › *H3: regional edit with source_video and a
78+
mask* and *Choosing refine_strength for a regional edit* (the measured
79+
bands and the shifted-σ note). Also the `ARCHITECTURE.md` H3 row.
80+
- Tests: `tests/test_h3_mask.py` (arithmetic, insertion, the blend ending
81+
at the source, rules, static media, strength 1.0, the job event) and
82+
additions to `tests/test_h3_hold_audio.py`.
83+
84+
The output is the decode as-is (Q3: no pixel paste-back), so kept pixels
85+
are near-identical to the source, not bit-identical. Outside-mask SSIM
86+
measured about 0.98 at every strength.
87+
88+
### Plan v2: the measured strength bands
89+
90+
v1 promised that `refine_strength: 0.8` changes the masked region "per the
91+
prompt". On the M-F218 take (`ep6-cold-open.mp4` + `h3-mask/region.png`,
92+
`mask_grow` 1, 5 turbo steps, `video_shift` 6, seed 1062; prompt asking for
93+
a bright red raincoat) it didn't. Don ruled for a sweep:
94+
95+
| strength | inside SSIM | outside SSIM | inside the mask |
96+
|---|---|---|---|
97+
| 0.80 / 0.85 | 0.86 / 0.83 | 0.978 | re-detail only: the source wins |
98+
| 0.90 | 0.78 | 0.977 | a held object is replaced; same person, pose, clothing |
99+
| 0.95 | 0.65 | 0.978 | garments reworked too; face, hair and pose mostly hold |
100+
| 1.00 | 0.44 | 0.975 | the prompt lands, but a fresh generation (identity lost) |
101+
102+
The numbers sit high because `refine_strength` is σ₀ on the shifted grid.
103+
At shift 6 the unshifted start is u = s / (6 − 5s): 0.8 → 0.40,
104+
0.9 → 0.60, 0.95 → 0.76. Q6 kept the semantics (re-scaling would move
105+
`refine_tiles` too). A big recolour of a large region needs 1.0, which is
106+
a documented limit, not a promise.
107+
108+
### Stage B, #1078: template and skill branch
109+
110+
- `workflows/templates/minimax/regional-edit.json`: shape `shot`, traits
111+
`needs-input-media` and `has-audio`. Defaults are 960×544×124,
112+
`refine_strength` 0.9, 5 turbo steps, shift 6, and
113+
`hold_audio: variable:source_video`. Cost **2.1 min on an RTX 3090**,
114+
measured: a cold run took 120-123 s (encode 45 s, denoise 45-47 s,
115+
decode 19 s).
116+
- `minimax-h3` skill: the "fix one region of a good take" branch in
117+
`SKILL.md`, with `references/regional-edit.md`. It covers making the
118+
mask (segment on a frame, or upload), choosing the strength from the
119+
bands, the recolour limit, and the patch-cell grain.
120+
121+
## Deviations from the plan
122+
123+
- **Every refusal is at validate.** The first build checked media only at
124+
the call, so an all-black mask queued a job (the first tester bounce).
125+
`media_problems` now runs at validate on the files that resolve.
126+
- **Strength 1.0 is accepted with a `source_video`**, following "1.0
127+
regenerates from noise". It stays refused without one.
128+
- **The template runs 5 steps, not the sketch's 8**, matching the turbo
129+
LoRA and the measured bands.
130+
- **The acceptance strength moved from 0.8 to 0.9** (plan v2). The red
131+
raincoat is no longer required inside the mask.
132+
133+
## Bounces per stage
134+
135+
- **#1077: one architecture bounce, four tester bounces, then a park.**
136+
- architecture: the mask arithmetic was hand-written instead of on
137+
Pillow, torch and scipy, and `mask_image` took an RGB round trip that
138+
could binarize differently from `mask_video`;
139+
- tester: refusals fired at the run rather than at validate (M-F216), and
140+
M-F214/M-F215 (a)(d) were short. Fixed by validate-time media checks,
141+
holding the source mp4's own track, and strength 1.0 with a source;
142+
- tester: M-F218, the region changed but not per the prompt at 0.8, and
143+
no job event named the edit. The driver parked the stage after four
144+
hand-offs. Don ruled: add the event, and re-plan (v2) on a measured
145+
sweep;
146+
- tester: M-F216 (k), a string `"0.8"` validated clean. Not a code
147+
defect: the carrier's variable default was a number, so the argument
148+
was converted before the rule saw it. Tests were added that pin where
149+
a string reaches the rule;
150+
- tester: M-F217, the latent-refine range message had changed wording
151+
for the no-source case. Restored verbatim, and the exact sentence is
152+
pinned.
153+
154+
Twice an architecture review sent the stage back with no findings and no
155+
record, which looked like a failed reviewer session. Nothing was changed
156+
for either. It verified at `a34b4d15`.
157+
- **#1078: one.** M-F249: the listed cost of 3.0 min was 32% over the
158+
measured 2.05 min. It was recalibrated to 2.1, and the stage verified at
159+
`5e04cab0`.
160+
161+
No `usage:` figures were recorded on the stages, so cost is left out. The
162+
plan's estimate was $6-9.
163+
164+
## Deferred and left open
165+
166+
- **Audio intervals and "audio follows video mask".** Cut until a field
167+
report wants a line re-sung or re-spoken inside a kept take. The
168+
mechanism is the same blend on audio rows, with `refine_problems`'
169+
"every audio row held" rule relaxed.
170+
- **Mask + `refine_tiles`.** Refused (`TILE_CONFLICTS`). It comes back when
171+
a 1080p regional fix is asked for, by cropping the mask per tile the way
172+
`latents` already is.
173+
- **Mask on a chained step.** Refused, until a chained regional-edit
174+
request comes in.
175+
- **The big-recolour case**, where identity holds at strength 1.0. Revisit
176+
with an H3 refiner LoRA (#1065) or an upstream mask input.
177+
- **Strength generality.** The bands come from one take, one seed and 5
178+
turbo steps at shift 6. The guide states those conditions, and the u
179+
formula says which way another shift moves them.
180+
- **Upstream `denoise_mask`** on `MiniMaxH3LoopSchedulerStep`: an
181+
optional diffusers PR. It would reduce the blend block to a
182+
pass-through.
183+
- **Q1's alternative** (pin kept rows clean at t = 1.0) was not tried.
184+
- **Stale v1 cases.** M-F218, M-F222 and the v1 lines of M-F220/M-F221
185+
expect v1's 0.8 / 0.7 / "0.5 / 1.0". The tester asked the curator to
186+
retire or amend them in dkackman/harnest#158. M-F244 to M-F249 are
187+
authoritative.

‎docs/proposals/todo.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,16 @@ remaining deferred fix is recorded in
383383
template were not built. Record, with the design and what would reopen it
384384
(a lighter H3 build, or #1045's guide path):
385385
`complete/chain-follow-video-complete.md`.
386+
- **H3 regional edit** (#1062, stages #1077-#1078), shipped 2026-10-31. An
387+
H3 step takes a `source_video` and a `mask_image` or `mask_video`, with
388+
`mask_grow`. Two dw blocks (a source encode, and a mask blend inside the
389+
denoise loop) regenerate the masked patch cells and keep the rest of the
390+
take and its soundtrack. Strength bands are measured: 0.9 replaces an
391+
object, 0.95 reworks clothing, and 1.0 is a fresh generation. Ships the
392+
`templates/minimax/regional-edit` template and a `minimax-h3` skill
393+
branch. Record, including what was deferred (audio intervals, masked
394+
tiles, chains, upstream `denoise_mask`):
395+
`complete/h3-regional-edit-complete.md`.
386396

387397
## Declined
388398

0 commit comments

Comments
 (0)