diff --git a/README.md b/README.md index e8d016926..5b7e950e9 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ two front ends on it: an **MCP server**, so Claude Code (or any MCP client) can author, run and inspect generations; and a **web UI** for doing the same by hand. A CLI sits underneath for when you want neither. -**Python 3.10-3.14 | CUDA (NVIDIA) | MPS (Apple Silicon) | CPU** +**Python 3.12-3.14 | CUDA (NVIDIA) | MPS (Apple Silicon) | CPU** @@ -245,3 +245,4 @@ frame interpolation and more. - [Testing](docs/TESTING.md) — Running the test suite - [Releasing](docs/RELEASING.md) — Cutting a release from a version tag - [Agent Loop](docs/AGENT_LOOP.md) — What the automated implementer/tester ticket labels mean, for anyone joining an Issue +- [Attributions](docs/ATTRIBUTIONS.md) — Code included from other projects, and the projects whose ideas dw reimplemented diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index ab1063ff5..15023b2de 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -33,15 +33,16 @@ one to open. | Step progress | `dw/pipeline_processors/progress.py`: `reported_progress_bars` | A pipeline with no step callback reports its denoise steps through its progress bars. | `tests/test_modular_progress.py::test_every_advance_of_the_bar_is_reported` | | Step timing | `dw/pipeline_processors/timing.py`: `StepTimer`, `step_timer`, `timed_step`, `mark` | Only a pipeline step that ran is timed, on the worker's monotonic clock: `step_timer` opens a `StepTimer` for a step with a `pipeline` or `pipeline_reference` and None for any other, which also hides an outer step's timer from a composed child. Load, call, denoise start, each pass and save reach the open timer through `mark`, and nowhere else. Denoise start is never taken inside a compiled region: a whole-compiled denoiser (`_compiled_call_impl` set) has that callable wrapped for the call instead of a forward pre-hook, which `fullgraph: true` cannot trace. A span with a missing mark is null, never guessed. `record_step` puts the reading on the step's manifest entry and `step_end`, and a reused step gets no `timing`. A chain sums its pipeline calls into one `timing` with `segments: N`. `ManifestEntry.timing` declares the shape (`docs/WORKSPACES.md` *Step timing*). | `tests/test_step_timing.py::test_a_missing_denoise_start_nulls_encode_and_first_pass`, `tests/test_step_timing.py::test_a_reused_step_carries_no_timing_even_when_one_is_passed`, `tests/test_step_timing.py::test_a_chain_sums_its_segments_and_takes_one_median_over_both`, `tests/test_step_timing.py::test_a_task_step_run_through_the_step_loop_has_no_timing`, `tests/test_step_timing.py::test_the_denoise_hook_marks_the_first_forward_once_and_removes_itself`, `tests/test_step_timing.py::test_a_fullgraph_compiled_denoiser_runs_and_still_gets_the_start_mark` | | Step-cache skip counts | `dw/pipeline_processors/timing.py`: `CacheCounter`, `counting_forwards`, `report_cache_stats`, `StepTimer.cache_counted`; `dw/pipeline_processors/components.py`: `stateful_cache_context`; `dw/pipeline_processors/mag_calibration.py`: `get_cache_transformer`, `captured_mag_calibration` | For `first_block` and `mag`, a forward hook on the cache transformer counts computed or skipped once per forward, reading the leader hook's `should_compute` inside the cond/uncond `cache_context` the pipeline opened, so classifier-free guidance counts both forwards. `stateful_cache_context` registers the hook for one pipeline call; its `finally` emits `cache_stats` before `_reset_stateful_cache()` and removes the hook after the event. Private diffusers state that has moved gives null counts and one `cache_stats_unavailable` warning, and never fails the run; `taylorseer`, `faster` and `text_kv` report type and config only, with no warning. The denoiser a cache attaches to (`get_cache_transformer`) and the MagCache calibration capture (`captured_mag_calibration`, the end-of-run log record turned into `cache_stats`-side job events, #891/#927) live in `mag_calibration.py`, which `components.py` imports and re-exports for `pipeline.py`, never the reverse. The counts reach the step's `timing.cache` only through `StepTimer.cache_counted`; a chain sums its calls' counts, and a null in any call nulls the sum. A step with no `cache` has no `cache` key and no event (`docs/ACCELERATION.md` *Reading the skip counts*). | `tests/test_cache_stats.py::test_cfg_counts_two_forwards_per_step_and_they_add_up`, `tests/test_cache_stats.py::test_the_event_names_the_step_and_segment_and_goes_out_before_the_reset`, `tests/test_cache_stats.py::test_moved_hook_state_nulls_the_counts_warns_once_and_the_run_finishes`, `tests/test_cache_stats.py::test_the_private_diffusers_names_are_where_the_counter_reads_them`, `tests/test_cache_stats.py::test_the_timer_sums_counts_and_nulls_on_any_null`, `tests/test_cache_stats.py::test_types_without_a_decision_report_type_only_and_no_warning` | +| Vendored model ports | `dw/vendored/` (`audio8_tts_model.py`, `mmaudio_model.py`, `mmaudio_sync_model.py`, `mmaudio_clip.py`) | Upstream model code dw carries is one module per port, pinned to the commit or Hub revision its docstring names, so it can be diffed against its source and re-pinned as a unit; a port reaches dw only for the validated media loaders and is imported only by the task that runs it. Like `dw/community_pipelines/`, the package is outside the architecture ratchet (`EXCLUDED` in `scripts/arch_metrics.py`): a port keeps upstream's shape, which the module and function ceilings are not written for. | `tests/test_arch_metrics.py::test_vendored_model_ports_are_not_counted`, `tests/test_mmaudio_model.py::test_modules_are_imported_only_by_the_sound_effects_task`, `tests/test_mmaudio_sync_model.py::test_module_is_imported_only_by_the_sound_effects_task` | | Community pipelines | `dw/community_pipelines/` (e.g. `pipeline_ltx2_refine.py`: `LTX2RefinePipeline`), `dw/introspection.py`: `community_pipelines`, `load_allowed_class` | A pipeline dw ships is named by its dotted path (`dw.community_pipelines..`), loads untrusted because it is under `dw`, and is listed by `list_pipelines` and resolved by `get_pipeline_signature` - only the modules in that directory, never a caller-chosen import. `LTX2RefinePipeline` VAE-encodes `video` at `width` x `height` and passes it unnormalized as `latents`, because `LTX2Pipeline.prepare_latents` normalizes 5-D latents itself. | `tests/test_ltx2_refine_pipeline.py::TestIntrospection::test_only_the_shipped_modules_resolve`, `tests/test_ltx2_refine_pipeline.py::TestEncode::test_the_latents_are_left_unnormalized_for_the_parent` | | Adding a task | `dw/tasks/registry.py`: `register_command`, `RegistryTable` | A task is a function registered with `@register_command`, and the signature of its implementation is its argument schema. Its validate-time rules are declared on the same decorator - `domains`, `choices`, `static_check`, `whole_numbers` and `media_arguments` - and `TASK_ARGUMENT_DOMAINS`, `TASK_ARGUMENT_CHOICES`, `TASK_STATIC_CHECKS`, `TASK_WHOLE_NUMBER_ARGUMENTS` and `TASK_MEDIA_ARGUMENTS` are read-only views derived from them, so there is no second table to forget (#692). The registry lives in `dw/tasks/registry.py`; `dw/tasks/task.py` re-exports it, and a task module may register its own handler, as `dw/tasks/beats.py` does, provided `dw/tasks/task.py` imports it (`docs/TASKS.md` *Adding a task*). The built-in handlers are grouped by family the same way - `dw/tasks/audio_handlers.py`, `dw/tasks/video_handlers.py`, `dw/tasks/finish_handlers.py` and `dw/tasks/model_handlers.py` (#790) - and `dw/tasks/task.py` keeps the general-purpose commands, the processor dispatch and `Task`. | `tests/test_task_discovery.py::TestDescribeTask::test_signature_becomes_the_schema`, `tests/test_task_registry_rules.py::test_every_derived_table_is_the_registrys_view`, `tests/test_task_registry_rules.py::test_every_static_check_is_registered_to_exactly_one_command`, `tests/test_task_registry_rules.py::test_every_media_argument_is_a_parameter_of_its_command` | -| Task argument domains | `dw/task_domains.py`: `check_arguments`, `task_argument_errors`; `dw/task_problems.py`: `face_track_errors`, `beats_errors`, `cuts_errors`, `lut_errors` | A numeric domain that a signature cannot express is declared on the command's `@register_command(domains=...)`, and it is checked at validation and again at run time. A numeric argument is read by `whole_number` or `real_number`, the only numeric coercion in `dw/tasks/`, and validation applies the same rule (`number_problem`), so a value is refused at both or at neither: `"3.0"` is a whole number, `"3.5"` for one of `whole_numbers` is not, and `true` is no number (#774). `Task.run` coerces every declared argument through `coerce_arguments` before dispatch, so a handler sees a number, and an argument that has no range besides being a number declares `finite`. A command's cross-argument rules are its registered `static_check`, which `task_argument_errors` runs at validation (`attribute_voices`' `voices_argument_errors`, kept in `dw/tasks/voice_attribution.py`, among them). A literal enum choice (`choice_errors`) is refused at validation by the same module. The per-family rules past a single domain live in `dw/task_problems.py`, which reads `task_domains`' helpers and is never imported by it (`task_argument_errors` reaches them only through the registered `static_check`, so `import_cycles` stays 0; #790): the `ingredients_grid` image count against `max_images` (`ingredients_grid_errors`), and `crop_face_track`'s cross-argument rules (`face_track_problems`, `face_track_errors`: crop size a multiple of the step's `multiple` (32 by default), `remainder` below `modulus`, `gate_zero` above `gate_full`, padding and confidence limits), which the task checks again at run time, as `analyze_beats`' are too (`beats_problems`, `beats_errors`: `min_bpm` below `max_bpm`, and anchors that are one kind per list, ascending, with whole beat indexes; `tests/test_analyze_beats.py`), and `plan_cuts`' (`cuts_problems`, `cuts_errors`: a bare-string transcript (`transcript_problem`, which `plan_cuts` also calls at run time), `min_scene_s` above `max_scene_s`, `segment_by: "beat"` without beats), and `apply_lut`'s (`lut_source_problem`, `palette_problem`, `lut_errors`: exactly one of `lut` or `palette`, and a palette of 2..16 `#rrggbb` colours, re-checked at run time by `check_lut_source`), and `check_face_detector_source` vets its Hub repo and `.onnx` file before any download. `select`'s index stays in `task_domains` with the other media-timing rules (`select_index_problem`, `select_rule_problems`: a whole number, at least 0, and below the candidate count when the count is known, i.e. at run time or when `candidates` is a written list). | `tests/test_task_domains.py::TestTheStaticPass::test_a_negative_frame_count_is_refused_at_its_path`, `tests/test_numeric_coercion.py::TestGradeExposure::test_a_string_number_grades_like_the_number`, `tests/test_task_domains.py::TestAtRunTime::test_slice_audio_refuses_a_negative_count`, `tests/test_ingredients_grid.py::TestStaticValidation::test_bad_layout_and_fit`, `tests/test_face_track.py::TestStaticValidation::test_bad_literal_is_an_error_at_its_argument`, `tests/test_face_track.py::TestCropFaceTrack::test_bad_arguments_are_refused_at_run_time`, `tests/test_face_track.py::TestStaticValidation::test_a_declared_grid_is_accepted`, `tests/test_face_track.py::TestDetectorSource::test_traversing_repo_is_refused`, `tests/test_select_validation.py::TestSelectErrors::test_an_out_of_range_literal_index_names_both_ends`, `tests/test_lut.py::TestPaletteAtValidate::test_both_refuse_naming_the_two`, `tests/test_lut.py::TestPalette::test_a_bad_palette_refuses_naming_the_entry` | +| Task argument domains | `dw/task_domains.py`: `check_arguments`, `task_argument_errors`; `dw/task_problems.py`: `face_track_errors`, `beats_errors`, `cuts_errors`, `lut_errors` | A numeric domain that a signature cannot express is declared on the command's `@register_command(domains=...)`, and it is checked at validation and again at run time. A numeric argument is read by `whole_number` or `real_number`, the only numeric coercion in `dw/tasks/`, and validation applies the same rule (`number_problem`), so a value is refused at both or at neither: `"3.0"` is a whole number, `"3.5"` for one of `whole_numbers` is not, and `true` is no number (#774). `Task.run` coerces every declared argument through `coerce_arguments` before dispatch, so a handler sees a number, and an argument that has no range besides being a number declares `finite`. A command's cross-argument rules are its registered `static_check`, which `task_argument_errors` runs at validation (`attribute_voices`' `voices_argument_errors`, kept in `dw/tasks/voice_attribution.py`, among them). A literal enum choice (`choice_errors`) is refused at validation by the same module. The per-family rules past a single domain live in `dw/task_problems.py`, which reads `task_domains`' helpers and is never imported by it (`task_argument_errors` reaches them only through the registered `static_check`, so `import_cycles` stays 0; #790): the `ingredients_grid` image count against `max_images` (`ingredients_grid_errors`), and `crop_face_track`'s cross-argument rules (`face_track_problems`, `face_track_errors`: crop size a multiple of the step's `multiple` (32 by default), `remainder` below `modulus`, `gate_zero` above `gate_full`, `max_frames` at or above `min_frames`, padding and confidence limits), which the task checks again at run time, as `analyze_beats`' are too (`beats_problems`, `beats_errors`: `min_bpm` below `max_bpm`, and anchors that are one kind per list, ascending, with whole beat indexes; `tests/test_analyze_beats.py`), and `plan_cuts`' (`cuts_problems`, `cuts_errors`: a bare-string transcript (`transcript_problem`, which `plan_cuts` also calls at run time), `min_scene_s` above `max_scene_s`, `segment_by: "beat"` without beats), and `apply_lut`'s (`lut_source_problem`, `palette_problem`, `lut_errors`: exactly one of `lut` or `palette`, and a palette of 2..16 `#rrggbb` colours, re-checked at run time by `check_lut_source`), and `check_face_detector_source` vets its Hub repo and `.onnx` file before any download. `select`'s index stays in `task_domains` with the other media-timing rules (`select_index_problem`, `select_rule_problems`: a whole number, at least 0, and below the candidate count when the count is known, i.e. at run time or when `candidates` is a written list). | `tests/test_task_domains.py::TestTheStaticPass::test_a_negative_frame_count_is_refused_at_its_path`, `tests/test_numeric_coercion.py::TestGradeExposure::test_a_string_number_grades_like_the_number`, `tests/test_task_domains.py::TestAtRunTime::test_slice_audio_refuses_a_negative_count`, `tests/test_ingredients_grid.py::TestStaticValidation::test_bad_layout_and_fit`, `tests/test_face_track.py::TestStaticValidation::test_bad_literal_is_an_error_at_its_argument`, `tests/test_face_track.py::TestCropFaceTrack::test_bad_arguments_are_refused_at_run_time`, `tests/test_face_track.py::TestStaticValidation::test_a_declared_grid_is_accepted`, `tests/test_face_track.py::TestDetectorSource::test_traversing_repo_is_refused`, `tests/test_select_validation.py::TestSelectErrors::test_an_out_of_range_literal_index_names_both_ends`, `tests/test_lut.py::TestPaletteAtValidate::test_both_refuse_naming_the_two`, `tests/test_lut.py::TestPalette::test_a_bad_palette_refuses_naming_the_entry` | | Variables | `dw/variables.py`: `set_variables`, `argument_errors`, `resolve_variable_values` | `argument_errors` folds a caller's arguments in exactly as `set_variables` does when a run starts, so a bad name or value is refused before anything is queued. | `tests/test_variables.py::test_argument_errors_reports_a_dict_passed_for_a_string_variable` | | List variables from the CLI | `dw/run.py`, `dw/variables.py`: `get_value` | A `name=value` string given for a list variable is split on commas, so a list of objects can only be passed as JSON over the API or MCP. | `tests/test_variables.py::test_set_variables_list_default_splits_on_comma` | | `for_each` expansion | `dw/for_each.py`: `expand_for_each`, `group_of` | Each entry becomes an ordinary step named `@`, after substitution and before the reference check. | `tests/test_for_each.py::TestNaming::test_member_name_joins_with_at` | | Optional entry field | `dw/for_each.py`: `_optional_item`, `_item_fields` | `{"item": "field", "default": X}` reads an entry field with a fallback; without `default` it is the plain required read, and `list_fields` / `entry_field_warnings` count the field as one entries take. The editor's string scan needs no twin code, only the shared case file. | `tests/test_for_each.py::TestOptionalItem::test_a_present_field_wins_and_an_absent_one_takes_the_default`, `tests/test_ui_twins.py::test_the_engine_decides_each_shared_reference_case` | | Previous results | `dw/previous_results.py`: `get_iterations`, `previous_result_reference_errors` | Several `previous_result:` references in one step multiply into a cartesian product. | `tests/test_previous_results.py::TestGetIterations::test_multiple_references_create_cartesian_product` | -| Previous-result check | `dw/previous_results.py`: `previous_result_reference_errors` | A literal reference to a step that is not earlier in the workflow is a validation error. | `tests/test_previous_results.py::TestStaticReferenceChecking::test_a_step_cannot_reference_itself_or_a_later_one` | +| Previous-result check | `dw/previous_results.py`: `previous_result_reference_errors`, `cross_entry_reference_errors`; the sibling rule is `dw/for_each.py`: `cross_entry_key` (beside `group_of`, which it calls), which the cross-entry check, the earlier-step check and `dw/video_extensions.py`: `video_into_image_errors` all ask | A literal reference to a step that is not earlier in the workflow is a validation error. A for_each member reading another entry of its own list (`shot@a` inside `shot@b`, either spelling, any `.property`) is a validation error too, `cross_entry_reference_errors` (#1101); same-key pairing, other groups and `gather:` stay legal. | `tests/test_previous_results.py::TestStaticReferenceChecking::test_a_step_cannot_reference_itself_or_a_later_one`, `tests/test_previous_results.py::TestCrossEntryReferences::test_the_prefix_spelled_in_an_entry_is_refused` | | Type conversion | `dw/arguments.py`: `realize_args`, `dw/type_helpers.py`: `get_type` | A key ending `_type` or `_dtype` loads a Python object at realize time. The other spellings, and `{}` escaping, are in `docs/WORKFLOW_GUIDE.md`. | `tests/test_type_helpers.py::TestGetType::test_get_type_from_diffusers`, `tests/test_arguments.py::TestRealizeArgs::test_realize_escaped_type_reference` | ## References and libraries @@ -75,6 +76,7 @@ one to open. | Result subfolders | `dw/subfolders.py`: `subfolder_errors`, `dw/security.py`: `SUBFOLDER_PATTERN` | A step's `result.subfolder` is checked for shape at validation. | `tests/test_subfolders.py::TestValidationErrorsIntegration::test_validation_errors_reports_a_bad_subfolder` | | Template subfolder roles | `workflows/templates/`, `dw/workflows/` | Every saving step of a template is marked `final` or `intermediate`, with at least one `final`, and packaged builtins stay unmarked. | `tests/test_template_subfolders.py::test_every_saving_step_of_a_template_names_its_role`, `tests/test_template_subfolders.py::test_the_packaged_builtins_stay_unmarked` | | Frames carrying their rate | `dw/media_types.py`: `FrameList` (re-exported by `dw/tasks/video_utils.py`), `rated_videos` | The one list of frames (or of videos) that carries `.fps` and `.shots`: a video loaded from a file (`dw/argument_media.py`) and a step that knows its rate but would return bare frame lists both answer it - `h3_tiles`' tiled refine `videos`, and every pipeline step whose pipeline reports its own `fps` (MiniMax-H3's property; `dw/output_extraction.py`: `attach_video_frame_rate`, called by `pipeline.py`'s `_run_once` beside `attach_audio_sample_rate`, #1056) - through `rated_videos`, the batch and each video in it, so `pair_audio` and the result writer read `getattr(video, "fps", None)` instead of fitting and playing at the default 8 fps. A pipeline that reports no rate leaves its lists bare. No second class of that name. | `tests/test_h3_tiles.py::TestTileLoopReporting::test_the_videos_are_dws_one_frame_list`, `tests/test_pipeline_components.py::TestVideoFrameRate::test_a_dict_outputs_videos_carry_the_pipelines_rate`, `tests/test_pipeline_components.py::TestVideoFrameRate::test_a_pipeline_with_no_rate_leaves_the_lists_bare`, `tests/test_modular_output_properties.py::TestTheBasesOwnTakeSavesBesideIt::test_the_base_take_pairs_at_its_own_rate_with_no_padding`, `tests/test_video_utils.py` (the `FrameList` cases) | +| Writing a video at its rate | `dw/writers.py`: `encode_video`, `exact_frame_rate` | A fractional rate is written as the exact rational (30000/1001, 24000/1001, 60000/1001), never truncated to a whole rate, and no `dw/` or `dw_mcp/` module imports `encode_video` from `diffusers.utils` - every audio-carrying video write goes through `dw.writers.encode_video` (`dw/result.py`, `dw/pipeline_processors/chain.py`). A declared fractional `result.fps` is therefore accepted by `fps_errors`. | `tests/test_fractional_fps.py::test_encode_video_with_audio_keeps_fractional_rate`, `tests/test_fractional_fps.py::test_no_dw_module_calls_the_diffusers_encode_video_directly`, `tests/test_fractional_fps.py::test_the_diffusers_hook_the_wrapper_swaps_still_exists` | | JSON records beside media | `dw/media_types.py`: `JsonRecord`, `dw/result.py`: `Result.save_artifact` | A dict marked `JsonRecord` is saved whole as one `.json`, whatever content type the step declares, and is never split into one file per key. | `tests/test_face_track.py::test_json_record_is_saved_whole_under_a_video_content_type`, `tests/test_face_track.py::test_the_task_result_saves_crops_as_video_and_track_as_one_json` | | Reading a JSON record back | `dw/locations.py`: `load_json_record` | A task reads a saved JSON record (`fit_to_model`'s, `crop_face_track`'s) only through `load_json_record`: the path is confined like any media path, then the extension must be `.json`, then the size is checked against `MAX_JSON_SIZE`, and only then is it parsed. `fit._read_fit` and `face_track._read_track` call it, never `open`. | `tests/test_locations.py::TestLoadJsonRecord::test_traversal_out_of_the_roots_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_a_non_json_extension_is_refused`, `tests/test_locations.py::TestLoadJsonRecord::test_an_oversized_file_is_refused_before_it_is_read`, `tests/test_fit_to_model.py::TestRestore::test_fit_path_with_traversal_is_refused` | | Shots in a joined video | `dw/shots.py` | Every `AudioVideo` constructor decides what to do with `shots`, and the sample spans are measured from the joined waveform. | `tests/test_shots.py::test_every_audio_video_constructor_site_is_decided`, `tests/test_shots.py::TestConcatVideosShots::test_an_overrun_track_is_measured_not_derived` | @@ -98,7 +100,7 @@ one to open. | Variable constraints | `dw/variable_constraints.py`: `apply_constraints`; the grid arithmetic `aligned` / `aligned_down` | A model's rule about a value is declared in the workflow's `variable_constraints`, and it is checked at validation and again before anything loads. `plan_cuts` (and its static check) rounds render lengths through `aligned` / `aligned_down`, never its own copy, so a planned `num_frames` is one the template's constraint accepts; `crop_face_track` pads its crops to the step's `modulus`/`remainder` grid through `aligned` too (`padding_to_grid`). | `tests/test_variable_constraints.py::TestTheGrid::test_without_snap_an_off_grid_value_is_refused_not_rounded`, `tests/test_variable_constraints.py::TestTheGrid::test_plan_cuts_rounds_through_the_constraint_owner`, `tests/test_face_track.py::TestPadding::test_padding_to_grid_rounds_through_the_constraint_owner`, `tests/test_variable_constraints.py::TestTheRunTimePass::test_an_illegal_value_is_refused_before_anything_loads` | | Bound acknowledgement | `dw/plan.py`: `fingerprint`, `dw/server/admission.py`: `check_bound_acknowledgement` | A bound `acknowledged_cost` is refused with 409 when the plan's fingerprint or its required downloads changed. The fingerprint is keyed only by what can change the output: `DOCUMENTATION_KEYS` (`cost`, `cost_drivers`, `vram_estimate`, descriptions, catalog shape/traits), a constraint's `reason` and a result's role, subfolder and file name are left out, of the workflow and of each composed child; every schema key is sorted into left out or kept. A number digests by value, not spelling (`12.0` as `12`), and the seed stays out (#85). | `tests/test_plan.py::TestFingerprintKeys::test_the_exclusion_lists_are_pinned`, `tests/test_plan.py::TestTheCatalogH3Template::test_a_client_round_trip_keeps_it`, `tests/test_plan.py::TestFingerprintKeys::test_every_top_level_schema_key_is_sorted`, `tests/test_server.py::TestBoundRerun::test_a_rerun_bound_to_a_stale_plan_is_refused`, `tests/test_server.py::TestBoundAcknowledgement::test_a_matching_fingerprint_queues` | | Per-repo gating | `dw/plan_downloads.py`: `_model_info`, `_probe_gate_blocked`, assembled by `dw/plan.py`: `build_plan` | Each gated repo, including a `loras` entry's repo, is probed and reported on its own. | `tests/test_plan.py::TestDownloadsRequired::test_a_gated_repo_this_token_lacks_access_to_is_blocked` | -| Download sources | `dw/plan_downloads.py`: `downloads_required`, `_collect_sources`, `_collect_component_overrides`, `_sub_workflow_paths` (which `plan.estimate` walks children by too; the module never imports `dw/plan.py` back, and `plan.py` re-exports `downloads_required` and `gate_warnings`) | `downloads_required` lists every `model_name`, `loras` repo, single-file URL and `load_components.pretrained_model_name_or_path` override (string, per-component dict, `default`), once each. The base repo stays listed whole: an override's replaced subfolder is not subtracted, so the list is a floor. | `tests/test_plan.py::TestDownloadsRequired::test_a_load_components_repo_override_is_listed` | +| Download sources | `dw/plan_downloads.py`: `downloads_required`, `_collect_sources`, `_collect_component_overrides`, `_sub_workflow_paths` (which `plan.estimate` walks children by too; the module never imports `dw/plan.py` back, and `plan.py` re-exports `downloads_required` and `gate_warnings`) | `downloads_required` lists every `model_name`, `loras` repo (a `loras` entry naming a resolved `weight_name` is listed instead as a `{repo, filename, revision}` file entry built by `file_entries`, sized as that file), single-file URL and `load_components.pretrained_model_name_or_path` override (string, per-component dict, `default`), once each. The base repo stays listed whole: an override's replaced subfolder is not subtracted, so the list is a floor. | `tests/test_plan.py::TestDownloadsRequired::test_a_load_components_repo_override_is_listed` | | Task weights | each fetching task's `*_source(arguments)` (e.g. `dw/tasks/remove_object.py`: `remover_source`); `dw/tasks/weights.py`: `argument_or_default`, `file_entries`, `package_dir`; `dw/tasks/registry.py`: `required_weights_hook`, `register_weights_fallback`; `dw/tasks/image_utils.py`: `processor_weights_hook`; `dw/security.py`: `validate_weight_name`; `dw/plan_downloads.py`: `_collect_task_weights`, `_file_info` | A model-backed task declares `required_weights(arguments)` on its `register_command`, and `downloads_required` lists those repos beside the pipeline `model_name`s (a shared repo once; a reference or local-path argument none). A single-file task lists `{repo, filename, revision}` entries, checked per file against the cache; `restore_faces` also lists facexlib's weights as URL entries. A task's declaration and its loader read the same resolution of which argument wins (a single-file task's `*_source(arguments)`, an overridable whole-repo task's `argument_or_default` call, a fixed repo's module constant), so they cannot differ; `argument_or_default` is the shared helper only where `None` means the default (`remover_source` and `upscaler_source` treat any falsy value as absent; `spandrel_source` has no default). `file_entries` builds a file entry (its name checked by `validate_weight_name`), `package_dir` finds facexlib's weights dir without importing it, and `_file_info` sizes one file. The lookup path is acyclic: `dw/locations.py` reads the registry's tables and the task modules import the location policy and `weights.py`, so the registry imports no task module (an image processor's hook reaches it through `register_weights_fallback`, which `image_utils` calls at import) and `weights.py` imports only `dw/security.py`, where `validate_weight_name` lives for that reason. A task module the hook lookup imports keeps `transformers` out of its module top (`hf_pipeline` is a lazy wrapper, patched by name in the tests), so a validate never pays for it. | `tests/test_task_weights.py`, `tests/test_task_weights.py::TestTheHookPathStaysAcyclic::test_weights_imports_only_the_security_leaf_from_dw`, `tests/test_task_weights.py::TestTheHookPathStaysAcyclic::test_the_registry_imports_nothing_from_dw` | | Observed cost | `dw/server/observed_cost.py`: `declared_drivers`, `ObservedCosts._on_this_card`, `ObservedCosts.card` | `observed` comes only from this box's finished jobs on the named card (the one `route` names, for `plan.estimate`), else the server's own card; `ObservedCosts.card` owns a card's (kind, name, default). Rows are matched by card name whatever index they ran on (a job from before jobs carried a `device` counts only while the server is on the default card), and never writes `cost`. Cold and warm runs are reported separately. | `tests/test_observed_cost.py::TestColdIsNotWarm::test_the_two_are_reported_separately_each_with_its_runs`, `tests/test_observed_cost.py::TestByCard::test_only_rows_from_this_card_count_whichever_index_they_had`, `tests/test_observed_cost.py::TestByCard::test_rows_from_before_jobs_carried_a_device_do_not_count_off_the_default_card` | | The estimate | `dw/plan.py`: `estimate`, `_tempered`; the driver scaling is `dw/cost_scaling.py`: `scale_curated`, `_driver_scaling` | `plan.estimate` quotes the observed figure ahead of the curated one, and blends it toward the curated figure below three runs. A `cost` entry is measured at the workflow's variable defaults; a run moving `num_inference_steps`, `width`/`height` or `num_frames` (per list entry, summed) scales it in proportion (`scale_curated`, which `dw/server/deps.py` reuses for an inherited figure), basis `derived` with `scaled` naming each axis, times the list-length re-price, and that scaled figure is what a thin observed one blends toward (#866). | `tests/test_plan.py::TestObservedEstimate::test_history_beats_a_curated_figure`, `tests/test_plan.py::TestLowConfidenceObservedEstimate::test_a_single_run_blends_toward_the_curated_figure`, `tests/test_plan.py::TestDriverScaling::test_steps_canvas_and_frames_multiply`, `tests/test_plan.py::TestScalingAccuracy::test_within_thirty_percent` | @@ -112,8 +114,8 @@ one to open. | Acceleration warnings | `dw/acceleration_warnings.py`: `acceleration_warnings`, `CACHE_MIN_USEFUL_STEPS`, `CUDA_ONLY_ATTENTION_BACKENDS` (the last entry of `WARNING_CHECKS`) | An acceleration setting that can't pay off (a `first_block`/`mag` cache under `CACHE_MIN_USEFUL_STEPS`, any `taylorseer` cache, `compile.fullgraph` with a cache, a cache on a pipeline class with no transformer or whose transformer lacks `enable_cache()`, a CUDA-only `attention_backend` on a non-CUDA device) is a warning and never a refusal, and an unresolvable value (steps, backend, device) leaves it silent. | `tests/test_validation.py::TestAccelerationWarnings::test_warnings_never_refuse`, `tests/test_validation.py::TestAccelerationWarnings::test_steps_that_resolve_to_nothing_are_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unknown_device_is_silent`, `tests/test_validation.py::TestAccelerationWarnings::test_an_unresolvable_backend_is_silent_on_mps` | | H3 sparse attention | `dw/sparse_attention.py`: `sparse_call`, `install_sparse`, `install_sparse_for_call`, `apply_sparse_settings`, `SparseH3AttnProcessor`; wired from `dw/pipeline_processors/components.py`: `configure_components` and `dw/pipeline_processors/pipeline.py`: `apply_runtime_settings` | Whether a pipeline call runs sparse is decided once per call, all or nothing (the call's first kernel invocation included); it never changes diffusers' process-wide backend and never reaches `AttentionBackendName`; it installs by a processor swap plus forward hooks, never a `forward` wrapper; and `sparse_tau` is a runtime setting, not part of the weights identity (#931). | `tests/test_sparse_attention_processor.py`, `tests/test_sparse_attention_wiring.py`, `tests/test_sparse_attention_tiles.py`, `tests/test_sparse_attention_backends.py` | | H3 adapter partition | `dw/adapter_compatibility.py`; the server's index is `dw/server/deps.py` `lora_index` | The LoRA catalog decides first, on the server only: a `proven` entry's `workflow` refuses a step in the other partition, a `trial`/`rejected` entry's only warns. A file no entry places falls to its name: an FL2VA LoRA on a reference step is refused, and one whose name says neither `ref2v` nor `fl2v` is warned. | `tests/test_h3_adapters.py::TestWhatIsRefused::test_a_keyframe_adapter_on_the_reference_path`, `tests/test_h3_adapters.py::TestWhatIsRefused::test_an_unrecognised_name_is_a_warning_not_a_refusal`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_proven_entry_refuses_the_other_partition`, `tests/test_h3_adapters.py::TestTheCatalogDecidesFirst::test_a_weaker_entry_only_warns`, `tests/test_server_loras.py::TestTheCatalogPlacesAnAdapter::test_a_trial_entry_warns_and_promoting_it_refuses` | -| H3 audio hold, refine, tiled refine and guide layout | Six modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants), `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`) and `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`), which import diffusers when they load, so the two owners import them only inside `blocks()` and `guide_blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline.py` adds its blocks for a `guides` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors`, `refine_strength_errors` | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and they are no-ops without `hold_audio` / `refine_strength`. The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers` | -| IC-LoRA reference scale | `workflows/templates/ltx2/` | The three conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | +| H3 audio hold, refine, tiled refine, guide layout and regional edit | Nine modules, imported one way: `dw/pipeline_processors/h3_rules.py` holds the names and rules, torch- and diffusers-free (`refine_problems`, the per-argument refine rules, `refine_tiles` and the explicit `refine_sigmas` schedule among them, asked before validation and before the call; `tile_plan`, the tile layout on the 32 px grid, shared by validate and the run; `not_h3`, the step rule; `snap_guide_length`, `guide_frame_problem`, `guide_chain_problems`, `guide_end_problem`; `CHAIN_CONTINUITY_MODES`; the chunk constants; and the regional-edit rules `mask_problems`, `media_problems` and `edits_region`, the argument rules shared by validate and `_with_mask`; `media_problems` is asked by validate too, on the files that resolve then, and refuses a `source_video` off the `17n + 5` grid, which the pipeline would otherwise snap up over a padded source), `dw/pipeline_processors/h3_mask_cells.py` holds the mask-to-rows arithmetic (`binarized`, `regenerate_cells`, `keep_rows`), which `h3_mask.py` re-exports and `h3_mask_steps.py` reads from there so the blocks never import their importer, `dw/pipeline_processors/h3_hold.py` puts the hold and refine blocks in (`insert_audio_hold`, `core_denoise_sequences`, `refines`; `HELD_LATENTS_OUTPUT` and `encoded_hold_audio`, the hold block's encode handed back to it), and `dw/pipeline_processors/h3_guides.py` the guide layout (`LAYOUT_ANCHORS`, `insert_guides`), which imports `h3_hold` and never the reverse, and `dw/pipeline_processors/h3_mask.py` the regional edit (`insert_mask_blocks`, `edits_regions`, and the mask-to-rows arithmetic `binarized`, `regenerate_cells` and `keep_rows`, on Pillow's L conversion, torch's adaptive max pool and scipy's dilation, each imported where it runs), which imports `h3_hold` and never the reverse; the blocks themselves, and the arithmetic they run, are `dw/pipeline_processors/h3_hold_steps.py` (`encode_audio_span`, `refine_sigmas`, and `explicit_sigmas`, a step's `refine_sigmas` list run verbatim in place of the shift grid) `dw/pipeline_processors/h3_guide_steps.py` (`splice_guide_rows`) and `dw/pipeline_processors/h3_mask_steps.py` (the source-latents and mask-blend blocks), which import diffusers when they load, so the three owners import them only inside `blocks()`, `guide_blocks()` and `h3_mask.blocks()` (#790), and `h3_guide_steps` imports `h3_hold_steps` and never the reverse (`_with_guides` in `dw/pipeline_processors/pipeline.py` adds its blocks for a `guides` call, and `_with_mask` checks a regional edit's arguments and media and builds the regenerate mask for a `source_video` call), `dw/guides.py`: the `guides` validation check and the chain `continuity: "guide"` check, `guide_chain_errors`, which asks the `guides` check's own step rule (`_takes_guides_problem`, built on `h3_rules.not_h3` and `REFERENCE_WORKFLOWS`) and reads the mode names from `h3_rules.CHAIN_CONTINUITY_MODES`, the names `chain.py`'s `CONTINUITY_MODES` is zipped against (both registered in `dw/validation.py`; a `previous_result:` guide is checked at run time), `dw/pipeline_processors/h3_tiles.py`: the tiled refine (the tile loop, the latent-space feather blend, the per-tile crop of one whole-canvas noise draw, the decode-free graph variant, and the one spatially tiled decode), which imports `h3_rules`, `h3_hold` and `media_types` (for `FrameList`; and `events`, `progress`) and is imported by `pipeline.py` and, lazily, `components.py` (which attaches the variant at load, behind `h3_hold.core_denoise_sequences`, and calls `insert_mask_blocks` at load), never by `h3_rules` or `h3_hold`, `dw/hold_audio.py`: `hold_audio_errors` (a video is held by its soundtrack), `refine_strength_errors`, `mask_errors` (Check `regional_edit`, which probes the source and mask through `ValidationContext.probe` the way `dw/guides.py` probes guide videos) | The one place dw modifies a diffusers modular pipeline's block graph: three blocks go in, before `set_timesteps`, `denoise` and `after_denoise`, in every H3 core-denoise sequence (the whole graph and the `workflow=`-pruned flat graph), and two more for a regional edit: the source block before `set_timesteps` and the blend block inside the `denoise` loop after `update`. All are no-ops without their arguments (`hold_audio` / `refine_strength`, `source_video`). The tiled refine adds a second graph built at load beside them (`h3_tiles.decode_free_blocks`: the decode and text-encoder blocks dropped, every kept block the graph's own instance, so hold and refine stay in) and swaps it into `pipeline._blocks` only for the duration of the one text-encode call and the tile calls (`swapped_blocks`, restored on failure). Every tile runs the hold block, but the track is encoded once per refine: the first tile is asked for the block's encode (`HELD_LATENTS_OUTPUT`) and every later tile's `hold_audio` carries it (`encoded_hold_audio`, an `EncodedAudioReference`), which the block writes in place of running the audio VAE. A `hold_audio` path is opened only after `locations.validate_media_path` against the step's workflow directory. `dw/introspection.py` builds the same graph without weights, hold included, so `get_pipeline_signature` lists a modular pipeline's block inputs. The copies of diffusers behaviour are pinned to the installed diffusers: `shifted_sigma_grid` (the spacing `refine_sigmas` rests on) to stock `set_timesteps`, `GUIDE_FRAMES_PER_CHUNK`/`GUIDE_LATENTS_PER_CHUNK` to the H3 video VAE's `clip_length` and latents per clip, and every name in `LAYOUT_ANCHORS` (`_fill_audio_positions` among them) to `before_denoise`. | `tests/test_h3_hold_audio.py::TestAnchors::test_inserted_next_to_the_anchors_in_every_shape`, `tests/test_h3_hold_audio.py::TestHoldAudioLocation::test_a_path_outside_the_roots_is_refused_at_the_call`, `tests/test_h3_hold_audio.py::TestRefusals::test_a_non_h3_step_is_refused`, `tests/test_h3_hold_audio.py::TestSignature::test_the_h3_signature_shows_hold_audio`, `tests/test_h3_refine.py::TestAnchors::test_inserted_before_denoise_in_every_shape`, `tests/test_h3_tiles.py::TestDecodeFree::test_variant_has_no_decode_and_shares_blocks`, `tests/test_h3_tiles.py::TestTileLoop::test_blocks_are_restored_when_a_tile_fails`, `tests/test_h3_tiles.py::TestTileLoop::test_the_track_is_encoded_by_one_tile_and_handed_to_the_rest`, `tests/test_h3_hold_audio.py::TestHoldBlock::test_a_carried_encode_is_written_without_encoding_again`, `tests/test_h3_tiles.py`, `tests/test_h3_rules.py` (`tile_plan` and the `refine_tiles` rules), `tests/test_h3_mask.py`, `tests/test_h3_refine.py::TestSigmaGridDrift::test_the_full_grid_is_stock_set_timesteps`, `tests/test_h3_refine.py::TestSigmaListOneOwner::test_the_runtime_message_is_the_owners`, `tests/test_h3_refine.py::TestSigmaListSchedule::test_the_block_runs_the_list_unshifted`, `tests/test_h3_guides.py::TestChunkDrift::test_frames_per_chunk_is_the_vae_clip_length`, `tests/test_h3_guides.py::TestChunkDrift::test_latents_per_chunk_is_the_vae_tokens_per_clip`, `tests/test_h3_guides.py::TestAnchors::test_installed_diffusers_has_every_anchor`, `tests/test_guide_chain_validation.py::TestRefused::test_the_step_rule_is_the_guides_rule`, `tests/test_guide_chain_validation.py::TestRefused::test_every_run_mode_is_known_to_validate`, `tests/test_h3_rules.py::test_h3_rules_imports_neither_torch_nor_diffusers`, `tests/test_h3_rules.py::test_the_block_owners_import_no_diffusers` | +| IC-LoRA reference scale | `workflows/templates/ltx2/` | The five conditioning templates run at `reference_downscale_factor: 1`, and the two upscalers at 2. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_reference_is_encoded_at_the_output_resolution`, `tests/test_ltx2_ic_loras.py::TestTheGenerativeUpscaleMatchesTheSameCard::test_it_loads_the_upscaler_at_factor_two` | | IC-LoRA numbers | `workflows/templates/ltx2/` | Every number in the IC-LoRA templates comes from the vendor's model card. | `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_strength_is_the_cards_default`, `tests/test_ltx2_ic_loras.py::TestEachTemplateMatchesItsCard::test_the_defaults_are_the_trained_bucket` | | IC-LoRA prompt genre | `prompts/ltx2/` | A stored prompt tagged `ic-lora` is checked against its trained caption form, not against the T2V paragraph rule. | `tests/test_ltx_prompt_library.py::test_an_ic_lora_prompt_is_in_its_trained_form` | @@ -135,13 +137,14 @@ one to open. | Concept | Owner | Rule | Enforced by | | --- | --- | --- | --- | -| Opening media | `dw/media.py` | This is the only module that calls `av.open`. It imports no torch and no step types. | `tests/test_media_layering.py::test_av_open_appears_only_in_media`, `tests/test_media_layering.py::test_media_imports_no_step_types` (no torch: —) | +| Opening media | `dw/media.py`, `dw/media_readers.py` (the frame readers, re-exported by `dw/media.py`, which imports from it and never the reverse) | These are the only modules that call `av.open`. They import no torch and no step types. | `tests/test_media_layering.py::test_av_open_appears_only_in_media`, `tests/test_media_layering.py::test_media_imports_no_step_types` (no torch: —) | | Frame moments and crops | `dw/media_frames.py`: `moment_index`, `parse_crop` | One `at` grammar ("frame:N", or seconds at the clip's fps; negative counts back from the end for `get_output_frames` and is refused by `compare_outputs`) and one `[x, y, width, height]` crop parse serve `get_output_frames` and `compare_outputs`. Only the bounds policy is the caller's: `get_output_frames` refuses a moment past the end and clamps a crop (`_moment_to_index`, `resolve_crop_box`); `compare_outputs` blanks a past-end cell and refuses a crop that doesn't fit an output. | `tests/test_media_compare.py::test_resolve_moments_refuses_what_get_output_frames_refuses`, `tests/test_media_compare.py::test_crop_box_refuses_a_malformed_box_as_resolve_crop_box_does` | | Output comparison | `dw/media_compare.py`: `compare`, `psnr`, `ssim`, `mean_abs_diff`, `check_request`, `crop_box` | Metrics always compare the same pixels: one shared crop is the measured region, and per-output crops are display only, leaving metrics on the full frame. An output whose size differs from the baseline's, or a moment past its end, gets null metrics with a reason; nothing is resized to force a number. SSIM is on Rec. 601 luma (SSIM's convention, not `image_ops.luma`). It runs in the server process and decodes through `dw/media.py` and `open_bounded`. | `tests/test_compare_route.py::test_a_shared_crop_is_the_region_measured`, `tests/test_compare_route.py::test_per_output_crops_are_display_only`, `tests/test_compare_route.py::test_a_size_mismatch_gets_no_numbers_and_says_why`, `tests/test_compare_route.py::test_a_moment_past_the_shorter_clip_is_a_blank_cell` | | Signal processing | `dw/dsp.py` | It measures and transforms waveforms, decides nothing, and imports nothing from `dw`. | `tests/test_media_layering.py::test_dsp_imports_nothing_from_dw` | | Assessment probes | `dw/tasks/assess.py`, `dw/assessment_rules.py` | Exactly three registered commands are assessment probes. Each measures a finished file, and every rule names a real probe field. | `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_assessment_is_exactly_the_three_probes`, `tests/test_assessment_rules.py::TestRuleProbesAreRealCommands::test_every_rule_names_a_registered_json_command` | -| Script check | `dw/tasks/script_check.py`: `check`; `dw/tasks/script_take.py`: `guard_words`, `word_level`, `shot_spans`; `dw/script_lines.py`: `parse_lines`, `parse_shots`, `shot_names_error` | `check_script` aligns a take's heard words to its expected lines and guards out words heard over silence and repetition loops (Whisper's hallucination over music). It builds its findings with `assessment_rules.finding()` and reads the guard's floor and window (`DEAD_AIR_FLOOR_DBFS`, `DEAD_AIR_WINDOW`) rather than copying them; the loudest-window measure is `dsp.loudest_window_dbfs` (`script_take.word_level` wraps it; the guard, the transcription levelling and the shot placement are `dw/tasks/script_take.py`, which `script_check` reads one way - nothing there reads the rule tables), which `transcribe_audio`'s `suspect: "silent"` mark reads with the same floor and window; the repetition-loop detector (`script_lines.looped_texts`, `REPEAT_RUN_MIN`, `REPEAT_MAX_PERIOD`) lives in `dw/script_lines.py` and backs both `check_script`'s `repetition` discard and `transcribe_audio`'s `suspect: "repetition"`, and `script_lines.suspect_runs`/`SUSPECT_LEFT_OUT` are what `plan_cuts` groups and leaves out suspect chunks by - so neither task imports the other. It is not an assessment probe and adds nothing to `RULES`. Its literal `lines` is refused at validation through `task_problems` (`script_lines_errors`), which calls the same `parse_lines` the task runs, and a literal `shots` through the same `parse_shots` and `shot_names_error`; the parsers live in `dw/script_lines.py`, not the task, so that `task_problems` reaches them without importing `script_check` (the `import_cycles` ratchet in `scripts/arch_metrics.py`). A line's `shot` may be the short `@name` of the one recorded shot that carries it (`script_lines.matching_shot_names`, `with_full_shot_names`, shared by the task and the literal-`shots` check); which member key a recorded name carries comes from `references.member_keys`, beside `MEMBER_SEPARATOR` and `for_each.member_name`, never a re-derived pattern: a key ends at the first character outside `references.VARIABLE_NAME_CHARACTERS`, the set `security.validate_variable_name` (which checks a for_each entry's name) builds its pattern from. With `audio` an `output:`/`asset:`/path whose run manifest records shots, `dw/shot_span_preflight.py: script_shot_name_errors` (the `script_shot_names` error check) runs `shot_names_error` at validation, reading the map through `runs.shots_beside`. Its shot map comes from `assess.resolve_shots`, the one the probes use, never a copy; it places each shot on the soundtrack with `assess.sample_span`, decides repeated shot names with `shots.duplicate_shot_names`, and asks `locations.is_http_url` whether the take is a URL. | `tests/test_script_check.py::TestRegistration::test_registered_as_json_command_without_assessment_flag`, `tests/test_script_check.py::TestEmptyLines::test_words_only_over_silence_are_discarded_not_found`, `tests/test_script_check.py::TestLinesErrors::test_literal_with_a_colon_is_not_a_reference`, `tests/test_script_check.py::TestShotValidation::test_a_short_member_name_matches_the_one_shot_carrying_it`, `tests/test_script_check.py::TestShotValidation::test_a_valid_variable_name_comes_back_whole_as_a_member_key`, `tests/test_shot_span_preflight.py::TestScriptShotNames::test_a_wrong_shot_name_is_an_error_at_validate_time` | +| Script check | `dw/tasks/script_check.py`: `check`; `dw/tasks/script_findings.py`: the rule constants and the finding builders `check` calls (imported one way, re-exported by `script_check`); `dw/tasks/script_take.py`: `guard_words`, `word_level`, `shot_spans`; `dw/script_lines.py`: `parse_lines`, `parse_shots`, `shot_names_error` | `check_script` aligns a take's heard words to its expected lines and guards out words heard over silence and repetition loops (Whisper's hallucination over music). It builds its findings with `assessment_rules.finding()` and reads the guard's floor and window (`DEAD_AIR_FLOOR_DBFS`, `DEAD_AIR_WINDOW`) rather than copying them; the loudest-window measure is `dsp.loudest_window_dbfs` and the looped-bed self-correlation `dsp.repeat_correlation` (`script_findings.is_looped_bed` only decides on it) (`script_take.word_level` wraps it; the guard, the transcription levelling, the floor-gated `audible_runs` and the shot placement (with `clamp_to_shot`, a line's timing kept inside its shot) are `dw/tasks/script_take.py`, which `script_check` reads one way - nothing there reads the rule tables), which `transcribe_audio`'s `suspect: "silent"` mark reads with the same floor and window; the repetition-loop detector (`script_lines.looped_texts`, `REPEAT_RUN_MIN`, `REPEAT_MAX_PERIOD`) lives in `dw/script_lines.py` and backs both `check_script`'s `repetition` discard and `transcribe_audio`'s `suspect: "repetition"`, and `script_lines.suspect_runs`/`SUSPECT_LEFT_OUT` are what `plan_cuts` groups and leaves out suspect chunks by - so neither task imports the other. It is not an assessment probe and adds nothing to `RULES`. Its literal `lines` is refused at validation through `task_problems` (`script_lines_errors`), which calls the same `parse_lines` the task runs, and a literal `shots` through the same `parse_shots` and `shot_names_error`; the parsers live in `dw/script_lines.py`, not the task, so that `task_problems` reaches them without importing `script_check` (the `import_cycles` ratchet in `scripts/arch_metrics.py`). A line's `shot` may be the short `@name` of the one recorded shot that carries it (`script_lines.matching_shot_names`, `with_full_shot_names`, shared by the task and the literal-`shots` check); which member key a recorded name carries comes from `references.member_keys`, beside `MEMBER_SEPARATOR` and `for_each.member_name`, never a re-derived pattern: a key ends at the first character outside `references.VARIABLE_NAME_CHARACTERS`, the set `security.validate_variable_name` (which checks a for_each entry's name) builds its pattern from. With `audio` an `output:`/`asset:`/path whose run manifest records shots, `dw/shot_span_preflight.py: script_shot_name_errors` (the `script_shot_names` error check) runs `shot_names_error` at validation, reading the map through `runs.shots_beside`. Its shot map comes from `assess.resolve_shots`, the one the probes use, never a copy; it places each shot on the soundtrack with `assess.sample_span`, decides repeated shot names with `shots.duplicate_shot_names`, and asks `locations.is_http_url` whether the take is a URL. | `tests/test_script_check.py::TestRegistration::test_registered_as_json_command_without_assessment_flag`, `tests/test_script_check.py::TestEmptyLines::test_words_only_over_silence_are_discarded_not_found`, `tests/test_script_check.py::TestLinesErrors::test_literal_with_a_colon_is_not_a_reference`, `tests/test_script_check.py::TestShotValidation::test_a_short_member_name_matches_the_one_shot_carrying_it`, `tests/test_script_check.py::TestShotValidation::test_a_valid_variable_name_comes_back_whole_as_a_member_key`, `tests/test_shot_span_preflight.py::TestScriptShotNames::test_a_wrong_shot_name_is_an_error_at_validate_time` | | Audio sync against a source slice | `dw/tasks/measure_sync.py`: `measure_sync`; `dw/dsp.py`: `envelope_lag` | `measure_sync` cross-correlates the onset envelopes of a shot's audio and its source slice and answers the lag and a 0-1 confidence. A positive lag means the audio is LATE (its events fall after the reference's), negative early; `lag_seconds` is None for a silent or flat track. It builds its findings (`sync_unmeasurable`, `sync_low_confidence`, `audio_out_of_sync`) with `assessment_rules.finding()` over module-local rule dicts and adds nothing to `RULES`; it reads the silence floor from `beats.SILENT_DBFS`. `envelope_lag` imports nothing from `dw`. | `tests/test_measure_sync.py::TestLag::test_late_audio_is_a_positive_lag`, `tests/test_measure_sync.py::TestLag::test_early_audio_is_a_negative_lag`, `tests/test_measure_sync.py::TestUnmeasurable::test_a_silent_reference_has_no_lag` | +| Sung phrases from voiced energy | `dw/tasks/phrases.py`: `analyze_phrases`, `voiced_power`, `voiced_runs`; `dw/task_problems.py`: `phrases_problems`, `phrases_errors` | `analyze_phrases` times a song's sung phrases with no model: a stereo file's centre-weighted band power (max(0, P_mid - P_side), 250-3500 Hz, 20 ms frames) or, for mono, `centre: false` or identical channels, the downmix's band power (`method` `band`). The voiced floor is `voice_attribution.voiced_floor_dbfs`, the rule `attribute_voices` uses, unless `floor_db` is given. It answers a dict - `{phrases, duration_seconds, method, warnings}` - so a step saves one JSON artifact. Its ranges are the registration's domains; `phrases_problems` (a real boolean `centre`) runs at validate and at run time. | `tests/test_analyze_phrases.py::TestCentre::test_centred_bursts_under_hard_panned_noise_are_found`, `tests/test_analyze_phrases.py::TestCentre::test_bursts_panned_hard_to_one_side_are_not_centre`, `tests/test_analyze_phrases.py::TestRefusals::test_validate_refuses_a_missing_audio_and_a_negative_gap` | ## Security and architecture guardrails diff --git a/docs/ATTRIBUTIONS.md b/docs/ATTRIBUTIONS.md new file mode 100644 index 000000000..02d07b21b --- /dev/null +++ b/docs/ATTRIBUTIONS.md @@ -0,0 +1,111 @@ +# Attributions + +diffusers-workflow is Apache-2.0. This page credits the work it builds on that +a dependency list does not show: code that was copied in, and other projects +whose ideas were reimplemented here. Python dependencies are in +[requirements.txt](../requirements.txt) and [pyproject.toml](../pyproject.toml). +Model and LoRA weights keep their own licences, and each catalog entry +([workflows/models/](../workflows/models/), [loras/](../loras/)) names the +repository it downloads from. + +## Code included in this repository + +Each file keeps its upstream licence notice in its header. + +| File | Source | Licence | +|---|---|---| +| [dw/tasks/rife_model.py](../dw/tasks/rife_model.py) | The RIFE interpolation network from [hzwer/Practical-RIFE](https://github.com/hzwer/Practical-RIFE) | MIT, Copyright (c) Megvii Inc. | +| [dw/tasks/h3_latent_upscaler_model.py](../dw/tasks/h3_latent_upscaler_model.py) | The MiniMax H3 3D-convolution latent upscaler from [LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler](https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler) | MIT, Copyright (c) LBH-123-AI | +| [dw/vendored/audio8_tts_model.py](../dw/vendored/audio8_tts_model.py) | The Audio8-TTS (ArkTTS) dual-AR speech model, codec and processor from [Edge0/Audio8-TTS-Realistic-0.6b](https://huggingface.co/Edge0/Audio8-TTS-Realistic-0.6b/tree/1e857568bfd4f02931aca3988ee773a44d2d26d3), ported (upstream NOTICE kept in the file header) | Apache-2.0 | +| [dw/vendored/mmaudio_model.py](../dw/vendored/mmaudio_model.py), [dw/vendored/mmaudio_clip.py](../dw/vendored/mmaudio_clip.py) | The MMAudio flow-matching network, its sequence lengths and classifier-free guidance, and the CLIP conditioning from [hkchengrex/MMAudio](https://github.com/hkchengrex/MMAudio/tree/974010a026c731054592d8f777218bd9d85a6c24), ported (the sampler is dw's own Euler loop, and the DFN5B towers are transformers' CLIP models filled by a key map) | MIT, Copyright (c) 2024 Sony Research Inc. | +| [dw/vendored/mmaudio_sync_model.py](../dw/vendored/mmaudio_sync_model.py) | MMAudio's Synchformer visual encoder (a MotionFormer feature extractor) and its 16-frame segmenting, from [hkchengrex/MMAudio](https://github.com/hkchengrex/MMAudio/tree/974010a026c731054592d8f777218bd9d85a6c24)'s `mmaudio/ext/synchformer/`, ported without timm, einops or omegaconf | MIT, Copyright (c) 2024 Vladimir Iashin | +| [dw/tasks/mmaudio_decoder.py](../dw/tasks/mmaudio_decoder.py) | Loads diffusers' `MMAudioVAE` and `MMAudioVocoder`; the vocoder is [BigVGAN](https://github.com/NVIDIA/BigVGAN) (no BigVGAN code is copied here: the classes are diffusers') | BigVGAN: MIT, Copyright (c) 2024 NVIDIA CORPORATION | +| [dw/community_pipelines/pipeline_flux_rf_inversion.py](../dw/community_pipelines/pipeline_flux_rf_inversion.py) | The diffusers community RF-Inversion pipeline, after [RF Inversion](https://rf-inversion.github.io/) by Litu Rout, Yujia Chen, Nataniel Ruiz, Constantine Caramanis, Sanjay Shakkottai and Wen-Sheng Chu | Apache-2.0, Copyright 2024 Black Forest Labs and The HuggingFace Team | +| [dw/prompt_weighting.py](../dw/prompt_weighting.py) | Based on [sd_embed](https://github.com/xhinker/sd_embed) by Andrew Zhu | Apache-2.0 | + +## Model weights with their own terms + +Most weights are named only by their catalog entry. These are listed here as +well, because their terms are narrower than this repository's licence and +apply to anything generated with them. + +| Weights | Used by | Licence | +|---|---|---| +| [hkchengrex/MMAudio](https://huggingface.co/hkchengrex/MMAudio) (`weights/mmaudio_large_44k_v2.pth` and its `ext_weights/`) | `generate_sound_effects`: [dw/vendored/mmaudio_model.py](../dw/vendored/mmaudio_model.py) | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): non-commercial use only | +| [hkchengrex/MMAudio](https://huggingface.co/hkchengrex/MMAudio) `ext_weights/synchformer_state_dict.pth` | `generate_sound_effects`, MMAudio's sync features, [dw/vendored/mmaudio_sync_model.py](../dw/vendored/mmaudio_sync_model.py) | [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/): non-commercial use only | +| [kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers](https://huggingface.co/kandinskylab/Kandinsky-6.0-Lite-distill-5s-Diffusers/tree/4c487e8b4aeeb9059e5a77905aa2ddb59ec28f81) `audio_vae/` and `vocoder/` | `generate_sound_effects`, MMAudio's decoder, [dw/tasks/mmaudio_decoder.py](../dw/tasks/mmaudio_decoder.py) | The MMAudio VAE tensors: [CC-BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/), non-commercial use only, as MMAudio's weights (K6's model card states MIT; dw follows the stricter MMAudio terms). The BigVGAN vocoder: MIT. | +| [apple/DFN5B-CLIP-ViT-H-14-384](https://huggingface.co/apple/DFN5B-CLIP-ViT-H-14-384) | `generate_sound_effects`, MMAudio's CLIP conditioning, [dw/vendored/mmaudio_clip.py](../dw/vendored/mmaudio_clip.py) | [Apple Machine Learning Research Model License](https://huggingface.co/apple/DFN5B-CLIP-ViT-H-14-384/blob/main/LICENSE): research purposes only. Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement. dw maps the weights into transformers' layout in memory at load time and never writes or ships converted tensors. | + +## Ideas reimplemented here + +These projects showed an approach that dw rebuilt in its own code for +diffusers. No code, prompt text or data was copied from them; where a licence +does not allow reuse, only the concept was taken. The issue numbers link to the +design discussion, which names the source files consulted. + +### [vrgamegirl19/comfyui-vrgamedevgirl](https://github.com/vrgamegirl19/comfyui-vrgamedevgirl) + +A ComfyUI node pack and set of workflows for MiniMax H3 and LTX-2 film making, +assessed in October 2026. It is source-available under a licence that does not +allow reuse here, so only concepts were taken. It inspired: + +- holding the reference audio during H3 denoising, and a refine pass after the + latent upscale ([#598](https://github.com/dkackman/diffusers-workflow/issues/598)) +- tracked, distance-gated face repair with LTX on face crops ([#599](https://github.com/dkackman/diffusers-workflow/issues/599)) +- the music-video timing stack: beat analysis, a lyric- and beat-aware cut + planner, pad-then-trim shots ([#600](https://github.com/dkackman/diffusers-workflow/issues/600)) +- overlapping windows for long video sources, joined with a cosine blend + (`window_video`, `join_windows`; [#601](https://github.com/dkackman/diffusers-workflow/issues/601)) +- the exact size and frame-count round trip for video-to-video templates + (`fit_to_model`, `restore_to_source`; [#602](https://github.com/dkackman/diffusers-workflow/issues/602)) +- the finishing tasks: LUT apply and make, film grain, sharpen, tonal controls + in `grade` ([#603](https://github.com/dkackman/diffusers-workflow/issues/603)) +- stem separation as its own task (`separate_stems`; [#604](https://github.com/dkackman/diffusers-workflow/issues/604)) +- strength-selectable same-size LTX refine and per-segment strength decay in + chains ([#606](https://github.com/dkackman/diffusers-workflow/issues/606)) +- laying out a reference sheet from individual images (`ingredients_grid`; [#607](https://github.com/dkackman/diffusers-workflow/issues/607)) +- the H3 dialogue-budget warnings ([#608](https://github.com/dkackman/diffusers-workflow/issues/608)) +- checking a take's transcript against the expected lines (`check_script`; [#609](https://github.com/dkackman/diffusers-workflow/issues/609)) +- the script-to-video shot-list contract and voice bible ([#610](https://github.com/dkackman/diffusers-workflow/issues/610)) +- H3 chunk continuation in latent space ([#611](https://github.com/dkackman/diffusers-workflow/issues/611)) +- trying the Alissonerdx H3 LoRAs, which its workflows use ([#612](https://github.com/dkackman/diffusers-workflow/issues/612)) + +### [Kablex/ComfyUI-Ref2VA-VSA](https://github.com/Kablex/ComfyUI-Ref2VA-VSA) + +Apache-2.0. Showed that MiniMax H3's Ref2VA attention could run block-sparse, +building on the [FastVideo](https://github.com/hao-ai-lab/FastVideo) team's +VSA-H3 sparse-attention work. dw's `sparse` attention backend +([dw/sparse_attention.py](../dw/sparse_attention.py), +[#931](https://github.com/dkackman/diffusers-workflow/issues/931)) runs on +[comfy-kitchen](https://pypi.org/project/comfy-kitchen/)'s Sol-Attn kernel, a +dependency. + +### [mo-akkakk/ComfyUI-MiniMaxH3-TemporalWindow-SpatialTiles](https://github.com/mo-akkakk/ComfyUI-MiniMaxH3-TemporalWindow-SpatialTiles) + +MIT. The tiled low-sigma refine after an H3 upscale (`refine_tiles`, +[#1043](https://github.com/dkackman/diffusers-workflow/issues/1043)), and the +approach of the test that checks a one-segment H3 chain is bit-identical to a +native run on a tiny random-weight model on CPU +([#1046](https://github.com/dkackman/diffusers-workflow/issues/1046)). + +### [harlixay7/Kinoloom](https://github.com/harlixay7/Kinoloom) + +MIT. Its seamless-chain research gave `analyze_seams` its saturation, +lightness, contrast and detail measures +([#1031](https://github.com/dkackman/diffusers-workflow/issues/1031)), and the +`script-to-video` skill its guidance on re-rolling a chained segment whose seam +fails ([#1034](https://github.com/dkackman/diffusers-workflow/issues/1034)). + +### [r34vtraining/Longshot_Studio](https://github.com/r34vtraining/Longshot_Studio) + +A ComfyUI studio for making long MiniMax H3 films shot by shot, assessed in +October 2026. It has no licence, so only concepts are taken. Its ideas are +filed as dialogue/effects stem separation +([#1091](https://github.com/dkackman/diffusers-workflow/issues/1091)), an H3 +face refine with the audio held +([#1092](https://github.com/dkackman/diffusers-workflow/issues/1092)), cutaway +and lead-in joins ([#1093](https://github.com/dkackman/diffusers-workflow/issues/1093)), +rendering the odd shots of a long take first and bridging the gaps +([#1094](https://github.com/dkackman/diffusers-workflow/issues/1094)), and +finding phrase starts from the centre of the stereo mix +([#1095](https://github.com/dkackman/diffusers-workflow/issues/1095)). diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index 088443bc4..27152ad84 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -16,7 +16,7 @@ source ./activate .\venv\scripts\activate ``` -The install scripts handle everything: Python version detection (3.10-3.14), virtual environment creation, PyTorch, diffusers, and all dependencies. +The install scripts handle everything: Python version detection (3.12-3.14), virtual environment creation, PyTorch, diffusers, and all dependencies. ### Manual Installation diff --git a/docs/MCP.md b/docs/MCP.md index cbda32031..2466bca73 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -424,11 +424,18 @@ one phase, so two polls otherwise come back identical: read `denoise_step` moving (slow but healthy) against a `denoise_step` that is a number and stays put while `seconds_since_event` climbs (nothing is happening). A null `denoise_step` under `generating` is neither - it is the lead-in the -pipeline runs before the loop, encoding the prompt and every reference, with -nothing emitted, so silence there is expected. Its length follows what it -encodes: ~90 s on MiniMax H3 for a prompt with an image or audio reference, -~10 min once a *video* reference is among them (measured: 629 s for one 5 s -960x544 clip on an RTX 3090). `get_job_events` names the block it is in +pipeline runs before the loop, encoding the prompt and every reference and +then the first denoise pass, with nothing emitted, so silence there is +expected. On MiniMax H3 the first pass dominates the lead-in, not the encode, +and it varies by card. The reference encode (`vae_encoder`) is about 6 s for +an image reference and about 50-53 s for a 5 s 960x544 video reference; +reference tokens make the first pass slower. Measured on lem: an image +reference's lead-in was 187 s (job `f90a041af26a`: `timing.encode_s` 26 s, +first pass 161 s; of the encode, `vae_encoder` was 6 s); the same image + +video job took 508 s on cuda:1 (`f5aefcdd5e26`: `encode_s` 86 s, first pass +422 s; `vae_encoder` 59 s) and 165 s on cuda:0 (`6961876f11d3`: `encode_s` +79 s, first pass 86 s; `vae_encoder` 56 s). Plan for several minutes, up to +~8.5 min measured (508 s, `f5aefcdd5e26`), with a video reference. `get_job_events` names the block it is in while that runs - a `log` line per top-level block of a modular pipeline (`MiniMaxAI/MiniMax-H3: vae_encoder`), which is the difference between silence and knowing it is encoding the reference. And once the counter is a number the gaps diff --git a/docs/RELEASING.md b/docs/RELEASING.md index 41d0efbf4..1ca2351ad 100644 --- a/docs/RELEASING.md +++ b/docs/RELEASING.md @@ -7,6 +7,136 @@ notes from commits at tag time (see below). This section is a scratch pad for items a branch's author wants the next release note to name; clear it when a release ships. +### 0.17.0 + + + +A release about fixing a good take instead of re-rolling it, and about +sound that is not the model's own. MiniMax H3 edits one region of a take +behind a mask, redraws a soft face and takes an explicit refine schedule; +LTX-2.5 pulls an alpha matte and colourises black-and-white footage; a +picture-synced sound-effects task (MMAudio) and a zero-shot voice clone +(Audio8-TTS) join the catalog as vendored ports that run no Hub code; and +every one-for-one video task now writes at its source's frame rate, exactly. + +**H3 regional edit, face repair and refine schedule** (#1062, #1077, #1078, #1082, #1087, #1092, #1098, #1099) + +- `templates/minimax/regional-edit`: an H3 step takes `source_video` and a + `mask_image` or `mask_video` (white regenerates) with `refine_strength`, + re-denoises the masked rows and writes every other row back to the source + at each step, soundtrack held. The guide's *H3: regional edit with + source_video and a mask* has the measured strength bands (0.9 replaces an + object, 0.6 redraws a face); the `minimax-h3` skill routes "fix one region + of a good take" to it (`references/regional-edit.md`). Validate and the call + refuse the same things: a mask with no source, a source on a chained step, + a size or frame-count mismatch, an all-black mask, a source whose frame + count is off the `17n + 5` grid (the pipeline would snap it up over a + padded source and the mask would drift). +- `templates/minimax/face-repair`: `crop_face_track` cuts the face out on H3's + grid carrying the take's padded soundtrack, a t2va refine at 0.6 redraws it + with `hold_audio` so the mouth stays on the speech, and `paste_face_track` + puts it back at the source's frame rate. The crop refuses a window outside + 124-345 frames or a rate other than 24 before H3 loads. +- `refine_sigmas` on an H3 refine step gives the schedule as the sigma points + themselves (first = `refine_strength`, last = 0, strictly decreasing as + float32, finite, `num_inference_steps` = the point count), the guide's + *Explicit refine sigmas*. A full-frame refine whose `width`/`height` + disagree with the upscaled latents is refused at validate (#1087), and a + single-file LoRA is priced as its file, not its repo (#1088). +- The `minimax-h3` skill gains the style pointer (`references/style.md`, + #1070), the AISkills and odd-then-bridge probes recorded as "probed, not a + rule" (#1072, #1103), the re-measured lead-in (#1089), and routes for the + past-768p latent refine and for replacing a shot's own sound. + +**Sound effects and voice cloning** (#1112, #1120, #1121, #1124, #1131-#1133, #1134) + +- `generate_sound_effects` (MMAudio, vendored at a pinned commit with its + Synchformer and DFN5B CLIP conditioning, decoder weights read from + Kandinsky 6's safetensors) makes picture-synced effects for a clip and/or + a prompt: `templates/video-to-sound-effects` and + `templates/text-to-sound-effect`. Duration is the clip's (0.64-20 s), the + model is trained around 8 s, noise-like output is warned about, and the + output runs 7-25 dB hot: `match_levels` before mixing. The weights are + CC-BY-NC and Apple research-only; `docs/ATTRIBUTIONS.md` lists them. +- `generate_speech` with `model_name: Edge0/Audio8-TTS-Realistic-0.6b` clones + a voice from a short `reference_audio` (a path, an `asset:`, or an earlier + audio step's output) and its `reference_text`; `templates/voice-clone` runs + it. The model is a vendored port pinned to one revision; its Hub repo's + code never runs. `attribute_voices` takes an audio step's output as a voice + reference too (#1134). +- The vendored ports live in `dw/vendored/`, outside the architecture ratchet + like `dw/community_pipelines/`. + +**LTX-2.5: alpha matte, colourisation, and the source's frame rate** (#1064, #1079, #1080, #1090, #1106-#1109, #1114) + +- `templates/ltx2/alpha-matte` (Alpha-Gen IC-LoRA) pulls a grey matte from a + clip, one shot at a time (a hard cut leaks), with the decoders kept off the + card; `templates/ltx2/restore-colorize` colourises black-and-white footage + from a prompt that describes the clip, with its colour limit on low-rate + and portrait sources documented. +- Every one-for-one template (`upscale-clip`, `refine-clip`, + `refine-in-place`, `restore-deblur`, `restore-decompression`, + `restore-long`, `face-repair`, `alpha-matte`, `restore-colorize`) writes at + the source's frame rate (`frame_rate: "source"`), never relabelled to 24, + and a fractional rate (29.97, 23.976) is written as its exact rational + (#1109). `restore-deblur` and `restore-decompression` fit the clip to the + model, then restore to the source's size. + +**Cuts, phrases and the script check** (#1095, #1104, #1105, #1115-#1119, #1129, #1130, #1135-#1138) + +- `analyze_phrases` times sung phrases from centre-weighted voiced energy, + with no transcript, and `plan_cuts` takes its result: lyric lines pair with + phrases one to one when the counts match (a disproportionate pairing is + warned about) and are spread evenly when they do not, with the warning + naming which way to move `min_gap_seconds` or `floor_db`. The music-video + templates' README shows the Whisper-free chain. +- `transcribe_audio` marks Whisper's hallucinations: chunks of a near-silent + clip (`silent`), its stock outro lines (`stock_phrase`), 5-6 word loops and + duplicate overlapping chunks (`repetition`); `plan_cuts` and `check_script` + leave suspect chunks out and say so. +- `check_script`: an untranscribed block crossing a shot seam is credited to + each shot it covers (a shot too short to stand alone is folded into its + neighbour, never dropped); a line's timing stays inside its shot; a take + with no shots is one implicit shot for `unscripted_speech`; a looped + room-tone bed is not untranscribed speech; speech outside a partial shot + map is named in `rules_skipped`; an unheard line's clip is checked against + its shot's end. +- `separate_stems` no longer shift-averages a clip shorter than the htdemucs + segment, which clipped the stems 9-17 dB hot (#1130). + +**Skills and composition** (#1093, #1100, #1122, #1125, #1123, #1126-#1128, #1091, #1096) + +- `script-to-video`: cutaway and lead-in joins (`references/joins.md`), beat + authoring and the beat review (`references/beat-review.md`), shared with + `series-episodes`; `series-episodes` gains the dialogue/rest stem recipe + via `separate_stems`, kept balanced for headroom. +- The story-forge lessons that held on lem (framing, no invented camera, no + all-caps) landed as guidance in the `ltx-2.5` skill; the ones that did not + are recorded as probed. + +**Validation, server and security** (#1086, #1101, #1111, #1116, #1119, GHSA-hhfj-p9xj-xjmr, GHSA-75fh-g499-53cf, GHSA-9597-5qpq-55x6, GHSA-w9qv-2pg5-j79m, GHSA-75cg-7964-pjwx) + +- A task step's `trust_remote_code`, `custom_pipeline` or `custom_generate` + is refused at validate and again at `Task.run`, for every task; a `videos` + list and the active result types are gated at validate; a reference error + no longer names a server path. +- `for_each` refuses a literal cross-entry reference inside one group; + `list_jobs` refuses a workspace that does not exist; a cached pipeline's + minutes-long decode and a slow first pass are warned about as measurements, + with no cause asserted. +- Python 3.12 or newer is required (#1084); the install scripts and + classifiers say so. + +**Architecture** + +- `dw/media.py`, `dw/tasks/cuts.py`, `dw/tasks/script_check.py` and + `dw/pipeline_processors/pipeline.py` each hand a concern to a companion + module (`media_readers`, `cuts_lines`, `script_findings`, + `pipeline_outputs`), and the regional edit's mask arithmetic moves to + `h3_mask_cells` so the blocks never import their importer. The ratchet's + module count rises 226 -> 236 for these and the new tasks; every other + metric holds. + ### 0.16.0 diff --git a/docs/SECURITY.md b/docs/SECURITY.md index dd576e779..55a93e4e1 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -135,6 +135,42 @@ both its `generate_kwargs` and its `forward_params` to `generate()` for a generative model (Bark, VibeVoice). Each must be an object, and `custom_generate` / `trust_remote_code` in either are refused untrusted, before the model loads (released versions through 0.14.0 passed them through). +`trust_remote_code`, `custom_pipeline` or `custom_generate` given as a step +argument of their own, outside those two objects, is refused too, for every +model: untrusted by the trust gate, trusted as an argument the task doesn't +take. The task never read them there, but it used to drop them silently, so a +step setting one ran as if it had taken effect. + +Untrusted, every task step is checked the same way at validate: +`trust_remote_code`, `custom_pipeline` or `custom_generate` set anywhere in its +`arguments`, top level or nested, is an error naming the key and its path, so +the run is refused before it queues. `Task.run` repeats the check for every +command before its handler runs (GHSA-hhfj-p9xj-xjmr). + +`speech_generation`'s Audio8-TTS branch (`Edge0/Audio8-TTS-Realistic-0.6b`, +`dw/vendored/audio8_tts_model.py`) runs no code from its repo: the model is +vendored, so there is no `trust_remote_code` load. Its five files are fetched +with `hf_hub_download` at one pinned 40-hex revision, never a branch, and the +weights are safetensors. The audio codec, `codec.pth`, is a pickle checkpoint; +it is read with `torch.load(weights_only=True)`, which refuses any object that +isn't a tensor or a plain container, and only at that pinned revision. Its +`generate_kwargs` / `forward_params` pass the same trust gate as above, then an +allowlist (`temperature`, `top_p`, `top_k`, `do_sample`, `max_new_tokens`): +any other key is refused by name before the model loads. + +MMAudio (`dw/vendored/mmaudio_model.py`, `dw/vendored/mmaudio_sync_model.py`, +`dw/tasks/mmaudio_decoder.py`) runs no code from its repo either: the network +and the Synchformer visual encoder are vendored at upstream commit +`974010a0`. Their two checkpoints, `hkchengrex/MMAudio`'s +`weights/mmaudio_large_44k_v2.pth` and `ext_weights/synchformer_state_dict.pth`, +are pickles, pinned to one 40-hex revision; each is read with +`torch.load(weights_only=True)` and loaded strictly, so any object that isn't a +tensor or a plain container is refused and an unexpected key fails the load. +The decoder's VAE and vocoder are Kandinsky 6's safetensors at a pinned full +sha, built from their `config.json` read as JSON (only the class's own +constructor arguments are taken), with no pickle and no `trust_remote_code`. +The modules take local paths and never fetch: tests pin that none of them +calls the Hub or writes a checkpoint. `text_generation`'s `system_prompt` may be read from a Hub repo file (`{"from_hub": {...}}`). That fetch is anonymous only: the Hub is asked @@ -325,8 +361,9 @@ SecurityError - **Arbitrary file read through a media argument** — a location a workflow supplies is confined to the roots it may read (`dw/locations.py`) - **Script on the UI's origin through an output** — a step's - `result.content_type` may not be `text/html` or `text/xml` (compared - without case or parameters): validation refuses it at + `result.content_type` may not be an active document type - `text/html`, + `application/xhtml+xml`, `text/xml`, `application/xml` or `image/svg+xml` + (compared without case or parameters): validation refuses it at `steps[i].result.content_type` and the writer refuses it again (`dw/content_types.py`). A file of an active type that reaches `/outputs` or `/inputs` anyway is served with `Content-Security-Policy: sandbox` diff --git a/docs/SERVER.md b/docs/SERVER.md index 9c636d98f..1690f8398 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -327,10 +327,13 @@ to learn where a long render is: A null `denoise_step` under `generating` is the pipeline's lead-in - encoding the prompt and every reference - which emits nothing and runs well over a -minute on a large video model. How long it runs follows what it has to -encode: on MiniMax H3, ~90 s for a prompt with an image or audio reference, -but ~10 min once a *video* reference is among them - a measured run encoding -one 5 s 960x544 clip on an RTX 3090 sat silent from 94 s to 723 s. The keys +minute on a large video model. It is the encode plus the first denoise +pass, and on MiniMax H3 the first pass dominates and varies by card. The +reference encode (`vae_encoder`) is about 6 s for an image reference and +about 50-53 s for a 5 s 960x544 video reference. Measured on lem: 187 s for +an image reference (job `f90a041af26a`), and 508 s on cuda:1 +(`f5aefcdd5e26`, first pass 422 s) against 165 s on cuda:0 +(`6961876f11d3`, first pass 86 s) for the same image + video job. The keys are always present so that lead-in can be told from a loop that has stopped advancing: `seconds_since_event` is a stall signal once `denoise_step` is a number, or in any phase other than `generating`. @@ -937,8 +940,8 @@ network: (`text/html`, `application/xhtml+xml`, `text/xml`, `application/xml`, `image/svg+xml`) carries `Content-Security-Policy: sandbox`: it opens under an opaque origin with no script. Range and ETag answers are - unchanged. The engine does not write `text/html` or `text/xml` results - at all (below), so such a file is one planted on disk. + unchanged. The engine does not write a result of any of those types + (below), so such a file is one planted on disk. - Requests carrying a `Host` header that names neither a loopback address nor the configured `--host` are rejected (400). A wildcard bind (`--host 0.0.0.0` or `::`) skips this check - clients reach such a diff --git a/docs/TASKS.md b/docs/TASKS.md index 7372bfbd8..96b593237 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -529,7 +529,7 @@ add grain last so it is not itself sharpened: | `window_video` | One overlapping, fixed-length window of a long video, with its audio | `index`, `num_frames`, `overlap`, `fps` - see [window_video](#window_video) | | `join_windows` | Blend processed windows back into one video the source's length, with the source's audio | `videos`, `source`, `num_frames`, `overlap`, `curve`, `fps` - see [join_windows](#join_windows) | | `fit_to_model` | Fit a video into a model's working size and exact frame count; returns `{video, fit}` | `width`, `height`, `num_frames`, `mode` (`letterbox`, `stretch`, `crop`), `downscale` - see [fit_to_model and restore_to_source](#fit_to_model-and-restore_to_source) | -| `restore_to_source` | Put a model's output back at its source's size and length, from the `fit` record | `fit` - see [fit_to_model and restore_to_source](#fit_to_model-and-restore_to_source) | +| `restore_to_source` | Put a model's output back at its source's size and length, from the `fit` record | `fit`, `fps` (omitted: the input's rate; `"source"`: the source's; or a number) - see [fit_to_model and restore_to_source](#fit_to_model-and-restore_to_source) | The frame commands accept videos in any shape a result carries them: PIL frame lists, numpy or torch frame arrays, and audio+video pairs (LTX-2, MiniMax H3). @@ -973,7 +973,7 @@ that step instead. | `num_frames` | Yes | Frames per window, as given to `window_video` | | `overlap` | Yes | Frames each window shares with the one before, as given to `window_video` | | `curve` | No | The blend's shape across a seam: `cosine` (the default, `(1 - cos πt) / 2`), `smoothstep` (`3t² - 2t³`) or `linear` (`t`) | -| `fps` | No | The source's frame rate. Defaults to the rate its file was read at; needed only to put its audio back | +| `fps` | No | The joined video's frame rate: omitted or `"source"` for the rate the source's file was read at, or a number above zero for that rate. No frames are added or dropped; the source's audio is put back over the frames at this rate | The result has exactly the source's frame count, each source frame once: window 0's real frames, then each later window's first `overlap` frames @@ -1002,7 +1002,8 @@ check is the task's (`dw/window_count_errors.py`), so any `join_windows` step gets it, and it applies the same `window_count_problem` the task does. It also refuses a window whose frame count is not `num_frames` (named by position) and windows of different sizes. An unknown `curve` and an `overlap` that is not below `num_frames` are refused by `validate_workflow` -on literal values as well. +on literal values as well, and so is an `fps` that is neither `"source"` +nor a number above zero. The output's audio is the source's own track over exactly `frames_to_samples(source_frames)`, and none when the source has none; the @@ -1049,6 +1050,38 @@ and exactly `num_frames`, the pipeline runs on the fitted video, and ] ``` +#### Reframe to another aspect + +To turn a 16:9 clip into 9:16, 4:5 or 1:1 there are two ways today: + +- **Crop.** `fit_to_model` with `mode: "crop"` and the target `width`/`height` + scales each frame to fill and centre-crops it. It works at full resolution + with no model, and it loses the sides. `num_frames` is required: set it to + the clip's own length to keep every frame. `restore_to_source` is the wrong + follow-up for a reframe, because it undoes the fit. + + ```json + { + "name": "reframe", + "task": { + "command": "fit_to_model", + "arguments": { + "video": "asset:clip.mp4", + "width": 432, + "height": 768, + "num_frames": 121, + "mode": "crop" + } + } + } + ``` + +- **Re-render natively.** A generated clip is usually better re-made at the + portrait size. Any H3 template takes any 32-multiple `width`/`height` (for + example 544x960), and that keeps the audio. + +Generative outpaint of video (filling the new sides) is not supported; see #1066. + `mode` is `letterbox` (default: scale to fit, centred on black), `stretch` (resize to fill exactly) or `crop` (scale to fill, centre-crop). The frame count is exact: a longer source is cut to its first `num_frames` frames and a @@ -1063,8 +1096,10 @@ it goes straight to a pipeline's `video` or a reference condition's `frames`; saved, it is written as an mp4 at that rate. The `fit` record holds `mode`, `source_width`, `source_height`, `source_frames`, `model_width`, `model_height`, `model_frames`, `content_box` (where the source -sits in the model frame) and `source_box` (the part of the source kept - all -of it unless `mode` is `crop`), each box `{x, y, w, h}`. +sits in the model frame), `source_box` (the part of the source kept - all +of it unless `mode` is `crop`), each box `{x, y, w, h}`, and `source_fps` +(the source's frame rate, `null` when it carries none). A record written +before `source_fps` existed still restores. `restore_to_source` takes the pipeline's output and that record; it has no `scale` argument. The scale is the output's size over the fitted size and must @@ -1076,6 +1111,17 @@ the source's pixel density times the scale: 640x480 fitted into 512x288 restores to 640x360. A saved video keeps its exact size; only an odd side grows by one pixel, since the encoder needs even sides. +`fps` sets the restored clip's frame rate; it never adds or drops a frame. +Omitted, the clip keeps the rate it came in at - the pipeline's, which for a +model that runs at a fixed rate is not the source's. `"source"` takes the +record's `source_fps`, falling back to the input's rate when the record has +none. A positive number sets that rate. Anything else is refused, at +validation for a literal. A model that runs at 24 fps on an 8 fps clip +returns one frame per source frame, so `"fps": "source"` writes it back at 8 +and it stays frame-aligned with the source in time as well as in count +(`templates/ltx2/alpha-matte`, whose `frame_rate` variable defaults to +`"source"` and passes through to this argument). + Neither task carries a soundtrack. Pair the source's with `pair_audio` after the restore. @@ -1108,6 +1154,9 @@ face-detail pass that needs the face large and still: | `modulus` | No | The crop count is padded to `modulus * n + remainder` frames - the frame grid of the model the crops feed. At least `1`. Default `8` | | `remainder` | No | The padded count's remainder, `0` up to `modulus - 1`. Default `1` | | `multiple` | No | `crop_size` must be a multiple of this - the model's frame-size step. At least `1`. Default `32` | +| `min_frames` | No | Refuse the clip when its padded crop count is below this - the shortest clip the model the crops feed takes. A whole number, at least `1`. No check when unset | +| `max_frames` | No | Refuse the clip when its padded crop count is above this - the longest the model takes. A whole number, at least `min_frames`. No check when unset | +| `fps` | No | Refuse the clip unless its frame rate is this, within `0.01` (23.976 passes for `23.98`, not for `24`). Above `0`. No check when unset | | `detector_repo` | No | Hub repo holding the YuNet weights. Default `opencv/face_detection_yunet`, read at a pinned revision; any other must be a Hub repo id | | `detector_file` | No | The detector file in that repo, a bare `.onnx` name. Default `face_detection_yunet_2023mar.onnx` | | `device` | No | Where detection runs | @@ -1118,7 +1167,8 @@ A validate's `downloads_required` lists the detector file as a file entry when i the workflow's to declare. The defaults (8, 1, 32) are LTX's 8n+1 frames and 32-pixel steps, so a workflow that names none gets the output it always did; the `face-repair` template passes `modulus: 8, remainder: 1, multiple: 32` -explicitly. A `modulus` below 1, a `remainder` below 0 or not below `modulus`, +explicitly. A model on another grid names its own: MiniMax H3's is `modulus: +17, remainder: 5, multiple: 32` (17n+5 frames, 32-pixel steps). A `modulus` below 1, a `remainder` below 0 or not below `modulus`, and a `crop_size` that is not a multiple of `multiple` are refused at validation. @@ -1145,8 +1195,21 @@ change it. It is 1 when face width over frame width is at or below `gate_full`, 0 at or above `gate_zero`, linear between, and multiplied by the hold decay on frames the detector missed. -It returns `{crops, track}`. `crops` is a video (no audio) at the clip's frame -rate. `track` is a JSON record saved as its own `.json` file, readable by +`min_frames`, `max_frames` and `fps` refuse a clip the model the crops feed +cannot take, before the detector loads: the window is checked against the +padded crop count, and the message names both the clip's frame count and the +count it pads to (`the clip's 346 frames pad to 362 crops on the 17n+5 grid, +above max_frames 345`). A clip that records no frame rate is refused when +`fps` is set. + +It returns `{crops, track}`. `crops` is a video at the clip's frame rate. When +the clip has a soundtrack, `crops` carries it padded to match: `pad_before` +frames of silence, the clip's own sound, then `pad_after` frames of silence, so +the track is exactly `crop_frames / fps` seconds long and a step holding the +crops' audio (`hold_audio: previous_result:.crops`) lines up with them +frame for frame. A silent clip gives crops with no audio; a soundtrack on a +clip with no frame rate to place it by is dropped, with a `crop_audio_dropped` +warning. `track` is a JSON record saved as its own `.json` file, readable by `previous_result:.track`: | Field | Description | @@ -1164,8 +1227,10 @@ rate. `track` is a JSON record saved as its own `.json` file, readable by A clip with no face still succeeds: every strength is 0, the crops are the centre square, and a `no_face_found` warning is raised. `validate_workflow` refuses a `crop_size` that is not a multiple of 32, a `gate_zero` at or below -`gate_full`, a `padding` out of range, a `detector_repo` that is not a repo id -and a `detector_file` that is not a `.onnx` name, before anything downloads. +`gate_full`, a `padding` out of range, a `detector_repo` that is not a repo id, +a `detector_file` that is not a `.onnx` name, a `min_frames`, `max_frames` or +`fps` at or below 0, and a `max_frames` below `min_frames`, before anything +downloads. ### paste_face_track @@ -1193,6 +1258,7 @@ using the `track` record `crop_face_track` returned: | `track` | Yes | The `track` record `crop_face_track` returned (`previous_result:.track`), or the path of its saved `.json`, such as an `output:` reference | | `feather` | No | Fraction of the paste's radius that fades out, `0` to `1`. Default `0.3` | | `color_match` | No | Shift each crop's mean colour onto the source's inside the mask before blending. Default `true` | +| `fps` | No | The output's frame rate: omitted or `"source"` for the clip's own, or a number above zero for that rate. No frames are added or dropped | The pad frames are dropped: the crop at `pad_before + i` goes back on source frame `i`. Each crop is resized to the square it was cut from, then blended @@ -1207,11 +1273,12 @@ through exactly. With `color_match`, the crop's mean colour is shifted onto the source's inside the mask, so a repair pass that drifted in tone does not show as a patch. -It returns the source clip with its audio, frame rate and shots carried, the -same frames one for one. It refuses a track whose frame count or frame size +It returns the source clip with its audio and shots carried, the same frames +one for one, at the clip's frame rate unless `fps` sets another. It refuses a track whose frame count or frame size does not match the clip, a `repaired` with fewer frames than the track's `crop_frames`, and a `track` that is not a `crop_face_track` record. -`validate_workflow` refuses a `feather` past `1` before the job runs. +`validate_workflow` refuses a `feather` past `1`, and an `fps` that is +neither `"source"` nor a number above zero, before the job runs. **Example:** [face-repair.json](../workflows/templates/ltx2/face-repair.json) — the repaired crops pasted back over the source, its soundtrack kept. @@ -1979,6 +2046,65 @@ out of order, a non-whole `beat_index`), at `validate_workflow`; an anchor past the song's end, or a `beat_index` past the detected count, at run start. It returns JSON, so `result` may only be `application/json`. +### analyze_phrases + +Find when a song's sung phrases start and stop, from voiced energy alone - no +Whisper, no model download, seconds of runtime. Where Whisper cannot hear the +words (a non-English verse, a hallucinated transcript, wordless vocals) this +still says when the lines are sung. Feed it the vocal stem where there is one; +the result is JSON and nothing is built: + +```json +{ + "name": "phrases", + "task": { + "command": "analyze_phrases", + "arguments": { + "audio": "previous_result:stems.vocals" + } + }, + "result": { "content_type": "application/json" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `audio` | Yes | The song or its vocal stem: a path, `asset:`/`output:` reference of an audio or video file, a waveform, or an earlier step's generated audio (`previous_result:.vocals`) | +| `min_gap_seconds` | No | A silence shorter than this between voiced runs is bridged into one phrase (default `0.35`). Raise it to merge phrases split by breaths, lower it to split lines sung close together. At least 0 | +| `min_phrase_seconds` | No | A phrase shorter than this, after bridging, is dropped as a click or breath (default `0.3`). Above 0 | +| `floor_db` | No | The level, in dBFS of a 20 ms frame, a frame must reach to count as voiced. Unset, it is the track's 95th-percentile frame less 35 dB, never below -60 dBFS (the floor `attribute_voices` uses). It is the knob for a fused legato or reverberant verse: when the voice never drops below the relative floor between lines, raising it (less negative, e.g. `-30`) splits them where `min_gap_seconds` can't - then check the phrases by ear before trusting a one-to-one pairing: a raised floor can make the counts match on wrong boundaries. At or below 0 | +| `centre` | No | `true` (the default) weights a stereo file to its centre, where a lead vocal is mixed; `false` uses the band energy of the downmix. A real boolean - `"false"` is refused | +| `sample_rate` | With a waveform | Rate of a directly passed waveform (files carry their own) | + +Returns `{phrases: [{start, end}], duration_seconds, method, warnings}`, in +seconds rounded to 0.01; the phrases ascend and do not overlap. + +- **`method: "centre"`** (stereo, `centre: true`): mid = (L+R)/2 and side = + (L-R)/2 are both bandpassed to 250-3500 Hz, and a 20 ms frame's power is + max(0, P_mid - P_side), smoothed over three frames. What is panned away from + the centre cancels, so a hard-panned guitar or noise bed does not read as + singing - and a vocal panned hard to one side does not either: use + `centre: false` for it. +- **`method: "band"`**: the band energy of the downmix, for a mono file (which + also warns), for `centre: false`, and for a stereo file whose channels are + the same (it has no side to cancel, so no warning). + +A frame is voiced at or above the floor; gaps shorter than `min_gap_seconds` +are bridged, then runs shorter than `min_phrase_seconds` are dropped. + +Warnings: no phrases found (a silent track, or one whose loudest frame is +under -50 dBFS in the voice band - room tone, hiss - unless `floor_db` is +given); and one phrase covering more than 90% of the track. The task can't tell +a stem from a mix, so it names both remedies: on a mix whose centred bass and +kick fill the voice band, feed it `separate_stems`' `vocals` instead; on a +vocal stem already (legato or reverberant singing), raise `floor_db` and check +the phrases by ear. `warnings` repeats what was emitted. + +Refused at `validate_workflow`: a missing `audio`, a negative +`min_gap_seconds`, a `min_phrase_seconds` at or below 0, a `floor_db` above 0, +and a `centre` that is not `true` or `false`; the run refuses the same. +It returns JSON, so `result` may only be `application/json`. + ### plan_cuts Plan a music video's cuts from a song's lyrics and beats. `transcribe_audio` @@ -2011,12 +2137,12 @@ the list rendered. A transcript chunk `transcribe_audio` marked `suspect` `no_wo | Argument | Required | Description | | -------- | -------- | ----------- | -| `transcript` | Yes | `transcribe_audio`'s result with `timestamps` set (`"segment"` or `"word"`): a `{text, chunks}` dict of `{start, end, text}` chunks in seconds. A null `end` runs to the next chunk's start, or the song's end. Its `result` must be `application/json` | -| `lyrics` | No | The song's own lyrics, one line per line with a blank line or a lone section tag (`[Chorus]`) between stanzas, or a list of lines. The shots carry these lines verbatim and in order; the transcript only times them. `null` or `""` means none: the transcript's text is used | +| `transcript` | Yes | `transcribe_audio`'s result with `timestamps` set (`"segment"` or `"word"`): a `{text, chunks}` dict of `{start, end, text}` chunks in seconds. A null `end` runs to the next chunk's start, or the song's end. Its `result` must be `application/json`. Or an [`analyze_phrases`](#analyze_phrases) result (recognised by its `phrases` key), for a song Whisper can't time: each phrase is one line, and suspect filtering and word alignment don't apply. Without `lyrics` a phrase is a line with no lyric (shot `lyric: null`, kind vocal), and `segment_by` `line`, `stanza` (phrases a silence of at least `min_gap_seconds` apart are grouped) and `beat` all work. With `lyrics` of as many lines as phrases they map one to one, in order; a warning names any line whose phrase is under 1/4 or over 4x the median phrase length, since its boundaries may be wrong (the pairing stands - check it by ear). With a different count a warning names both counts and the knob: raise `analyze_phrases`' `min_gap_seconds` when there are more phrases than lyric lines (it bridges more pauses), lower it when fewer, or, for fused legato or reverberant lines, raise its `floor_db` (e.g. `-30`) and check the phrases by ear before trusting the pairing; the lines are then spread evenly from the first phrase's start to the last phrase's end, never paired silently. A long phrase split into pieces gives every piece its whole lyric | +| `lyrics` | No | The song's own lyrics, one line per line with a blank line or a lone section tag (`[Chorus]`) between stanzas, or a list of lines. The shots carry these lines verbatim and in order; the transcript only times them (an `analyze_phrases` transcript: one line per phrase, see `transcript`). `null` or `""` means none: the transcript's text is used (a phrase result has none: `lyric` is `null`) | | `beats` | For `segment_by: "beat"` | `analyze_beats`' result, or a list of beat times in seconds | | `segment_by` | No | `line` (default), `stanza` or `beat` | | `fps` | No | Frames per second the shots are counted in (default `24`) | -| `duration_s` | No | The song's length; `analyze_beats`' `duration_seconds` when `beats` is that whole dict, else the transcript's last end (with a warning). `"beats": "previous_result:beats"` passes only the result's `beats` list - the engine takes the key an argument is named for - so name the length too: `"duration_s": "previous_result:beats.duration_seconds"` | +| `duration_s` | No | The song's length; `analyze_beats`' `duration_seconds` when `beats` is that whole dict, else a phrase result's `duration_seconds`, else the transcript's last end (with a warning). `"beats": "previous_result:beats"` passes only the result's `beats` list - the engine takes the key an argument is named for - so name the length too: `"duration_s": "previous_result:beats.duration_seconds"` | | `min_scene_s` | No | The shortest a shot may be (default `1.0`) | | `max_scene_s` | No | The longest a shot may be (default none) | | `vocal_tail_s` | No | Seconds a sung shot holds past its last word (default `0`) | @@ -2086,7 +2212,7 @@ on) and `lyrics` with no sung lines. [music-video-cuts.json](../workflows/templates/minimax/music-video-cuts.json) chains `separate_stems`, `transcribe_audio` (of the vocal stem, so an instrumental intro is not folded into the first word), `analyze_beats` (of the full mix) and `plan_cuts`, passing H3's grid (`17`, `5`, `124`, `345`) and a `lead_s` variable (default `0.5`, 12 -frames at 24 fps). Its shots are what +frames at 24 fps). When Whisper can't time the words (a non-English verse, a hallucinated transcript, wordless vocals), swap the `transcribe` step for an `analyze_phrases` step on `previous_result:stems.vocals`, keep `plan_cuts`' `transcript` on that step and pass `lyrics`, one line per sung phrase; the default chain is unchanged. Its shots are what [music-video.json](../workflows/templates/minimax/music-video.json) renders: the agent writes a prompt per shot, drops `lyric` and `kind`, and passes the list as `shots`. @@ -2284,6 +2410,19 @@ The model is fixed (`htdemucs`, run in fp32), its weights download on first use, and it needs the `demucs` package. On MPS a failed separation falls back to the CPU and warns (`separation_cpu_fallback`). +**Dialogue and the rest.** The same split takes a dialogue shot apart from a +score mixed under it: `vocals` is the dialogue and `drums` + `bass` + `other`, +summed with `mix_audio`, is the rest. Measured on H3 shots under a Music 3 +score, the `vocals` stem matched the dry shot within 0.1 dB rms, and the +score's leak into it sat about 30 dB under the bed +([record](proposals/audits/2026-11-dialogue-stems.md)). The rest is not a +clean score, though: htdemucs puts a shot's own SFX and foley in `drums` and +`other`, so they come out with the music. Keep the score file when the goal is +to replace the music. Transcribing the `vocals` stem also gives the line where +transcribing the dry shot can loop on a hallucination; a speechless shot gives +a near-silent `vocals` stem and an invented transcript, so judge "no dialogue" +by the stem's level. Single-speaker dialogue doesn't raise `stacked_vocals`. + **Stacked vocals.** A chorus is often recorded as layered stereo doubles - every Music 3 chorus is - and htdemucs returns all of them in the one `vocals` stem. When the stem's left and right channels sing different lines over a sung span, @@ -2396,9 +2535,10 @@ picking better-separated reference spans is the fix, not reading past it. Both models are fixed - there is no model-name argument - and both run in fp32 (their STFT front ends are fp32-only in practice, and both are small -enough that it costs nothing). Weights download on first use - htdemucs from -Hugging Face (`adefossez/HTDemucs` via demucs 4.1), ECAPA from speechbrain - -and are cached between calls like any other model. On MPS a `separate: true` +enough that it costs nothing). Weights download on first use - htdemucs +through demucs' own downloader (from `dl.fbaipublicfiles.com` into the torch +hub cache, outside the Hugging Face cache), ECAPA from speechbrain - and are +cached between calls like any other model. On MPS a `separate: true` run that fails in htdemucs falls back to the CPU and warns (`separation_cpu_fallback`) rather than failing the step. @@ -3029,6 +3169,7 @@ Speak a line of text with a local text-to-speech model. A validate's `downloads_ | Bark | `suno/bark-small` (the command's default) | Distinct speakers by `voice_preset` | | MMS / VITS | `facebook/mms-tts-eng`, and `mms-tts-` per language (the template's default) | One voice, a quarter Bark's size; a multi-speaker VITS takes `speaker_id` in `forward_params` | | SpeechT5 | `microsoft/speecht5_tts` | The voice of a reference clip, given as `speaker_embedding` | +| Audio8-TTS | `Edge0/Audio8-TTS-Realistic-0.6b` | A cloned voice from `reference_audio` + `reference_text`, or its default voice; 44.1 kHz mono, many languages from one checkpoint | The result is a waveform carrying the rate its model generated at, so it composes with [`slice_audio`](#slice_audio), [`fade_audio`](#fade_audio) and [`pair_audio`](#pair_audio) directly ([`concat_videos`](#concat_videos) and `dissolve_videos` join videos — pair the track onto a video first). @@ -3052,6 +3193,8 @@ The result is a waveform carrying the rate its model generated at, so it compose | `model_name` | No | HuggingFace model ID (default: `suno/bark-small`) | | `voice_preset` | No | The speaker, for a model with presets — `v2/en_speaker_0` through `v2/en_speaker_9` for Bark. A model with no processor (a single-voice model such as `facebook/mms-tts-eng`) refuses a `voice_preset` with an error rather than ignoring it | | `speaker_embedding` | No | A reference audio file (typically an `asset:` reference) whose voice a SpeechT5 model should speak in. Reduced to an x-vector with speechbrain's `spkrec-xvect-voxceleb` and injected into `forward_params` as `speaker_embeddings`. A model that isn't SpeechT5 refuses it the same way a single-voice model refuses `voice_preset` | +| `reference_audio` | No | Audio8-TTS only: a clip of the voice to clone (typically an `asset:` reference), up to 20 s; 5-10 s of clean speech is plenty. Needs `reference_text` | +| `reference_text` | No | Audio8-TTS only: the words spoken in `reference_audio`, verbatim. Needs `reference_audio` | | `forward_params` | No | Passed to the model's forward/generate call | | `generate_kwargs` | No | Ad-hoc generation settings for a generative model — `temperature`, `do_sample` | @@ -3061,6 +3204,23 @@ The command's own default is Bark because its voice presets give distinct speake The result needs no `sample_rate`. A generated track carries the rate its model produced it at, and that beats the 44100 default; declaring one still wins over both, for a track whose rate was reported wrong. Every TTS model runs at a different rate, so a declared rate that does not match plays the speech at the wrong speed and pitch without ever failing. +### Cloning a voice (Audio8-TTS) + +`model_name: "Edge0/Audio8-TTS-Realistic-0.6b"` takes its own branch. The model is vendored (`dw/vendored/audio8_tts_model.py`), so it loads no code from its repo, and its files - `config.json`, `generation_config.json`, `model.safetensors`, `tokenizer.json` and the codec `codec.pth` - are fetched at one pinned revision; a validate's `downloads_required` lists all five. The result is a 44.1 kHz mono track. + +Give `reference_audio` and `reference_text` together and it speaks `text` in the reference's voice; leave both out and it speaks in its default voice. The language follows `text` - there is no `language` argument. Every refusal fires before the model loads: + +- `reference_audio` without `reference_text` (an empty one counts as missing), or the reverse; +- either one on any other model; +- `speaker_embedding`, `voice_preset` or `messages` on Audio8; +- a reference longer than 20 s; +- text that, with the reference, won't fit the model's 2,048-position context. It is refused, never truncated: split the line; +- a `generate_kwargs` / `forward_params` key outside `temperature`, `top_p`, `top_k`, `do_sample`, `max_new_tokens`. `max_new_tokens` defaults to the room left in the context. + +The fit check is an estimate from the text's length; a line that passes it but still runs out of room before the model finishes is refused after generation rather than returned cut off. The workflow's `seed` fixes the take: the same seed, text and reference give the same audio. + +[`voice-clone.json`](../workflows/templates/voice-clone.json) is the template. **Clone only a voice you have the speaker's consent to use.** + ### Generating a voice to condition on The role this earns its place in is voice *timbre reference*, not the track a mouth follows. A `MiniMaxH3AudioReference` is audio H3 reads, not audio it plays: it takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line, and a mouth asked to follow a whole track through a reference follows it loosely. `music-video` and the `match_audio` chain templates pass each shot's slice as one, and a reference alone can re-sing the track shifted by whole beats (#795). Holding the track with `hold_audio` (the `workflows` guide, "H3: generating to a held soundtrack") keeps it exact: `music-video` holds each slice by default, the chains leave it null. #619 found hold with image-only references worse than the reference alone on a sung track; #808 measured hold plus the reference at 0.00 s offset on every shot against up to 1.37 s for the reference alone, audio only, mouth onsets not counted. Build the reference with `from_previous_result` and the clip's own sample rate comes across with it: @@ -3081,8 +3241,49 @@ A speech model is worth releasing before a video model loads — set `release_mo **Examples:** - [generate-speech.json](../workflows/templates/generate-speech.json) — Speak a line and save it as a `.wav` +- [voice-clone.json](../workflows/templates/voice-clone.json) — Speak a line in the voice of a reference clip (Audio8-TTS) - [voice-timbre-reference.json](../workflows/templates/minimax/voice-timbre-reference.json) — Generate a voice, then condition H3's `