Skip to content

Pass DIVE camera registrations to 2-cam/3-cam VIAME pipelines - #1780

Open
romleiaj wants to merge 9 commits into
mainfrom
sealtk/registration-json
Open

Pass DIVE camera registrations to 2-cam/3-cam VIAME pipelines#1780
romleiaj wants to merge 9 commits into
mainfrom
sealtk/registration-json

Conversation

@romleiaj

Copy link
Copy Markdown
Collaborator

Multicam pipelines previously received no transform information from DIVE; the old h5 workflow relied on paths hardcoded inside the pipes. Desktop and web now hand the dataset's camera registration to the pipeline's warp processes, mirroring the stereo calibration flow:

  • One standard to_registration.json per non-reference camera is written into the job work dir; warpN (matching the pipeline camera position) receives its file via -s warpN:transformation_file=..., with the pair and direction pinned through the reader's from_camera/to_camera config since a pair may be stored in either orientation.
  • Pipeline camera order is the registration reference first (the pipes warp everything onto camera 1's frame), then display order. Which detector a pipe runs on which input is the pipe's documented contract, not something DIVE infers.
  • Only pairs registering a camera directly onto the reference are supported; pairs between two non-reference cameras are explicitly unsupported for pipelines and never reach the job. Cameras with points-only (unfitted) pairs get no settings.
  • Web sends the registration as a multicam_registration job param built from dataset meta; the worker writes the files.

Requires VIAME's dive transform_2d_io reader and warp_detections process.

@romleiaj
romleiaj marked this pull request as draft July 21, 2026 19:21
@romleiaj

Copy link
Copy Markdown
Collaborator Author

Paired with VIAME/VIAME#278.

romleiaj and others added 9 commits August 19, 2026 12:53
Multicam pipelines previously received no transform information from
DIVE; the old h5 workflow relied on paths hardcoded inside the pipes.
Desktop and web now hand the dataset's camera registration to the
pipeline's warp processes, mirroring the stereo calibration flow:

- One standard <camera>_to_<reference>_registration.json per
  non-reference camera is written into the job work dir; warpN
  (matching the pipeline camera position) receives its file via
  -s warpN:transformation_file=..., with the pair and direction pinned
  through the reader's from_camera/to_camera config since a pair may be
  stored in either orientation.
- Pipeline camera order is the registration reference first (the pipes
  warp everything onto camera 1's frame), then display order. Which
  detector a pipe runs on which input is the pipe's documented
  contract, not something DIVE infers.
- Only pairs registering a camera directly onto the reference are
  supported; pairs between two non-reference cameras are explicitly
  unsupported for pipelines and never reach the job. Cameras with
  points-only (unfitted) pairs get no settings.
- Web sends the registration as a multicam_registration job param built
  from dataset meta; the worker writes the files.

Requires VIAME's dive transform_2d_io reader and warp_detections
process.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The web path built its registration files from the pre-v2 correspondence
shape: a flat list of {id, a, b} per camera pair, emitted as pairs[].points
and stamped version 1. Format v2 replaced that with observations -- one
entry per image pair, each carrying its own points -- so this did not
merely write an outdated file, it raised KeyError: 'a' on the first
correspondence and failed the job in the worker. Every web multicam
pipeline run on a v2-registered dataset would have died there.

Build observations instead, as the inverse of
registration_output._from_registration_pairs: the store's imageA/imageB
become the file's imageLeft/imageRight, frame/enabled/source/stats carry
through, and each point's a/b pair becomes one `leftX leftY rightX rightY`
row. Stamp version 2.

The desktop path was never affected -- it writes through
buildPerCameraRegistrationFiles, which the format-v2 work already moved --
which is exactly why this was easy to miss: desktop produces correct v2
files while the worker crashes.

VIAME's dive transform reader only consumes the matrices, so the
observations travel for provenance and so a file round-trips back into
DIVE without losing which frame contributed what. Tests now assert the
version, which nothing did before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S8CVLjQDzaBGbCSKL7FiF
- cameraRegistration.spec.ts: JsonMeta does not exist (use JsonConfig);
  type the fs-extra mock the way common.spec.ts does so vue-tsc accepts
  the default export and statSync parameter types.
- crud_rpc.py: drop unused pathlib import.
- multicam_pipeline.py: black formatting.
- test_multicam_pipeline.py: VIAME's dive reader ignores the file version;
  it is DIVE's format-v2 loader that rejects other versions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
Which dataset camera feeds which inputN of a multicam pipe is the pipe's
contract -- the arctic seal 3-cam pipe runs the thermal detector on input3
and projects optical boxes onto input2 -- but DIVE was inferring it from
display order (reference camera first, then the rest), so a dataset
imported as EO, IR, UV silently ran the thermal detector on UV.

Pipes now state their slots with a `# Camera Order: EO, UV, IR` header,
parsed into pipeline metadata on desktop and web. At run time each slot
is matched to a dataset camera by name: an exact camera name, or a name
segment sharing the slot's role (EO ~ eo/rgb/optical/color/vis, IR ~
ir/thermal/lwir/flir, UV ~ uv/ultraviolet); other tokens match literally.
A slot matching no camera or several refuses the run with a message that
names the pipe's slots and the dataset's cameras. Camera 1 of the
resolved order is the warp target the registration files are built onto.
Pipes without the header keep the reference-first fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
… datasets

Which dataset camera feeds which inputN of a 2-cam/3-cam pipe was decided
silently -- from the pipe's `# Camera Order:` header matched against
camera names, and before that from display order. Make it explicit and
visible:

- Datasets carry a sensor role per camera (`cameraRoles`: eo / ir / uv),
  inferred once at import from the camera name and, failing that, from
  tokens in its image file names (KAMERA style _rgb / _ir / _uv). Only a
  unanimous answer is recorded. Desktop and web import both set it; it is
  a mutable config key so it round-trips through load/save on both.
- Running a 2-cam/3-cam pipe opens a camera-assignment step: one row per
  pipeline camera (the header's slots, or bare input1..N) with the dataset
  camera DIVE proposes -- by role when both sides have one, else by name --
  which the user confirms or changes. Unfilled or duplicated slots block
  the run. Confirming role-labelled slots saves the roles back onto the
  dataset (opt-out checkbox), so a corrected role wins over a misleading
  name next time and for every other pipe.
- The confirmed order travels as pipelineParams.cameraOrder; desktop and
  web validate it against the dataset's cameras and use it verbatim.
  Runs without it (CLI) still resolve the header by role then name; pipes
  with no header keep the reference-first fallback.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
…re time

A 2-cam/3-cam pipe whose warpN camera had no registration onto camera 1
launched anyway and died inside kwiver with "Path does not exist:
registration_camera2_to_camera1.json". Do the check before the job exists:

- Pipeline discovery (desktop + web) records registrationWarps: the input
  positions of `process warpN :: warp_detections | warp_image` in the pipe
  body, e.g. [2, 3].
- The camera-assignment dialog shows, for each warped row, whether the
  chosen camera has a fitted Aligned View registration onto camera 1, and
  blocks Run with "Register X -> Y in Aligned View first" when it does not.
- Desktop (viame.ts) and web (crud_rpc) refuse the run with the same
  message before creating the job, so CLI and API callers get it too.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
…stration-missing message

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
The design settled on the camera-assignment step always supplying the
input1..N order, which left the earlier automatic header resolution as a
dead path with a lot of code behind it. Remove it and flatten what is
left:

- Drop resolvePipelineCameraOrder / candidatesForSlot / camerasMatchingSlot
  / pipelineCameraNames / pipelineOrderedCameraNames on the client and
  resolve_pipeline_camera_order / candidates_for_slot /
  cameras_matching_slot / pipeline_camera_order on the server. Without a
  confirmed cameraOrder (API callers) both backends now use the dataset's
  stored camera order, exactly as main does. One matcher, camerasForSlot,
  remains for the dialog prefill.
- buildRegistrationPipelineArgs takes the pipe's warp positions and throws
  the register-first error itself, instead of viame.ts reverse-engineering
  the fitted set from the -s keys it had just built; its three guards for
  "this camera has a fitted pair onto camera 1" become one.
- Web import infers camera roles from names only (no per-folder item
  scan); the server's infer_camera_role drops its image-name fallback
  accordingly. Desktop keeps the image-name fallback since it has the
  list in hand.

No change on the dialog path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MD6kx1uDv4NuiKdFNnYpG2
@romleiaj

Copy link
Copy Markdown
Collaborator Author

Pass DIVE camera registrations to 2-cam/3-cam VIAME pipes

Multicam pipes got no transform info from DIVE; paths were hard-coded in the
pipes. Which dataset camera fed which inputN was guessed from display order,
so the thermal detector could silently run on the wrong camera.

This PR sends the dataset's registration to the pipe's warp processes and makes
the camera order an explicit, confirmed step.

Registration hand-off

  • One <camera>_to_<camera1>_registration.json per non-reference camera is
    written into the job dir and passed as -s warpN:transformation_file=….
  • Camera 1 of the run order is the warp target.
  • Only pairs registered directly onto camera 1 are used. Unfitted pairs are
    skipped.
  • Files are format v2. VIAME reads only the matrices; the rest is provenance.
  • Web sends the registration as a multicam_registration job param; the worker
    writes the files. (Before this, the worker crashed with KeyError: 'a' on v2
    datasets.)
  • Needs VIAME dev/dive-registration-json (dive transform reader,
    warp_detections).

Camera assignment

  • Pipes can declare slots with # Camera Order: EO, UV, IR. Tokens EO/IR/UV
    match by role (with aliases like rgb/thermal); anything else matches a camera
    name.
  • Datasets get cameraRoles (eo/ir/uv), inferred at import from camera names
    (desktop also checks image file names).
  • Running a 2-cam/3-cam pipe opens an Assign cameras dialog: one row per
    slot, prefilled by role or name. Empty or duplicate slots block Run.
    Confirming saves roles back to the dataset (checkbox to opt out).
  • The confirmed order goes as pipelineParams.cameraOrder. API calls without
    it use the dataset's stored order, as before.

Registration check before the run

  • Discovery records which inputs the pipe warps (registrationWarps).
  • The dialog shows per row whether that camera is registered onto camera 1 and
    blocks Run with "Register X → Y in the Camera Registration tab first".
  • Desktop and web refuse the run with the same message before making a job.

Cleanup

  • Removed the old automatic header-matching path (dialog always supplies the
    order now).
  • docs/Pipeline-Import-Export.md documents the header and the dialog.

Manual tests

Needs VIAME from dev/dive-registration-json and a registered multicam
dataset. utility_warp_annotations_2-cam.pipe is a model-free test pipe.

  • Assign dialog: run a 2-cam pipe with a # Camera Order: header. Verify
    slots prefill by role, swapping works, empty/duplicate blocks Run, roles are
    saved to the dataset.
  • Hand-off: with IR registered onto EO, run. Verify the job dir has
    <IR>_to_<EO>_registration.json ("version": 2) and the command has
    -s warp2:transformation_file=….
  • Missing registration: clear the IR→EO registration, run. Verify the
    dialog blocks with "Register IR → EO … first"; the API returns the same.
  • Web: same hand-off on web-girder. Verify the worker writes the file and
    the pipe finishes.
  • No header: run a pipe with no # Camera Order:. Verify bare
    input1/input2 rows in the dataset's stored order.

Stack

Standalone on main. Uses format v2 from dev/auto-register-pipeline.
Companion VIAME branch: dev/dive-registration-json.

Overall: 21 files, +1537/-10. Commits: 8bb338f through 58e1959.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant