VitPose / EKF Playground

Multi-view pose reconstruction, model building, and kinematic analysis tools for trampoline sequences.

This repository contains:

• a desktop GUI to inspect 2D detections, reconstructions, and analyses
• command-line tools to generate reconstruction bundles and run named profiles
• analysis utilities for root kinematics, DD estimation, execution deductions, trampoline displacement, observability, and 3D segment analysis

Repository Overview

Main entry points:

• pipeline_gui.py: main graphical interface
• vitpose_ekf_pipeline.py: end-to-end pipeline and core algorithms
• export_reconstruction_bundle.py: generate one standardized reconstruction bundle
• run_reconstruction_profiles.py: run a set of named reconstruction profiles

Main packages:

• reconstruction: bundle generation, dataset handling, timings, profiles, naming
• kinematics: root kinematics and 3D analysis
• camera_tools: camera metrics and camera selection helpers
• judging: DD analysis, trampoline displacement, reference codes
• preview: preview bundle loading and frame navigation
• observability: Jacobian-rank analysis
• analysis: standalone plotting and exploration scripts
• animation: GIF export scripts

Installation

1. Create the environment from the YAML file

The simplest setup is:

cd /Users/mickaelbegon/Documents/Playground
conda env create -f environment.vitpose-ekf.yml
conda activate vitpose-ekf

If the environment already exists:

conda env update -f environment.vitpose-ekf.yml --prune
conda activate vitpose-ekf

These environment files already include the developer tools used in the repo:

• black
• isort
• flake8

2. Install the project in editable mode

Minimal install:

pip install -e .

With test dependencies:

pip install -e .[test]

With a few extra utilities:

pip install -e .[full]

3. Optional: install non-PyPI dependencies manually

Some parts of the project depend on biorbd, and optionally on OpenSim-related tooling depending on your workflow.

For biorbd, a Conda install is the safest route:

conda install -c conda-forge biorbd

If you use the GUI, make sure your Python installation has Tk support.

4. Optional developer tools

Formatting and tests:

pip install black isort flake8 pytest

Input Data

Typical inputs are organized under inputs/:

• calibration file: inputs/calibration/Calib.toml
• 2D detections: inputs/keypoints/<trial>_keypoints.json
• optional TRC file: inputs/trc/<trial>.trc
• optional DD reference file: inputs/dd/<trial>_DD.json
• optional images or extracted frames: typically inputs/images/<trial>/... or another sibling folder inferred from the keypoint file

Example:

• inputs/keypoints/1_partie_0429_keypoints.json
• inputs/trc/1_partie_0429.trc
• inputs/dd/1_partie_0429_DD.json

Outputs are typically written under:

• output/<dataset>/models
• output/<dataset>/reconstructions
• output/<dataset>/figures

Launching the GUI

Run:

python /Users/mickaelbegon/Documents/Playground/pipeline_gui.py

The GUI now uses a shared reconstruction selector at the top of the window. Most analysis tabs reuse that selector instead of maintaining their own reconstruction table.

Typical workflow:

1. Choose the 2D input in 2D explorer
2. Inspect cameras, flips, and candidate issues in Caméras
3. Generate models in Modèle
4. Define named reconstruction profiles in Profiles
5. Run and inspect bundles in Reconstructions
6. Compare outputs in the analysis tabs

Main analysis tabs:

• 3D animation: export comparative 3D GIFs
• 2D multiview: export multi-camera 2D GIFs
• DD: jump segmentation and DD estimation
• Execution: localized execution deductions with 3D and 2D overlays, ready for future image overlays
• Toile: horizontal displacement scoring on the trampoline bed
• Racine: root translations, rotations, or rotation matrices
• Autres DoF: left/right joint comparison
• 3D analysis: segment-length boxplots and angular momentum
• Observabilité: Jacobian rank across frames

Running the Main CLI Tools

Run one reconstruction bundle

Example:

python /Users/mickaelbegon/Documents/Playground/export_reconstruction_bundle.py \
  --name triangulation_exhaustive_flip_rotfix \
  --family triangulation \
  --calib inputs/calibration/Calib.toml \
  --keypoints inputs/keypoints/1_partie_0429_keypoints.json \
  --output-dir output/1_partie_0429/reconstructions/triangulation_exhaustive_flip_rotfix \
  --pose-data-mode cleaned \
  --triangulation-method exhaustive \
  --flip-method epipolar_fast \
  --flip-left-right \
  --initial-rotation-correction \
  --fps 120 \
  --triangulation-workers 6

Supported families:

• pose2sim (TRC file import)
• triangulation
• ekf_3d
• ekf_2d

Run a list of named profiles

Example:

python /Users/mickaelbegon/Documents/Playground/run_reconstruction_profiles.py \
  --config reconstruction_profiles.json \
  --dataset-name 1_partie_0429 \
  --calib inputs/calibration/Calib.toml \
  --keypoints inputs/keypoints/1_partie_0429_keypoints.json \
  --trc-file inputs/trc/1_partie_0429.trc \
  --fps 120 \
  --triangulation-workers 6

To run only some profiles:

python /Users/mickaelbegon/Documents/Playground/run_reconstruction_profiles.py \
  --config reconstruction_profiles.json \
  --dataset-name 1_partie_0429 \
  --calib inputs/calibration/Calib.toml \
  --keypoints inputs/keypoints/1_partie_0429_keypoints.json \
  --trc-file inputs/trc/1_partie_0429.trc \
  --profile ekf_2d_acc_rootq0_boot15_flip_rotfix \
  --profile triangulation_exhaustive_flip_rotfix

Main Algorithms

1. 2D pose preprocessing

The pipeline can work from:

• raw detections
• filtered detections
• cleaned detections

Cleaning includes temporal smoothing and outlier rejection based on a robust motion amplitude estimate.

2. Left/right flip detection

The project includes several strategies to detect left/right label swaps:

• epipolar Sampson-based scoring
• fast epipolar symmetric-distance scoring
• triangulation + reprojection scoring with once, greedy, or exhaustive variants

Corrected 2D variants are cached, so downstream stages can reuse:

• raw
• cleaned without flip
• cleaned + epipolar flip
• cleaned + fast epipolar flip
• cleaned + triangulation-based flip

For epipolar-family methods, the current implementation also applies a simple 2-state Viterbi decoding (normal / flipped) only when you explicitly pick epipolar_viterbi or epipolar_fast_viterbi.

3. Triangulation

Three triangulation modes are available:

• once: one weighted triangulation pass using the currently available views.
• greedy: starts from all available views and removes the worst ones step by step.
• exhaustive: tests more camera combinations and is the most robust, but also the slowest.

The triangulation stage also stores:

• per-frame reprojection error
• view usage
• excluded-camera patterns
• coherence scores

4. Root orientation extraction

For geometric reconstructions such as triangulation or a TRC file import:

• the trunk frame is built from hips and shoulders
• the root orientation is expressed with the YXZ Euler sequence
• an optional initial yaw correction (rotfix) aligns the trunk to the global frame by snapping to the nearest right angle

For model-based reconstructions:

• root rotations are read from the model generalized coordinates
• the GUI can also display the corresponding rotation matrix components

5. EKF 3D

The 3D EKF uses model-based kinematics driven by 3D marker trajectories.

Recent initialization strategies include:

• triangulation-based initialization
• root-only initialization with zero rest of the body (root_pose_zero_rest)

6. EKF 2D

The 2D EKF combines:

• the articulated model
• 2D observations in all cameras
• multiview coherence weighting
• configurable predictor (acc or dyn)

Important improvements already integrated in the codebase:

• sequential camera updates
• vectorized measurement assembly
• root-pose bootstrap initialization (root_pose_bootstrap)
• configurable coherence families: epipolar, epipolar_fast, triangulation_once, triangulation_greedy, triangulation_exhaustive

7. DD estimation

The DD tab and judging/dd_analysis.py provide:

• jump segmentation from root height
• salto / tilt / twist analysis
• DD code inference
• comparison with expected codes loaded from *_DD.json
• comparison of each reconstruction against the expected DD reference with color-coded status

8. Execution deductions

The Execution tab and judging/execution.py provide:

• per-jump localized deductions
• a synchronized 3D view and 2D camera overlay
• session-level time-of-flight scoring
• a structure ready for direct image overlays as soon as camera frames are available

9. Trampoline displacement

The Toile tab estimates horizontal displacement penalties:

• contact windows are inferred between jumps segmented in the DD analysis
• contact position currently uses the feet as a proxy
• the bed geometry is based on a calibrated set of trampoline reference markers

10. Observability analysis

The Observabilité tab computes frame-wise ranks of:

• J_markers_3D(q)
• J_obs_2D(q)

This helps visualize when the marker or image Jacobians lose rank.

Project Conventions

• default worker count is 6
• default output root is output/
• formatting uses isort + black
• linting uses flake8
• tests live in tests
• the project uses a local Matplotlib cache under .cache/matplotlib

Development

Run tests:

pytest -q

Format code:

isort . --profile black
black .

Lint code:

flake8 .

The repository contains a single CI workflow under:

• .github/workflows/ci.yml

Notes

• The GUI is the easiest entry point if you want to explore reconstructions interactively.
• The CLI tools are better when you want reproducible named runs and cached bundles.
• Some advanced analyses require reconstructions with q, qdot, and an associated .bioMod.