OTC Multicam Fusion (Hoyerswerda)

Open-source, geometry-first multi-camera trajectory fusion for traffic management — designed to work within the OpenTrafficCam ecosystem (OTCamera, OTVision, OTAnalytics). The approach projects detections/tracks into a shared metric world frame (planar Z=0), associates them across cameras, and yields quality-assured fused trajectories plus stream-level event summaries for traffic analysis and planning.

Status: This repository is a documentation snapshot accompanying the submitted Diplomarbeit. It preserves the method, code structure, and references for readers and reviewers.

Table of Contents

• Why
• Key Features
• How It Works (System Overview)
• Assumptions & Scope
• What’s in this Repository
• Quick Start
• Installation
• Configuration
• Inputs & Outputs
• Design Choices & Limitations
• Repository Structure
• Citing
• License
• Acknowledgements

Why

Single-camera tracking breaks down under occlusion, shadows, and adverse weather. Multi-camera fusion recovers completeness and robustness by associating consistent world-space footprints of the same vehicle across overlapping fields of view. This project packages a practical, geometry-first method that is explainable, FGSV-aligned, and compatible with OTC pipelines.

Key Features

• Geometry-first fusion: project per-camera pixel tracks to a metric world CRS (e.g., EPSG:25833) on Z = 0, estimate BEV footprints, and associate across cameras using gated costs (IoU, distance, heading) with a greedy solver for clarity and speed.
• State fusion: maintain fused object lifecycles and UKF-based states, close gaps, and emit canonical trajectories for downstream analytics.
• OTC-native I/O: consume OTVision/OTAnalytics track exports and OTFlow sections, aggregate events, and write GeoPackage/CSV for further use.
• Operational guidance: includes tools for calibration validation, time synchronization, and privacy-aware region-of-interest handling in the validation/ toolkit.

How It Works (System Overview)

1. Per-camera preprocessing: FrameProcessor ingests videos, detections, and calibration to produce synchronized world-space observations.
2. Cross-camera association: configurable cost components gate and score candidate merges before greedy assignment.
3. Fusion & lifecycle: the FusionEngine updates multi-camera tracks, manages lifetimes, and maintains unscented Kalman filter (UKF) state estimates.
4. Event derivation: EventLogger densifies trajectories, intersects ROIs/sections, and emits CSV/GPKG outputs for OTAnalytics workflows.

Assumptions & Scope

• Static, fixed cameras with planar projection (Z = 0) and consistent world CRS (e.g., EPSG:25833).
• Timestamp alignment via GNSS/PTP or post-hoc offsets (time_sync/).
• Focus on vehicular traffic; person/bike support exists but has not been benchmarked in the thesis submission.
• Offline or pseudo-stream processing; real-time execution is a follow-up task.

What’s in this Repository

This repository is intended for reading and reference:

• Method description, code layout, and implementation hints for researchers/engineers.
• Inline implementation references (markers) pointing to source regions for context.
• No supported CLI or “one-click run”; scripts are illustrative and may reference non-public data.

Quick Start

Workflow:

1. Configure paths (config.py): set BASE_DIR, SAVE_DIR, CAMERA_LIST; point to tracks (.ottrk/.zst), ROIs (.otflow/GeoJSON), calibration (*.h5), optional time-offset CSV; review CRS/fps/thresholds.
2. Calibrate intrinsics (calibrate_intrinsics.py): record checkerboard/charuco, run intrinsic calibration, verify low reprojection error, save per-camera calibration
3. Calibrate extrinsics (calibrate_extrinsics.py): collect GCPs, solve pose to world CRS (Z=0), validate projections/residuals, store with intrinsics.
4. Run multicam (run_multicam.py): ensure inputs exist, execute runner, then check outputs (tracks.gpkg, detections.gpkg, events.csv) and logs; if off, revisit time sync/calibration/matching params.

Example outputs:

• out/..._geo_tracks.gpkg — fused trajectories in world coordinates (LineString)
• out/..._geo_detections.gpkg — per-observation footprints (Point)
• out/..._events.csv — section-level events for OTAnalytics
• out/... .log — quality log for traceability

Installation

./install.sh

Configuration

Fusion behavior is controlled via dataclass-backed configs. Use the field names directly if you materialise them in YAML/JSON.

key	type	default	description
fps	float	20.0	Target processing framerate (frames per second).
max_missed_frames	int	5	Frames a track may be absent before pruning.
match_threshold	float	0.6	Maximum composite cost for accepting a merge (lower = stricter).
window_frames	int	20	History window when evaluating merges.
disallow_same_camera_merge	bool	False	Prevent merging tracks originating from the same camera.
max_position_distance_m	float	2.0	Gating distance for centroid proximity (meters).
max_heading_diff_rad	float	0.785	Maximum heading deviation (radians).
w_dist, w_head, w_iou	float	1.0 / 1.0 / 2.0	Relative weights for cost aggregation.
meas_sigma_pos	float	0.5	Position measurement noise (meters).
meas_sigma_theta	float	0.087	Heading measurement noise (radians).
min_confidence	float	0.1	Minimum detection confidence accepted.
ukf.*	see below	—	Unscented Kalman filter process/measurement noise and scaling.

UKF defaults:

key	default	units
alpha	0.5	—
beta	2.0	—
kappa	0.0	—
q_pos	0.5	meters
q_theta	0.087	radians
q_v	0.5	m/s
q_omega	0.087	rad/s
r_pos	0.5	meters
r_theta	0.087	radians

Inputs & Outputs

Inputs

• Per-camera tracks (*.ottrk.zst / .ottrk): exported detections and track histories from OTCamera/OTVision.
• ROI/Sections (*.otflow, *.gpkg): defines counting sections and privacy masks consumed by SectionIndex.
• Calibration (*.h5): camera intrinsics/extrinsics and projection metadata used by the pose estimator.
• Time offsets (*.csv): video start offsets per camera to align frames globally (optional but recommended).

Outputs

• Fused trajectories (*_geo_tracks.gpkg): canonical tracks as LineStrings in the project CRS, offset back to local coordinates before export.
• Per-observation points (*_geo_detections.gpkg): densified observation history for QA and debugging.
• Events (*_events.csv): section entries/exits with timestamp, speed, and heading.
• Logs (*.log): processing KPI snapshots, fusion stats, and debug output for audits.

Expected File Structure

PROJECT_DATA_DIR/
├─ intrinsics/
│  ├─ calibration_images/              # Input images for intrinsic calibration (*.png, *.jpg)
│  └─ OTC_ww_intrinsics_v01*           # Saved intrinsic calibration result (.h5/.npz)
├─ extrinsics/
│  ├─ OTCameraXX_gcp_v01.csv           # GCPs per camera (image_x,image_y,x_world,y_world[,z_world])
│  └─ OTCameraXX_calibration_v01.h5    # Saved extrinsic+intrinsic bundle per camera
├─ sync/
│  └─ video_time_offsets.csv           # Per-camera time offsets (date,camera,offset_frames/seconds)
├─ output/                             # Auto-created outputs (logs, videos, gpkg, csv)
├─ validation_sections.gpkg            # Sections/ROIs for analytics (project CRS)
└─ map.tiff                            # Optional background GeoTIFF for BEV visualization

VIDEO_BASE_DIR/
└─ OTCameraXX/
   └─ videos/
      ├─ *.mp4                         # Input videos
      └─ OTCameraXX.otflow             # Sections/ROIs for that camera

Data contracts

CanonicalObservation (per fused sample)

field	type	description
id	FusedId	Global fused identifier (0001, 0002, …).
classification	str	Dominant vehicle class across members.
track_ids	Dict[CameraID, TrackId]	Source track references by camera.
timestamp	MultiCamTimeStamp	Global time/frame indices.
world_xy	(float, float)	Projected centroid in meters (local CRS).
speed_mps	float	Instantaneous speed in meters/second.
heading	float	Heading in degrees clockwise from north.

CanonicalTrack (exported trajectory)

field	type	description
id	FusedId	Canonical track identifier.
history	List[CanonicalObservation]	Ordered observations across fusion lifetime.

SectionEvent (stream-level analytics)

field	type	description
fused_id	FusedId	Track responsible for the event.
track_ids	Dict[CameraID, TrackId]	Contributing camera tracks.
timestamp	MultiCamTimeStamp	Event timestamp (global).
section	Section	Section metadata (id, name).
x, y	float	Intersection point in CRS units.
speed	Optional[float]	Speed at the section crossing (m/s).
direction	Optional[float]	Heading at the section crossing (degrees).

Design Choices & Limitations

• Appearance-free association (no Re-ID embeddings) keeps the system explainable and aligns with FGSV documentation requirements.
• Planar projection assumes minimal vertical relief; accuracy degrades on ramps/bridges — extend the transformer for full 3D if needed.
• Clock alignment quality critically impacts fusion; review time_sync/ outputs regularly.
• Real-time deployment requires further engineering around IO throughput and process supervision; current focus is offline batch processing.

Repository Structure

.
├── bounding_box_estimation/     # Vehicle footprint projection utilities
├── camera/                      # Camera configuration helpers and defaults
├── coordinate_transformer/      # Pixel→world transformers and CRS helpers
├── importer/                    # Parsers for OTC track/event exports
├── logger/                      # Structured logging setup used across modules
├── multicam/                    # Core fusion package (processing, fusion, viz)
├── patterns/                    # YAML/CSV patterns for OTCamera pipelines
├── time_sync/                   # Tools for GPS/PTP offset estimation
├── trackers/                    # Single-camera tracker implementations
├── validation/                  # Evaluation + QA scripts for the thesis
├── run_multicam.py              # End-to-end fusion script (prototype)
├── run_multicam_parallel.py     # Experimental parallel runner
└── calibrate_*.py               # Intrinsic/extrinsic calibration utilities

Citing

If you use this work in research or deployments, please cite the associated thesis:

@thesis{gerken2025multicam,
  author    = {Sebastian Gerken},
  title     = {Automatisierte Trajektorien-Fusion in Multikamerasystemen zum Verkehrsmanagement},
  school    = {Technische Universit{"a}t Dresden, Fakult{"a}t Verkehrswissenschaften "Friedrich List"},
  year      = {2025},
  address   = {Dresden, Germany},
  type      = {Diplomarbeit},
  note      = {Reallabor Hoyerswerda / OpenTrafficCam},
}

The latest BibTeX is stored in docs/citation.bib.

License

This project is released under GNU General Public License v3.0 (GPL-3.0). Make sure the repository contains a matching LICENSE file.

Acknowledgements

• TU Dresden — Chair of Mobility System Planning
• platomo GmbH
• OpenTrafficCam community & contributors
• Reallabor Hoyerswerda partners and data providers