[manupawickramasinghe]

Performance and code-quality pass over the Python per-frame paths and the four Rust crates. All the numbers below come from running the code in the frigate-ncnn-vulkan-npu image on the dev host. Frigate was running on the same host, so expect some noise, but every gain here is well beyond it.

Summary

1. The ctypes call cost more than the work for several Rust helpers, so they now use Python/OpenCV. In two cases the Rust path also gave different results from the fallback.
2. Removed FFI exports that nothing calls. The PR is +1,483 / −2,498 lines in total; most of the net removal is dead Rust kernels, and a large share of the Rust churn is the separate cargo fmt commit.
3. Clippy now reports 0 warnings, down from 77, across all four crates, with no allow attributes. Every pub unsafe extern "C" fn now has a # Safety section.
4. Birdseye only redraws cameras whose frame changed: about 6x cheaper per composed frame, and the output is byte-identical.
5. Smaller cleanups: lazy logging in TrackedObject.update, rust_yolo prototypes declared once, and two Rust-only tests now skip when Rust is missing.

1. Rust dispatch that was slower than the fallback

I timed each helper that dispatches to Rust against its own fallback, with per-frame-sized inputs (min of 5 repeats):

helper (input)	via Rust	Python / OpenCV	same result?
intersection_over_union (1 pair)	1.71 µs	1.11 µs	no: Rust worked in f32 (0.638741672 vs 0.6387416736)
intersects_any (3 / 20 boxes)	3.37 / 9.32 µs	0.56 / 1.24 µs	yes
inside_any (3 / 20 boxes)	3.43 / 9.43 µs	0.61 / 1.38 µs	yes
average_boxes (10)	5.55 µs	1.71 µs	yes
median_of_boxes (10)	5.71 µs	1.44 µs	Rust returned a new list, not the input box
create_tensor_input rgb 320→320	1025 µs	179 µs	no (see below)
create_tensor_input rgb 640→320	1885 µs	376 µs	no
zone point_in_polygon (never wired)	1.86 µs (0.41 µs with a pre-built ctypes array)	cv2.pointPolygonTest 0.25 µs	–

These helpers now always use the Python/OpenCV implementation. The Rust kernels and bindings behind them are removed.

create_tensor_input is the one that matters most. It runs for every detection region on every frame, and the fused Rust crop+convert+resize took 5x longer than yuv_crop_and_resize + cv2.cvtColor + cv2.resize. It also used full-range BT.601 coefficients, while OpenCV's COLOR_YUV2RGB_I420 uses video-range. As a result, the detector saw a lower-contrast image whenever Rust was present: video white (Y=235) came out as 235 instead of 255, and black (Y=16) as 16 instead of 0. frigate/test/test_box_helpers.py now checks that create_tensor_input matches the OpenCV conversion and maps video-range white and black to 255 and 0. 14 of its checks fail against the old code.

I did not wire the zone point_in_polygon / polygon_box_overlap bindings into tracked_object.py. cv2.pointPolygonTest is already 7x faster than the binding, and the Rust ray-cast does not count points on an edge as inside, while pointPolygonTest(...) >= 0 does.

Kept, because Rust is faster and gives the same result: reduce_boxes (7.1 vs 15.0 µs), get_cluster_candidates (7.4 vs 60.9 µs; same clusters, deterministic order), the norfair distance (2.1 vs 21.5 µs) and its 5x5 matrix (6.6 vs 560 µs), percentile_15_85 (4.3 vs 110 µs), ncc_u8 (16 vs 52 µs), yuv_crop_and_resize (38 vs 42 µs), yuv_to_3_channel_yuv (158 vs 213 µs), reduce_detections (it scales better: 27 vs 62 µs at 25 detections, and is even at small counts).

Exports removed because nothing calls them: point_in_polygon, polygon_box_overlap, fast_shm_copy (only a throughput test used it; it was a plain memmove), preprocess_detect_input, yuv420_to_rgb and nhwc_to_nchw_f32 in frame-rs. In motion-rs: motion_detect_full, motion_detect, motion_init_average, motion_accumulate_weighted, plus the kernels only they used (contrast stretch, histogram percentile, masked average update, accumulate). get_anchor_free_grid_points is still used by anchor_free_post_process, so it stays.

2. Clippy and fmt

crate	before	after
frame-rs	40	0
motion-rs	15	0
yolo-rs	13	0
detector-rs	9	0

These are unique cargo clippy --release --all-targets warnings. About half disappeared with the dead code. For the rest:

• # Safety sections now state the actual contract: buffer lengths in elements, aliasing, and what the null/zero guards do.
• Argument-count warnings were fixed by refactoring. The 8-argument internal helpers were refactored, so no allow(clippy::too_many_arguments) was needed:
  • calculate_region_i32 takes a frame tuple and a bounds array.
  • decode_yolo_scale appends into a Candidates struct instead of returning a 3-tuple of fresh 1024-capacity Vecs per scale.
• Iterator rewrites in the anchor and DFL loops keep the arithmetic order, so results are unchanged.
• detector-rs:
  • Dropped the never-constructed Msg::Shutdown; the threads already stop when the channel closes.
  • Switched to io::Error::other and is_multiple_of.
  • The f16 conversion test now compares bits exactly. Its old abs() <= f32::EPSILON tolerance accepted any value below 1.2e-7, so it could not catch a wrong subnormal.
• Doc fixes: the crate docs are now inner //! comments, and the yuv420_crop_repack doc no longer says the chroma output is interleaved (it is planar I420).

cargo fmt is in its own commit (e1cccb0a4, formatting only) so it can be skipped when reviewing.

3. Birdseye: redraw only changed cameras

Each composed frame used to resize all five YUV planes of every camera in the layout, even though usually only one camera had a new frame. Tiles only move when the layout is reset, and that path clears the canvas. Each camera now carries a frame sequence number and the number last painted. The copy is skipped when they match, and both are forgotten on clear_frame().

Measured with testing-scripts/benchmark_birdseye.py (added; 8 cameras at 1280x720, 5 fps each):

	before	after
composing update()	2267–2870 µs	381–413 µs
40 frames/s in, 10 fps out	46–50 ms CPU/s	28–29 ms CPU/s

The canvas checksum is identical. frigate/test/test_birdseye_redraw.py checks three things: an unchanged camera is not redrawn, the incrementally built canvas equals a full redraw, and a layout reset redraws everything.

4. Smaller changes

• TrackedObject.update now uses %-style lazy logging, as the project's logging guidelines ask. The speed-zone debug message, which runs ~2 µs of numpy to format the velocity, is only built when debug logging is on. It also uses the loop's zone instead of looking the zone up again by name. There is no measurable change for a normal object (11.6 µs per update with 4 zones, before and after), so this is consistency, not a speed claim.
• rust_yolo now declares its ctypes prototypes once at load, like frame_rs and rust_engine already do. Previously argtypes was re-assigned on every call, which raced across threads. A library missing a symbol is now treated as unavailable. Timings: yolo26_post_process (2100 cells) 105.0 → 97.9 µs, nms_boxes (50) 8.8 → 8.4 µs.
• box_inside had its condition duplicated; it is now a single expression. Type hints were added to the box helpers touched here.
• Two Rust-only parity tests (test_transform_detect_input_and_normalize_parity, test_yolo26_and_anchor_free_and_nms_post_process) now skip instead of erroring when the libraries are missing.

Test results

• Python suite with Rust (freshly built libs mounted at /opt/frigate): Ran 1004 tests, OK (skipped=1). Baseline dev was 991 tests: this PR adds 14 and removes 1 (the fast_shm_copy throughput test).
• Python suite without Rust (an empty file mounted over each /opt/frigate/libfrigate_*_rs.so, and no .so anywhere on the search path): Ran 1004 tests, FAILED (failures=1, skipped=5). The only failure is test_rust_libraries_loaded, which by design asserts that the libraries load. The new test_box_helpers and test_birdseye_redraw pass both ways (14/14).
• cargo test --release: frame 22, motion 15, yolo 12, detector 2, all passing. cargo fmt --check is clean.
• ruff 0.15.20: ruff check frigate migrations docker *.py reports "All checks passed!", and ruff format --check reports 403 files already formatted.

Not changed, and why

• The motion-rs pixel pipeline is not equivalent to the OpenCV fallback, and I left its behaviour alone. I renamed dilate_3x3 to dilate_cross_3x3 and documented the differences on motion_pixel_pipeline:

  • a [1 2 1]/4 blur instead of gaussian_filter(sigma=1);
  • >= instead of > for the threshold;
  • a 3x3 cross dilation instead of a 3x3 rectangle;
  • 4-connected pixel-count components with area >= min instead of 8-connected findContours/contourArea with area > min.

  On a synthetic 720p scene (noise plus three moving rectangles, 270 frames after calibration), the two paths gave identical box lists on only 3% of frames, with an average of 38.6 vs 8.0 boxes per frame. Making Rust match OpenCV would change motion results in production, so it needs its own PR and a decision. Open question: should the Rust path be made to match the OpenCV fallback, or the other way round?

• transform_detect_input / normalize_u8_to_f32 dispatch is kept. Rust is about even with numpy (976 vs 918 µs NCHW 320, 80 vs 108 µs NHWC 320). It returns a contiguous NCHW tensor where numpy returns a strided view-copy, so the net effect depends on the detector.

• Untouched: frigate/embeddings/onnx/face_embedding.py and the face kernels (bilinear_resize_u8, laplacian_variance_u8, batch_cosine_similarity_f32), read_ffmpeg_frame, VAAPI/hwaccel, and test/upstream-0.18-rust-integration / feat: sync upstream 0.18 beta with custom Rust accelerators & security patches #244.

• detector-rs only post-processes YOLO26; other architectures return 0 detections. The Python side only spawns it for arch == "yolo26", so I left it.

• The Python post-processing fallback (frigate/util/model.py) can't handle the anchor-free tensor layouts that only the Rust path supports. That's why the second parity test above skips without Rust. Adding a fallback is a separate change.

🤖 Generated with Claude Code