Add offline object inventory and localize perception pipeline DIM1343 - #3422
Open
bogwi wants to merge 2 commits into
Open
Add offline object inventory and localize perception pipeline DIM1343#3422bogwi wants to merge 2 commits into
bogwi wants to merge 2 commits into
Conversation
…43 PR1). Text prompt to 2D masks and memory search to 3D point clouds, with OWLv2, SigLIP, MPS support, LocalizePolicy, and related dependency pins.
bogwi
requested review from
Dreamsorcerer,
leshy,
mustafab0,
paul-nechifor and
spomichter
as code owners
August 10, 2026 17:56
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## main #3422 +/- ##
==========================================
- Coverage 76.09% 76.07% -0.02%
==========================================
Files 1190 1190
Lines 115295 115338 +43
Branches 10367 10372 +5
==========================================
+ Hits 87729 87746 +17
- Misses 24554 24580 +26
Partials 3012 3012
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What features we implement
From the issue feature list, this branch covers A, D, E, and F.
inventory()discovers every in-scope support on the table without a text prompt, then names each instance.localize()runs SigLIP retrieval, OWLv2 boxes, and EdgeTAMsegment()so a query string becomes masks on ranked frames.localize()lifts those masks through depth, clusters them, verifies across views, and returns the newest verified cloud and position. Offline over a recording, no live robot.inventory()merges same-frame fragments and associates tracks with geometry only (Hungarian assignment plus fragment merge). Labels are applied after identity is fixed.Not in this branch: (B) exact dimensions, (C) 360-degree clouds, (G) re-ID.
What is the proposed stack
Both paths open a memory2
SqliteStore, readStreamTFfrom it, and call intodimos.perception.memory.gatesfor poses and stillness. Both fit a support plane withfit_support_planeand lift masks withImageDetections3DPC.from_depth.Localize
(
tool_localize->localize()): SigLIP embeds and ranks frames. Retrieval keeps onlygates.camera_stillframes. Thenfit_support_plane. Pass 2 loadsOwlv2DetectorandEdgeTAMImageSegmentertogether._DetectionCache.detectruns OWLv2 boxes then EdgeTAMsegment()on the same frame._liftbuilds world clouds. Clusters form,sees()finds other viewpoints, those frames re-enter the same detect/lift path. The winner is the verified cluster with the newest observation timestamp.Inventory
(
tool_inventory->inventory()):gates.still_intervalsthengates.keyframes(camera-still and scene-still). Thenfit_support_plane.EdgeTAM
propose_alldiscovers masks with no text query._lift_frame, plane_in_scope, and_merge_same_framerun per keyframe._associatethen_merge_tracksbuild identity._name_and_suppressloads OWLv2 only after association. No SigLIP.Models in the code:
SigLIPModel(google/siglip-base-patch16-224) in localize only,Owlv2Detector(google/owlv2-base-patch16-ensemble) for localize boxes and inventory naming,EdgeTAMImageSegmenter(edgetam.pt) for localizesegment()and inventorypropose_all(). Inventory never constructs SigLIP. Localize never callspropose_all().Prerequisites
recording
Both CLIs default to the stationery calibrated memory2 db via
get_data(...):xarm6_worldbelief_realsense_d435i_stationery_calibrated/xarm6_worldbelief_20260729_203624_161992.dbget_dataresolves under the dimosdata/tree. If that path is already present, it returns it. If not, it pullsdata/.lfs/xarm6_worldbelief_realsense_d435i_stationery_calibrated.tar.gzwith git-lfs and decompresses it. Override with--datasetfor another sqlite recording. The recording must have a tf stream, orlocalize/inventoryraise.models
Three weight sets, two loading paths.
EdgeTAMImageSegmenterloadsget_data("models_edgetam") / "edgetam.pt". Same LFS rule as the recording: first miss pullsdata/.lfs/models_edgetam.tar.gz, later runs reuse the decompressed tree. Needs CUDA or MPS. CPU raises.SigLIPModelloadsgoogle/siglip-base-patch16-224.Owlv2Detectorloadsgoogle/owlv2-base-patch16-ensemble. Both callfrom_pretrained(...)withoutlocal_files_only. If the Hugging Face hub cache already has the weights, they load from cache. If not, the first run downloads them into that cache and later runs reuse it.transformersPython package itself is a dependency (dimos[perception]/sentencepiecefor the SigLIP tokenizer). That is the library install through uv, not the model weights.How to test
Localize
Group letters A-F below are command groups in this document. They are not the issue's feature letters A-G.
Template:
Omit
--from/--durationfor full recording. On expected None, drop the&& viewerpart (there should be no successful hit).A. S1 positives (all baseline objects present)
... "red marker" --from 53 --duration 76... "black pen" --from 53 --duration 76... "book" --from 53 --duration 76... "yellow sticky-note" --from 53 --duration 76"post-it pad"if this misses)Commands
B. Absences (CLI enough; rerun only if it wrongly succeeds)
... "coke can" --from 53 --duration 76no verified detection/ exit 1... "yellow banana" --from 53 --duration 76... "roll of black tape" --from 0 --duration 417... "red marker" --from 135 --duration 75Commands
C. Tape after it appears
... "roll of black tape" --from 427 --duration 76Commands
D. Latest position (marker moved S1 -> S3)
Run both; keep the printed
positionlines (and/or both.rrdfiles under different names).... "red marker" --from 53 --duration 76... "red marker" --from 218 --duration 76... "red marker"(no from/duration)Pass: D1 and D2 positions clearly differ (order of centimeters+); D3 matches D2's place.
Fail: D3 still on D1's place (stale).
Commands
E. Optional hard distractors
"black tape"(shorter phrasing) before S5--from 0 --duration 417"red marker"in S1--from 53 --duration 76Commands
F. Twin ambiguity (S4, two identical post-its)
From
inventory.yaml,duplicate_of: post_itadds a second identical post-it pad during the 417.9-427.2 gap. Both are on the table and visible in the same frames from S4 onward. No appearance model can tell them apart, so a confident single answer is wrong even when it happens to pick a real post-it. This check does not requiretool_inventoryor aninstance_idfield on localize.... "yellow sticky-note" --from 342 --duration 76ambiguity_marginbelow the refusal threshold, orNonewith an ambiguity reason... "yellow sticky-note" --from 53 --duration 76F1 and F2 are a pair. F1 alone can be passed by a system that refuses everything; F2 is what stops that.
Commands
Inventory
Headless-first. Counts, pairwise distances, and name similarity come from CLI / pytest stdout. Inventory.yaml pins which descs must be present (and, for I2, that each pinned desc stays at exactly one instance when the window grows). It does not pin the total
instances: Nagainst table clutter that sits in the same support-plane band.Windows (same as the localize half; from
events.jsonlrelative to color-image startlo). Verified on the db: colorlo= 1785382584.90; S1 53.3-128.8, S4 341.7-417.9, S5 427.2-502.7.--duration 76covers S1 and S5 fully;--from 342clips the first 0.3 s of S4 (scan starts at 341.7), kept anyway for consistency withtest_localize_replay.py.PR1-inventory.md's dedup row uses the same 76 s span.--from--durationinventory.yaml)duplicate_of)roll of black tape(new_class; notes pin 6)Ground-truth names for matching use inventory
desctext (notnamekeys likered_marker):red markeryellow sticky-note / post-it pad(match either phrasing / soft embedding)black penbookroll of black tape(S5+)I1. S1 baseline count (A)
I2. Dedup invariant (F) - short S1 vs full S1
Requiring equal total
Nacross 30 s and 76 s conflates F fragmentation with newly visible in-scope clutter (laptop, foam blocks, mouse, phone,unknown-*).Example
membersis not object count. Each keyframe that proposes a mask, lifts it to a world cloud, and associates that cloud to the same track appends oneSupportObservation. Across the window the book was accepted on 8 keyframes, somembers=8. The print still shows onexyzbecause it only reports the latest member’s centroid (scene state at the newest sighting), not 8 positions.I3. S4 identical post-its (F hard case)
Expect exactly 2 instances name=sticky notes.