clip_classifier node issue

[Methasit-Pun]

Also, there's a problem with clip_classifier node. I get the error:
Find object error: AttributeError: 'BaseModelOutputWithPooling' object has no attribute 'norm'
and also from simple_sam_detector node:
[ERROR] NoneType: None
When calling /vision/find_object

[Methasit-Pun]

In EVERY terminal before running any ROS2 commands:

cd /home/group11/final_project_ws
source /opt/ros/humble/setup.bash
source install/setup.bash

Then you can run nodes or call services:

ros2 service call /vision/detect_grasp_bb custom_interfaces/srv/DetectGraspBBox "{x1: 100, y1: 100, x2: 200, y2: 300}"

[Methasit-Pun]

Summary of what was changed:
No versions or libraries were changed! The only changes made were:

Fixed Python dependencies in virtual environment:

Installed catkin_pkg (for ROS2 package.xml parsing)
Installed pytest (for testing)
Installed empy<4 (ROS2 Humble requires version 3.x, not 4.x)
Installed lark (parser library)
Fixed code issues:

Changed AutoProcessor → CLIPProcessor in clip_classifier.py
Simplified find_object_grasp_service_node.py to only return bbox
To use custom_interfaces, you MUST source the workspace in every terminal:

[Methasit-Pun]

1. Transformers Library Version Change
   Different versions of transformers handle get_text_features() differently:

Older versions: Returned the tensor directly
Newer versions: Return the full BaseModelOutputWithPooling object
2. Different CLIP Implementation
You might have previously used:

Original OpenAI CLIP (pip install git+https://github.com/openai/CLIP.git)
Now using HuggingFace CLIP

┌─────────────────────────────────────────────────────────────┐
│ OLD (Broken) Approach │
├─────────────────────────────────────────────────────────────┤
│ model.get_text_features() → BaseModelOutputWithPooling │
│ ↓ │
│ F.normalize() ❌ ERROR! │
│ (Can't normalize an object!) │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│ NEW (Fixed) Approach │
├─────────────────────────────────────────────────────────────┤
│ model.text_model() → BaseModelOutputWithPooling │
│ ↓ │
│ .pooler_output → Tensor [1, 512] │
│ ↓ │
│ F.normalize() ✅ Works! │
└─────────────────────────────────────────────────────────────┘

┌──────────────┐
│ User Input │ ros2 service call /find_object "{label: 'beer_can'}"
└──────┬───────┘
│
▼
┌──────────────────────────────────────────────────────────────┐
│ FindObjectServiceNode │
├──────────────────────────────────────────────────────────────┤
│ Step 1: Detect All Objects │
│ ├─► /vision/detect_objects │
│ └─► Returns: 5 objects detected │
│ │
│ Step 2: Find "beer_can" using CLIP │
│ ├─► /vision/find_object │
│ └─► Returns: bbox=[100,150,200,250], confidence=0.85 │
│ │
│ Step 3: Calculate Center │
│ └─► u=150, v=200 (pixel coordinates) │
│ │
│ Step 4: Convert to 3D World Coordinates │
│ ├─► /pixel_to_real │
│ └─► Returns: (0.123, -0.045, 0.452) meters │
│ │
│ Step 5: Return Complete Result │
│ └─► Success + bbox + world coords │
└──────────────────────────────────────────────────────────────┘
│
▼
┌──────────────┐
│ User Gets: │ object_id, bbox, confidence, x, y, z
└──────────────┘

[Methasit-Pun]

[Methasit-Pun]

ros2 service call /find_object custom_interfaces/srv/FindObjectReal "{label: 'bowl'}"

[Methasit-Pun]

Solve by Changing from hugging face clip to Original OpenAI CLIP (pip install git+https://github.com/openai/CLIP.git)