GraspNet and Scene Understanding Fix

[Methasit-Pun]

Already Try and not working

https://docs.google.com/document/d/17GwvMOmayvMuHfTUS6llQyY0fnNHNQoeMANG2dcPGJc/edit?usp=sharing

Issue:

1. Graspnet window Black & not response window
2. Scene Understanding Black & not response window

Already Check

• ROS2 Topic list /camera/color/image_raw -> Depth cam /camera/image_raw. No problem for this. It is correctly list
• Qos Check -> Already check for incompatible for reliable and best effort
• Depth RGB Image call in SAM -> Working but not in graspnet (not sure if it depth RGB issue see debug)
• Try to checkout to the original file before merge 2 times -> Still black window for graspnet

Not Check Yet

• image conversion with CvBridge and self.latest_rgb (Done but not sure)
• No RGB image received yet

Noted from gemini

• opencv issue

[Methasit-Pun]

README: Fixing OpenCV & cv_bridge Version Mismatch

This guide provides a step-by-step walkthrough to resolve conflicts between pip-installed OpenCV and **ROS 2 cv_bridge**. By symlinking the system-level OpenCV into your virtual environment, you ensure compatibility with ROS 2 nodes like GraspNet or Scene Understanding.

---

🛠 Problem Overview

The issue usually arises because cv_bridge is compiled against the system version of OpenCV (e.g., v4.5.4), but your virtual environment has a newer version installed via pip. This mismatch causes segmentation faults or import errors.

---

🚀 The Fix

Step 1: Activate Your Virtual Environment

First, ensure you are working within the specific environment where the issue exists.

source ~/final_project_ws/vision_venv/bin/activate

Note: If your venv path is different, adjust the command above accordingly.

Step 2: Uninstall "Bad" OpenCV Versions

Remove any Pip-installed versions of OpenCV that are overriding the system libraries.

pip uninstall opencv-python opencv-contrib-python opencv-python-headless -y

It is okay if the terminal says "Skipping" or "Not found" for some of these packages; we just need to ensure they are completely removed.

Step 3: Verify System OpenCV Location

Confirm that the system-level OpenCV (installed via apt) is present in the standard Ubuntu directory.

ls /usr/lib/python3/dist-packages/cv2

• If you see files (e.g., __init__.py, cv2.abi3.so): Proceed to Step 4.
• If you get "No such file or directory": Run sudo apt update && sudo apt install python3-opencv before continuing.

Step 4: Create the Symlink

We will now "trick" the virtual environment by creating a symbolic link (symlink) that points to the system's OpenCV installation.

1. Navigate to the venv site-packages folder:

cd ~/final_project_ws/vision_venv/lib/python3.10/site-packages/

2. Create the link:

ln -s /usr/lib/python3/dist-packages/cv2 cv2

Step 5: Verification (The "Moment of Truth")

Run this diagnostic command to confirm that Python is now pulling OpenCV from the system directory.

python3 -c "import cv2, cv_bridge, sys; print(f'\nPython: {sys.executable}'); print(f'OpenCV Path: {cv2.__file__}'); print(f'OpenCV Version: {cv2.__version__}')"

Expected Result:

• OpenCV Path: Should point to /usr/lib/python3/dist-packages/cv2/...
• OpenCV Version: Should be 4.5.x (matching the version cv_bridge expects).

---

🧪 Testing

Once verified, you can proceed to launch your ROS 2 nodes:

# Example
ros2 run your_package_name graspnet_node

---

Would you like me to help you create a Bash script to automate these steps for future environment setups?

[Methasit-Pun]

Resolve by Pointing opencv path

1. The "Clean Slate" Approach

Before diving into code, ensure you aren't fighting cached build files. If you've changed versions recently, your build and install folders are likely contaminated.

• Action: Delete your build/, install/, and log/ directories in your workspace.

2. Compile vision_opencv from Source

The most robust fix is to ignore the pre-compiled apt version of cv_bridge and build it yourself using your specific OpenCV version.

1. Navigate to your workspace src folder:

   Bash

   cd ~/your_ws/src
   

2. Clone the vision_opencv repository (ensure you switch to the branch matching your ROS2 distro, e.g., humble, jazzy):

   Bash

   git clone https://github.com/ros-perception/vision_opencv.git -b <your-distro>
   

3. Point to your OpenCV path:

   If you have a custom OpenCV install, you need to tell colcon where it is. In your terminal, set:

   Bash

   export OpenCV_DIR=/path/to/your/custom/opencv/build
   

3. Update your CMakeLists.txt

Make sure your project is explicitly looking for the same version. A common mistake is letting find_package(OpenCV) run without specifying the version that cv_bridge was built against.

CMake

# Be explicit!
find_package(OpenCV 4.x REQUIRED) 
find_package(cv_bridge REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS} ${cv_bridge_INCLUDE_DIRS})

# ...

target_link_libraries(your_node ${OpenCV_LIBRARIES} cv_bridge::cv_bridge)

4. The Python "SO" Hack (If using Python)

If you are getting a Symbol not found or ImportError in Python, it's often because cv_bridge is linked against a different .so file than your cv2 python module.

• Check links: Run ldd on the cv_bridge library file to see which OpenCV it's pointing to.

• The Fix: Ensure your PYTHONPATH isn't mixing a pip-installed opencv-python with the system-installed python3-opencv. Usually, uninstalling the pip version (pip uninstall opencv-python) and sticking to the system version solves the conflict.

---

Comparison of Solutions

Method	Effort	Reliability
Apt Install	Low	Low (Version Mismatch common)
Source Build	Medium	High (Guarantees compatibility)
Docker	High	Highest (Total isolation)

Quick Tip: If you are seeing errors specifically about shared_ptr or sensor_msgs, double-check that you aren't accidentally mixing ROS1 and ROS2 headers—it happens to the best of us when copy-pasting from StackOverflow!