Team: Prompt Predators

Team Members:

• Ajay Kumar Krishnaiahgari
• Adbhutha Reddy Chinnarappagari
• Ponnam Sairam
• Venkata Akhila Bhanavathu

Project Folder Structure

hackathon2/
├── docs/                  # System-level documentation, architecture, and flowcharts
├── ros2_ws/               # ROS2 workspace
│   ├── src/               # Source packages
│   │   ├── sensor_sim/            # Simulated sensor node
│   │   ├── depth_estimator/       # Depth refinement node
│   │   ├── perception_stack/      # Perception and occupancy grid
│   │   ├── slam_sim/              # SLAM/odometry simulation
│   │   ├── vlm_obstacle_manager/  # Obstacle categorization/avoidance
│   │   ├── mobility_controller/   # Mobility command logger
│   │   └── slam_system_bringup/   # Launch and bringup files
│   └── ...
├── scripts/               # Utility scripts
├── testing_videos/        # (Optional) Test videos
└── ...

This system demonstrates a modular ROS2-based perception and control pipeline using fully simulated data:

1. sensor_sim: Publishes synthetic RGB and depth images plus camera info, conceptually mimicking an Intel RealSense sensor.
2. depth_estimator: Refines depth (Gaussian blur) or synthesizes depth from RGB luminance when real depth is absent.
3. perception_stack: Converts refined depth into an OccupancyGrid, extracts obstacle positions as PoseArray, and publishes visualization MarkerArray.
4. slam_sim: Simulates Visual Odometry and SLAM outputs by integrating nominal linear and angular velocities to produce Odometry, PoseStamped, and an accumulated Path.
5. vlm_obstacle_manager: Simulates a Visual Language Model that categorizes obstacles (e.g., box/chair/cone) and computes a simple avoidance strategy, publishing geometry_msgs/Twist to steer.
6. mobility_controller: Receives vector commands and logs them, simulating a mobility processing controller.

All components are connected via ROS2 topics. No physical hardware is required.

ROS2 Nodes & Topics

• sensor_sim/realsense_sim_node

  • Publishes:
    • /camera/color/image_raw (sensor_msgs/Image, bgr8)
    • /camera/depth/image_rect_raw (sensor_msgs/Image, mono16, mm)
    • /camera/color/camera_info (sensor_msgs/CameraInfo)
  • Parameters: width, height, fps, max_depth_mm, scene_seed

• depth_estimator/depth_estimator_node

  • Subscribes: /camera/color/image_raw, /camera/depth/image_rect_raw
  • Publishes: /perception/depth_refined (sensor_msgs/Image, mono16)
  • Parameters: blur_kernel, synth_from_rgb

• perception_stack/perception_node

  • Subscribes: /perception/depth_refined
  • Publishes:
    • /perception/occupancy (nav_msgs/OccupancyGrid)
    • /perception/obstacles (geometry_msgs/PoseArray)
    • /perception/markers (visualization_msgs/MarkerArray)
  • Parameters: grid_resolution, grid_width, grid_height, depth_threshold_mm

• slam_sim/slam_node

  • Publishes: /slam/pose (geometry_msgs/PoseStamped), /slam/odom (nav_msgs/Odometry), /slam/path (nav_msgs/Path)
  • Parameters: linear_speed, angular_speed

• vlm_obstacle_manager/vlm_obstacle_node

  • Subscribes: /perception/obstacles (geometry_msgs/PoseArray)
  • Publishes: /vlm/obstacles_json (std_msgs/String), /cmd_vel (geometry_msgs/Twist)

• mobility_controller/mobility_controller_node

  • Subscribes: /cmd_vel (geometry_msgs/Twist)
  • Logs received commands

Core Algorithms (Simulated)

• RGB-D Generation: Color images are procedurally drawn with geometric shapes; depth maps encode distance in millimeters, with shapes rendered as nearer obstacles.
• Depth Refinement: Gaussian blur reduces noise. Optionally synthesize depth from RGB luminance normalized to mm range.
• Occupancy Grid: Downsample refined depth to a grid; mark cells with depth below a threshold as occupied (100) and otherwise free (0).
• Obstacle Extraction: Occupied cells are converted to obstacle poses at grid cell centers for downstream processing and visualization.
• SLAM Simulation: Integrate constant linear/angular velocities to produce odometry and path. This mimics a VO/SLAM pipeline producing Odometry and Path.
• VLM Simulation: Assign obstacle categories via a deterministic hash of positions; avoidance steers left when obstacles are detected ahead, otherwise moves forward.

Prerequisites

1. Install ROS2 Humble (or compatible)

• ROS2 Installation Guide

2. Install Python 3.8+ and pip
3. Install ROS2 build tools:

  sudo apt update
  sudo apt install python3-colcon-common-extensions python3-rosdep python3-vcstool
  sudo rosdep init
  rosdep update
  sudo apt install ros-humble-depth-image-proc
  sudo apt install ros-humble-slam-toolbox

4. Install Python dependencies:

  follow requirements.txt to install the necessary libraries

Video Input Configuration

The video file used for simulation can be changed by editing the YAML config file:

• Location: ros2_ws/src/sensor_sim/config/sensor_params.yaml
• Parameter: video_path (set this to your desired video file path)

How to Execute This Project

1. Build the workspace:

   cd ros2_ws
   colcon build --symlink-install

2. Source the workspace:

   source install/setup.bash

3. Launch the full system:

   ros2 launch slam_system_bringup bringup_rviz.launch.py

Configuration:

• Each package has a config/ folder with YAML files for tuning parameters.
• See docs/ and each package's docs/ for detailed architecture and flowcharts.

Extending the System

• Replace simulated generation with Gazebo/Ignition or other simulators while keeping the same topics.
• Swap the simulated VLM with an actual VLM service and feed detections back into avoidance.
• Integrate a motion planner node to compute smoother paths using the occupancy grid.

Design Notes

• Uses only standard ROS2 message types to avoid custom interfaces and keep builds simple.
• Avoids cv_bridge to minimize ROS-specific binary dependencies; images are encoded/decoded directly.
• Keeps modules decoupled via topics to enable independent testing and replacement.