VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech)

Table of Contents

• VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception (BeamNG.tech)
  • Table of Contents
  • Overview
  • Demos
    • Emergency Braking (AEB) Demo
    • Sign Detection & Detection and classification
    • Traffic Light Detection & Classification Demo
    • Latest Lane Detection Demo (v2)
      • Previous Lane Detection Demo (v1)
    • Foxglove Visualization Demo
    • Segmentation Demo
  • Sensor Suite
  • Roadmap
    • Perception
    • Sensor Fusion & Calibration
    • Control & Planning
    • Simulation & Scenarios
    • Visualization & Logging
    • Deployment & Infrastructure
    • README To-Dos
    • Other
  • Note on Installation
  • Known Limitations
    • Simulator-Specific Limitations
  • Credits
  • Citation
    • BeamNG.tech Citation
  • License

Overview

A modular Python project for autonomous driving research and prototyping, fully integrated with the BeamNG.tech simulator and Foxglove visualization. This system combines traditional computer vision and state-of-the-art deep learning (CNN, U-Net, YOLO, SCNN) with real-time sensor fusion and autonomous vehicle control to tackle:

• Lane detection (Traditional CV & SCNN)
• Traffic sign classification & detection (CNN, YOLO)
• Traffic light detection & classification (YOLO)
• Vehicle & pedestrian detection and recognition (YOLO)
• Multi-sensor fusion (Camera, LiDAR, Radar, GPS, IMU)
• Multi-model inference, real-time simulation, autonomous driving with PID control (BeamNG.tech)
• Containerized multi-model architecture (Docker-based), orchestrated via a central inference aggregator service
• Cruise control
• Automatic Emergency Breaking AEB
• Real-time visualization and monitoring (Foxglove WebSocket)
• Modular configuration system (YAML-based)
• Drive logging and telemetry

Demos

Emergency Braking (AEB) Demo

Watch the Emergency Braking System (AEB) in action with real-time radar filtering and collision avoidance:

Extended Demo: Watch the full video here

---

Sign Detection & Detection and classification

This demo shows real-time traffic sign detection and classification:

Extended Demo: Watch the full video here

VisionPilot does not yet support multi-camera. This is for demonstration purposes only.

---

Traffic Light Detection & Classification Demo

This demo shows real-time traffic light detection and classification:

No extended Demo avaliable yet.

---

Latest Lane Detection Demo (v2)

Watch the improved autonomous lane keeping demo (v2) in BeamNG.tech, featuring smoother fused CV+SCNN lane detection, stable PID steering, and robust cruise control:

Extended Demo: Watch the full video here

Note: Very low-light (tunnel) scenarios are not yet supported.

Previous Lane Detection Demo (v1)

The original demo is still available for reference:

Lane Keeping & Multi-Model Detection Demo (v1)

---

Foxglove Visualization Demo

See real-time LiDAR point cloud streaming and autonomous vehicle telemetry in Foxglove Studio:

Extended Demo: Watch the full video here

---

Segmentation Demo

See real-time image segmentation using front and rear cameras:

Extended Demo: Watch the full video here

More demo videos and visualizations will be added as features are completed.

Sensor Suite

The vehicle is equipped with a comprehensive multi-sensor suite for autonomous perception and control:

Sensor	Specification	Purpose
Front Camera	1920x1080 @ 50Hz, 70° FOV, Depth enabled	Lane detection, traffic signs, traffic lights, object detection
LiDAR (Top)	80 vertical lines, 360° horizontal, 120m range, 20Hz	Obstacle detection, 3D scene understanding
Front Radar	200m range, 128×64 bins, 50Hz	Collision avoidance, adaptive cruise control
Rear Left & Right Radar	30m range, 64×32 bins, 50Hz	Blindspot monitoring, rear object detection
Dual GPS	Front & rear positioning @ 50Hz	Localization
IMU	100Hz update rate	Vehicle dynamics, pose estimation

Sensor Array	Front Radar	Lidar Visualization

Configuration files are located in the /config directory:

Roadmap

Perception

• Sign classification & Detection (CNN / YOLOv11m)

• Traffic light classification & Detection (CNN / YOLOv11m)

• Lane detection Fusion (SCNN / CV)

• 🔥🔥 YOLOP integration

  • Drivable are segmenatation
  • Lane detection
  • Object detection

• Contextual lane detection (Use road curvature to predict lane continuity)

• Advanced lane detection using OpenCV (robust highway, lighting, outlier handling)

• Integrate Majority Voting system for CV

• Lighting Condition Detection

• ⭐ Semantic Segmentatation (Already built not implemented here yet)

  • Panoptic segmentation (instance + semantic)

• Depth Estimation (Monocular for obstacle distance)

• ⭐ Real-Time Object Detection (Cars, Trucks, Buses, Pedestrians, Cyclists) (Trained)

• 🔥 Speed Estimation using detection from camera and lidar

  • Multiple Object Tracking (MOT)

• 🔥 Handle dashed lines better in lane detection

• Road Marking Detection (Arrows, Crosswalks, Stop Lines)

• 🔥🔥 Lidar Object Detection 3D

• Ocluded Object Detection (Detect objects that are partially blocked or not visible in the camera view using radar/lidar)

• Detect multiple lanes

• 💤 Classify lane types

• 💤 Multi Camera Setup (Will implement after all other camera-based features are finished)

• 💤 Overtaking, Merging (Will be part of Path Planning)

Sensor Fusion & Calibration

• 🔥 Kalman Filtering

  • Extended

• Integrate Radar

• Integrate Lidar

• Integrate GPS

• Integrate IMU

• Ultrasonic Sensor Integration

• 💤 SLAM (simultaneous localization and mapping)

  • Build HD Map of the BeamNG.tech map
  • Localize Vehicle on HD Map

Control & Planning

• Integrate vehicle control (Throttle, Steering, Braking Implemented) (PID needs further tuning)

• Integrate PIDF controller

• ⭐ Adaptive Cruise Control (Currently only basic Cruise Control implemented)

• ⭐ Automatic Emergency Braking AEB (Still an issue with crashing after EB activated)

  • Obstacle Avoidance (Steering away from obstacles instead of just braking)

• Model Predictive Control MPC (More advanced control strategy that optimizes control inputs over a future time horizon)

• Curve Speed Optimization (Slow down for sharp curves based on lane curvature)

• Trajectory Predcition for surrounding vehicles

• Blindspot Monitoring (Using left/right rear short range radars)

• Traffic Rule Enforcement (Stop at red lights, stop signs, yield signs)

• Dynamic Target Speed based on Speed Limit Signs

• Global Path planning

• Local Path planning

• Lane Change Logic

  • Change Blindspots before lane change
  • Signal Lane Change

• Parking Logic (Path finding / Parallel or Perpendicular)

• 💤💤 U-Turn Logic (3-point turn)

• 💤💤 Advanced traffic participant prediction (trajectory, intent)

Simulation & Scenarios

• Integrate and test in BeamNG.tech simulation (replacing CARLA)

• Modularize and clean up BeamNG.tech pipeline

• Tweak lane detection parameters and thresholds

• Fog Weather conditions (Rain or snow not supported in BeamNG.tech)

• Traffic scenarios: driving in heavy, moderate, and light traffic

• Test all Systems in different lighting conditions (Day, Night, Dawn/Dusk, Tunnel)

• Construction Zones (temporary lanes, cones, barriers)

• 💤💤 Test using actual RC car

Visualization & Logging

• ⭐ Full Foxglove visualization integration (Overhaul needed)

• Modular YAML configuration system

• Real-time drive logging and telemetry

• Birds eye view BEV (Top down view of vehicle and surroundings)

• Real time Annotations Overlay in Foxglove

• Show predicted trajectories in Foxglove

• Show Global and local path plans in Foxglove

• Live Map Visualization

Deployment & Infrastructure

• Containerize Models for easy deployment and scalability (Also eliminates dependency issues)

  • Message Broker (redis/rabbitmq)

  • Create docker compose

  • Aggregator service

  • Refactor beamng.py

README To-Dos

• Add demo images and videos to README

• Add performance benchmarks section

• Add Table of Contents for easier navigation

Other

• Vibe-Code a website for the project

• Redo project structure for better modularity

Driver Monitoring System would've been pretty cool but human drivers are not implemented in BeamNG.tech

Legend

🔥 = High Priority

⭐ = Complete but still being improved/tuned/changed (not final version)

💤 = Minimal Priority, can be addressed later

💤💤 = Very Low Priority, may not be implement

Note on Installation

Status: This project is currently in active development. A stable, production-ready release with pre-trained models and complete documentation will be available eventually.

Known Limitations

• Tunnel/Low-Light Scenarios: Camera depth perception fails below certain lighting thresholds
• Multi-Camera Support: Single front-facing camera only (future roadmap)
• Dashed Lane Detection: Requires improvement for better accuracy
• PID Controller Tuning: May oscillate on aggressive maneuvers
• Real-World Testing: Only validated in simulation (BeamNG.tech), for now...

Simulator-Specific Limitations

• Rain/snow physics not supported in BeamNG.tech
• No native ROS2 support (custom bridge required)
• Pedestrians
• Human Drivers

Credits

Datasets:

• CU Lane, LISA, GTSRB, Mapillary, BDD100K

Simulation & Tools:

• BeamNG.tech by BeamNG GmbH
• Foxglove Studio for visualization

Special Thanks:

• Kaggle for free GPU resources (model training)
• Mr. Pratt (teacher/supervisor) for guidance

Citation

If you use VisionPilot in your research, please cite:

@software{visionpilot2025,
  title={VisionPilot: Autonomous Driving Simulation, Computer Vision & Real-Time Perception},
  author={Julian Stamm},
  year={2025},
  url={https://github.com/visionpilot-project/VisionPilot}
}

BeamNG.tech Citation

Title: BeamNG.tech
Author: BeamNG GmbH
Address: Bremen, Germany
Year: 2025
Version: 0.35.0.0
URL: https://www.beamng.tech/

License

This project is licensed under the MIT License - see LICENSE file for details.