Camera-LiDAR fusion with neural networks for obstacle detection and tracking on autonomous vehicles

The goal of this project is to develop a pipeline for merging images from Cameras and point clouds from LiDAR sensors in order to realize precise 3D obstacle detection and tracking using deep learning techniques.

Camera sensors are common sensor for the perception of autonomous agents. Providing dense color information, they can’t however give precise 3D spatial information. On the other hand, LiDAR sensor can create accurate representations of the surroundings through 3D point clouds. However the quantity of available data is low and sparse. Deep learning have illustrated impressive results at encoding large and complex information for difficult computer vision tasks The main goal of this project is to create algorithms for merging and encoding complementary information from both modality thanks to artificial neural networks methods. The studied tasks are 3D obstacle detection (localize the obstacle and estimate its dimensions) and tracking (follow the obstacle through sequence). Every kind of obstacle is not considered, we mainly focus on cars, pedestrians and cyclists.

Ruddy Théodose, Dieumet Denis, Christophe Blanc, Thierry Chateau, Paul Checchin. Détection de véhicules fondée sur l’estimation de carte de probabilités. ORASIS 2019, May 2019, Saint-Dié-des-Vosges, France

Ruddy Théodose, Dieumet Denis, Christophe Blanc, Thierry Chateau, Paul Checchin. Vehicle Detection based on Deep Learning Heatmap Estimation. IEEE IV 2019, June 2019, Paris, Franc