Target detection and localization using thermal camera, mmWave radar and deep learning.

Abstract

Reliable detection, and localization of tiny unmanned aerial vehicles (UAVs), birds, and other aerial vehicles with small cross-sections is an ongoing challenge. The detection task becomes even more challenging in harsh weather conditions such as snow, fog, and dust. RGB camera-based sensing is widely used for some tasks, especially navigation. However, the RGB camera's performance degrades in poor lighting conditions. On the other hand, mmWave radars perform very well in harsh weather conditions also. Additionally, thermal cameras perform reliably in low lighting conditions too. The combination of these two sensors makes an excellent choice for many of these applications. In this work, a model to detect and localize UAVs is made using an integrated system of a thermal camera and mmWave radar. Data collected with the integrated sensors are used to train a model for object detection using the yolov5 algorithm. The model detects and classifies objects such as humans, cars and UAVs. The images from the thermal camera are used in combination with the trained model to localize UAVs in the cameras Field of View(FOV)