Spatiotemporal Analysis of RGB-D-T Facial Images for Multimodal Pain Level Recognition

Abstract

Pain is a vital sign of human health and its automatic detectioncan be of crucial importance in many different contexts,including medical scenarios. While most availablecomputer vision techniques are based on RGB, in this paper,we investigate the effect of combining RGB, depth, and thermalfacial images for pain detection and pain intensity levelrecognition. For this purpose, we extract energies releasedby facial pixels using a spatiotemporal filter. Experimentson a group of 12 elderly people applying the multimodal approachshow that the proposed method successfully detectspain and recognizes between three intensity levels in 82%of the analyzed frames improving more than 6% over RGBonly analysis in similar conditions