Deep audio and video emotion detection

Abstract

Human beings rely on two capacities for successful social interactions Cowie et al. (2001). The first is more obvious and explicitly conveys messages which may be about anything or nothing and the other is more subtl and transmits implicit messages about the speakers themselves. In the last few years with the advancement of technology, interpretation of the first channel becomes more feasible. For instance, speech processing systems can easily convert a voice to text or computer vision systems can detect a face in an image. The second channel is still not as well understood. One of the key elements for exploiting the second one is interpreting human emotion. To solve the problem, earlier works in emotion recognition have relied on handcrafted features by incorporating domain knowledge into the underlying system. However, in the last few years, deep neural networks have proven to be effective models for tackling a variety of tasks. In this dissertation, we explore the effects of applying deep learning methods to the emotion recognition task. We demonstrate these methods by learning rich representations achieve superior accuracy over traditional techniques. Moreover, we demonstrate our methods are not bound to emotion recognition task and other classes of tasks such as multi-label classification can get benefit from our approaches. The first part of this work focuses only on the task of video-based emotion recognition using only visual inputs. We show that by exploiting information from the spatial and temporal aspects of input data we can get promising results. In the second part, we move our attention to multimodal data. Particularly we focus on how to fuse multimodal data. We introduce a new architecture that incorporates the best features from early and late fusion architecture.

Date	20 Jul 2020
Original language	American English
Awarding Institution	École de technologie supérieure
Supervisor	Patrick Cardinal (Supervisor) & Marco Pedersoli (Co-supervisor)