Doublage visuel des vidéos monoculaires

Abstract

Audio dubbing is a technique for reproducing actors’ dialogue in a language different from the original language. Despite this technique’s success, it requires immense work to find the best words and phrases that synchronize the dubbed phonemes with the actor’s visemes. To overcome this limitation, several researchers have worked on the visual dubbing of videos. This method consists of modifying the actor’s face to synchronize the expressions with the dubbed voice. However, existing methods have requirements that limit their use in our context. For example, the necessity of the actor’s presence to digitize his face, the availability of long videos with different expressions, or the use of complex and expensive equipment. In this thesis, the primary purpose is to perform visual dubbing of short advertising videos. To do so, we propose two methods of visual dubbing of monocular videos. The first approach is mainly based on 3D faces. In this approach, we reconstruct the 3D faces of the actor and the dubber. Using a bilinear face model, we extract the identity weights of the actor’s reconstructed sequence and the expression weights of the dubber’s reconstructed sequence. Then, using these two weights, we recreate the 3D animation of the dubbed actor’s face. The second approach is based on machine learning with 2D images. We use a machine learning model to extract the actor’s identity and then transfer it to the dubber’s face. Then, we merge the result of the dubbing with the images of the original sequence. Both approaches have been tested on a variety of actors and dubbers reciting different speeches. The results are encouraging and demonstrate progress in automatic visual dubbing with minimal equipment on pre-recorded RGB monocular videos.

Date	16 Apr 2021
Original language	French
Awarding Institution	École de technologie supérieure
Supervisor	Eric Paquette (Supervisor)