VDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track

Simple item page
dc.contributor.authorGarrido, Pabloen_US
dc.contributor.authorValgaerts, Levien_US
dc.contributor.authorSarmadi, Hamiden_US
dc.contributor.authorSteiner, Ingmaren_US
dc.contributor.authorVaranasi, Kiranen_US
dc.contributor.authorPerez, Patricken_US
dc.contributor.authorTheobalt, Christianen_US
dc.contributor.editorOlga Sorkine-Hornung and Michael Wimmeren_US
dc.date.accessioned2015-04-16T07:43:56Z
dc.date.available2015-04-16T07:43:56Z
dc.date.issued2015en_US
dc.description.abstractIn many countries, foreign movies and TV productions are dubbed, i.e., the original voice of an actor is replaced with a translation that is spoken by a dubbing actor in the country's own language. Dubbing is a complex process that requires specific translations and accurately timed recitations such that the new audio at least coarsely adheres to the mouth motion in the video. However, since the sequence of phonemes and visemes in the original and the dubbing language are different, the video-to-audio match is never perfect, which is a major source of visual discomfort. In this paper, we propose a system to alter the mouth motion of an actor in a video, so that it matches the new audio track. Our paper builds on high-quality monocular capture of 3D facial performance, lighting and albedo of the dubbing and target actors, and uses audio analysis in combination with a space-time retrieval method to synthesize a new photo-realistically rendered and highly detailed 3D shape model of the mouth region to replace the target performance. We demonstrate plausible visual quality of our results compared to footage that has been professionally dubbed in the traditional way, both qualitatively and through a user study.en_US
dc.description.number2en_US
dc.description.sectionheadersAll About Facesen_US
dc.description.seriesinformationComputer Graphics Forumen_US
dc.description.volume34en_US
dc.identifier.doi10.1111/cgf.12552en_US
dc.identifier.pages193-204en_US
dc.identifier.urihttps://doi.org/10.1111/cgf.12552en_US
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.titleVDub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Tracken_US
Files
Collections
34-Issue 2
EG 2015 - Full Papers - CGF 34-Issue 2