Personalized Visual Dubbing through Virtual Dubber and Full Head Reenactment
Loading...
Date
2025
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Visual dubbing aims to modify facial expressions to ''lip-sync'' a new audio track. While person-generic talking head generation methods achieve expressive lip synchronization across arbitrary identities, they usually lack person-specific details and fail to generate high-quality results. Conversely, person-specific methods require extensive training. Our method combines the strengths of both methods by incorporating a virtual dubber, a person-generic talking head, as an intermediate representation. We then employ an autoencoder-based person-specific identity swapping network to transfer the actor identity, enabling fullhead reenactment that includes hair, face, ears, and neck. This eliminates artifacts while ensuring temporal consistency. Our quantitative and qualitative evaluation demonstrate that our method achieves a superior balance between lip-sync accuracy and realistic facial reenactment.
Description
CCS Concepts: Computing methodologies → Image manipulation; Animation
@inproceedings{10.2312:egs.20251034,
booktitle = {Eurographics 2025 - Short Papers},
editor = {Ceylan, Duygu and Li, Tzu-Mao},
title = {{Personalized Visual Dubbing through Virtual Dubber and Full Head Reenactment}},
author = {Jeon, Bobae and Paquette, Eric and Mudur, Sudhir and Popa, Tiberiu},
year = {2025},
publisher = {The Eurographics Association},
ISSN = {1017-4656},
ISBN = {978-3-03868-268-4},
DOI = {10.2312/egs.20251034}
}