ShellNeRF: Learning a Controllable High-resolution Model of the Eye and Periocular Region

Abstract
Eye gaze and expressions are crucial non-verbal signals in face-to-face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate-based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation - ShellNeRF - that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable 'periocular' region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light-transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi-view video input, we demonstrate significant improvements over state-of-the-art in expression re-enactment and transfer for high-resolution close-up views of the eye region.
Description

CCS Concepts: Computing methodologies -> Motion capture; Physical simulation; Image-based rendering; Mixed / augmented reality; Volumetric models; Parametric curve and surface models; Appearance and texture representations; Shape representations; 3D imaging; Reconstruction

        
@article{
10.1111:cgf.15041
, journal = {Computer Graphics Forum}, title = {{
ShellNeRF: Learning a Controllable High-resolution Model of the Eye and Periocular Region
}}, author = {
Li, Gengyan
and
Sarkar, Kripasindhu
and
Meka, Abhimitra
and
Buehler, Marcel
and
Mueller, Franziska
and
Gotardo, Paulo
and
Hilliges, Otmar
and
Beeler, Thabo
}, year = {
2024
}, publisher = {
The Eurographics Association and John Wiley & Sons Ltd.
}, ISSN = {
1467-8659
}, DOI = {
10.1111/cgf.15041
} }
Citation
Collections