ShellNeRF: Learning a Controllable High-resolution Model of the Eye and Periocular Region

Simple item page
dc.contributor.authorLi, Gengyanen_US
dc.contributor.authorSarkar, Kripasindhuen_US
dc.contributor.authorMeka, Abhimitraen_US
dc.contributor.authorBuehler, Marcelen_US
dc.contributor.authorMueller, Franziskaen_US
dc.contributor.authorGotardo, Pauloen_US
dc.contributor.authorHilliges, Otmaren_US
dc.contributor.authorBeeler, Thaboen_US
dc.contributor.editorBermano, Amit H.en_US
dc.contributor.editorKalogerakis, Evangelosen_US
dc.date.accessioned2024-04-30T09:09:29Z
dc.date.available2024-04-30T09:09:29Z
dc.date.issued2024
dc.description.abstractEye gaze and expressions are crucial non-verbal signals in face-to-face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate-based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation - ShellNeRF - that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable 'periocular' region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light-transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi-view video input, we demonstrate significant improvements over state-of-the-art in expression re-enactment and transfer for high-resolution close-up views of the eye region.en_US
dc.description.number2
dc.description.sectionheadersFace Modeling and Reconstruction
dc.description.seriesinformationComputer Graphics Forum
dc.description.volume43
dc.identifier.doi10.1111/cgf.15041
dc.identifier.issn1467-8659
dc.identifier.pages15 pages
dc.identifier.urihttps://doi.org/10.1111/cgf.15041
dc.identifier.urihttps://diglib.eg.org/handle/10.1111/cgf15041
dc.publisherThe Eurographics Association and John Wiley & Sons Ltd.en_US
dc.rightsAttribution 4.0 International License
dc.rights.urihttps://creativecommons.org/licenses/by/4.0/
dc.subjectCCS Concepts: Computing methodologies -> Motion capture; Physical simulation; Image-based rendering; Mixed / augmented reality; Volumetric models; Parametric curve and surface models; Appearance and texture representations; Shape representations; 3D imaging; Reconstruction
dc.subjectComputing methodologies
dc.subjectMotion capture
dc.subjectPhysical simulation
dc.subjectImage
dc.subjectbased rendering
dc.subjectMixed / augmented reality
dc.subjectVolumetric models
dc.subjectParametric curve and surface models
dc.subjectAppearance and texture representations
dc.subjectShape representations
dc.subject3D imaging
dc.subjectReconstruction
dc.titleShellNeRF: Learning a Controllable High-resolution Model of the Eye and Periocular Regionen_US
Files
Original bundle
Now showing 1 - 2 of 2
Thumbnail Image
Name:
v43i2_44_15041.pdf
Size:
34.43 MB
Format:
Adobe Portable Document Format
Download
No Thumbnail Available
Name:
website_shellnerf.zip
Size:
170.7 MB
Format:
Zip file
Download
Collections
43-Issue 2