Pixels2Points: Fusing 2D and 3D Features for Facial Skin Segmentation
| dc.contributor.author | Chen, Victoria Yue | en_US |
| dc.contributor.author | Wang, Daoye | en_US |
| dc.contributor.author | Garbin, Stephan | en_US |
| dc.contributor.author | Bednarik, Jan | en_US |
| dc.contributor.author | Winberg, Sebastian | en_US |
| dc.contributor.author | Bolkart, Timo | en_US |
| dc.contributor.author | Beeler, Thabo | en_US |
| dc.contributor.editor | Ceylan, Duygu | en_US |
| dc.contributor.editor | Li, Tzu-Mao | en_US |
| dc.date.accessioned | 2025-05-09T09:35:34Z | |
| dc.date.available | 2025-05-09T09:35:34Z | |
| dc.date.issued | 2025 | |
| dc.description.abstract | Face registration deforms a template mesh to closely fit a 3D face scan, the quality of which commonly degrades in non-skin regions (e.g., hair, beard, accessories), because the optimized template-to-scan distance pulls the template mesh towards the noisy scan surface. Improving registration quality requires a clean separation of skin and non-skin regions on the scan mesh. Existing image-based (2D) or scan-based (3D) segmentation methods however perform poorly. Image-based segmentation outputs multi-view inconsistent masks, and they cannot account for scan inaccuracies or scan-image misalignment, while scan-based methods suffer from lower spatial resolution compared to images. In this work, we introduce a novel method that accurately separates skin from non-skin geometry on 3D human head scans. For this, our method extracts features from multi-view images using a frozen image foundation model and aggregates these features in 3D. These lifted 2D features are then fused with 3D geometric features extracted from the scan mesh, to then predict a segmentation mask directly on the scan mesh. We show that our segmentations improve the registration accuracy over pure 2D or 3D segmentation methods by 8.89% and 14.3%, respectively. Although trained only on synthetic data, our model generalizes well to real data. | en_US |
| dc.description.sectionheaders | Short Paper 2 | |
| dc.description.seriesinformation | Eurographics 2025 - Short Papers | |
| dc.identifier.doi | 10.2312/egs.20251037 | |
| dc.identifier.isbn | 978-3-03868-268-4 | |
| dc.identifier.issn | 1017-4656 | |
| dc.identifier.pages | 4 pages | |
| dc.identifier.uri | https://doi.org/10.2312/egs.20251037 | |
| dc.identifier.uri | https://diglib.eg.org/handle/10.2312/egs20251037 | |
| dc.publisher | The Eurographics Association | en_US |
| dc.rights | Attribution 4.0 International License | |
| dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | |
| dc.title | Pixels2Points: Fusing 2D and 3D Features for Facial Skin Segmentation | en_US |