PG2024 Conference Papers and Posters

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/3607048

Browse

Now showing 1 - 20 of 57

3D-SSGAN: Lifting 2D Semantics for 3D-Aware Compositional Portrait Synthesis
(The Eurographics Association, 2024) Liu, Ruiqi; Zheng, Peng; Wang, Ye; Ma, Rui; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Existing 3D-aware portrait synthesis methods can generate impressive high-quality images while preserving strong 3D consistency. However, most of them cannot support the fine-grained part-level control over synthesized images. Conversely, some GAN-based 2D portrait synthesis methods can achieve clear disentanglement of facial regions, but they cannot preserve view consistency due to a lack of 3D modeling abilities. To address these issues, we propose 3D-SSGAN, a novel framework for 3D-aware compositional portrait image synthesis. First, a simple yet effective depth-guided 2D-to-3D lifting module maps the generated 2D part features and semantics to 3D. Then, a volume renderer with a novel 3D-aware semantic mask renderer is utilized to produce the composed face features and corresponding masks. The whole framework is trained end-to-end by discriminating between real and synthesized 2D images and their semantic masks. Quantitative and qualitative evaluations demonstrate the superiority of 3D-SSGAN in controllable part-level synthesis while preserving 3D view consistency.
3DStyleGLIP: Part-Tailored Text-Guided 3D Neural Stylization
(The Eurographics Association, 2024) Chung, SeungJeh; Park, JooHyun; Kang, HyeongYeop; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
3D stylization, the application of specific styles to three-dimensional objects, offers substantial commercial potential by enabling the creation of uniquely styled 3D objects tailored to diverse scenes. Recent advancements in artificial intelligence and textdriven manipulation methods have made the stylization process increasingly intuitive and automated. While these methods reduce human costs by minimizing reliance on manual labor and expertise, they predominantly focus on holistic stylization, neglecting the application of desired styles to individual components of a 3D object. This limitation restricts the fine-grained controllability. To address this gap, we introduce 3DStyleGLIP, a novel framework specifically designed for text-driven, parttailored 3D stylization. Given a 3D mesh and a text prompt, 3DStyleGLIP utilizes the vision-language embedding space of the Grounded Language-Image Pre-training (GLIP) model to localize individual parts of the 3D mesh and modify their appearance to match the styles specified in the text prompt. 3DStyleGLIP effectively integrates part localization and stylization guidance within GLIP's shared embedding space through an end-to-end process, enabled by part-level style loss and two complementary learning techniques. This neural methodology meets the user's need for fine-grained style editing and delivers high-quality part-specific stylization results, opening new possibilities for customization and flexibility in 3D content creation. Our code and results are available at https://github.com/sj978/3DStyleGLIP.
Audio-Driven Speech Animation with Text-Guided Expression
(The Eurographics Association, 2024) Jung, Sunjin; Chun, Sewhan; Noh, Junyong; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
We introduce a novel method for generating expressive speech animations of a 3D face, driven by both audio and text descriptions. Many previous approaches focused on generating facial expressions using pre-defined emotion categories. In contrast, our method is capable of generating facial expressions from text descriptions unseen during training, without limitations to specific emotion classes. Our system employs a two-stage approach. In the first stage, an auto-encoder is trained to disentangle content and expression features from facial animations. In the second stage, two transformer-based networks predict the content and expression features from audio and text inputs, respectively. These features are then passed to the decoder of the pre-trained auto-encoder, yielding the final expressive speech animation. By accommodating diverse forms of natural language, such as emotion words or detailed facial expression descriptions, our method offers an intuitive and versatile way to generate expressive speech animations. Extensive quantitative and qualitative evaluations, including a user study, demonstrate that our method can produce natural expressive speech animations that correspond to the input audio and text descriptions.
Biophysically-based Simulation of Sun-induced Skin Appearance Changes
(The Eurographics Association, 2024) He, Xueyan; Huang, Minghao; Fu, Ruoyu; Guo, Jie; Yuan, Junping; Wang, Yanghai; Guo, Yanwen; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Skin appearance modeling plays a crucial role in various fields such as healthcare, cosmetics and entertainment. However, the structure of the skin and its interaction with environmental factors like ultraviolet radiation are very complex and require further detailed modeling. In this paper, we propose a biophysically-based model to illustrate the changes in skin appearance under ultraviolet radiation exposure. It takes ultraviolet doses and specific biophysical parameters as inputs, leading to variations in melanin and blood concentrations, as well as the growth rate of skin cells. These changes bring alteration of light scattering, which is simulated by random walk method, and result in observable erythema and tanning. We showcase effects of various skin tones, comparisons across different body parts, and images illustrating the impact of occlusion. It demonstrates superior quality to the the commonly used method with more convincing skin details and bridges biological insights with visual simulations.
CKD-LQPOSE: Towards a Real-World Low-quality Cross-Task Distilled Pose Estimation Architecture
(The Eurographics Association, 2024) Liu, Tao; Yao, Beiji; Huang, Jun; Wang, Ya; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Although human pose estimation (HPE) methods have achieved promising results, they remain challenging in real-world lowquality (LQ) scenarios. Moreover, due to the general lack of modeling of LQ information in currently public HPE datasets, it is difficult to accurately evaluate the performance of the HPE methods in LQ scenarios. Hence, we propose a novel CKD-LQPose architecture, which is the first architecture fusing cross-task feature information in HPE that uses a cross-task distillation method to merge HPE information and well-quality (WQ) information. The CKD-LQPose architecture effectively enables adaptive feature learning from LQ images and improves their quality to enhance HPE performance. Additionally, we introduce the PatchWQ-Gan module to obtain WQ information and the refined transformer decoder (RTD) module to refine the features further. In the inference stage, CKD-LQPose removes the PatchWQ-Gan and RTD modules to reduce the computational burden. Furthermore, to accurately evaluate the HPE methods in LQ scenarios, we develop an RLQPose-DS test benchmark. Extensive experiments on RLQPose-DS, real-world images, and LQ versions of well-known datasets such as COCO, MPII, and CrowdPose demonstrate CKD-LQPose outperforms state-of-the-art approaches by a large margin, demonstrating its effectiveness in realworld LQ scenarios.
CNCUR : A simple 2D Curve Reconstruction Algorithm based on constrained neighbours
(The Eurographics Association, 2024) Antony, Joms; Reghunath, Minu; Muthuganapathy, Ramanathan; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Given a planar point set S ∈ R2 (where S = {v1, . . . , vn}) sampled from an unknown curve Σ, the goal is to obtain a piece-wise linear reconstruction the curve from S that best approximates Σ. In this work, we propose a simple and intuitive Delaunay triangulation(DT)-based algorithm for curve reconstruction. We start by constructing a Delaunay Triangulation (DT) of the input point set. Next, we identify the set of edges, ENp in the natural neighborhood of each point p in the DT. From the set of edges in ENp, we retain the first two shorter edges connected to each point. To take care of open curves, one of the retained edges has to be removed based on a parameter δ. Here, δ is a parameter used to eliminate the longer edge based on the allowable ratio between the maximum and minimum edge lengths. Our algorithm inherently handles self-intersections, multiple components, sharp corners, and different levels of Gaussian noise, all without requiring any parameters, pre-processing, or post-processing.
Colorectal Protrusions Detection based on Conformal Colon Flattening
(The Eurographics Association, 2024) Ren, Yuxue; Hu, Wei; Li, Zhengbin; Chen, Wei; Lei, Na; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
We propose an innovative approach to automatically detect colorectal protrusions on the colon surface. In the colon, these protrusions include polyps. This approach comprises two successive stages. In the first stage, we identify single protrusions and extract folds containing suspected protrusions in the flattened colon image by integrating shape analysis with curvature rendering and conformal colon flattening. This stage enables accurate and rapid detection of single protrusions, especially flat protrusions, since the 3D protrusion detection problem is converted into a 2D pattern recognition problem. To detect protrusions on folds, the folds containing suspected protrusions is inversely mapped back to 3D colon surface in the second stage. We detect protrusions in the 3D surface area by curvature-based analysis and reduce the false positives by quadratic surface fitting. We evaluated our method via real colon data from the National CT Colonography Trial of the American College of Radiology Imaging Network (ACRIN, 6664). Experimental results show that our method can efficiently and accurately identify protrusion lesions, is robust to noise, and is suitable for implementation within CTC-CAD systems.
Computational Mis-Drape Detection and Rectification
(The Eurographics Association, 2024) Shin, Hyeon-Seung; Ko, Hyeong-Seok; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
For various reasons, mis-drapes occur in physically-based clothing simulation. Therefore, when developing a virtual try-on system that works without any human operators, a technique to algorithmically detect and rectify mis-drapes has to be developed. This paper makes a first attempt in that direction, by defining two mis-drape determinants, namely, the Gaussian and crease mis-drape determinants. According to the experiments performed to various avatar-garment combinations, the proposed determinants identify mis-drapes pretty accurately. This paper also proposes a treatment that can be applied to rectify the mis-drapes. The proposed treatment successfully resolves the mis-drapes without unnecessarily destroying the original drape.
Continuous Representation based Internal Self-supporting Structure via Ellipsoid Hollowing for 3D Printing
(The Eurographics Association, 2024) Wang, Shengfa; Yang, Jun; Hu, Jiangbei; Lei, Na; Luo, Zhongxuan; Liu, Ligang; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Hollowing is an effective way to achieve lightweight objectives by removing material from the interior volume while maintaining feasible mechanical properties. However, hollowed models often necessitate the use of additional support materials to prevent collapse during the printing process, which can substantially negate the benefits of weight reduction. We introduce a framework for designing and optimizing self-supporting infill cavities, which are represented and optimized directly using continuous functions based on ellipsoids. Ellipsoids are favored as filling structures due to their advantageous properties, including their self-supporting nature, precise mathematical definability, variable controllability, and stress concentration mitigation capabilities. Thanks to the explicit definability, we formulate the creation of self-supporting infill cavities as a structural stiffness optimization problem using function representations. The utilization of function representation eliminates the necessity for remeshing to depict structures and shapes, thereby enabling the direct computation of integrals and gradients on the functions. Based on the representations, we propose an efficient optimization strategy to determine the shapes, positions, and topology of the infill cavities, with the goal of achieving multiple objectives, including minimizing material cost, maximizing structural stiffness, and ensuring self-supporting. We perform various experiments to validate the effectiveness and convergence of our approach. Moreover, we demonstrate the self-supporting and stability of the optimized structures through actual 3D printing trials and real mechanical testing.
A Contrastive Unified Encoding Framework for Sticker Style Editing
(The Eurographics Association, 2024) Ni, Zhihong; Li, Chengze; Liu, Hanyuan; Liu, Xueting; Wong, Tien-Tsin; Wen, Zhenkun; Wu, Huisi; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Stickers are widely used in digital communication to enhance emotional and visual expressions. The conventional process of creating new sticker pack images involves time-consuming manual drawing, including meticulous color coordination and shading techniques for visual harmony. Learning the visual styles of distinct sticker packs would be critical to the overall process; however, existing solutions usually learn this style information within a limited number of style ''domains'', or per image. In this paper, we propose a contrastive learning framework that allows the style editing of an arbitrary sticker based on one or a number of style references with a continuous manifold to encapsulate all styles across sticker packs. The key to our approach is the encoding of styles into a unified latent space so that each sticker pack correlates with a unique style latent encoding. The contrastive loss ensures identical style latents within the same sticker pack, while distinct styles diverge. Through exposure to diverse sticker sets during training, our model crafts a consolidated continuous latent style space with strong expressive power, fostering seamless style transfer, interpolation, and mixing across sticker sets. Experiments show compelling style transfer results, with both qualitative and quantitative evaluations confirming the superiority of our method over existing approaches.
Convex Hull Computation in a Grid Space: A GPU Accelerated Parallel Filtering Approach
(The Eurographics Association, 2024) Antony, Joms; Mukundan, Manoj Kumar; Thomas, Mathew; Muthuganapathy, Ramanathan; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Many real-world applications demand the computation of a convex hull (CH) when the input points originate from structured configurations such as two-dimensional (2D) or three-dimensional (3D) grids. Convex hull in grid space has found applications in geographic information systems, medical data analysis, path planning for robots/autonomous vehicles etc. Conventional as well as existing GPU-accelerated algorithms available for CH computation cannot operate directly on 2D or 3D grids represented in matrix format and do not exploit the inherent sequential ordering in such rasterized representations. This work introduces novel filtering algorithms, initially developed for a 2D grid space and subsequently extended to 3D to speed up the hull computation. They are further extended as GPU-CPU hybrid algorithms and are implemented and evaluated on a commercial NVIDIA GPU. For a 2D grid, the number of contributing pixels is always restricted to ≤ 2n for an (n×n) grid. Moreover, they are extracted in lexicographic order, ensuring an efficient O(n) computation of CH. Similarly, in 3D, the number of contributing voxels is always limited to ≤ 2n2 for an (n×n×n) voxel matrix. Additionally, 2D CH filtering is enabled across all slices of the 3D grid in parallel, leading to a further reduction in the number of contributing voxels to be fed to the 3D CH computation procedure. Comparison with the state of the art indicated that our method is superior, especially for large and sparse point clouds.
Data Parallel Ray Tracing of Massive Scenes based on Neural Proxy
(The Eurographics Association, 2024) Xu, Shunkang; Xu, Xiang; Xu, Yanning; Wang, Lu; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Data-parallel ray tracing is an important method for rendering massive scenes that exceed local memory. Nevertheless, its efficacy is markedly contingent upon bandwidth owing to the substantial ray data transfer during the rendering process. In this paper, we advance the utilization of neural representation geometries in data-parallel rendering to reduce ray forwarding and intersection overheads. To this end, we introduce a lightweight geometric neural representation, denoted as a ''neural proxy.'' Utilizing our neural proxies, we propose an efficient data-parallel ray tracing framework that significantly minimizes ray transmission and intersection overheads. Compared to state-of-the-art approaches, our method achieved a 2.29∼ 3.36× speedup with an almost imperceptible image quality loss.
Deep-PE: A Learning-Based Pose Evaluator for Point Cloud Registration
(The Eurographics Association, 2024) Gao, Junjie; Wang, Chongjian; Ding, Zhongjun; Chen, Shuangmin; Xin, Shiqing; Tu, Changhe; Wang, Wenping; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
In the realm of point cloud registration, the most prevalent pose evaluation approaches are statistics-based, identifying the optimal transformation by maximizing the number of consistent correspondences. However, registration recall decreases significantly when point clouds exhibit a low overlap ratio, despite efforts in designing feature descriptors and establishing correspondences. In this paper, we introduce Deep-PE, a lightweight, learning-based pose evaluator designed to enhance the accuracy of pose selection, especially in challenging point cloud scenarios with low overlap. Our network incorporates a Pose-Aware Attention (PAA) module to simulate and learn the alignment status of point clouds under various candidate poses, alongside a Pose Confidence Prediction (PCP) module that predicts the likelihood of successful registration. These two modules facilitate the learning of both local and global alignment priors. Extensive tests across multiple benchmarks confirm the effectiveness of Deep-PE. Notably, on 3DLoMatch with a low overlap ratio, Deep-PE significantly outperforms state-of-the-art methods by at least 8% and 11% in registration recall under handcrafted FPFH and learning-based FCGF descriptors, respectively. To the best of our knowledge, this is the first study to utilize deep learning to select the optimal pose without the explicit need for input correspondences.
Dense Crowd Motion Prediction through Density and Trend Maps
(The Eurographics Association, 2024) Wang, Tingting; Fu, Qiang; Wang, Minggang; Bi, Huikun; Deng, Qixin; Deng, Zhigang; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
In this paper we propose a novel density/trend map based method to predict both group behavior and individual pedestrian motion from video input. Existing motion prediction methods represent pedestrian motion as a set of spatial-temporal trajectories; however, besides such a per-pedestrian representation, a high-level representation for crowd motion is often needed in many crowd applications. Our method leverages density maps and trend maps to represent the spatial-temporal states of dense crowds. Based on such representations, we propose a crowd density map net that extracts a density map from a video clip, and a crowd prediction net that utilizes the historical states of a video clip to predict density maps and trend maps for future frames. Moreover, since the crowd motion consists of the motion of individual pedestrians in a group, we also leverage the predicted crowd motion as a clue to improve the accuracy of traditional trajectory-based motion prediction methods. Through a series of experiments and comparisons with state-of-the-art motion prediction methods, we demonstrate the effectiveness and robustness of our method.
DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping
(The Eurographics Association, 2024) Cai, Zeyu; Wang, Duotun; Liang, Yixun; Shao, Zhijing; Chen, Ying-Cong; Zhan, Xiaohang; Wang, Zeyu; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Score Distillation Sampling (SDS) has emerged as a prevalent technique for text-to-3D generation, enabling 3D content creation by distilling view-dependent information from text-to-2D guidance. However, they frequently exhibit shortcomings such as over-saturated color and excess smoothness. In this paper, we conduct a thorough analysis of SDS and refine its formulation, finding that the core design is to model the distribution of rendered images. Following this insight, we introduce a novel strategy called Variational Distribution Mapping (VDM), which expedites the distribution modeling process by regarding the rendered images as instances of degradation from diffusion-based generation. This special design enables the efficient training of variational distribution by skipping the calculations of the Jacobians in the diffusion U-Net. We also introduce timestep-dependent Distribution Coefficient Annealing (DCA) to further improve distilling precision. Leveraging VDM and DCA, we use Gaussian Splatting as the 3D representation and build a text-to-3D generation framework. Extensive experiments and evaluations demonstrate the capability of VDM and DCA to generate high-fidelity and realistic assets with optimization efficiency.
DViTGAN: Training ViTGANs with Diffusion
(The Eurographics Association, 2024) Tong, Mengjun; Rao, Hong; Yang, Wenji; Chen, Shengbo; Zuo, Fang; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Recent research findings indicate that injecting noise using diffusion can effectively improve the stability of GAN for image generation tasks. Although ViTGAN based on Vision Transformer has certain performance advantages compared to traditional GAN, there are still issues such as unstable training and generated image details are not rich enough. Therefore, in this paper, we propose a novel model, DViTGAN, which leverages the diffusion model to generate instance noise facilitating ViTGAN training. Specifically, we employ forward diffusion to progressively generate noise that follows a Gaussian mixture distribution, and then introduce the generated noise into the input image of the discriminator. The generator incorporates the discriminator's feedback by backpropagating through the forward diffusion process to improve its performance. In addition, we observe that the ViTGAN generator lacks positional information, leading to a decreased context modeling ability and slower convergence. To this end, we introduce Fourier embedding and relative positional encoding to enhance the model's expressive ability. Experiments on multiple popular benchmarks have demonstrated the effectiveness of our proposed model.
Editing Compact Voxel Representations on the GPU
(The Eurographics Association, 2024) Molenaar, Mathijs; Eisemann, Elmar; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
A Sparse Voxel Directed Acyclic Graph (SVDAG) is an efficient representation to display and store a highly-detailed voxel representation in a very compact data structure. Yet, editing such a high-resolution scene in real-time is challenging. Existing solutions are hybrid, involving the CPU, and are restricted to small local modifications. In this work, we address this bottleneck and propose a solution to perform edits fully on the graphics card, enabled by dynamic GPU hash tables. Our framework makes large editing operations possible, such as 3D painting, at real-time frame rates.
Enhancing Human Optical Flow via 3D Spectral Prior
(The Eurographics Association, 2024) Mao, Shiwei; Sun, Mingze; Huang, Ruqi; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
In this paper, we consider the problem of human optical flow estimation, which is critical in a series of human-centric computer vision tasks. Recent deep learning-based optical flow models have achieved considerable accuracy and generalization by incorporating various kinds of priors. However, the majority either rely on large-scale 2D annotations or rigid priors, overlooking the 3D non-rigid nature of human articulations. To this end, we advocate enhancing human optical flow estimation via 3D spectral prior-aware pretraining, which is based on the well-known functional maps formulation in 3D shape matching. Our pretraining can be performed with synthetic human shapes. More specifically, we first render shapes to images and then leverage the natural inclusion maps from images to shapes to lift 2D optical flow into 3D correspondences, which are further encoded as functional maps. Such lifting operation allows to inject the intrinsic geometric features encoded in the spectral representations into optical flow learning, leading to improvement of the latter, especially in the presence of non-rigid deformations. In practice, we establish a pretraining pipeline tailored for triangular meshes, which is general regarding target optical flow network. It is worth noting that it does not introduce any additional learning parameters but only require some pre-computed eigen decomposition on the meshes. For RAFT and GMA, our pretraining task achieves improvements of 12.8% and 4.9% in AEPE on the SHOF benchmark, respectively.
Fast Approximation to Large-Kernel Edge-Preserving Filters by Recursive Reconstruction from Image Pyramids
(The Eurographics Association, 2024) Xu, Tianchen; Yang, Jiale; Qin, Yiming; Sheng, Bin; Wu, Enhua; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
Edge-preserving filters, as known as bilateral filters, are fundamental to graphics rendering techniques, providing greater generality and capability of edge preservation than pure convolution filters. However, sampling with a large kernel per pixel for these filters can be computationally intensive in real-time rendering. Existing acceleration methods for approximating edgepreserving filters still struggle to balance blur controllability, edge clarity, and runtime efficiency. In this paper, we propose a novel scheme for approximating edge-preserving filters with large anisotropic kernels by recursively reconstructing them from multi-image pyramid (MIP) layers that are weightedly filtered in a dual 3×3 kernel space. Our approach introduces a concise unified processing pipeline independent of kernel size, which includes upsampling and downsampling on MIP layers and enables the integration of custom edge-stopping functions. We also derive the implicit relations of the sampling weights and formulate a weight template model for inference. Furthermore, we convert the pipeline into a lightweight neural network for numerical solutions through data training. Consequently, our image post-processors achieve high-quality and high-performance edgepreserving filters in real-time, using the same control parameters as the original bilateral filters. These filters are applicable for depth-of-fields, global illumination denoising, and screen-space particle rendering. The simplicity of the reconstruction process in our pipeline makes it user-friendly and cost-effective, saving both runtime and implementation costs.
Fast Wavelet-domain Smoke Guiding
(The Eurographics Association, 2024) Lyu, Luan; Ren, Xiaohua; Wu, Enhua; Yang, Zhi-Xin; Chen, Renjie; Ritschel, Tobias; Whiting, Emily
We propose a simple and efficient wavelet-based method to guide smoke simulation with specific velocity fields. This method primarily uses wavelets to combine low-resolution velocities with high-resolution details for smoke guiding. Due to the natural ability of wavelets to divide data into different frequency bands, we can merge low and high-resolution velocities by replacing wavelet coefficients. Compared to Fourier methods, the wavelet transform can use wavelets with shorter, compact supports, making the transformation faster and more adaptable to various boundary conditions. The method has a time complexity of O(n) and a memory complexity of n. Additionally, wavelets are compactly supported, which allows us to locally filter out or retain details by editing the wavelet coefficients. This enables us to locally edit smoke. Moreover, to accelerate the performance of wavelet transforms on GPUs, we propose a technique implemented in CUDA called in-kernel warp-level wavelet transform computation. This technique utilizes warp-level CUDA intrinsic functions to reduce data read times during computations, thus enhancing the efficiency of the wavelet transform. The experiments demonstrate that our proposed wavelet-based method achieves an approximate 5x speedup in 3D on GPUs compared to the Fourier methods, resulting in an overall improvement of around 40% in the smoke-guided simulation.

Browse

Browsing PG2024 Conference Papers and Posters by Title

Results Per Page

Sort Options