38-Issue 7

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/2632813

Browse

Now showing 1 - 20 of 70

Style Mixer: Semantic-aware Multi-Style Transfer Network
(The Eurographics Association and John Wiley & Sons Ltd., 2019) HUANG, Zixuan; ZHANG, Jinghuai; LIAO, Jing; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Recent neural style transfer frameworks have obtained astonishing visual quality and flexibility in Single-style Transfer (SST), but little attention has been paid to Multi-style Transfer (MST) which refers to simultaneously transferring multiple styles to the same image. Compared to SST, MST has the potential to create more diverse and visually pleasing stylization results. In this paper, we propose the first MST framework to automatically incorporate multiple styles into one result based on regional semantics. We first improve the existing SST backbone network by introducing a novel multi-level feature fusion module and a patch attention module to achieve better semantic correspondences and preserve richer style details. For MST, we designed a conceptually simple yet effective region-based style fusion module to insert into the backbone. It assigns corresponding styles to content regions based on semantic matching, and then seamlessly combines multiple styles together. Comprehensive evaluations demonstrate that our framework outperforms existing works of SST and MST.
Image Composition of Partially Occluded Objects
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Tan, Xuehan; Xu, Panpan; Guo, Shihui; Wang, Wencheng; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Image composition extracts the content of interest (COI) from a source image and blends it into a target image to generate a new image. In the majority of existing works, the COI is manually extracted and then overlaid on top of the target image. However, in practice, it is often necessary to deal with situations in which the COI is partially occluded by the target image content. In this regard, both tasks of extracting the COI and cropping its occluded part require intensive user interactions, which are laborious and seriously reduce the composition efficiency. This paper addresses the aforementioned challenges by proposing an efficient image composition method. First, we extract the semantic contents of the images by using state-of-the-art deep learning methods. Therefore, the COI can be selected with clicks only, which can greatly reduce the demanded user interactions. Second, according to the user's operations (such as translation or scale) on the COI, we can effectively infer the occlusion relationships between the COI and the contents of the target image. Thus, the COI can be adaptively embedded into the target image without concern about cropping its occluded part. Therefore, the procedures of content extraction and occlusion handling can be significantly simplified, and work efficiency is remarkably improved. Experimental results show that compared to existing works, our method can reduce the number of user interactions to approximately one-tenth and increase the speed of image composition by more than ten times.
Learning to Predict Image-based Rendering Artifacts with Respect to a Hidden Reference Image
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Bemana, Mojtaba; Keinert, Joachim; Myszkowski, Karol; Bätz, Michel; Ziegler, Matthias; Seidel, Hans-Peter; Ritschel, Tobias; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Image metrics predict the perceived per-pixel difference between a reference image and its degraded (e. g., re-rendered) version. In several important applications, the reference image is not available and image metrics cannot be applied. We devise a neural network architecture and training procedure that allows predicting the MSE, SSIM or VGG16 image difference from the distorted image alone while the reference is not observed. This is enabled by two insights: The first is to inject sufficiently many un-distorted natural image patches, which can be found in arbitrary amounts and are known to have no perceivable difference to themselves. This avoids false positives. The second is to balance the learning, where it is carefully made sure that all image errors are equally likely, avoiding false negatives. Surprisingly, we observe that the resulting no-reference metric, subjectively, can even perform better than the reference-based one, as it had to become robust against mis-alignments. We evaluate the effectiveness of our approach in an image-based rendering context, both quantitatively and qualitatively. Finally, we demonstrate two applications which reduce light field capture time and provide guidance for interactive depth adjustment.
Two-phase Hair Image Synthesis by Self-Enhancing Generative Model
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Qiu, Haonan; Wang, Chuan; Zhu, Hang; zhu, xiangyu; Gu, Jinjin; Han, Xiaoguang; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Generating plausible hair image given limited guidance, such as sparse sketches or low-resolution image, has been made possible with the rise of Generative Adversarial Networks (GANs). Traditional image-to-image translation networks can generate recognizable results, but finer textures are usually lost and blur artifacts commonly exist. In this paper, we propose a two-phase generative model for high-quality hair image synthesis. The two-phase pipeline first generates a coarse image by an existing image translation model, then applies a re-generating network with self-enhancing capability to the coarse image. The selfenhancing capability is achieved by a proposed differentiable layer, which extracts the structural texture and orientation maps from a hair image. Extensive experiments on two tasks, Sketch2Hair and Hair Super-Resolution, demonstrate that our approach is able to synthesize plausible hair image with finer details, and reaches the state-of-the-art.
Topology Preserving Simplification of Medial Axes in 3D Models
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Chu, Yiyao; Hou, Fei; Wang, Wencheng; Li, Lei; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We propose an efficient method for topology-preserving simplification of medial axes of 3D models. Existing methods either cannot preserve the topology during medial axes simplification or have the problem of being geometrically inaccurate or computationally expensive. To tackle these issues, we restrict our topology-checking to the areas around the topological holes to avoid unnecessary checks in other areas. Our algorithm can keep high precision even when the medial axis is simplified to be in very few vertices. Furthermore, we parallelize the medial axes simplification procedure to enhance the performance significantly. Experimental results show that our method can preserve the topology with highly efficient performance, much superior to the existing methods in terms of topology preservation, accuracy and performance.
ManyLands: A Journey Across 4D Phase Space of Trajectories
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Amirkhanov, Aleksandr; Kosiuk, Ilona; Szmolyan, Peter; Amirkhanov, Artem; Mistelbauer, Gabriel; Gröller, Eduard; Raidou, Renata Georgia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Mathematical models of ordinary differential equations are used to describe and understand biological phenomena. These models are dynamical systems that often describe the time evolution of more than three variables, i.e., their dynamics take place in a multi-dimensional space, called the phase space. Currently, mathematical domain scientists use plots of typical trajectories in the phase space to analyze the qualitative behavior of dynamical systems. These plots are called phase portraits and they perform well for 2D and 3D dynamical systems. However, for 4D, the visual exploration of trajectories becomes challenging, as simple subspace juxtaposition is not sufficient. We propose ManyLands to support mathematical domain scientists in analyzing 4D models of biological systems. By describing the subspaces as Lands, we accompany domain scientists along a continuous journey through 4D HyperLand, 3D SpaceLand, and 2D FlatLand, using seamless transitions. The Lands are also linked to 1D TimeLines. We offer an additional dissected view of trajectories that relies on small-multiple compass-alike pictograms for easy navigation across subspaces and trajectory segments of interest. We show three use cases of 4D dynamical systems from cell biology and biochemistry. An informal evaluation with mathematical experts confirmed that ManyLands helps them to visualize and analyze complex 4D dynamics, while facilitating mathematical experiments and simulations.
Field-aligned Quadrangulation for Image Vectorization
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Wei, Guangshun; Zhou, Yuanfeng; Gao, Xifeng; Ma, Qian; Xin, Shiqing; He, Ying; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Image vectorization is an important yet challenging problem, especially when the input image has rich content. In this paper, we develop a novel method for automatically vectorizing natural images with feature-aligned quad-dominant meshes. Inspired by the quadrangulation methods in 3D geometry processing, we propose a new directional field optimization technique by encoding the color gradients, sidestepping the explicit computing of salient image features. We further compute the anisotropic scales of the directional field by accommodating the distance among image features. Our method is fully automatic and efficient, which takes only a few seconds for a 400x400 image on a normal laptop. We demonstrate the effectiveness of the proposed method on various image editing applications.
Automatic Modeling of Cluttered Multi-room Floor Plans From Panoramic Images
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Pintore, Giovanni; Ganovelli, Fabio; Villanueva, Alberto Jaspe; Gobbetti, Enrico; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We present a novel and light-weight approach to capture and reconstruct structured 3D models of multi-room floor plans. Starting from a small set of registered panoramic images, we automatically generate a 3D layout of the rooms and of all the main objects inside. Such a 3D layout is directly suitable for use in a number of real-world applications, such as guidance, location, routing, or content creation for security and energy management. Our novel pipeline introduces several contributions to indoor reconstruction from purely visual data. In particular, we automatically partition panoramic images in a connectivity graph, according to the visual layout of the rooms, and exploit this graph to support object recovery and rooms boundaries extraction. Moreover, we introduce a plane-sweeping approach to jointly reason about the content of multiple images and solve the problem of object inference in a top-down 2D domain. Finally, we combine these methods in a fully automated pipeline for creating a structured 3D model of a multi-room floor plan and of the location and extent of clutter objects. These contribution make our pipeline able to handle cluttered scenes with complex geometry that are challenging to existing techniques. The effectiveness and performance of our approach is evaluated on both real-world and synthetic models.
A Stationary SVBRDF Material Modeling Method Based on Discrete Microsurface
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Zhu, Junqiu; Xu, Yanning; Wang, Lu; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Microfacet theory is commonly used to build reflectance models for surfaces. While traditional microfacet-based models assume that the distribution of a surface's microstructure is continuous, recent studies indicate that some surfaces with tiny, discrete and stochastic facets exhibit glittering visual effects, while some surfaces with structured features exhibit anisotropic specular reflection. Accordingly, this paper proposes an efficient and stationary method of surface material modeling to process both glittery and non-glittery surfaces in a consistent way. Our method comprises two steps: in the preprocessing step, we take a fixed-size sample normal map as input, then organize 4D microfacet trees in position and normal space for arbitrary-sized surfaces; we also cluster microfacets into 4D K-lobes via the adaptive k-means method. In the rendering step, moreover, surface normals can be efficiently evaluated using pre-clustered microfacets. Our method is able to efficiently render any structured, discrete and continuous micro-surfaces using a precisely reconstructed surface NDF. Our method is both faster and uses less memory compared to the state-of-the-art glittery surface modeling works.
RodSteward: A Design-to-Assembly System for Fabrication using 3D-Printed Joints and Precision-Cut Rods
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Jacobson, Alec; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We present RodSteward, a design-to-assembly system for creating furniture-scale structures composed of 3D-printed joints and precision-cut rods. The RodSteward systems consists of: RSDesigner, a fabrication-aware design interface that visualizes accurate geometries during edits and identifies infeasible designs; physical fabrication of parts automatically generated 3Dprintable joint geometries and cutting plans for rods; and RSAssembler, a guided-assembly interface that prompts the user to place parts in order while showing a focus+context visualization of the assembly in progress. We demonstrate the effectiveness of our tools with a number of example constructions of varying complexity, style and parameter choices.
Pyramid Multi-View Stereo with Local Consistency
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Liao, Jie; Fu, Yanping; Yan, Qingan; Xiao, Chunxia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we propose a PatchMatch-based Multi-View Stereo (MVS) algorithm which can efficiently estimate geometry for the textureless area. Conventional PatchMatch-based MVS algorithms estimate depth and normal hypotheses mainly by optimizing photometric consistency metrics between patch in the reference image and its projection on other images. The photometric consistency works well in textured regions but can not discriminate textureless regions, which makes geometry estimation for textureless regions hard work. To address this issue, we introduce the local consistency. Based on the assumption that neighboring pixels with similar colors likely belong to the same surface and share approximate depth-normal values, local consistency guides the depth and normal estimation with geometry from neighboring pixels with similar colors. To fasten the convergence of pixelwise local consistency across the image, we further introduce a pyramid architecture similar to previous work which can also provide coarse estimation at upper levels. We validate the effectiveness of our method on the ETH3D benchmark and Tanks and Temples benchmark. Results show that our method outperforms the state-of-the-art.
Succinct Palette and Color Model Generation and Manipulation Using Hierarchical Representation
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Jeong, Taehong; Yang, Myunghyun; Shin, Hyun Joon; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We propose a new method to obtain the representative colors and their distributions of an image. Our intuition is that it is possible to derive the global model from the local distributions. Beginning by sampling pure colors, we build a hierarchical representation of colors in the image via a bottom-up approach. From the resulting hierarchy, we can obtain satisfactory palettes/color models automatically without a predefined size. Furthermore, we provide interactive operations to manipulate the results which allow the users to reflect their intention directly. In our experiment, we show that the proposed method produces more succinct results that faithfully represent all the colors in the image with an appropriate number of components. We also show that the proposed interactive approach can improve the results of applications such as recoloring and soft segmentation.
Figure Skating Simulation from Video
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Yu, Ri; Park, Hwangpil; Lee, Jehee; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Figure skating is one of the most popular ice sports at the Winter Olympic Games. The skaters perform several skating skills to express the beauty of the art on ice. Skating involves moving on ice while wearing skate shoes with thin blades; thus, it requires much practice to skate without losing balance. Moreover, figure skating presents dynamic moves, such as jumping, artistically. Therefore, demonstrating figure skating skills is even more difficult to achieve than basic skating, and professional skaters often fall during Winter Olympic performances. We propose a system to demonstrate figure skating motions with a physically simulated human-like character. We simulate skating motions with non-holonomic constraints, which make the skate blade glide on the ice surface. It is difficult to obtain reference motions from figure skaters because figure skating motions are very fast and dynamic. Instead of using motion capture data, we use key poses extracted from videos on YouTube and complete reference motions using trajectory optimization. We demonstrate figure skating skills, such as crossover, three-turn, and even jump. Finally, we use deep reinforcement learning to generate a robust controller for figure skating skills.
Shadow Inpainting and Removal Using Generative Adversarial Networks with Slice Convolutions
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Wei, Jinjiang; Long, Chengjiang; Zou, Hua; Xiao, Chunxia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we propose a two-stage top-down and bottom-up Generative Adversarial Networks (TBGANs) for shadow inpainting and removal which uses a novel top-down encoder and a bottom-up decoder with slice convolutions. These slice convolutions can effectively extract and restore the long-range spatial information for either down-sampling or up-sampling. Different from the previous shadow removal methods based on deep learning, we propose to inpaint shadow to handle the possible dark shadows to achieve a coarse shadow-removal image at the first stage, and then further recover the details and enhance the color and texture details with a non-local block to explore both local and global inter-dependencies of pixels at the second stage. With such a two-stage coarse-to-fine processing, the overall effect of shadow removal is greatly improved, and the effect of color retention in non-shaded areas is significant. By comparing with a variety of mainstream shadow removal methods, we demonstrate that our proposed method outperforms the state-of-the-art methods.
Imitating Popular Photos to Select Views for an Indoor Scene
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Su, Rung-De; Liao, Zhe-Yo; Chen, Li-Chi; Tung, Ai-Ling; Wang, Yu-Shuen; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Selecting informative and visually appealing views for 3D indoor scenes is beneficial for the housing, decoration, and entertainment industries. A set of views that exhibit comfort, aesthetics, and functionality of a particular scene can attract customers and facilitate business transactions. However, selecting views for an indoor scene is challenging because the system has to consider not only the need to reveal as much information as possible, but also object arrangements, occlusions, and characteristics. Since there can be many principles utilized to guide the view selection, and various principles to follow under different circumstances, we achieve the goal by imitating popular photos on the Internet. Specifically, we select the view that can optimize the contour similarity of corresponding objects to the photo. Because the selected view can be inadequate if object arrangements in the 3D scene and the photo are different, our system imitates many popular photos and selects a certain number of views. After that, it clusters the selected views and determines the view/cluster centers by the weighted average to finally exhibit the scene. Experimental results demonstrate that the views selected by our method are visually appealing.
Specular Highlight Removal for Real-world Images
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Fu, Gang; Zhang, Qing; Song, Chengfang; Lin, Qifeng; Xiao, Chunxia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Removing specular highlight in an image is a fundamental research problem in computer vision and computer graphics. While various methods have been proposed, they typically do not work well for real-world images due to the presence of rich textures, complex materials, hard shadows, occlusions and color illumination, etc. In this paper, we present a novel specular highlight removal method for real-world images. Our approach is based on two observations of the real-world images: (i) the specular highlight is often small in size and sparse in distribution; (ii) the remaining diffuse image can be represented by linear com- bination of a small number of basis colors with the sparse encoding coefficients. Based on the two observations, we design an optimization framework for simultaneously estimating the diffuse and specular highlight images from a single image. Specif- ically, we recover the diffuse components of those regions with specular highlight by encouraging the encoding coefficients sparseness using L0 norm. Moreover, the encoding coefficients and specular highlight are also subject to the non-negativity according to the additive color mixing theory and the illumination definition, respectively. Extensive experiments have been performed on a variety of images to validate the effectiveness of the proposed method and its superiority over the previous methods.
Visibility-Aware Progressive Farthest Point Sampling on the GPU
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Brandt, Sascha; Jähn, Claudius; Fischer, Matthias; Heide, Friedhelm Meyer auf der; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we present the first algorithm for progressive sampling of 3D surfaces with blue noise characteristics that runs entirely on the GPU. The performance of our algorithm is comparable to state-of-the-art GPU Poisson-disk sampling methods, while additionally producing ordered sequences of samples where every prefix exhibits good blue noise properties. The basic idea is, to reduce the 3D sampling domain to a set of 2.5D images which we sample in parallel utilizing the rasterization hardware of current GPUs. This allows for simple visibility-aware sampling that only captures the surface as seen from outside the sampled object, which is especially useful for point-based level-of-detail rendering methods. However, our method can be easily extended for sampling the entire surface without changing the basic algorithm. We provide a statistical analysis of our algorithm and show that it produces good blue noise characteristics for every prefix of the resulting sample sequence and analyze the performance of our method compared to related state-of-the-art sampling methods.
Practical Foldover-Free Volumetric Mapping Construction
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Su, Jian-Ping; Fu, Xiao-Ming; Liu, Ligang; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we present a practically robust method for computing foldover-free volumetric mappings with hard linear constraints. Central to this approach is a projection algorithm that monotonically and efficiently decreases the distance from the mapping to the bounded conformal distortion mapping space. After projection, the conformal distortion of the updated mapping tends to be below the given bound, thereby significantly reducing foldovers. Since it is non-trivial to define an optimal bound, we introduce a practical conformal distortion bound generation scheme to facilitate subsequent projections. By iteratively generating conformal distortion bounds and trying to project mappings into bounded conformal distortion spaces monotonically, our algorithm achieves high-quality foldover-free volumetric mappings with strong practical robustness and high efficiency. Compared with existing methods, our method computes mesh-based and meshless volumetric mappings with no prescribed conformal distortion bounds. We demonstrate the efficacy and efficiency of our method through a variety of geometric processing tasks.
Generating 3D Faces using Multi-column Graph Convolutional Networks
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Li, Kun; Liu, Jingying; Lai, Yu-Kun; Yang, Jingyu; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this work, we introduce multi-column graph convolutional networks (MGCNs), a deep generative model for 3D mesh surfaces that effectively learns a non-linear facial representation. We perform spectral decomposition of meshes and apply convolutions directly in the frequency domain. Our network architecture involves multiple columns of graph convolutional networks (GCNs), namely large GCN (L-GCN), medium GCN (M-GCN) and small GCN (S-GCN), with different filter sizes to extract features at different scales. L-GCN is more useful to extract large-scale features, whereas S-GCN is effective for extracting subtle and fine-grained features, and M-GCN captures information in between. Therefore, to obtain a high-quality representation, we propose a selective fusion method that adaptively integrates these three kinds of information. Spatially non-local relationships are also exploited through a self-attention mechanism to further improve the representation ability in the latent vector space. Through extensive experiments, we demonstrate the superiority of our end-to-end framework in improving the accuracy of 3D face reconstruction. Moreover, with the help of variational inference, our model has excellent generating ability.
Pacific Conference on Computer Graphics and Applications 2019 - CGF38-7: Frontmatter
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Lee, Jehee; Theobalt, Christian; Wetzstein, Gordon; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon

Browse

Browsing 38-Issue 7 by Issue Date

Results Per Page

Sort Options