38-Issue 7

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/2632813

Browse

Now showing 1 - 20 of 70

Light Field Video Compression and Real Time Rendering
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Hajisharif, Saghi; Miandji, Ehsan; Larsson, Per; Tran, Kiet; Unger, Jonas; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Light field imaging is rapidly becoming an established method for generating flexible image based description of scene appearances. Compared to classical 2D imaging techniques, the angular information included in light fields enables effects such as post-capture refocusing and the exploration of the scene from different vantage points. In this paper, we describe a novel GPU pipeline for compression and real-time rendering of light field videos with full parallax. To achieve this, we employ a dictionary learning approach and train an ensemble of dictionaries capable of efficiently representing light field video data using highly sparse coefficient sets. A novel, key element in our representation is that we simultaneously compress both image data (pixel colors) and the auxiliary information (depth, disparity, or optical flow) required for view interpolation. During playback, the coefficients are streamed to the GPU where the light field and the auxiliary information are reconstructed using the dictionary ensemble and view interpolation is performed. In order to realize the pipeline we present several technical contributions including a denoising scheme enhancing the sparsity in the dataset which enables higher compression ratios, and a novel pruning strategy which reduces the size of the dictionary ensemble and leads to significant reductions in computational complexity during the encoding of a light field. Our approach is independent of the light field parameterization and can be used with data from any light field video capture system. To demonstrate the usefulness of our pipeline, we utilize various publicly available light field video datasets and discuss the medical application of documenting heart surgery.
Learning Style Compatibility Between Objects in a Real-World 3D Asset Database
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Liu, Yifan; Tang, Ruolan; Ritchie, Daniel; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Large 3D asset databases are critical for designing virtual worlds, and using them effectively requires techniques for efficient querying and navigation. One important form of query is search by style compatibility: given a query object, find others that would be visually compatible if used in the same scene. In this paper, we present a scalable, learning-based approach for solving this problem which is designed for use with real-world 3D asset databases; we conduct experiments on 121 3D asset packages containing around 4000 3D objects from the Unity Asset Store. By leveraging the structure of the object packages, we introduce a technique to synthesize training labels for metric learning that work as well as human labels. These labels can grow exponentially with the number of objects, allowing our approach to scale to large real-world 3D asset databases without the need for expensive human training labels. We use these synthetic training labels in a metric learning model that analyzes the in-engine rendered appearance of an object—-combining geometry, material, and texture-whereas prior work considers only object geometry, or disjoint geometry and texture features. Through an ablation experiment, we find that using this representation yields better results than using renders which lack texture, materiality, or both.
Wavelet Flow: Optical Flow Guided Wavelet Facial Image Fusion
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Ding, Hong; Yan, Qingan; Fu, Gang; Xiao, Chunxia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Estimating the correspondence between the images using optical flow is the key component for image fusion, however, computing optical flow between a pair of facial images including backgrounds is challenging due to large differences in illumination, texture, color and background in the images. To improve optical flow results for image fusion, we propose a novel flow estimation method, wavelet flow, which can handle both the face and background in the input images. The key idea is that instead of computing flow directly between the input image pair, we estimate the image flow by incorporating multi-scale image transfer and optical flow guided wavelet fusion. Multi-scale image transfer helps to preserve the background and lighting detail of input, while optical flow guided wavelet fusion produces a series of intermediate images for further fusion quality optimizing. Our approach can significantly improve the performance of the optical flow algorithm and provide more natural fusion results for both faces and backgrounds in the images. We evaluate our method on a variety of datasets to show its high outperformance.
Generic Interactive Pixel-level Image Editing
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Liang, Yun; Gan, Yibo; Chen, Mingqin; Gutierrez, Diego; Muñoz Orbañanos, Adolfo; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Several image editing methods have been proposed in the past decades, achieving brilliant results. The most sophisticated of them, however, require additional information per-pixel. For instance, dehazing requires a specific transmittance value per pixel, or depth of field blurring requires depth or disparity values per pixel. This additional per-pixel value is obtained either through elaborated heuristics or through additional control over the capture hardware, which is very often tailored for the specific editing application. In contrast, however, we propose a generic editing paradigm that can become the base of several different applications. This paradigm generates both the needed per-pixel values and the resulting edit at interactive rates, with minimal user input that can be iteratively refined. Our key insight for getting per-pixel values at such speed is to cluster them into superpixels, but, instead of a constant value per superpixel (which yields accuracy problems), we have a mathematical expression for pixel values at each superpixel: in our case, an order two multinomial per superpixel. This leads to a linear leastsquares system, effectively enabling specific per-pixel values at fast speeds. We illustrate this approach in three applications: depth of field blurring (from depth values), dehazing (from transmittance values) and tone mapping (from brightness and contrast local values), and our approach proves both favorably interactive and accurate in all three. Our technique is also evaluated with a common dataset and compared favorably.
RodSteward: A Design-to-Assembly System for Fabrication using 3D-Printed Joints and Precision-Cut Rods
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Jacobson, Alec; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We present RodSteward, a design-to-assembly system for creating furniture-scale structures composed of 3D-printed joints and precision-cut rods. The RodSteward systems consists of: RSDesigner, a fabrication-aware design interface that visualizes accurate geometries during edits and identifies infeasible designs; physical fabrication of parts automatically generated 3Dprintable joint geometries and cutting plans for rods; and RSAssembler, a guided-assembly interface that prompts the user to place parts in order while showing a focus+context visualization of the assembly in progress. We demonstrate the effectiveness of our tools with a number of example constructions of varying complexity, style and parameter choices.
Pyramid Multi-View Stereo with Local Consistency
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Liao, Jie; Fu, Yanping; Yan, Qingan; Xiao, Chunxia; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we propose a PatchMatch-based Multi-View Stereo (MVS) algorithm which can efficiently estimate geometry for the textureless area. Conventional PatchMatch-based MVS algorithms estimate depth and normal hypotheses mainly by optimizing photometric consistency metrics between patch in the reference image and its projection on other images. The photometric consistency works well in textured regions but can not discriminate textureless regions, which makes geometry estimation for textureless regions hard work. To address this issue, we introduce the local consistency. Based on the assumption that neighboring pixels with similar colors likely belong to the same surface and share approximate depth-normal values, local consistency guides the depth and normal estimation with geometry from neighboring pixels with similar colors. To fasten the convergence of pixelwise local consistency across the image, we further introduce a pyramid architecture similar to previous work which can also provide coarse estimation at upper levels. We validate the effectiveness of our method on the ETH3D benchmark and Tanks and Temples benchmark. Results show that our method outperforms the state-of-the-art.
Automatic Modeling of Cluttered Multi-room Floor Plans From Panoramic Images
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Pintore, Giovanni; Ganovelli, Fabio; Villanueva, Alberto Jaspe; Gobbetti, Enrico; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We present a novel and light-weight approach to capture and reconstruct structured 3D models of multi-room floor plans. Starting from a small set of registered panoramic images, we automatically generate a 3D layout of the rooms and of all the main objects inside. Such a 3D layout is directly suitable for use in a number of real-world applications, such as guidance, location, routing, or content creation for security and energy management. Our novel pipeline introduces several contributions to indoor reconstruction from purely visual data. In particular, we automatically partition panoramic images in a connectivity graph, according to the visual layout of the rooms, and exploit this graph to support object recovery and rooms boundaries extraction. Moreover, we introduce a plane-sweeping approach to jointly reason about the content of multiple images and solve the problem of object inference in a top-down 2D domain. Finally, we combine these methods in a fully automated pipeline for creating a structured 3D model of a multi-room floor plan and of the location and extent of clutter objects. These contribution make our pipeline able to handle cluttered scenes with complex geometry that are challenging to existing techniques. The effectiveness and performance of our approach is evaluated on both real-world and synthetic models.
A Stationary SVBRDF Material Modeling Method Based on Discrete Microsurface
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Zhu, Junqiu; Xu, Yanning; Wang, Lu; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Microfacet theory is commonly used to build reflectance models for surfaces. While traditional microfacet-based models assume that the distribution of a surface's microstructure is continuous, recent studies indicate that some surfaces with tiny, discrete and stochastic facets exhibit glittering visual effects, while some surfaces with structured features exhibit anisotropic specular reflection. Accordingly, this paper proposes an efficient and stationary method of surface material modeling to process both glittery and non-glittery surfaces in a consistent way. Our method comprises two steps: in the preprocessing step, we take a fixed-size sample normal map as input, then organize 4D microfacet trees in position and normal space for arbitrary-sized surfaces; we also cluster microfacets into 4D K-lobes via the adaptive k-means method. In the rendering step, moreover, surface normals can be efficiently evaluated using pre-clustered microfacets. Our method is able to efficiently render any structured, discrete and continuous micro-surfaces using a precisely reconstructed surface NDF. Our method is both faster and uses less memory compared to the state-of-the-art glittery surface modeling works.
Style Mixer: Semantic-aware Multi-Style Transfer Network
(The Eurographics Association and John Wiley & Sons Ltd., 2019) HUANG, Zixuan; ZHANG, Jinghuai; LIAO, Jing; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Recent neural style transfer frameworks have obtained astonishing visual quality and flexibility in Single-style Transfer (SST), but little attention has been paid to Multi-style Transfer (MST) which refers to simultaneously transferring multiple styles to the same image. Compared to SST, MST has the potential to create more diverse and visually pleasing stylization results. In this paper, we propose the first MST framework to automatically incorporate multiple styles into one result based on regional semantics. We first improve the existing SST backbone network by introducing a novel multi-level feature fusion module and a patch attention module to achieve better semantic correspondences and preserve richer style details. For MST, we designed a conceptually simple yet effective region-based style fusion module to insert into the backbone. It assigns corresponding styles to content regions based on semantic matching, and then seamlessly combines multiple styles together. Comprehensive evaluations demonstrate that our framework outperforms existing works of SST and MST.
Field-aligned Quadrangulation for Image Vectorization
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Wei, Guangshun; Zhou, Yuanfeng; Gao, Xifeng; Ma, Qian; Xin, Shiqing; He, Ying; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Image vectorization is an important yet challenging problem, especially when the input image has rich content. In this paper, we develop a novel method for automatically vectorizing natural images with feature-aligned quad-dominant meshes. Inspired by the quadrangulation methods in 3D geometry processing, we propose a new directional field optimization technique by encoding the color gradients, sidestepping the explicit computing of salient image features. We further compute the anisotropic scales of the directional field by accommodating the distance among image features. Our method is fully automatic and efficient, which takes only a few seconds for a 400x400 image on a normal laptop. We demonstrate the effectiveness of the proposed method on various image editing applications.
Succinct Palette and Color Model Generation and Manipulation Using Hierarchical Representation
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Jeong, Taehong; Yang, Myunghyun; Shin, Hyun Joon; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
We propose a new method to obtain the representative colors and their distributions of an image. Our intuition is that it is possible to derive the global model from the local distributions. Beginning by sampling pure colors, we build a hierarchical representation of colors in the image via a bottom-up approach. From the resulting hierarchy, we can obtain satisfactory palettes/color models automatically without a predefined size. Furthermore, we provide interactive operations to manipulate the results which allow the users to reflect their intention directly. In our experiment, we show that the proposed method produces more succinct results that faithfully represent all the colors in the image with an appropriate number of components. We also show that the proposed interactive approach can improve the results of applications such as recoloring and soft segmentation.
Subdivision Schemes for Quadrilateral Meshes with the Least Polar Artifact in Extraordinary Regions
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Ma, Yue; Ma, Weiyin; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
This paper presents subdivision schemes with subdivision stencils near an extraordinary vertex that are free from or with substantially reduced polar artifact in extraordinary regions while maintaining the best possible bounded curvature at extraordinary positions. The subdivision stencils are firstly constructed to meet tangent plane continuity with bounded curvature at extraordinary positions. They are further optimized towards curvature continuity at an extraordinary position with additional measures for removing or for minimizing the polar artifact in extraordinary regions. The polar artifact for subdivision stencils of lower valences is removed by applying an additional constraint to the subdominant eigenvalue to be the same as that of subdivision at regular vertices, while the polar artifact for subdivision stencils of higher valances is substantially reduced by introducing an additional thin-plate energy function and a penalty function for maintaining the uniformity and regularity of the characteristic map. A new tuned subdivision scheme is introduced by replacing subdivision stencils of Catmull-Clark subdivision with that from this paper for extraordinary vertices of valences up to nine. We also compare the refined meshes and limit surface quality of the resulting subdivision scheme with that of Catmull-Clark subdivision and other tuned subdivision schemes. The results show that subdivision stencils from our method produce well behaved subdivision meshes with the least polar artifact while maintaining satisfactory limit surface quality.
Inertia-based Fast Vectorization of Line Drawings
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Najgebauer, Patryk; Scherer, Rafal; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Image vectorisation is a fundamental method in graphic design and is one of the tools allowing to transfer artist work into computer graphics. The existing methods are based mainly on segmentation, or they analyse every image pixel; thus, they are relatively slow. We introduce a novel method for fast line drawing image vectorisation, based on a multi-scale second derivative detector accelerated by the summed-area table and an auxiliary grid. Image is scanned initially along the grid lines, and nodes are added to improve accuracy. Applying inertia in the line tracing allows for better junction mapping in a single pass. Our method is dedicated to grey-scale sketches and line drawings. It works efficiently regardless of the thickness of the line or its shading. Experiments show it is more than two orders of magnitude faster than the existing methods, not sacrificing accuracy.
Offline Deep Importance Sampling for Monte Carlo Path Tracing
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Bako, Steve; Meyer, Mark; DeRose, Tony; Sen, Pradeep; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Although modern path tracers are successfully being applied to many rendering applications, there is considerable interest to push them towards ever-decreasing sampling rates. As the sampling rate is substantially reduced, however, even Monte Carlo (MC) denoisers-which have been very successful at removing large amounts of noise-typically do not produce acceptable final results. As an orthogonal approach to this, we believe that good importance sampling of paths is critical for producing betterconverged, path-traced images at low sample counts that can then, for example, be more effectively denoised. However, most recent importance-sampling techniques for guiding path tracing (an area known as ''path guiding'') involve expensive online (per-scene) training and offer benefits only at high sample counts. In this paper, we propose an offline, scene-independent deeplearning approach that can importance sample first-bounce light paths for general scenes without the need of the costly online training, and can start guiding path sampling with as little as 1 sample per pixel. Instead of learning to ''overfit'' to the sampling distribution of a specific scene like most previous work, our data-driven approach is trained a priori on a set of training scenes on how to use a local neighborhood of samples with additional feature information to reconstruct the full incident radiance at a point in the scene, which enables first-bounce importance sampling for new test scenes. Our solution is easy to integrate into existing rendering pipelines without the need for retraining, as we demonstrate by incorporating it into both the Blender/Cycles and Mitsuba path tracers. Finally, we show how our offline, deep importance sampler (ODIS) increases convergence at low sample counts and improves the results of an off-the-shelf denoiser relative to other state-of-the-art sampling techniques.
A Unified Neural Network for Panoptic Segmentation
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Yao, Li; Chyau, Ang; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
In this paper, we propose a unified neural network for panoptic segmentation, a task aiming to achieve more fine-grained segmentation. Following existing methods combining semantic and instance segmentation, our method relies on a triple-branch neural network for tackling the unifying work. In the first stage, we adopt a ResNet50 with a feature pyramid network (FPN) as shared backbone to extract features. Then each branch leverages the shared feature maps and serves as the stuff, things, or mask branch. Lastly, the outputs are fused following a well-designed strategy. Extensive experimental results on MS-COCO dataset demonstrate that our approach achieves a competitive Panoptic Quality (PQ) metric score with the state of the art.
FontRNN: Generating Large-scale Chinese Fonts via Recurrent Neural Network
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Tang, Shusen; Xia, Zeqing; Lian, Zhouhui; Tang, Yingmin; Xiao, Jianguo; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Despite the recent impressive development of deep neural networks, using deep learning based methods to generate largescale Chinese fonts is still a rather challenging task due to the huge number of intricate Chinese glyphs, e.g., the official standard Chinese charset GB18030-2000 consists of 27,533 Chinese characters. Until now, most existing models for this task adopt Convolutional Neural Networks (CNNs) to generate bitmap images of Chinese characters due to CNN based models' remarkable success in various applications. However, CNN based models focus more on image-level features while usually ignore stroke order information when writing characters. Instead, we treat Chinese characters as sequences of points (i.e., writing trajectories) and propose to handle this task via an effective Recurrent Neural Network (RNN) model with monotonic attention mechanism, which can learn from as few as hundreds of training samples and then synthesize glyphs for remaining thousands of characters in the same style. Experimental results show that our proposed FontRNN can be used for synthesizing large-scale Chinese fonts as well as generating realistic Chinese handwritings efficiently.
Scale-adaptive Structure-preserving Texture Filtering
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Song, Chengfang; Xiao, Chunxia; Lei, Ling; Sui, Haigang; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
This paper proposes a scale-adaptive filtering method to improve the performance of structure-preserving texture filtering for image smoothing. With classical texture filters, it usually is challenging to smooth texture at multiple scales while preserving salient structures in an image. We address this issue in the concept of adaptive bilateral filtering, where the scales of Gaussian range kernels are allowed to vary from pixel to pixel. Based on direction-wise statistics, our method distinguishes texture from structure effectively, identifies appropriate scope around a pixel to be smoothed and thus infers an optimal smoothing scale for it. Filtering an image with varying-scale kernels, the image is smoothed according to the distribution of texture adaptively. With commendable experimental results, we show that, needing less iterations, our proposed scheme boosts texture filtering performance in terms of preserving the geometric structures of multiple scales even after aggressive smoothing of the original image.
A PatchMatch-based Approach for Matte Propagation in Videos
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Backes, Marcos; Menezes de Oliveira Neto, Manuel; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Despite considerable advances in natural image matting over the last decades, video matting still remains a difficult problem. The main challenges faced by existing methods are the large amount of user input required, and temporal inconsistencies in mattes between pairs of adjacent frames. We present a temporally-coherent matte-propagation method for videos based on PatchMatch and edge-aware filtering. Given an input video and trimaps for a few frames, including the first and last, our approach generates alpha mattes for all frames of the video sequence. We also present a user scribble-based interface for video matting that takes advantage of the efficiency of our method to interactively refine the matte results. We demonstrate the effectiveness of our approach by using it to generate temporally-coherent mattes for several natural video sequences. We perform quantitative comparisons against the state-of-the-art sparse-input video matting techniques and show that our method produces significantly better results according to three different metrics. We also perform qualitative comparisons against the state-of-the-art dense-input video matting techniques and show that our approach produces similar quality results while requiring only about 7% of the amount of user input required by such techniques. These results show that our method is both effective and user-friendly, outperforming state-of-the-art solutions.
Procedural Riverscapes
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Peytavie, Adrien; Dupont, Thibault; Guérin, Eric; Cortial, Yann; Benes, Bedrich; Gain, James; Galin, Eric; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
This paper addresses the problem of creating animated riverscapes through a novel procedural framework that generates the inscribing geometry of a river network and then synthesizes matching real-time water movement animation. Our approach takes bare-earth heightfields as input, derives hydrologically-inspired river network trajectories, carves riverbeds into the terrain, and then automatically generates a corresponding blend-flow tree for the water surface. Characteristics, such as the riverbed width, depth and shape, as well as elevation and flow of the fluid surface, are procedurally derived from the terrain and river type. The riverbed is inscribed by combining compactly supported elevation modifiers over the river course. Subsequently, the water surface is defined as a time-varying continuous function encoded as a blend-flow tree with leaves that are parameterized procedural flow primitives and internal nodes that are blend operators. While river generation is fully automated, we also incorporate intuitive interactive editing of both river trajectories and individual riverbed and flow primitives. The resulting framework enables the generation of a wide range of river forms, ranging from slow meandering rivers to rapids with churning water, including surface effects, such as foam and leaves carried downstream.
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction
(The Eurographics Association and John Wiley & Sons Ltd., 2019) Guo, Yi; Zhang, Zhuming; Han, Chu; Hu, Wenbo; Li, Chengze; Wong, Tien-Tsin; Lee, Jehee and Theobalt, Christian and Wetzstein, Gordon
Vectorizing line drawing is necessary for the digital workflows of 2D animation and engineering design. But it is challenging due to the ambiguity of topology, especially at junctions. Existing vectorization methods either suffer from low accuracy or cannot deal with high-resolution images. To deal with a variety of challenging containing different kinds of complex junctions, we propose a two-phase line drawing vectorization method that analyzes the global and local topology. In the first phase, we subdivide the lines into partial curves, and in the second phase, we reconstruct the topology at junctions. With the overall topology estimated in the two phases, we can trace and vectorize the curves. To qualitatively and quantitatively evaluate our method and compare it with the existing methods, we conduct extensive experiments on not only existing datasets but also our newly synthesized dataset which contains different types of complex and ambiguous junctions. Experimental statistics show that our method greatly outperforms existing methods in terms of computational speed and achieves visually better topology reconstruction accuracy.

Browse

Browsing 38-Issue 7 by Issue Date

Results Per Page

Sort Options