40-Issue 2

Permanent URI for this collection

https://diglib.eg.org/handle/10.2312/2633016

Browse

Now showing 1 - 20 of 48

Interactive Photo Editing on Smartphones via Intrinsic Decomposition
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Shekhar, Sumit; Reimann, Max; Mayer, Maximilian; Semmo, Amir; Pasewaldt, Sebastian; Döllner, Jürgen; Trapp, Matthias; Mitra, Niloy and Viola, Ivan
Intrinsic decomposition refers to the problem of estimating scene characteristics, such as albedo and shading, when one view or multiple views of a scene are provided. The inverse problem setting, where multiple unknowns are solved given a single known pixel-value, is highly under-constrained. When provided with correlating image and depth data, intrinsic scene decomposition can be facilitated using depth-based priors, which nowadays is easy to acquire with high-end smartphones by utilizing their depth sensors. In this work, we present a system for intrinsic decomposition of RGB-D images on smartphones and the algorithmic as well as design choices therein. Unlike state-of-the-art methods that assume only diffuse reflectance, we consider both diffuse and specular pixels. For this purpose, we present a novel specularity extraction algorithm based on a multi-scale intensity decomposition and chroma inpainting. At this, the diffuse component is further decomposed into albedo and shading components. We use an inertial proximal algorithm for non-convex optimization (iPiano) to ensure albedo sparsity. Our GPUbased visual processing is implemented on iOS via the Metal API and enables interactive performance on an iPhone 11 Pro. Further, a qualitative evaluation shows that we are able to obtain high-quality outputs. Furthermore, our proposed approach for specularity removal outperforms state-of-the-art approaches for real-world images, while our albedo and shading layer decomposition is faster than the prior work at a comparable output quality. Manifold applications such as recoloring, retexturing, relighting, appearance editing, and stylization are shown, each using the intrinsic layers obtained with our method and/or the corresponding depth data.
MultiResGNet: Approximating Nonlinear Deformation via Multi-Resolution Graphs
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Li, Tianxing; Shi, Rui; Kanai, Takashi; Mitra, Niloy and Viola, Ivan
This paper presents a graph-learning-based, powerfully generalized method for automatically generating nonlinear deformation for characters with an arbitrary number of vertices. Large-scale character datasets with a significant number of poses are normally required for training to learn such automatic generalization tasks. There are two key contributions that enable us to address this challenge while making our network generalized to achieve realistic deformation approximation. First, after the automatic linear-based deformation step, we encode the roughly deformed meshes by constructing graphs where we propose a novel graph feature representation method with three descriptors to represent meshes of arbitrary characters in varying poses. Second, we design a multi-resolution graph network (MultiResGNet) that takes the constructed graphs as input, and end-to-end outputs the offset adjustments of each vertex. By processing multi-resolution graphs, general features can be better extracted, and the network training no longer heavily relies on large amounts of training data. Experimental results show that the proposed method achieves better performance than prior studies in deformation approximation for unseen characters and poses.
Levitating Rigid Objects with Hidden Rods and Wires
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Kushner, Sarah; Ulinski, Risa; Singh, Karan; Levin, David I. W.; Jacobson, Alec; Mitra, Niloy and Viola, Ivan
We propose a novel algorithm to efficiently generate hidden structures to support arrangements of floating rigid objects. Our optimization finds a small set of rods and wires between objects and each other or a supporting surface (e.g., wall or ceiling) that hold all objects in force and torque equilibrium. Our objective function includes a sparsity inducing total volume term and a linear visibility term based on efficiently pre-computed Monte-Carlo integration, to encourage solutions that are as-hiddenas- possible. The resulting optimization is convex and the global optimum can be efficiently recovered via a linear program. Our representation allows for a user-controllable mixture of tension-, compression-, and shear-resistant rods or tension-only wires. We explore applications to theatre set design, museum exhibit curation, and other artistic endeavours.
Orthogonalized Fourier Polynomials for Signal Approximation and Transfer
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Maggioli, Filippo; Melzi, Simone; Ovsjanikov, Maks; Bronstein, Michael M.; Rodolà, Emanuele; Mitra, Niloy and Viola, Ivan
We propose a novel approach for the approximation and transfer of signals across 3D shapes. The proposed solution is based on taking pointwise polynomials of the Fourier-like Laplacian eigenbasis, which provides a compact and expressive representation for general signals defined on the surface. Key to our approach is the construction of a new orthonormal basis upon the set of these linearly dependent polynomials. We analyze the properties of this representation, and further provide a complete analysis of the involved parameters. Our technique results in accurate approximation and transfer of various families of signals between near-isometric and non-isometric shapes, even under poor initialization. Our experiments, showcased on a selection of downstream tasks such as filtering and detail transfer, show that our method is more robust to discretization artifacts, deformation and noise as compared to alternative approaches.
A Multiscale Microfacet Model Based on Inverse Bin Mapping
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Atanasov, Asen; Wilkie, Alexander; Koylazov, Vladimir; Krivánek, Jaroslav; Mitra, Niloy and Viola, Ivan
Accurately controllable shading detail is a crucial aspect of realistic appearance modelling. Two fundamental building blocks for this are microfacet BRDFs, which describe the statistical behaviour of infinitely small facets, and normal maps, which provide user-controllable spatio-directional surface features. We analyse the filtering of the combined effect of a microfacet BRDF and a normal map. By partitioning the half-vector domain into bins we show that the filtering problem can be reduced to evaluation of an integral histogram (IH), a generalization of a summed-area table (SAT). Integral histograms are known for their large memory requirements, which are usually proportional to the number of bins. To alleviate this, we introduce Inverse Bin Maps, a specialised form of IH with a memory footprint that is practically independent of the number of bins. Based on these, we present a memory-efficient, production-ready approach for filtering of high resolution normal maps with arbitrary Beckmann flake roughness. In the corner case of specular normal maps (zero, or very small roughness values) our method shows similar convergence rates to the current state of the art, and is also more memory efficient.
SnakeBinning: Efficient Temporally Coherent Triangle Packing for Shading Streaming
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Hladky, Jozef; Seidel, Hans-Peter; Steinberger, Markus; Mitra, Niloy and Viola, Ivan
Streaming rendering, e.g., rendering in the cloud and streaming via a mobile connection, suffers from increased latency and unreliable connections. High quality framerate upsampling can hide these issues, especially when capturing shading into an atlas and transmitting it alongside geometric information. The captured shading information must consider triangle footprints and temporal stability to ensure efficient video encoding. Previous approaches only consider either temporal stability or sample distributions, but none focuses on both. With SnakeBinning, we present an efficient triangle packing approach that adjusts sample distributions and caters for temporal coherence. Using a multi-dimensional binning approach, we enforce tight packing among triangles while creating optimal sample distributions. Our binning is built on top of hardware supported real-time rendering where bins are mapped to individual pixels in a virtual framebuffer. Fragment shader interlock and atomic operations enforce global ordering of triangles within each bin, and thus temporal coherence according to the primitive order is achieved. Resampling the bin distribution guarantees high occupancy among all bins and a dense atlas packing. Shading samples are directly captured into the atlas using a rasterization pass, adjusting samples for perspective effects and creating a tight packing. Comparison to previous atlas packing approaches shows that our approach is faster than previous work and achieves the best sample distributions while maintaining temporal coherence. In this way, SnakeBinning achieves the highest rendering quality under equal atlas memory requirements. At the same time, its temporal coherence ensures that we require equal or less bandwidth than previous state-of-the-art. As SnakeBinning outperforms previous approach in all relevant aspects, it is the preferred choice for texture-based streaming rendering.
Patch Erosion for Deformable Lapped Textures on 3D Fluids
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Gagnon, Jonathan; Guzmán, Julián E.; Mould, David; Paquette, Eric; Mitra, Niloy and Viola, Ivan
We propose an approach to synthesise a texture on an animated fluid free surface using a distortion metric combined with a feature map. Our approach is applied as a post-process to a fluid simulation. We advect deformable patches to move the texture along the fluid flow. The patches are covering the whole surface every frame of the animation in an overlapping fashion. Using lapped textures combined with deformable patches, we successfully remove blending artifact and rigid artifact seen in previous methods. We remain faithful to the texture exemplar by removing distorted patch texels using a patch erosion process. The patch erosion is based on a feature map provided together with the exemplar as inputs to our approach. The erosion favors removing texels toward the boundary of the patch as well as texels corresponding to more distorted regions of the patch. Where texels are removed leaving a gap on the surface, we add new patches below existing ones. The result is an animated texture following the velocity field of the fluid. We compared our results with recent work and our results show that our approach removes ghosting and temporal fading artifacts.
Semantics-Guided Latent Space Exploration for Shape Generation
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Jahan, Tansin; Guan, Yanran; Kaick, Oliver van; Mitra, Niloy and Viola, Ivan
We introduce an approach to incorporate user guidance into shape generation approaches based on deep networks. Generative networks such as autoencoders and generative adversarial networks are trained to encode shapes into latent vectors, effectively learning a latent shape space that can be sampled for generating new shapes. Our main idea is to enable users to explore the shape space with the use of high-level semantic keywords. Specifically, the user inputs a set of keywords that describe the general attributes of the shape to be generated, e.g., ''four legs'' for a chair. Then, our method maps the keywords to a subspace of the latent space, where the subspace captures the shapes possessing the specified attributes. The user then explores only this subspace to search for shapes that satisfy the design goal, in a process similar to using a parametric shape model. Our exploratory approach allows users to model shapes at a high level without the need for advanced artistic skills, in contrast to existing methods that allow to guide the generation with sketching or partial modeling of a shape. Our technical contribution to enable this exploration-based approach is the introduction of a label regression neural network coupled with shape encoder/decoder networks. The label regression network takes the user-provided keywords and maps them to distributions in the latent space. We show that our method allows users to explore the shape space and generate a variety of shapes with selected high-level attributes.
Geometric Construction of Auxetic Metamaterials
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Bonneau, Georges-Pierre; Hahmann, Stefanie; Marku, Johana; Mitra, Niloy and Viola, Ivan
This paper is devoted to a category of metamaterials called auxetics, identified by their negative Poisson's ratio. Our work consists in exploring geometrical strategies to generate irregular auxetic structures. More precisely we seek to reduce the Poisson's ratio n, by pruning an irregular network based solely on geometric criteria. We introduce a strategy combining a pure geometric pruning algorithm followed by a physics-based testing phase to determine the resulting Poisson's ratio of our structures. We propose an algorithm that generates sets of irregular auxetic networks. Our contributions include geometrical characterization of auxetic networks, development of a pruning strategy, generation of auxetic networks with low Poisson's ratio, as well as validation of our approach.We provide statistical validation of our approach on large sets of irregular networks, and we additionally laser-cut auxetic networks in sheets of rubber. The findings reported here show that it is possible to reduce the Poisson's ratio by geometric pruning, and that we can generate irregular auxetic networks at lower processing times than a physics-based approach.
Hierarchical Raster Occlusion Culling
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Lee, Gi Beom; Jeong, Moonsoo; Seok, Yechan; Lee, Sungkil; Mitra, Niloy and Viola, Ivan
This paper presents a scalable online occlusion culling algorithm, which significantly improves the previous raster occlusion culling using object-level bounding volume hierarchy. Given occluders found with temporal coherence, we find and rasterize coarse groups of potential occludees in the hierarchy. Within the rasterized bounds, per-pixel ray casting tests fine-grained visibilities of every individual occludees. We further propose acceleration techniques including the read-back of counters for tightly-packed multidrawing and occluder filtering. Our solution requires only constant draw calls for batch occlusion tests, while avoiding costly iteration for hierarchy traversal. Our experiments prove our solution outperforms the existing solutions in terms of scalability, culling efficiency, and occlusion-query performance.
Learning and Exploring Motor Skills with Spacetime Bounds
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Ma, Li-Ke; Yang, Zeshi; Tong, Xin; Guo, Baining; Yin, KangKang; Mitra, Niloy and Viola, Ivan
Equipping characters with diverse motor skills is the current bottleneck of physics-based character animation. We propose a Deep Reinforcement Learning (DRL) framework that enables physics-based characters to learn and explore motor skills from reference motions. The key insight is to use loose space-time constraints, termed spacetime bounds, to limit the search space in an early termination fashion. As we only rely on the reference to specify loose spacetime bounds, our learning is more robust with respect to low quality references. Moreover, spacetime bounds are hard constraints that improve learning of challenging motion segments, which can be ignored by imitation-only learning. We compare our method with state-of-the-art tracking-based DRL methods. We also show how to guide style exploration within the proposed framework.
Deep HDR Estimation with Generative Detail Reconstruction
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Zhang, Yang; Aydin, Tunc O.; Mitra, Niloy and Viola, Ivan
We study the problem of High Dynamic Range (HDR) image reconstruction from a Standard Dynamic Range (SDR) input with potential clipping artifacts. Instead of building a direct model that maps from SDR to HDR images as in previous work, we decompose an input SDR image into a base (low frequency) and detail layer (high frequency), and treat reconstructing these two layers as two separate problems. We propose a novel architecture that comprises individual components specially designed to handle both tasks. Specifically, our base layer reconstruction component recovers low frequency content and remaps the color gamut of the input SDR, whereas our detail layer reconstruction component, which builds upon prior work on image inpainting, hallucinates missing texture information. The output HDR prediction is produced by a final refinement stage. We present qualitative and quantitative comparisons with existing techniques where our method achieves state-of-the-art performance.
Spatiotemporal Texture Reconstruction for Dynamic Objects Using a Single RGB-D Camera
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Kim, Hyomin; Kim, Jungeon; Nam, Hyeonseo; Park, Jaesik; Lee, Seungyong; Mitra, Niloy and Viola, Ivan
This paper presents an effective method for generating a spatiotemporal (time-varying) texture map for a dynamic object using a single RGB-D camera. The input of our framework is a 3D template model and an RGB-D image sequence. Since there are invisible areas of the object at a frame in a single-camera setup, textures of such areas need to be borrowed from other frames. We formulate the problem as an MRF optimization and define cost functions to reconstruct a plausible spatiotemporal texture for a dynamic object. Experimental results demonstrate that our spatiotemporal textures can reproduce the active appearances of captured objects better than approaches using a single texture map.
RigidFusion: RGB-D Scene Reconstruction with Rigidly-moving Objects
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Wong, Yu-Shiang; Li, Changjian; Nießner, Matthias; Mitra, Niloy J.; Mitra, Niloy and Viola, Ivan
Although surface reconstruction from depth data has made significant advances in the recent years, handling changing environments remains a major challenge. This is unsatisfactory, as humans regularly move objects in their environments. Existing solutions focus on a restricted set of objects (e.g., those detected by semantic classifiers) possibly with template meshes, assume static camera, or mark objects touched by humans as moving. We remove these assumptions by introducing RigidFusion. Our core idea is a novel asynchronous moving-object detection method, combined with a modified volumetric fusion. This is achieved by a model-to-frame TSDF decomposition leveraging free-space carving of tracked depth values of the current frame with respect to the background model during run-time. As output, we produce separate volumetric reconstructions for the background and each moving object in the scene, along with its trajectory over time. Our method does not rely on the object priors (e.g., semantic labels or pre-scanned meshes) and is insensitive to the motion residuals between objects and the camera. In comparison to state-of-the-art methods (e.g., Co-Fusion, MaskFusion), we handle significantly more challenging reconstruction scenarios involving moving camera and improve moving-object detection (26% on the miss-detection ratio), tracking (27% on MOTA), and reconstruction (3% on the reconstruction F1) on the synthetic dataset. Please refer the supplementary and the project website for the video demonstration (geometry.cs.ucl.ac.uk/projects/2021/rigidfusion).
EUROGRAPHICS 2021: CGF 40-2 Frontmatter
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Mitra, Niloy; Viola, Ivan; Mitra, Niloy and Viola, Ivan
-
Real-Time Frequency Adjustment of Images and Videos
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Germano, Rafael L.; Oliveira, Manuel M.; Gastal, Eduardo S. L.; Mitra, Niloy and Viola, Ivan
We present a technique for real-time adjustment of spatial frequencies in images and videos. Our method allows for both decreasing and increasing of frequencies, and is orthogonal to image resizing. Thus, it can be used to automatically adjust spatial frequencies to preserve the appearance of structured patterns during image downscaling and upscaling. By pre-computing the image's space-frequency decomposition and its unwrapped phases, these operations can be performed in real time, thanks to our novel mathematical perspective on frequency manipulation of digital images: interpreting the problem through the theory of instantaneous frequencies and phase unwrapping. To make this possible, we introduce an algorithm for the simultaneous phase unwrapping of several unordered frequency components, which also deals with the frequency-sign ambiguity of real signals. As such, our method provides theoretical and practical improvements to the concept of spectral remapping, enabling real-time performance and improved color handling. We demonstrate its effectiveness on a large number of images subject to frequency adjustment. By providing real-time control over the spatial frequencies associated with structured patterns, our technique expands the range of creative and technical possibilities for image and video processing.
Practical Face Reconstruction via Differentiable Ray Tracing
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Dib, Abdallah; Bharaj, Gaurav; Ahn, Junghyun; Thébault, Cédric; Gosselin, Philippe; Romeo, Marco; Chevallier, Louis; Mitra, Niloy and Viola, Ivan
We present a differentiable ray-tracing based novel face reconstruction approach where scene attributes - 3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination - are estimated from unconstrained monocular images. The proposed method models scene illumination via a novel, parameterized virtual light stage, which in-conjunction with differentiable ray-tracing, introduces a coarse-to-fine optimization formulation for face reconstruction. Our method can not only handle unconstrained illumination and self-shadows conditions, but also estimates diffuse and specular albedos. To estimate the face attributes consistently and with practical semantics, a two-stage optimization strategy systematically uses a subset of parametric attributes, where subsequent attribute estimations factor those previously estimated. For example, self-shadows estimated during the first stage, later prevent its baking into the personalized diffuse and specular albedos in the second stage. We show the efficacy of our approach in several real-world scenarios, where face attributes can be estimated even under extreme illumination conditions. Ablation studies, analyses and comparisons against several recent state-of-the-art methods show improved accuracy and versatility of our approach. With consistent face attributes reconstruction, our method leads to several style - illumination, albedo, self-shadow - edit and transfer applications, as discussed in the paper.
Towards a Neural Graphics Pipeline for Controllable Image Generation
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Chen, Xuelin; Cohen-Or, Daniel; Chen, Baoquan; Mitra, Niloy J.; Mitra, Niloy and Viola, Ivan
In this paper, we leverage advances in neural networks towards forming a neural rendering for controllable image generation, and thereby bypassing the need for detailed modeling in conventional graphics pipeline. To this end, we present Neural Graphics Pipeline (NGP), a hybrid generative model that brings together neural and traditional image formation models. NGP decomposes the image into a set of interpretable appearance feature maps, uncovering direct control handles for controllable image generation. To form an image, NGP generates coarse 3D models that are fed into neural rendering modules to produce view-specific interpretable 2D maps, which are then composited into the final output image using a traditional image formation model. Our approach offers control over image generation by providing direct handles controlling illumination and camera parameters, in addition to control over shape and appearance variations. The key challenge is to learn these controls through unsupervised training that links generated coarse 3D models with unpaired real images via neural and traditional (e.g., Blinn- Phong) rendering functions, without establishing an explicit correspondence between them. We demonstrate the effectiveness of our approach on controllable image generation of single-object scenes. We evaluate our hybrid modeling framework, compare with neural-only generation methods (namely, DCGAN, LSGAN, WGAN-GP, VON, and SRNs), report improvement in FID scores against real images, and demonstrate that NGP supports direct controls common in traditional forward rendering. Code is available at http://geometry.cs.ucl.ac.uk/projects/2021/ngp.
Restricted Power Diagrams on the GPU
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Basselin, Justine; Alonso, Laurent; Ray, Nicolas; Sokolov, Dmitry; Lefebvre, Sylvain; Lévy, Bruno; Mitra, Niloy and Viola, Ivan
We propose a method to simultaneously decompose a 3D object into power diagram cells and to integrate given functions in each of the obtained simple regions.We offer a novel, highly parallel algorithm that lends itself to an efficient GPU implementation. It is optimized for algorithms that need to compute many decompositions, for instance, centroidal Voronoi tesselation algorithms and incompressible fluid dynamics simulations. We propose an efficient solution that directly evaluates the integrals over every cell without computing the power diagram explicitly and without intersecting it with a tetrahedralization of the domain. Most computations are performed on the fly, without storing the power diagram. We manipulate a triangulation of the boundary of the domain (instead of tetrahedralizing the domain) to speed up the process. Moreover, the cells are treated independently one from another, making it possible to trivially scale up on a parallel architecture. Despite recent Voronoi diagram generation methods optimized for the GPU, computing integrals over restricted power diagrams still poses significant challenges; the restriction to a complex simulation domain is difficult and likely to be slow. It is not trivial to determine when a cell of a power diagram is completely computed, and the resulting integrals (e.g. the weighted Laplacian operator matrix) do not fit into fast (shared) GPU memory. We address all these issues and boost the performance of the state-of-the-art algorithms by a factor 2 to 3 for (unrestricted) Voronoi diagrams and ax50 speed-up with respect to CPU implementations for restricted power diagrams. An essential ingredient to achieve this is our new scheduling strategy that allows us to treat each Voronoi/power diagram cell with optimal settings and to benefit from the fast memory.
Walk2Map: Extracting Floor Plans from Indoor Walk Trajectories
(The Eurographics Association and John Wiley & Sons Ltd., 2021) Mura, Claudio; Pajarola, Renato; Schindler, Konrad; Mitra, Niloy; Mitra, Niloy and Viola, Ivan
Recent years have seen a proliferation of new digital products for the efficient management of indoor spaces, with important applications like emergency management, virtual property showcasing and interior design. While highly innovative and effective, these products rely on accurate 3D models of the environments considered, including information on both architectural and non-permanent elements. These models must be created from measured data such as RGB-D images or 3D point clouds, whose capture and consolidation involves lengthy data workflows. This strongly limits the rate at which 3D models can be produced, preventing the adoption of many digital services for indoor space management. We provide a radical alternative to such data-intensive procedures by presentingWalk2Map, a data-driven approach to generate floor plans only from trajectories of a person walking inside the rooms. Thanks to recent advances in data-driven inertial odometry, such minimalistic input data can be acquired from the IMU readings of consumer-level smartphones, which allows for an effortless and scalable mapping of real-world indoor spaces. Our work is based on learning the latent relation between an indoor walk trajectory and the information represented in a floor plan: interior space footprint, portals, and furniture. We distinguish between recovering area-related (interior footprint, furniture) and wall-related (doors) information and use two different neural architectures for the two tasks: an image-based Encoder-Decoder and a Graph Convolutional Network, respectively. We train our networks using scanned 3D indoor models and apply them in a cascaded fashion on an indoor walk trajectory at inference time. We perform a qualitative and quantitative evaluation using both trajectories simulated from scanned models of interiors and measured, real-world trajectories, and compare against a baseline method for image-to-image translation. The experiments confirm that our technique is viable and allows recovering reliable floor plans from minimal walk trajectory data.

Browse

Browsing 40-Issue 2 by Issue Date

Results Per Page

Sort Options