Browsing by Author "Sitzmann, Vincent"

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Advances in Neural Rendering
    (The Eurographics Association and John Wiley & Sons Ltd., 2022) Tewari, Ayush; Thies, Justus; Mildenhall, Ben; Srinivasan, Pratul; Tretschk, Edith; Wang, Yifan; Lassner, Christoph; Sitzmann, Vincent; Martin-Brualla, Ricardo; Lombardi, Stephen; Simon, Tomas; Theobalt, Christian; Nießner, Matthias; Barron, Jon T.; Wetzstein, Gordon; Zollhöfer, Michael; Golyanik, Vladislav; Meneveaux, Daniel; Patanè, Giuseppe
    Synthesizing photo-realistic images and videos is at the heart of computer graphics and has been the focus of decades of research. Traditionally, synthetic images of a scene are generated using rendering algorithms such as rasterization or ray tracing, which take specifically defined representations of geometry and material properties as input. Collectively, these inputs define the actual scene and what is rendered, and are referred to as the scene representation (where a scene consists of one or more objects). Example scene representations are triangle meshes with accompanied textures (e.g., created by an artist), point clouds (e.g., from a depth sensor), volumetric grids (e.g., from a CT scan), or implicit surface functions (e.g., truncated signed distance fields). The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering. Neural rendering is closely related, and combines ideas from classical computer graphics and machine learning to create algorithms for synthesizing images from real-world observations. Neural rendering is a leap forward towards the goal of synthesizing photo-realistic image and video content. In recent years, we have seen immense progress in this field through hundreds of publications that show different ways to inject learnable components into the rendering pipeline. This state-of-the-art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D-consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling nonrigidly deforming objects and scene editing and composition. While most of these approaches are scene-specific, we also discuss techniques that generalize across object classes and can be used for generative tasks. In addition to reviewing these state-ofthe- art methods, we provide an overview of fundamental concepts and definitions used in the current literature. We conclude with a discussion on open challenges and social implications.
  • Thumbnail Image
    Item
    Neural Fields in Visual Computing and Beyond
    (The Eurographics Association and John Wiley & Sons Ltd., 2022) Xie, Yiheng; Takikawa, Towaki; Saito, Shunsuke; Litany, Or; Yan, Shiqin; Khan, Numair; Tombari, Federico; Tompkin, James; Sitzmann, Vincent; Sridhar, Srinath; Meneveaux, Daniel; Patanè, Giuseppe
    Recent advances in machine learning have led to increased interest in solving visual computing problems using methods that employ coordinate-based neural networks. These methods, which we call neural fields, parameterize physical properties of scenes or objects across space and time. They have seen widespread success in problems such as 3D shape and image synthesis, animation of human bodies, 3D reconstruction, and pose estimation. Rapid progress has led to numerous papers, but a consolidation of the discovered knowledge has not yet emerged. We provide context, mathematical grounding, and a review of over 250 papers in the literature on neural fields. In Part I, we focus on neural field techniques by identifying common components of neural field methods, including different conditioning, representation, forward map, architecture, and manipulation methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, and highlights the improved quality, flexibility, and capability brought by neural field methods. Finally, we present a companion website that acts as a living database that can be continually updated by the community.