Deep Learning for Vision and Graphics

The fields of 3D computer vision and graphics have been revolutionized by deep learning. For example, it is now possible to obtain detailed 3D reconstructions of humans and objects from single images, generate photo-realistic renderings of 3D scenes with neural networks, or manipulate and edit videos and images. In this seminar, we will cover the most recent publications and advances in the fields of neural rendering, 3D computer vision, 3D shape reconstruction, and representation learning for 3D shapes.

This is a Master's level course. Prerequisites:

• Knowledge of linear algebra and calculus, numerical optimization, probability theory.
• Prior participation in one of: Deep Learning, Probabilistic ML, Statistical ML is recommended.

Schedule

• April 20, 10:00-14:00
• Introduction
• April 27, 10:00-14:00
• DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• Occupancy Networks: Learning 3D Reconstruction in Function Space
• TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style
• May 11, 10:00-14:00
• D-NeRF: Neural Radiance Fields for Dynamic Scenes
• May 25, 10:00-14:00
• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields
• Action-Conditioned 3D Human Motion Synthesis with Transformer VAE
• gDNA : Towards Detailed Neural Avatars
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• June 22, 10:00-14:00
• Dynamical Graph CNN for Learning on Point Clouds
• VIBE: Video Inference for Human Body Pose and Shape Estimation
• Stochastic Scene-Aware Motion Prediction
• EMOCA: Emotion Driven Monocular Face Capture and Animation

The exact schedule will be decided with students. The scheduling is roughly bi-weekly, except for the first week. We will have a break at 12 to eat.

• Presentation/ Attendance
• Active participation in the entire event. We have 70% attendance policy for this seminar. You need to attend at least 5 of the 7 sessions.
• Short presentation on a selected topic/paper: 10 minutes talk + 10 minutes questions
• Long presentation on a selected topic/paper: 20 minutes talk + 20 minutes questions
• Read paper, which is to be presented and submit atleast 2 questions before the seminar to Garvita Tiwari (mail)

• Grading scheme:
• Students will be graded based on active participation(~40%) and presentations(~60%).
• Prior questions will also be considered as participation and so students who fail to submit any questions before seminar, will loose participation point.

The seminar will cover the following topics. Interested students are encouraged to check out some of the recent advances in these directions.

3D Implicit Shape Representation

• DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation* (paper)
• Occupancy Networks: Learning 3D Reconstruction in Function Space* (paper)
• Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (paper)
• Neural Unsigned Distance Fields for Implicit Function Learning(paper)
• Implicit Geometric Regularization for Learning Shapes(paper)

Neural Rendering

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis*(paper)
• Stereo Radiance Fields (SRF): Learning View Synthesis from Sparse Views of Novel Scenes(paper)
• D-NeRF: Neural Radiance Fields for Dynamic Scenes(paper)
• H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion(paper)
• GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields(paper)
• Efficient Geometry-aware 3D Generative Adversarial Networks(paper)
• Light Field Networks:Neural Scene Representations with Single-Evaluation Rendering(paper)
• Mixture of Volumetric Primitives for Efficient Neural Rendering(paper)

Modeling Human Appearance

• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization(paper)
• TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style(paper)
• SMPLicit: Topology-aware Generative Model for Clothed People (paper)
• Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Human (paper)
• Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing(paper)
• The Power of Points for Modeling Humans in Clothing(paper)
• gDNA: Towards Generative Detailed Neural Avatars(paper)
• Unsupervised Learning of Efficient Geometry-Aware Neural Articulated Representations (paper)

Modeling Human Motion

• On human motion prediction using recurrent neural networks(paper)
• Action-Conditioned 3D Human Motion Synthesis with Transformer VAE(paper)
• HuMoR: 3D Human Motion Model for Robust Pose Estimation(paper)
• Adversarial Parametric Pose Prior(paper)

Human Scene and Object Interaction

• Long-term Human Motion Prediction with Scene Context(paper)
• Neural state machine for character-scene interactions(paper)
• Grasping Field: Learning Implicit Representations for Human Grasps(paper)
• Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors(paper)
• Stochastic Scene-Aware Motion Prediction(paper)
• BEHAVE: Dataset and Method for Tracking Human Object Interactions(paper)
• GOAL: Generating 4D Whole-Body Motion for Hand-Object Grasping(paper)

• Please register at ILIAS.
• Note that the number of available presentation slots is limited to 10. These slots will be allocated in a first-come, first-serve basis.

19.04.2022 UPDATE: Location and list of papers updated.
10.04.2022 UPDATE: Registration closed.
21.03.2022 UPDATE: Registration open.

Prof. Dr. Gerard Pons-Moll (mail).

Garvita Tiwari (mail).