Virtual Humans

Course: Winter Semester 2024

Continuous Learning of Multimodal Data Streams

University of Tuebingen

appearance hps

Description

A virtual human is a digital representation of a real human. Virtual humans (VH) should look, move and eventually think like real humans. Building such VH is one of the long standing goals of Artificial Intelligence. Learning them requires techniques and algorithms at the intersection of Machine Learning, Computer Vision and Computer Graphics. In this course, we will cover the key mathematical foundations and computational tools to learn VH from 3D scans, images and video of real humans. The course will cover classical representations of humans based on 3D meshes and textures, as well as modern ones where the appearance and behavior of virtual humans are encoded in neural networks.
The following topics will be covered in the course:
  • Body and Clothing models.
  • Rotations, surface paramterization etc.
  • Neural Implicits and NeRF based human models.
  • Human Behaviour capture and synthesis.

Organization

Course module number is ML4508. This course is worth 9 ECTS points. There will be one lecture and one tutorial session per week. The TAs and their office hours are:

Prerequisites

Basic knowledge of linear algebra, probability theory, optimization, and programming skills in Python are required. Having experience with PyTorch, and knowledge of computer graphics concepts (geometry representation, rendering) is a plus.

Location


Assignments

There will be 3 bi-weekly assignments and a research project.

Exam

The final exams will take place on-site in Tuebingen, and you need to be physically present. There is going to be one exam at the beginning of the semester break and one at the end of the semester break. The exam format will be determined by the number of students attending the course (written in case of many people attending or oral in case of smaller attendance). The format will be announced at the start of the course.
Exam date (tentative): First exam: 26.02.2025, Second exam: 09.04.2025. Location: Lecture Hall (ground floor) Maria-von-Linden Strasse 6.


Schedule for Lectures

The lecture happens every Wednesday 12-14pm. Depends on the availability, lectures will take place either in Maria-von-Linder Strasse 6 (MvL6, preferred) or in Sand (Hoersaal 1 F119).
# Date & Time Location Title Links
01: Oct. 23, 1200-1400 MvL6 lecture hall (ground floor) Organization and Introduction to Body Models
02: Oct. 30, 1200-1400 Sand (Hoersaal 1 F119) Image formations and Rotations
03: Nov. 6, 1200-1400 TBD Surface Representations and Procrustes alignment
04: Nov. 13, 1200-1400 TBD ICP, Non rigid alignment & vertex based models
05: Nov. 20, 1200-1400 TBD Fitting SMPL to scans
06: Nov. 27, 1200-1400 TBD Fitting SMPL to images
07: Dec. 04, 1200-1400 TBD Fitting SMPL to IMUs
08: Dec. 11, 1600-1800 TBD Vertex based Clothing Models
09: Dec. 18, 1200-1400 TBD Neural Implicits and Point based Clothing Models
10: Jan. 8, 1200-1400 TBD NeRF, Gaussian Splatting and Humans
11: Jan. 15, 1200-1400 TBD Human behaviour capture
12: Jan. 22, 1200-1400 TBD Human behaviour synthesis
13: Jan. 29, 900-1000 TBD Diffusion model and human motion, reconstruction


Schedule for Tutorials

The tutorials will happen every Tuesday 16-18pm in the lecture hall of Maria-von-Linden Strasse 6.
# Date & Time Location Title
01: Oct. 29, 1600-1800 MvL6 Organization and Introduction to Body Models
02: Nov. 05, 1600-1800 MvL6 Image formations and Rotations
03: Nov. 12, 1600-1800 MvL6 Surface Representations and Procrustes alignment
04: Nov. 19, 1600-1800 MvL6 ICP, Non rigid alignment and vertex based body models
05: Nov. 26, 1600-1800 MvL6 Fitting SMPL to scans
06: Dec. 03, 1600-1800 MvL6 Fitting SMPL to images
07: Dec. 10, 1600-1800 MvL6 Fitting SMPL to IMUs
08: Dec. 17, 1600-1800 MvL6 Vertex based Clothing Models
09: Jan. 07, 1600-1800 MvL6 Neural Implicits and Point based Clothing Models
10: Jan. 14, 1600-1800 - Individual project discussion
11: Jan. 21, 1600-1800 - Individual project discussion
12: Jan. 28, 1600-1800 MvL6 Project presentations


Registration

  • Registration will be open from September 30, 12pm (noon). We plan to admit 30 students in total, so first come first serve.
  • Please register at ILIAS website.

Contact

Prof. Dr. Gerard Pons-Moll (mail).
Xianghui Xie (mail).
Yuxuan Xue (mail).
Berna Kabadayi (mail).