Virtual Humans
Course: Winter Semester 2024
Continuous Learning of Multimodal Data Streams
University of Tuebingen
|
|
Description
A virtual human is a digital representation of a real human. Virtual humans (VH) should look, move and eventually think like real humans.
Building such VH is one of the long standing goals of Artificial Intelligence. Learning them requires techniques and algorithms at the
intersection of Machine Learning, Computer Vision and Computer Graphics. In this course, we will cover the key mathematical foundations
and computational tools to learn VH from 3D scans, images and video of real humans. The course will cover classical representations of
humans based on 3D meshes and textures, as well as modern ones where the appearance and behavior of virtual humans are encoded in neural networks.
The following topics will be covered in the course:
The following topics will be covered in the course:
- Body and Clothing models.
- Rotations, surface paramterization etc.
- Neural Implicits and NeRF based human models.
- Human Behaviour capture and synthesis.
Organization
Course module number is ML4508. This course is worth 9 ECTS points. There will be one lecture and one tutorial session per week.
The TAs and their office hours are:
- Xianghui Xie (mail) [Friday, 15:00-17:00 (Maria-von-Linden Strasse 6, 2nd floor, 20-31/A12)]
- Yuxuan Xue (mail) [Friday, 10:00-12:00 (Maria-von-Linden Strasse 6, 2nd floor, 20-28/A15)]
- Berna Kabadayi (mail) [Thursday, 12:00-15:00 (S2.013, MPI-IS, Max Planck Ring 4)]
Prerequisites
Basic knowledge of linear algebra, probability theory, optimization, and programming skills in Python are required. Having experience with PyTorch, and knowledge of computer graphics concepts (geometry representation, rendering) is a plus.Location
- Lectures: TBD (the lectures might be moved to Lecture Hall (ground floor), Maria-von-Linden Strasse 6 based on its availability, the place for the lecture will be announced 1 week before on the website and ILIAS).
- Tutorials: Lecture Hall (ground floor), Maria-von-Linden Strasse 6
Assignments
There will be 3 bi-weekly assignments and a research project.Exam
The final exams will take place on-site in Tuebingen, and you need to be physically present. There is going to be one exam at the beginning of the semester break and one at the end of the semester break. The exam format will be determined by the number of students attending the course (written in case of many people attending or oral in case of smaller attendance). The format will be announced at the start of the course.Exam date (tentative): First exam: 26.02.2025, Second exam: 09.04.2025. Location: Lecture Hall (ground floor) Maria-von-Linden Strasse 6.
Schedule for Lectures
The lecture happens every Wednesday 12-14pm. Depends on the availability, lectures will take place either in Maria-von-Linder Strasse 6, ground floor lecture hall (MvL6, preferred) or in Sand (Hoersaal 1 F119).| # | Date & Time | Location | Title | Links |
|---|---|---|---|---|
| 01: | Oct. 23, 1200-1400 | MvL6 | Organization and Introduction to Body Models | YouTube |
| 02: | Oct. 30, 1200-1400 | Sand | Image formations and Rotations | YouTube | pdf 1 2 | pptx 1 2 |
| 03: | Nov. 6, 1200-1400 | TBD | Surface Representations and Procrustes alignment | YouTube | pdf 1 2 | pptx 1 2 |
| 04: | Nov. 13, 1200-1400 | TBD | ICP, Non rigid alignment & vertex based models | YouTube | pdf 1 2 | pptx 1 2 |
| 05: | Nov. 20, 1200-1400 | TBD | Fitting SMPL to scans | YouTube | pdf 1 2 | pptx 1 2 |
| 06: | Nov. 27, 1200-1400 | TBD | Fitting SMPL to images | YouTube | pdf 1 2 | pptx 1 2 |
| 07: | Dec. 04, 1200-1400 | TBD | Fitting SMPL to IMUs | YouTube | pdf 1 2 | pptx 1 2 |
| 08: | Dec. 11, 1600-1800 | TBD | Vertex based Clothing Models | YouTube | pdf 1 | pptx 1 |
| 09: | Dec. 18, 1200-1400 | TBD | Neural Implicits and Point based Clothing Models | YouTube | pdf 1 | pptx 1 |
| 10: | Jan. 8, 1200-1400 | TBD | NeRF, Gaussian Splatting and Humans | YouTube | pdf 1 2 | pptx 1 1 |
| 11: | Jan. 15, 1200-1400 | TBD | Human behaviour capture | YouTube | pdf 1 2 | pptx 1 2 |
| 12: | Jan. 22, 1200-1400 | TBD | Human behaviour synthesis | YouTube | pdf 1 | pptx 1 |
| 13: | Jan. 29, 900-1000 | TBD | Diffusion model and human motion, reconstruction | | pdf TBD | pptx TBD |
Schedule for Tutorials
The tutorials will happen every Tuesday 16-18pm in the lecture hall of Maria-von-Linden Strasse 6.| # | Date & Time | Location | Title |
|---|---|---|---|
| 01: | Oct. 29, 1600-1800 | MvL6 | Organization and Introduction to Body Models |
| 02: | Nov. 05, 1600-1800 | MvL6 | Image formations and Rotations |
| 03: | Nov. 12, 1600-1800 | MvL6 | Surface Representations and Procrustes alignment |
| 04: | Nov. 19, 1600-1800 | MvL6 | ICP, Non rigid alignment and vertex based body models |
| 05: | Nov. 26, 1600-1800 | MvL6 | Fitting SMPL to scans |
| 06: | Dec. 03, 1600-1800 | MvL6 | Fitting SMPL to images |
| 07: | Dec. 10, 1600-1800 | MvL6 | Fitting SMPL to IMUs |
| 08: | Dec. 17, 1600-1800 | MvL6 | Vertex based Clothing Models |
| 09: | Jan. 07, 1600-1800 | MvL6 | Neural Implicits and Point based Clothing Models |
| 10: | Jan. 14, 1600-1800 | - | Individual project discussion |
| 11: | Jan. 21, 1600-1800 | - | Individual project discussion |
| 12: | Jan. 28, 1600-1800 | MvL6 | Project presentations |
Registration
- Registration will be open from September 30, 12pm (noon). We plan to admit 30 students in total, so first come first serve.
- Please register at ILIAS website.