Generative AI - From Diffusion to World Models (IN2106, IN40038)

Welcome to the Generative AI - From Diffusion to World Models practical course!

Generative AI has rapidly evolved from image synthesis to full-fledged world modeling, enabling machines to generate, reconstruct, and reason about complex visual and physical environments. This practical course provides an in-depth, hands-on introduction to modern generative models, spanning diffusion models, autoregressive transformers, feed-forward reconstruction networks, and world models.

In the spirit of learning by doing, students will implement and extend state-of-the-art generative methods drawn from recent top-tier research. The course emphasizes both conceptual understanding and practical implementation, with a strong focus on visual computing, representation learning, and generation.

General Course Structure

Students will work in small teams on a semester-long research project. Each team is supervised by PhD students actively working in the field of generative AI. Projects are typically inspired by recent research papers and involve reproducing, analyzing, and extending published methods.

The course structure includes:

• Mandatory weekly meetings to report progress and discuss challenges
• Dedicated communication channels for technical and conceptual support
• Independent literature study and experimental work
• Final project presentation
• Submission of a technical report documenting methodology, experiments, and results

Implementation will primarily use Python and PyTorch.

Project Topics

Possible project topics include (but are not limited to):

• Diffusion Models for Image and Video Generation (e.g., DDPMs, Latent Diffusion, Stable Diffusion-style architectures)
• Conditional and Controllable Generation (e.g., text-to-image/video, layout- or geometry-conditioned diffusion)
• Generative Video Models (e.g., temporal diffusion, video transformers)
• World Models and Simulation Learning (e.g., Dreamer-style models, latent dynamics models)
• Generative Models for 3D Content (e.g., diffusion for NeRFs, 3D Gaussian Splatting, meshes, point clouds)
• Multimodal Generative Models (e.g., vision–language models, audio-visual generation)
• Scene Generation Methods (e.g., Text2Room, WorldExplorer)
• Face Generation, Reconstruction and Animation (e.g., FlexAvatar, Wan-Animate, NPGA, FaceAnything)
• Feed-Forward Reconstruction (e.g., VGGT, Depth Anything 3)

Objective

The goal of this practical course is to enable students to:

• Understand the theoretical foundations of modern generative models
• Implement and experiment with state-of-the-art diffusion and world models
• Critically read and analyze current research papers
• Develop research-oriented skills relevant for Master's thesis and PhD-level work
• Gain hands-on experience with large-scale deep learning systems

The course is particularly suited for students interested in research careers in AI, Machine Learning, Computer Vision, or Robotics.

Prerequisites

Previous knowledge expected:

• IN2346 - Introduction to Deep Learning
• Good knowledge of 3D computer vision is required; prior completion of related courses is strongly recommended.
• Solid background in Machine Learning and Linear Algebra
• Proficient programming skills in Python
• Prior experience with PyTorch (required)

Course Criteria and Registration

• For registration you have to be identified in TUMonline as a student.
• Registration via TUM's matching system: https://matching.in.tum.de/
• Additionally, fill out the application form by 15.07.2026. It is mandatory to fill out the form for acceptance: https://forms.gle/MkdQicwcveqEyA3R8

Contact Us

If you have any questions regarding the organization of the course, do not hesitate to contact us. For content-related questions, we will use the dedicated communication channels announced at the start of the course.