マルチモーダル知能構成論

Multimodal Intelligent System Design

We will cover a wide range of state-of-the-art artificial intelligence, especially multimodal intelligent systems based on cutting-edge deep learning techniques. We will discuss fundamental techniques to handle various modalities such as vision, language, audio, action, etc., as well as how to integrate them to realize multimodal foundation models and agentic systems.

MIMA Search

授業計画

(1) Oct. 7 Introduction, course design, environment setup (2) Oct. 14 Recap of the basics of deep learning (MLP, CNN, RNN, attention) (3) Oct. 21 Transformer, ViT, Mamba (4) Oct. 28 　Language model: history, principles (5) Nov. 4 Language model: pre-training (6) Nov. 11 Language model: post-training (7) Nov. 18　Language model: test-time scaling, RAG, agent (8) Dec. 9　 Other generative models (mainly for Vision) (9) Dec. 16 Cross-modal embedding (CLIP, ImageBind) (10) Dec. 23 Vision-Language Models, Multimodal LLMs (11) Jan. 6 Information theory for Multimodal Learning, VLA (12) Jan. 13 NVIDIA: Physical AI hands-on (13) Jan. 20 NVIDIA: talk session

授業の方法

Onsite lectures and hands-on. We plan to provide computing resources using UTokyo Azure.

成績評価方法

1. Attendance (10%) Will be checked via UTOL (from the next lecture) 2. Hands-on Assignments (60%) 6~7 times, Fill-in-the blanks coding tasks 3. Final Project (30%) Open-ended project: implement a multimodal AI to solve your own problems

履修上の注意

This course is advanced one intended for graduate students specializing in AI-related fields. We assume that course participants already have basic knowledge of machine learning or deep learning and experience in their implementation using Python and other languages.