Deep Learning and Related Methods for Large Dataset Information Processing

Deep learning in artificial neural networks is a collection of statistical methods that benefit from large datasets and parallel computing. Recently it led to remarkable progress in many domains of research. This course provides an introduction to the subject, including the latest research. The structure of the course is chosen with the aim to be useful to students with very different academic backgrounds.

Topics include: Optimization: backpropagation, stochastic gradient descent and its accelerated versions, second-order optimization methods. Supervised and semi-supervised machine learning: under-fitting and over-fitting, regularization, cross-validation, data augmentation. Neural network architecture: activation functions and their properties, layer patterns. Training neural networks: data preprocessing, weight initialization, gradient flow, batch normalization, regularization, practical aspects of GPU computing and distributed training. Hyper-parameter optimization, model ensembles, model compression. Transfer learning and fine-tuning. Spatial data modeling: convolutional networks, visualizing their internal data representations, susceptibility to adversarial examples. Sequence data modeling: recurrent networks, LSTMs, GRUs, and their convolutional alternatives, attention. Recursive data modeling: recursive neural networks. Natural language processing: word embedding and its visualization, neural machine translation, speech recognition and synthesis. Capsule networks. Unsupervised machine learning: variational autoencoders, adversarial networks, graphical models. Reinforcement learning: Q-learning, policy gradient methods and actor-critic methods, trust region policy optimization. Evolutionary strategies. Use of neural networks for designing and training other neural networks: neural architecture search, meta-learning. Hybrid computing combining advantages of neural networks and conventional computers. Use of deep learning for causal inference and counterfactual predictions. Privacy and ethical issues related to artificial intelligence.

Selected applications: econometric estimation of causal effects, solutions to game-theoretic models, economic time-series modeling, sentiment analysis, patient health outcome prediction, low-cost disease diagnosis, overcoming sensory loss with deep-learning technologies.

The course will include a first introduction to Python and to deep learning frameworks PyTorch, TensorFlow and Keras. The precise selection of topics for the course will be adjusted based on the students' interests.