Course Info

Deep learning is widely used in many market segments ranging from mobile devices to supercomputers. Recently many software packages as well as special hardware accelerators were developed to support deep learning. The course will focus on algorithms, programming frameworks and new software/hardware interfaces that aim to allow execution of deep learning algorithms in a productive and efficient way.

Learning Outcomes

At the end of the course, the student will:

  1. Understand and be able to apply notions in deep learning.
  2. Know how to effectively use leading python machine-learning and deep learning frameworks such as PyTorch.
  3. Know how to optimize software and hardware performance in deep neural network applications.
  4. Know how to program GPUs using CUDA.
  5. Perform a small research project using the studied notions and techniques.

Administration

Evaluation: 40% Homework assignments, 60% final project.

Language: The course will be taught in English.

Credits: 3.0.

Course Staff

Lecturers

Prof. Alex Bronstein

Prof. Alex Bronstein

Lecturer

Prof. Avi Mendelson

Prof. Avi Mendelson

Lecturer

TAs

Chaim Baskin

Chaim Baskin

Head TA

Aviv Rosenberg

Aviv Rosenberg

Assignments TA

Literature

  • Deep Learning

    Ian Goodfellow, Yoshua Bengio, Aaron Courville

    MIT Press, 2016

  • Neural Networks and Deep Learning

    Charu C. Aggarwal

    Springer, 2018

  • Deep Learning with PyTorch

    Vishnu Subramanian

    Packt, 2018

Detailed Syllabus

Date # Lecture Tutorial
21/10/2018 1 Introduction to machine learning and accelerators Environment set up, intro to notebooks, using course servers
28/10/2018 2 Supervised learning: linear classifiers, linear regression, loss functions, optimization, descent methods, SGD ML basics: Linear Classifiers, score & loss functions, data sets, cross validation
04/11/2018 3 Neural networks: multi-layered perceptrons, backpropagation and intro to CNNs, ResNets MLP, Optimization algorithms
11/11/2018 4 Training neural networks: Initialization, dropout, batch normalization, update rules, data augmentation Training convolution neural networks in PyTorch
18/11/2018 5 Recurrent neural networks: RNN, B-RNN, LSTM, GRU and applications Convolutional neural networks
25/11/2018 5
6		 RNNs (continued);
Unsupervised Learning: Generative Models, VAEs, GANs		 RNNs, LSTM training for NLP application
09/12/2018 6 Unsupervised Learning (continued) Unsupervised learning, GANs, Variational Auto Encoders
16/12/2018 7 Reinforcement learning, Deep Q-Learning, Policy Gradients Domain adaptation, style transfer
23/12/2018 8 Intro to Parallel architectures for DL Deep reinforcement learning
30/12/2018 9 Parallel architectures for Training Introduction to CUDA
06/01/2019 10
11		 Parallel architectures for Inference;
CNNs on non-euclidean domains: graphs and manifolds		 CUDA with PyTorch
13/01/2019 11 CNNs on non-euclidean domains (continued) CNN on graphs
20/01/2019 12 Project Presentations Project Presentations