The accompanying material for each lecture is posted here.
Origins of deep learning, course goals, overview of machine-learning paradigms, intro to computational acceleration.
Supervised learning problem statement, data sets, hypothesis classes, loss functions, basic examples of supervised machine learning models, adding non-linear...
Linear and multilayer Perceptron, loss functions, activation functions, pooling, weight sharing, convolutional layers, gradient descent, backpropagation.
Approximation, estimation and optimization errors, regularization, loss surface curvature, descent-based optimization methods, second-order methods.
RNN model, input-output sequences relationships, non-sequential input, layered RNN, backpropagation through time, word embeddings.
Subspace models, autoencoders, unsupervised loss, generative adversarial nets, domain adaptation.
Markov decision process, policies, rewards, value functions, the Bellman equation, q-learning, policy learning, actor-critic learning, AutoML.
Parallelism and multithreading, execution of DL tasks, CPU vs. GPU architectures, CUDA programming model.
NVidia GPU architectures, memory hierarchy, CUDA threads, unified memory, optimizations for CNNs, hardware architectures for training.
Challenges of inference, low-bit representations, pruning, GPU vs FPGA and ASIC, TPU architecture.
Toeplitz operators, graphs, fields, gradients, divergence, Laplace-Beltrami operator, non-euclidean convolution, spectral and spatial CNN for graphs.
SVD for linear layers, flattened and grouped convolutions, pruning and retraining, network weights quantization.