TCOM-5970/ECE-5973: Deep Learning

In a wide sense, deep learning is a study of machine learning models composed of multiple layers of representation (hence deep). In the last decade, there has been significant progress of using artificial neural networks for deep models. This leads to numerous breakthroughs on image, video, and natural language processing applications.

This course is aimed to be quite hands-on and should provide students with sufficient details for them to quickly apply to their own research. In particular, applications relating to computer vision and communications will be discussed. There may be some math but we probably won't spend much time going into proofs. Instead, we may try to go through (not exhasively) some of the free libraries such as Caffe and Torch. And you are definitely encouraged to explore and leverage them for your course project.

N.B. Because we will “borrow” programming assignment from Stanford 231n, Python prerequiste is needed.

Textbook

  • Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press.

It is not required but is a very good reference.

Reference

Some Deep Learning Toolboxes and Libraries

  • Tensorflow: From Google, probably most popular package. Not quite optimized for single PC

  • Caffe2: From Facebook

  • Caffe: From Berkeley

  • Torch 7: From NYU, and used by Facebook/Twitter

  • Theano: From Bengio's group in Montreal

  • Keras: High-level layer on top of Theano/Tensorflow

  • Lasagne: High-level layer on top of Theano

  • matconvnet: From Oxford, kind of restricted

  • mxnet: From Amazon

  • Neon: From Intel

  • Deeplearning4j

Office Hours

There are no “regular” office hours. And you are welcome to come catch me anytime or contact me through emails.

Course Syllabus (Tentative)

  • Overview of machine learning

  • History of artificial neural networks

  • Perceptrons

  • Backpropagation algorithms

  • Regularization and dropout

  • Weight initialization

  • Optimization methods

  • Convolutional neural networks (CNN)

  • R-CNN, faster R-CNN

  • Weight visualization, Deep visualization, t-SNE, deepdream

  • Recurrent neural networks

  • LSTM networks

  • Restricted boltzmann machines

  • Autoencoders

  • Deep belief networks

Projects

Your projects will be presented in class on May 5. You may include an additional written report if you would like to further clarify anything.

Grading

"Activities": 30%. Quizzes, paper review, presentations, etc.

Homework: 30%. Programming assignments.

Final Project: 40%.

Calendar

Topics Materials
1/20 Overview, AI, machine learning and its types, artificial neural networks and its history slides, Example: image classification
1/27 Machine learning basics, linear regression, regularization, ridge regression, lasso, logistic regression, softmax classifier, loss function, optimization basics slides, linear regression notebook
2/2 Back-propagation, activation functions, weight initialization, dropout, batch normalization, more on optimization tricks, debugging optimizers, hyperparameter optimization slides, optimizer comparison notebook
2/9 Convolutional neural networks (CNNs) slides
2/16 Object localization and detection, CNNs and arts, fooling CNNs slides
2/23 Recurrent neural networks (RNNs), long short-term memory (LSTM) networks slides
3/3 Echo state networks (ESNs), image captioning slides, project explanation, pyESN: Python implementation of ESN, neuraltalk2
3/10 NLP tasks, name entity recognition, recursive neural networks slides
3/23 Seq2seq models, neural machine translations, chatbots slides
3/30 Memory networks slides
4/7 Neural Turing machines slides
4/14 Restricted Boltzmann machines, deep belief networks slides, a practical guide of training RBM
4/21 Autoencoders, generative adversarial networks slides
4/28 Deep reinforcement learning slides, epilogue, AlphaGo Zero