ECE 5973-961/983: Artificial Neural Networks and Applications

Artificial neural networks was introduced in the 50’s of the last century. However, in the last decade, there has been strong resurgence of neural networks as processing techniques where they have been applied to many real-world problems. This leads to numerous breakthroughs on image, video, and natural language processing applications.

This course is aimed to be quite hands-on and should provide students with sufficient details for them to quickly apply to their own research. In particular, applications relating to computer vision and natural language processing will be discussed. There may be some math but we will not spend too much time going into proofs. Instead, we may try to go through (not exhaustively) some of the free libraries such as Caffe and Torch. And you are definitely encouraged to explore and leverage them for your course project.


  • Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press.

It is not required but is a very good reference.


Some Deep Learning Toolboxes and Libraries

  • Tensorflow: From Google, probably most popular package. Not quite optimized for single PC

  • Caffe2: From Facebook

  • Caffe: From Berkeley

  • Torch 7: From NYU, and used by Facebook/Twitter

  • PyTorch: The Python version of Torch

  • Theano: From Bengio's group in Montreal

  • Keras: High-level layer on top of Theano/Tensorflow

  • Lasagne: High-level layer on top of Theano

  • matconvnet: From Oxford, kind of restricted

  • mxnet: From Amazon

  • Neon: From Intel

  • Deeplearning4j

Office Hours

There are no “regular” office hours. And you are welcome to come catch me anytime or contact me through emails.

Course Syllabus (Tentative)

  • Overview of machine learning

  • History of artificial neural networks

  • Perceptrons

  • Backpropagation algorithms

  • Regularization and dropout

  • Weight initialization

  • Optimization methods

  • Convolutional neural networks (CNN)

  • R-CNN, faster R-CNN

  • Weight visualization, Deep visualization, t-SNE, deepdream

  • Recurrent neural networks

  • LSTM networks

  • Restricted boltzmann machines

  • Autoencoders

  • Deep belief networks


Video presentation due on May 9. Please read this for guideline. Written report is not mandatory but worth a maximum 20% extra credit.


Homework: 30%. Written and programming assignments

Presentation: 20%. Including presentation abstract and peer reviews.

Midterm: 20%.

Final Project: 30%. Including project proposal and progress report.

In-class participation and quizzes (extra credits): 10%.

Final grade:

  • A: above 90%

  • B: above 80% but not more than 90%

  • C: above 70% but not more than 80%

  • D: above 60% but not more than 70%

  • F: not more than 60%


Calculus (MATH 1914 or equivalent), linear algebra (MATH 3333 or equivalent), basic probability (MATH 4733 or equivalent), and intermediate programming skill (experience on Python/Numpy is preferred)

Note that we will “borrow” programming assignment from Stanford 231n. So ability to program in Python is expected. Python is not difficult if you are familiar with any other high level general programming languages such as C/C++/C#/Java/Javascript/Perl/Matlab etc. If you don't know anything about Python, I would recommend you to try out this app.

Late Policy

  • There will be 5% deduction per each late day for all submissions

  • The deduction will be saturated after 10 days. So you will get half of your marks even if you are super late

Reasonable Accommodation Policy

Any student in this course who has a disability that may prevent the full demonstration of his or her abilities should contact me personally as soon as possible so we can discuss accommodations necessary to ensure full participation and facilitate your educational opportunities.

Should you need modifications or adjustments to your course requirements because of documented pregnancy-related or childbirth-related issues, please contact me as soon as possible to discuss. Generally, modifications will be made where medically necessary and similar in scope to accommodations based on temporary disability. Please see this for commonly asked questions.

Title IX Resources

For any concerns regarding gender-based discrimination, sexual harassment, sexual misconduct, stalking, or intimate partner violence, the University offers a variety of resources, including advocates on-call 24.7, counseling services, mutual no contact orders, scheduling adjustments and disciplinary sanctions against the perpetrator. Please contact the Sexual Misconduct Office 405-325-2215 (8-5, M-F) or OU Advocates 405-615-0013 (24.7) to learn more or to report an incident.


Topics Materials Further reading/watching
1/18 Course overiew (video) overview Andrew Ng: Artificial Intelligence is the New Electricity
1/20 Machine learning basics, linear regression (video)
1/25 Ridge regression, Lasso, logistic regression, cross-entropy, softmax regression classifier (video) classification and regressionregression notebook, Low-Rank Simplicity Bias in Deep Networks, Predictive Model Degrees of Freedom in Linear Regression
1/27 SVM (video) neural networks SVM explained
2/01 Back-propagation (video)
2/03 Activation functions, Weight initialization (video) Activation functions explained, Activation function design for deep networks: linearity and effective initialisation, Data-driven Weight Initialization with Sylvester Solvers, Neuron Campaign for Initialization Guided by InformationBottleneck Theory, AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks
2/08 dropout, first-order optimizers, BFGS, LBFGS (video) optimizer comparison notebook, dropout paper, batch normalization paper, Normalization techniques in training DNNs, Neural Tangent Kernel:Convergence and Generalization in Neural Networks
2/10 PyTorch tutorial (video) jupyter notebook PRIME: A Few Primitives Can Boost Robustness to Common Corruptions
2/15 Convolutional neural networks (video) cnn neocognition
2/17 Advice on Training NN models (videos)
2/22 CNN architectures (video)
2/24 CNN tricks (video)
3/01 PyTorch Lightning (video) jupyter notebook
3/03 Presentation
3/08 Presentation
3/10 Presentation
3/15 spring break
3/17 spring break
3/22 computer vision applications with CNNs (video) CNN applications
3/24 t-SNE, visualizing CNN layers, CNNs and arts (video) Visualizing CNN
3/29 Self-supervised learning (video) Self-supervised learning
3/31 Noisy student, recurrent neural networks (RNNs), long short-term memory (LSTM) (video) RNN
4/05 Introdution to reinforcement learning (video) deep reinforcement learning, AlphaGo Zero slides, demo code
4/07 Policy-gradient, actor-critic (video)
4/12 DDPG, TD3, Soft actor-critic (video)
4/14 Graph neural networks (video)
4/19 Spectral and spatial GCN (video)
4/21 PyG and Capsule networks (video) PyG tutorial
4/26 GLOM and EM routing for capsules (video)
4/28 Autoencoder, generative adversarial network, Attention and transformer (video) Attention and transformers attention is all you need