ECE 5973-981: Artificial Neural Networks and Applications

A bit about this course

Artificial neural networks was introduced in the 50’s of the last century. However, in the last decade, there has been strong resurgence of neural networks as processing techniques where they have been applied to many real-world problems. This leads to numerous breakthroughs on image, video, and natural language processing applications.

This course is aimed to be quite hands-on and should provide students with sufficient details for them to quickly apply to their own research. In particular, applications relating to computer vision and natural language processing will be discussed. There may be some math but we will not spend too much time going into proofs. Instead, we may try to go through (not exhaustively) some of the free libraries (mostly PyTorch). And you are definitely encouraged to explore and leverage them for your course project.

Textbook

• Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press.

It is not required but is a very good reference.

Disscusion forum

Please sign up Discord through the link on Canvas. Please raise any questions, comments, or concerns there. You may contact me privately through Discord as well.

Reference

• Bengio's Textbook

• Lecun's NYU course

• Stanford CS231n

• Stanford CS224d

• Stanford CS20si

• Toronto CSC321

• Hinton's Coursera Course

• Fast AI course

• Oxford Course

• Berkeley CS 294

• MIT 6.S191

• MIT 6.S094

• Pythonprogramming.net Course

• Neural Network Course by Larochelle

• Princeton COS 598

• Video Recordings of the ICML’15 Deep Learning Workshop

• A good collection of papers

• Representation learning: A review and new perspectives

• Deep learning: a critical appraisal

• AI Playbook

• Machine learning for humans

• An AI reading list

Some Deep Learning Toolboxes and Libraries

• Tensorflow: From Google, probably most popular package. Not quite optimized for single PC

• JAX: Another experimental framework from Google

• Caffe2: From Facebook

• Caffe: From Berkeley

• Torch 7: From NYU, and used by Facebook/Twitter

• PyTorch: The Python version of Torch

• Theano: From Bengio's group in Montreal

• Keras: High-level layer on top of Theano/Tensorflow

• Lasagne: High-level layer on top of Theano

• matconvnet: From Oxford, kind of restricted

• mxnet: From Amazon

• Neon: From Intel

• Deeplearning4j

Office Hours

There are no “regular” office hours. And you are welcome to come catch me anytime or contact me through Discord. I am quite responsive and usually reply you within a day.

Course Syllabus (Tentative)

• Overview of machine learning

• History of artificial neural networks

• Perceptrons

• Backpropagation algorithms

• Regularization and dropout

• Weight initialization

• Optimization methods

• Convolutional neural networks (CNN)

• R-CNN, faster R-CNN

• Weight visualization, Deep visualization, t-SNE, deepdream

• Self-supervised learning

• Recurrent neural networks

• LSTM networks

• Autoencoders

• Deep belief networks

• Transformers

• Capsule networks

• Graph neural networks

Projects

Written report is due on May 11. Please read this for guideline. Video presnetation is worth a maximum 10% extra credit.

Grading

Homework: 30%. Written and programming assignments

Presentation: 20%. Including presentation abstract and peer reviews.

Literature survey: 10%.

Final Project: 40%. Including project proposal and progress report.

In-class participation and quizzes (extra credits): 10%.

Final grade:

• A: above 90%

• B: above 80% but not more than 90%

• C: above 70% but not more than 80%

• D: above 60% but not more than 70%

• F: not more than 60%

Prerequiste

Calculus (MATH 1914 or equivalent), linear algebra (MATH 3333 or equivalent), basic probability (MATH 4733 or equivalent), and intermediate programming skill (experience on Python/Numpy is preferred)

Note that the ability to program in Python is expected. Python is not difficult if you are familiar with any other high level general programming languages such as C/C++/C#/Java/Javascript/Perl/Matlab etc. If you don't know anything about Python, I would recommend you to try out this app.

Late Policy

• There will be 5% deduction per each late day for all submissions

• The deduction will be saturated after 10 days. So you will get half of your marks even if you are super late

Calendar

Acknowledgement

This course is partially supported by NSF NAIRR Pilot

University Policies

Mental Health Support Services

Support is available for any student experiencing mental health issues that are impacting their academic success. Students can either be seen at the University Counseling Center (UCC) located on the second floor of Goddard Health Center or receive 247365 crisis support from a licensed mental health provider through TELUS Health. To schedule an appointment or receive more information about mental health resources at OU, please call the UCC at 405-325-2911 or visit University Counseling Center. The UCC is located at 620 Elm Ave., Room 201, Norman, OK 73019.

Title IX Resources and Reporting Requirement

The University of Oklahoma faculty are committed to creating a safe learning environment for all members of our community, free from gender and sex-based discrimination, including sexual harassment, domestic and dating violence, sexual assault, and stalking, in accordance with Title IX. There are resources available to those impacted, including: speaking with someone confidentially about your options, medical attention, counseling, reporting, academic support, and safety plans. If you have (or someone you know has) experienced any form of sex or gender-based discrimination or violence and wish to speak with someone confidentially, please contact OU Advocates (available 24/7 at 405-615-0013) or University Counseling Center (M-F 8 a.m. to 5 p.m. at 405-325-2911).

Because the University of Oklahoma is committed to the safety of you and other students, and because of our Title IX obligations, I, as well as other faculty, Graduate Assistants, and Teaching Assistants, are mandatory reporters. This means that we are obligated to report gender-based violence that has been disclosed to us to the Institutional Equity Office. This includes disclosures that occur in: class discussion, writing assignments, discussion boards, emails, and during Student/Office Hours. You may also choose to report directly to the Institutional Equity Office. After a report is filed, the Title IX Coordinator will reach out to provide resources, support, and information, and the reported information will remain private. For more information regarding the University’s Title IX Grievance procedures, reporting, or support measures, please visit Institutional Equity Office at 405-325-3546.

Reasonable Accommodation Policy

The University of Oklahoma (OU) is committed to the goal of achieving equal educational opportunity and full educational participation for students with disabilities. If you have already established reasonable accommodations with the Accessibility and Disability Resource Center (ADRC), please submit your semester accommodation request through the ADRC as soon as possible and contact me privately, so that we have adequate time to arrange your approved academic accommodations.

If you have not yet established services through ADRC, but have a documented disability and require accommodations, please complete ADRC’s pre-registration form to begin the registration process. ADRC facilitates the interactive process that establishes reasonable accommodations for students at OU. For more information on ADRC registration procedures, please review their Register with the ADRC web page. You may also contact them at (405) 325-3852 or adrc@ou.edu, or visit www.ou.edu/adrc for more information.

Note: Disabilities may include, but are not limited to, mental health, chronic health, physical, vision, hearing, learning, and attention disabilities, pregnancy-related conditions. ADRC can also support students experiencing temporary medical conditions.

Religious Observance

It is the policy of the University to excuse the absences of students that result from religious observances and to reschedule examinations and additional required classwork that may fall on religious holidays, without penalty.

See Faculty Handbook 3.15.2

Adjustments for Pregnancy and Related Issues

Should you need modifications or adjustments to your course requirements because of pregnancy or a pregnancy-related condition, please request modifications via the Institutional Equity Office website or call the Institutional Equity Office at 405/325-3546 as soon as possible. Also, see the Institutional Equity Office FAQ on Pregnant and Parenting Students’ Rights for answers to commonly asked questions.

Final Exam Preparation Period

Pre-finals week will be defined as the seven calendar days before the first day of finals. Faculty may cover new course material throughout this week. For specific provisions of the policy please refer to OU’s Final Exam Preparation Period policy.

Emergency Protocol

During an emergency, there are official university procedures that will maximize your safety.

Severe Weather: If you receive an OU Alert to seek refuge or hear a tornado siren that signals severe weather.

1. Look for severe weather refuge location maps located inside most OU buildings near the entrances.

2. Seek refuge inside a building. Do not leave one building to seek shelter in another building that you deem safer. If outside, get into the nearest building.

3. Go to the building’s severe weather refuge location. If you do not know where that is, go to the lowest level possible and seek refuge in an innermost room. Avoid outside doors and windows.

4. Get in, Get Down, Cover Up.

5. Wait for official notice to resume normal activities.

Additional Weather Safety Information is available through the Department of Campus Safety.

The University of Oklahoma Active Threat Guidance

The University of Oklahoma embraces a Run, Hide, Fight strategy for active threats on campus. This strategy is well known, widely accepted, and proven to save lives. To receive emergency campus alerts, be sure to update your contact information and preferences in the account settings section at one.ou.edu.

RUN: Running away from the threat is usually the best option. If it is safe to run, run as far away from the threat as possible. Call 911 when you are in a safe location and let them know from which OU campus you’re calling and location of active threat.

HIDE: If running is not practical, the next best option is to hide. Lock and barricade all doors; turn off all lights; turn down your phone’s volume; search for improvised weapons; hide behind solid objects and walls; and hide yourself completely and stay quiet. Remain in place until law enforcement arrives. Be patient and remain hidden.

FIGHT: If you are unable to run or hide, the last best option is to fight. Have one or more improvised weapons with you and be prepared to attack. Attack them when they are least expecting it and hit them where it hurts most: the face (specifically eyes, nose, and ears), the throat, the diaphragm (solar plexus), and the groin.

’'Please save OUPD’s contact information in your phone.’’

NORMAN campus: For non-emergencies call (405) 325-1717. For emergencies call (405) 325-1911 or dial 911.

TULSA campus: For non-emergencies call (918) 660-3900. For emergencies call (918) 660-3333 or dial 911.

Fire Alarm/General Emergency:

If you receive an OU Alert that there is danger inside or near the building, or the fire alarm inside the building activates:

1. LEAVE the building. Do not use the elevators.

2. KNOW at least two building exits.

3. ASSIST those that may need help.

4. PROCEED to the emergency assembly area.

5. ONCE safely outside, NOTIFY first responders of anyone that may still be inside the building due to mobility issues.

6. WAIT for official notice before attempting to re-enter the building.

OU Fire Safety on Campus