Deep Learning

  • Home/
  • Deep Learning

CLASS HOURS

Monday:? 5:30 pm  7:00 pm,? Wednesday: 7:15 pm  8:45 pm
Location LT:4, 6th Floor.

OFFICE HOURS AND CONTACT INFO.

Instructor: Dr. Ali Ahmed
Office Hours: Tues, Thurs (2:00pm – 4:00pm)
Email:?ali.ahmed@itu.edu.pk

Teaching Assistant: Mubashir ul Islam
Office Hours: Thursday, Friday (3:30pm – 5:30pm)
Email:?msee19023@itu.edu.pk

COURSE BASICS

Core Course
Credit Hours: 3
Batches: MSDS, MSCS, MSEE @ ITU

Five Assignemnts

PREREQUISITE

It is recommended students have a strong mathematical background (Linear algebra, calculus especially taking partial derivatives, and probabilities & statistics) and atleast an introductory course in Machine Learning. Strong programming skills (specifically python) are necessary to complete the assignments.

COURSE OVERVIEW

Deep learning is a sub-field of machine learning that focuses on learning complex, hierarchical feature representations from raw data. The dominant method for achieving this, artificial neural networks, has revolutionized the processing of data (e.g., images, videos, text, and audio) as well as decision-making tasks (e.g., game-playing). Its success has enabled a tremendous amount of practical commercial applications and has had significant impact on society. Students will gain foundational knowledge of deep learning algorithms and get practical experience in building neural networks in TensorFlow.

COURSE OBJECTIVES

In this course, we’ll examine the history of neural networks and state-of-the-art approaches to deep learning. Students will learn to design neural network architectures and training procedures via hands-on assignments. Students will read current research articles to appreciate state-of-the-art approaches. Our main focus will be on introducing major deep learning algorithms, the problem settings, and their applications to solve real-world problems.

GRADING POLICY

  • 45% Assignments
  • 5% Class participation and Creating Notes
  • 20% Final Project
  • 10% Quizzes
  • 10% Midterm Exam
  • 10% Final Exam

HONOR CODE

All cases of academic misconduct will be forwarded to the disciplinary committee. All assignments are group-based unless explicitly specified by the instructor.

COURSE OUTLINE

Topics

Introduction to Deep Learning

  1. Forward propagation, Backpropagation
  2. Implementing Neural Network from Scratch in Python

Introduction to PyTorch

  1. The Basics of PyTorch
  2. Tensors
  3. Dynamic Computation Graph and Backpropagation
  4. Loss modules

Activation Functions

  1. Common activation function
  2. Visualizing activation functions

Optimization and Initialization

  1. Preparation
  2. Initialization
  3. Optimization

Convolutional Neural Network with PyTorch

  1. Hidden Layer Feedforward Neural Network
  2. Pooling Layers
  3. Paddings
Inception, ResNet and DenseNet

Transformers and Multi-Head Attention

  1. Transformers Architecture
  2. Experiments

Graph Neural Network

  1. PyTorch Geometric
  2. Experiments on Graph Structure

Deep Energy-Based Generative Models

  1. Energy Models
  2. Image Generation

Deep Autoencoders

  1. Building the Autoencoder
  2. Finding Visual Similar Images

Adversarial Attacks

  1. Deep CNNs on ImageNet
  2. White-Box Adversarial Attacks

COURSE NOTES

? Topics Notes / Reading Material / Comments
27-09-2021 Introduction to Deep Learning
  • Notes
29-09-2021

The Perceptron

(The structural building block of deep learning)
  • The Perceptron:
    • Forward Propagation, Activation Function
  • Common Activation Functions
    • Sigmoid, Hyperbolic Tangent
  • Importance of Activation Functions
04-10-2021 Building Neural Networks with Perceptron
  • The Perceptron:
    • Simplified,Multi Output Perceptron
  • Single Layer Neural Network
  • Deep Neural Network
  • Applying Neural Networks
06-10-2021 Training Neural Networks
  • Loss Optimization:
    • Empirical Loss, Binary Cross Entropy Loss, Mean Squared Error Loss
  • Gradient Decent Algorithm
  • Computing Gradients:
    • Backpropagation
11-10-2021 Neural Networks in Practice:
  • Optimization:
    • Loss Functions Can Be Difficult to Optimize,Setting the Learning Rate,Adaptive Learning Rate Algorithms.
  • Mini-batches:Mini-batches while training,Gradient Descent,Stochastic Gradient Descent.
  • Overfitting
    • The Problem of Overfitting,Regularization,Regularization 1: Dropout,Regularization 1: Early Stopping
18-10-2021 Neural Networks in Practice:
  • Notes
20-10-2021 Backpropogation?
  • Notes: FirstNeuralNet.pdf
25-10-2021 Backpropogation (Continued)
  • Notes: FirstNeuralNet.pdf
01-11-2021 PyTorch Tutorial

The Basics of PyTorch:

  • Tensors,
  • Backpropagation,
  • GPU support

Learning by Example:

  • The model
  • The data:
    • The dataset class,
    • The data loader class
  • Optimization:
    • Loss modules
    • Stochastics Gradient Decent
  • Training

Evaluation
03-11-2021 Convolutional Neural Network
  • Hyper parameters in CNN
    • Number of layers
    • Size of features
    • Pooling window size
    • Stride
    • Number of neurons in fully connected layers
  • 1?1 Convolution
    • Feature Fusion,
    • Dimensionality reduction or bottleneck layer.
  • Notes: ConvolutionalNeuralNetwork.pdf
08-11-2021 Activation Functions implementation in PyTorch
  • Common activation functions:
    • Sigmoid
    • Tanh
    • ReLU
    • LeakyReLU
    • ELU
    • Swish

    Analyzing the effect of activation functions:

    • Visualizing the gradient flow after initialization
    • Visualizing the activation distribution
    • Finding dead neurons in ReLU networks
10-11-2021 Activation Functions implementation in PyTorch (Continued)
  • Notes
15-11-2021

Dimensionality Reduction

Mutual Information

Feature Extraction
  • Notes
    • Curse of Dimensionality
17-11-2021 Principal Component Analysis (Derivation)
  • Notes

Assigned Reading:

https://medium.com/analytics-vidhya/understanding-principle-component-analysis-pca-step-by-step-e7a4bb4031d9?

https://builtin.com/data-science/step-step-explanation-principal-component-analysis
22-11-2021 Midterm
  • Notes
24-11-2021 Case Study: CNN Alex Net, VGG, Google Net and ResNet
  • Notes

Assigned Reading:

https://uvadlc.github.io/lectures/nov2016/lecture4.pdf
01-12-2021 Case Study: CNN Alex Net, VGG, Google Net and ResNet (Continued)
  • Notes
22-12-2021 Recurrent Neural Network (RNN)
  • RNN:

Computational Graph

  • Many to Many
  • Many to One
  • One to Many

Backpropagation through time

RNN tradeoffs

Assigned Reading:

http://cs231n.stanford.edu/slides/2020/lecture_10.pdf
10-01-2022 Long Short-Term Memory (LSTM)
  • Notes

Assigned Reading:

http://cs231n.stanford.edu/slides/2020/lecture_10.pdf
11-01-2022 Attention and Transformers
  • Notes

Assigned Reading:

http://cs231n.stanford.edu/slides/2020/lecture_10.pdf
12-01-2022 Attention and Transformers (Continued)
  • Notes
17-01-2022 Image Captioning with Attention
  • Notes

Assigned Reading:

http://cs231n.stanford.edu/slides/2020/lecture_10.pdf
19-01-2022 Implementing RNN in PyTorch from scratch
  • Assigned Reading:

https://cs231n.github.io/assignments2016/assignment3/
24-01-2022 Pixel CNN, Pixel RNN

Assigned Readings:

https://arxiv.org/abs/1606.05328?

https://towardsdatascience.com/auto-regressive-generative-models-pixelrnn-pixelcnn-32d192911173

https://arxiv.org/abs/1601.06759
26-01-2022

Pixel CNN, Autoencoders,

Variational Autoencoders

Assigned Readings:

https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture19.pdf
31-01-2022 Variational Autoencoders

Assigned Reading:

https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture20.pdf
02-02-2022 Implementation of Autoencoders and Variational Autoencoders in PyTorch

Assigned Readings:

https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial9/AE_CIFAR10.html
07-02-2022

Generative Adversarial Network

Generative vs Self-supervised Learning

Self-supervised Learning

Assigned Reading:

http://cs231n.stanford.edu/slides/2019/cs231n_2019_lecture13.pdf

?

TEXT BOOK

  • Text Book: Deep Learning by Ian Goodfellow?Link
  • Course notes will be posted on google classroom

Top Conferences to Follow

  • International Conference on Machine Learning (ICML)
  • Conference on Neural Information Processing Systems (NIPS)
  • International Joint Conference on Artificial Intelligence (ICAI)
  • Conference on Computer Vision and Pattern Recognition (CVPR)
  • International Conference on Computer Vision (ICCV)
  • British Machine Vision Conference (BMVC)

ASSIGNMENTS

  • ASSIGNMENT 1:

    • Implement Simple Neural Network with NumPy. One question assignment, required to implement neural network from scratch.

  • ASSIGNMENT 2:?

    • Driving expression for the backpropagation and implementing Neural Network for the regression task.

  • ASSIGNMENT 3:?

    • Classification of MNIST Digits using PyTorch. In this assignment the students will get hands on experience in building models in pytorch.

  • ASSIGNMENT 4:

    • Finding dead neuron in relu and mutual information between random variables in python.

  • ASSIGNMENT 5

    • Train Transformer on custom data and understanding the implementation of Recurrent Neural Network
  • ?

  • ASSIGNMENT 6

    • Image Captioning with Vanilla RNNs. In this assignment, you will implement a vanilla recurrent neural network and use it to train a model that can generate novel captions for images.