1 - Introduction to AI-Driven Digital Transformation

Last Semester’s Introductory Lecture with introduction to and Motivation for Big Data Applications and Analytics Class. See section G for material directly related to this lass but all sections are relevant

This Lecture is recorded in 8 parts and gives an introduction and motivation for the class. This and other lectures in class are divided into “bite-sized lessons” from 5 to 30 minutes in length; that’s why it has 8 parts.

Lecture explains what students might gain from the class even if they end up with different types of jobs from data engineering, software engineering, data science or a business (application) expert. It stresses that we are well into a transformation that impacts industry research and the way life is lived. This transformation is centered on using the digital way with clouds, edge computing and deep learning giving the implementation. This “AI-Driven Digital Transformation” is as transformational as the Industrial Revolution in the past. We note that deep learning dominates most innovative AI replacing several traditional machine learning methods.

The slides for this course can be found at E534-Fall2020-Introduction

A: Getting Started: BDAA Course Introduction Part A: Big Data Applications and Analytics

This lesson describes briefly the trends driving and consequent of the AI-Driven Digital Transformation. It discusses the organizational aspects of the class and notes the two driving trends are clouds and AI. Clouds are mature and a dominant presence. AI is still rapidly changing and we can expect further major changes. The edge (devices and associated local fog computing) has always been important but now more is being done there.

B: Technology Futures from Gartner’s Analysis: BDAA Course Introduction Part B: Big Data Applications and Analytics

This lesson goes through the technologies (AI Edge Cloud) from 2008-2020 that are driving the AI-Driven Digital Transformation. we use Hype Cycles and Priority Matrices from Gartner tracking importance concepts from the Innovation Trigger, Peak of Inflated Expectations through the Plateau of Productivity. We contrast clouds and AI.

  • This gives illustrations of sources of big data.
  • It gives key graphs of data sizes, images uploaded; computing, data, bandwidth trends;
  • Cloud-Edge architecture.
  • Intelligent machines and comparison of data from aircraft engine monitors compared to Twitter
  • Multicore revolution
  • Overall Global AI and Modeling Supercomputer GAIMSC
  • Moores Law compared to Deep Learning computing needs
  • Intel and NVIDIA status

E: Big Data and Science: BDAA Course Introduction Part E: Big Data Applications and Analytics

  • Applications and Analytics
  • Cyberinfrastructure, e-moreorlessanything.
  • LHC, Higgs Boson and accelerators.
  • Astronomy, SKA, multi-wavelength.
  • Polar Grid.
  • Genome Sequencing.
  • Examples, Long Tail of Science.
  • Wired’s End of Science; the 4 paradigms.
  • More data versus Better algorithms.

F: Big Data Systems: BDAA Course Introduction Part F: Big Data Applications and Analytics

  • Clouds, Service-oriented architectures, HPC High Performance Computing, Apace Software
  • DIKW process illustrated by Google maps
  • Raw data to Information/Knowledge/Wisdom/Decision Deluge from the EdgeInformation/Knowledge/Wisdom/Decision Deluge
  • Parallel Computing
  • Map Reduce

G: Industry Transformation: BDAA Course Introduction Part G: Big Data Applications and Analytics

AI grows in importance and industries transform with

  • Core Technologies related to
  • New “Industries” over the last 25 years
  • Traditional “Industries” Transformed; malls and other old industries transform
  • Good to be master of Cloud Computing and Deep Learning
  • AI-First Industries,

H: Jobs and Conclusions: BDAA Course Introduction Part H: Big Data Applications and Analytics

  • Job trends
  • Become digitally savvy so you can take advantage of the AI/Cloud/Edge revolution with different jobs
  • The qualitative idea of Big Data has turned into a quantitative realization as Cloud, Edge and Deep Learning
  • Clouds are here to stay and one should plan on exploiting them
  • Data Intensive studies in business and research continue to grow in importance

2 - AI-First Engineering Cybertraining Spring 2021

Updated On an ongoing Basis

Week 1

Lecture

Our first meeting is 01:10P-02:25P on Tuesday

The zoom will be https://iu.zoom.us/my/gc.fox

We will discuss how to interact with us. We can adjust the course somewhat.

Also as lectures are/will be put on YouTube, we will go to one lecture per week – we will choose day

The Syllabus has a general course description

Please communicate initially by email gcf@iu.edu

This first class discussed structure of class and agreed to have a section on deep learning technology.

We gave Introductory Lecture

Assignments

Week 2

Introduction

We gave an introductory lecture to optimization and deep learning. Unfortunately we didn’t record the zoom serssion but we did make an offline recording with slides IntroDLOpt: Introduction to Deep Learning and Optimization and YouTube

Google Colab

We also went through material on using Google Colab with examples. This is a lecture plus four Python notebooks

with recorded video

First DL: Deep Learning MNIST Example Spring 2021

We now have recorded all the introductory deep learning material

with recorded videos

IntroDLOpt: Introduction to Deep Learning and Optimization

  • Video: IntroDLOpt: Introduction to Deep Learning and Optimization

Opt: Overview of Optimization Spring2021

  • Video: Opt: Overview of Optimization Spring2021

DLBasic: Deep Learning - Some examples Spring 2021

  • Video: DLBasic: Deep Learning - Some examples Spring 2021

DLBasic: Components of Deep Learning Systems

  • Video: DLBasic: Components of Deep Learning Systems

DLBasic: Summary of Types of Deep Learning Systems

  • Video: DLBasic: Summary of Types of Deep Learning Systems

Week 3

Deep Learning Examples, 1

We discussed deep learning examples covering first half of slides DLBasic: Deep Learning - Some examples with recorded video

Week 4

Deep Learning Examples, 2 plus Components

We concluded deep learning examples and covered components with slides Deep Learning: More Examples and Components with recorded video

Week 5

Deep Learning Networks plus Overview of Optimization

We covered two topics in this weeks video

with recorded video

Week 6

Deep Learning and AI Examples in Health and Medicine

We went about 2/3rds of way through presentation AI First Scenarios: Health and Medicine

with recorded video

Week 7

Deep Learning and AI Examples

with recorded video

Week 8

Deep Learning and AI Examples

with recorded video

Week 9

Deep Learning and AI Examples

with recorded video

Week 10

GitHub for the Class project

  • We explain how to use GitHub for the class project. A video is available on YouTube. Please note that we only uploaded the relevant portion. The other half of the lecture went into individual comments for each student which we have not published. The comments are included in the GitHub repository.

Note project guidelines are given here

Video

Week 11

The Final Project

  • We described the gidelines of final projects in Slides
  • We were impressed by the seven student presentations describing their chosen project and approach.

Video

Week 12

Practical Issues in Deep Learning for Earthquakes

We used our research on Earthquake forecasting, to illustrate deep learning for Time Series with slides

Video

Week 13

Practical Issues in Deep Learning for Earthquakes

We continued discussion that illustrated deep learning for Time Series with the same slides as last week

Video

3 - Introduction to AI in Health and Medicine

This section discusses the health and medicine sector

Overview

This module discusses AI and the digital transformation for the Health and Medicine Area with a special emphasis on COVID-19 issues. We cover both the impact of COVID and some of the many activities that are addressing it. Parts B and C have an extensive general discussion of AI in Health and Medicine

The complete presentation is available at Google Slides while the videos are a YouTube playlist

Part A: Introduction

This lesson describes some overarching issues including the

  • Summary in terms of Hypecycles
  • Players in the digital health ecosystem and in particular role of Big Tech which has needed AI expertise and infrastructure from clouds to smart watches/phones
  • Views of Pataients and Doctors on New Technology
  • Role of clouds. This is essentially assumed throughout presentation but not stressed.
  • Importance of Security
  • Introduction to Internet of Medical Things; this area is discussed in more detail later in preserntation

slides

Part B: Diagnostics

This highlights some diagnostic appliocations of AI and the digital transformation. Part C also has some diagnostic coverage – especially particular applications

  • General use of AI in Diagnostics
  • Early progress in diagnostic imaging including Radiology and Opthalmology
  • AI In Clinical Decision Support
  • Digital Therapeutics is a recognized and growing activity area

slides

Part C: Examples

This lesson covers a broad range of AI uses in Health and Medicine

  • Flagging Issues requirng urgent attentation and more generally AI for Precision Merdicine
  • Oncology and cancer have made early progress as exploit AI for images. Avoiding mistakes and diagnosing curable cervical cancer in developing countries with less screening.
  • Predicting Gestational Diabetes
  • cardiovascular diagnostics and AI to interpret and guide Ultrasound measurements
  • Robot Nurses and robots to comfort patients
  • AI to guide cosmetic surgery measuring beauty
  • AI in analysis DNA in blood tests
  • AI For Stroke detection (large vessel occlusion)
  • AI monitoring of breathing to flag opioid-induced respiratory depression.
  • AI to relieve administration burden including voice to text for Doctor’s notes
  • AI in consumer genomics
  • Areas that are slow including genomics, Consumer Robotics, Augmented/Virtual Reality and Blockchain
  • AI analysis of information resources flags probleme earlier
  • Internet of Medical Things applications from watches to toothbrushes

slides

Part D: Impact of Covid-19

This covers some aspects of the impact of COVID -19 pandedmic starting in March 2020

  • The features of the first stimulus bill
  • Impact on Digital Health, Banking, Fintech, Commerce – bricks and mortar, e-commerce, groceries, credit cards, advertising, connectivity, tech industry, Ride Hailing and Delivery,
  • Impact on Restaurants, Airlines, Cruise lines, general travel, Food Delivery
  • Impact of working from home and videoconferencing
  • The economy and
  • The often positive trends for Tech industry

slides

Part E: Covid-19 and Recession

This is largely outdated as centered on start of pandemic induced recession. and we know what really happenmed now. Probably the pandemic accelerated the transformation of industry and the use of AI.

slides

Part F: Tackling Covid-19

This discusses some of AI and digital methods used to understand and reduce impact of COVID-19

  • Robots for remote patient examination
  • computerized tomography scan + AI to identify COVID-19
  • Early activities of Big Tech and COVID
  • Other early biotech activities with COVID-19
  • Remote-work technology: Hopin, Zoom, Run the World, FreeConferenceCall, Slack, GroWrk, Webex, Lifesize, Google Meet, Teams
  • Vaccines
  • Wearables and Monitoring, Remote patient monitoring
  • Telehealth, Telemedicine and Mobile Health

slides

Part G: Data and Computational Science and Covid-19

This lesson reviews some sophisticated high performance computing HPC and Big Data approaches to COVID

  • Rosetta volunteer computer to analyze proteins
  • COVID-19 High Performance Computing Consortium
  • AI based drug discovery by startup Insilico Medicine
  • Review of several research projects
  • Global Pervasive Computational Epidemiology for COVID-19 studies
  • Simulations of Virtual Tissues at Indiana University available on nanoHUB

slides

Part H: Screening Drug and Candidates

A major project involving Department of Energy Supercomputers

  • General Structure of Drug Discovery
  • DeepDriveMD Project using AI combined with molecular dynamics to accelerate discovery of drug properties

slides

Part I: Areas for Covid19 Study and Pandemics as Complex Systems

slides

  • Possible Projects in AI for Health and Medicine and especially COVID-19
  • Pandemics as a Complex System
  • AI and computational Futures for Complex Systems

4 - Mobility (Industry)

This section discusses the mobility in Industry

Overview

  1. Industry being transformed by a) Autonomy (AI) and b) Electric power
  2. Established Organizations can’t change
    • General Motors (employees: 225,000 in 2016 to around 180,000 in 2018) finds it hard to compete with Tesla (42000 employees)
    • Market value GM was half the market value of Tesla at the start of 2020 but is now just 11% October 2020
    • GM purchased Cruise to compete
    • Funding and then buying startups is an important “transformation” strategy
  3. Autonomy needs Sensors Computers Algorithms and Software
    • Also experience (training data)
    • Algorithms main bottleneck; others will automatically improve although lots of interesting work in new sensors, computers and software
    • Over the last 3 years, electrical power has gone from interesting to “bound to happen”; Tesla’s happy customers probably contribute to this
    • Batteries and Charging stations needed

Summary Slides

Full Slide Deck

Mobility Industry A: Introduction

  • Futures of Automobile Industry, Mobility, and Ride-Hailing
  • Self-cleaning cars
  • Medical Transportation
  • Society of Automotive Engineers, Levels 0-5
  • Gartner’s conservative View

Mobility Industry B: Self Driving AI

  • Image processing and Deep Learning
  • Examples of Self Driving cars
  • Road construction Industry
  • Role of Simulated data
  • Role of AI in autonomy
  • Fleet cars
  • 3 Leaders: Waymo, Cruise, NVIDIA

Mobility Industry C: General Motors View

  • Talk by Dave Brooks at GM, “AI for Automotive Engineering”
  • Zero crashes, zero emission, zero congestion
  • GM moving to electric autonomous vehicles

Mobility Industry D: Self Driving Snippets

  • Worries about and data on its Progress
  • Tesla’s specialized self-driving chip
  • Some tasks that are hard for AI
  • Scooters and Bikes

Mobility Industry E: Electrical Power

  • Rise in use of electrical power
  • Special opportunities in e-Trucks and time scale
  • Future of Trucks
  • Tesla market value
  • Drones and Robot deliveries; role of 5G
  • Robots in Logistics

5 - Space and Energy

This section discusses the space and energy.

Overview

  1. Energy sources and AI for powering Grids.
  2. Energy Solution from Bill Gates
  3. Space and AI

Full Slide Deck

A: Energy

  • Distributed Energy Resources as a grid of renewables with a hierarchical set of Local Distribution Areas
  • Electric Vehicles in Grid
  • Economics of microgrids
  • Investment into Clean Energy
  • Batteries
  • Fusion and Deep Learning for plasma stability
  • AI for Power Grid, Virtual Power Plant, Power Consumption Monitoring, Electricity Trading

Slides

B: Clean Energy startups from Bill Gates

  • 26 Startups in areas like long-duration storage, nuclear energy, carbon capture, batteries, fusion, and hydropower …
  • The slide deck gives links to 26 companies from their website and pitchbook which describes their startup status (#employees, funding)
  • It summarizes their products

Slides

C: Space

  • Space supports AI with communications, image data and global navigation
  • AI Supports space in AI-controlled remote manufacturing, imaging control, system control, dynamic spectrum use
  • Privatization of Space - SpaceX, Investment
  • 57,000 satellites through 2029

Slides

6 - AI In Banking

This section discusses AI in Banking

Overview

In this lecture, AI in Banking is discussed. Here we focus on the transition of legacy banks towards AI based banking, real world examples of AI in Banking, banking systems and banking as a service.

Slides

AI in Banking A: The Transition of legacy Banks

  1. Types of AI that is used
  2. Closing of physical branches
  3. Making the transition
  4. Growth in Fintech as legacy bank services decline

AI in Banking B: FinTech

  1. Fintech examples and investment
  2. Broad areas of finance/banking where Fintech operating

AI in Banking C: Neobanks

  1. Types and Examples of neobanks
  2. Customer uptake by world region
  3. Neobanking in Small and Medium Business segment
  4. Neobanking in real estate, mortgages
  5. South American Examples

AI in Banking D: The System

  1. The Front, Middle, Back Office
  2. Front Office: Chatbots
  3. Robo-advisors
  4. Middle Office: Fraud, Money laundering
  5. Fintech
  6. Payment Gateways (Back Office)
  7. Banking as a Service

AI in Banking E: Examples

  1. Credit cards
  2. The stock trading ecosystem
  3. Robots counting coins
  4. AI in Insurance: Chatbots, Customer Support
  5. Banking itself
  6. Handwriting recognition
  7. Detect leaks for insurance

AI in Banking F: As a Service

  1. Banking Services Stack
  2. Business Model
  3. Several Examples
  4. Metrics compared among examples
  5. Breadth, Depth, Reputation, Speed to Market, Scalability

7 - Cloud Computing

Cloud Computing

E534 Cloud Computing Unit

Full Slide Deck

Overall Summary

Video:

Defining Clouds I: Basic definition of cloud and two very simple examples of why virtualization is important

  1. How clouds are situated wrt HPC and supercomputers
  2. Why multicore chips are important
  3. Typical data center

Video:

Defining Clouds II: Service-oriented architectures: Software services as Message-linked computing capabilities

  1. The different aaS’s: Network, Infrastructure, Platform, Software
  2. The amazing services that Amazon AWS and Microsoft Azure have
  3. Initial Gartner comments on clouds (they are now the norm) and evolution of servers; serverless and microservices
  4. Gartner hypecycle and priority matrix on Infrastructure Strategies

Video:

Defining Clouds III: Cloud Market Share

  1. How important are they?
  2. How much money do they make?

Video:

Virtualization: Virtualization Technologies, Hypervisors and the different approaches

  1. KVM Xen, Docker and Openstack

Video:

  1. Clouds physically across the world
  2. Green computing
  3. Fraction of world’s computing ecosystem in clouds and associated sizes
  4. An analysis from Cisco of size of cloud computing

Video:

Cloud Infrastructure II: Gartner hypecycle and priority matrix on Compute Infrastructure

  1. Containers compared to virtual machines
  2. The emergence of artificial intelligence as a dominant force

Video:

Cloud Software: HPC-ABDS with over 350 software packages and how to use each of 21 layers

  1. Google’s software innovations
  2. MapReduce in pictures
  3. Cloud and HPC software stacks compared
  4. Components need to support cloud/distributed system programming

Video:

Cloud Applications I: Clouds in science where area called cyberinfrastructure; the science usage pattern from NIST

  1. Artificial Intelligence from Gartner

Video:

Cloud Applications II: Characterize Applications using NIST approach

  1. Internet of Things
  2. Different types of MapReduce

Video:

Parallel Computing Analogies: Parallel Computing in pictures

  1. Some useful analogies and principles

Video:

Real Parallel Computing: Single Program/Instruction Multiple Data SIMD SPMD

  1. Big Data and Simulations Compared
  2. What is hard to do?

Video:

Storage: Cloud data approaches

  1. Repositories, File Systems, Data lakes

Video:

HPC and Clouds: The Branscomb Pyramid

  1. Supercomputers versus clouds
  2. Science Computing Environments

Video:

Comparison of Data Analytics with Simulation: Structure of different applications for simulations and Big Data

  1. Software implications
  2. Languages

Video:

The Future I: The Future I: Gartner cloud computing hypecycle and priority matrix 2017 and 2019

  1. Hyperscale computing
  2. Serverless and FaaS
  3. Cloud Native
  4. Microservices
  5. Update to 2019 Hypecycle

Video:

Future and Other Issues II: Security

  1. Blockchain

Video:

Future and Other Issues III: Fault Tolerance

Video:

8 - Transportation Systems

This section discusses the transportation systems

Transportation Systems Summary

  1. The ride-hailing industry highlights the growth of a new “Transportation System” TS a. For ride-hailing TS controls rides matching drivers and customers; it predicts how to position cars and how to avoid traffic slowdowns b. However, TS is much bigger outside ride-hailing as we move into the “connected vehicle” era c. TS will probably find autonomous vehicles easier to deal with than human drivers
  2. Cloud Fog and Edge components
  3. Autonomous AI was centered on generalized image processing
  4. TS also needs AI (and DL) but this is for routing and geospatial time-series; different technologies from those for image processing

Slides

Transportation Systems A: Introduction

  1. “Smart” Insurance
  2. Fundamentals of Ride-Hailing

Transportation Systems B: Components of a Ride-Hailing System

  1. Transportation Brain and Services
  2. Maps, Routing,
  3. Traffic forecasting with deep learning

Transportation Systems C: Different AI Approaches in Ride-Hailing

  1. View as a Time Series: LSTM and ARIMA
  2. View as an image in a 2D earth surface - Convolutional networks
  3. Use of Graph Neural Nets
  4. Use of Convolutional Recurrent Neural Nets
  5. Spatio-temporal modeling
  6. Comparison of data with predictions
  7. Reinforcement Learning
  8. Formulation of General Geospatial Time-Series Problem

9 - Commerce

This section discusses Commerce

Overview

Slides

AI in Commerce A: The Old way of doing things

  1. AI in Commerce
  2. AI-First Engineering, Deep Learning
  3. E-commerce and the transformation of “Bricks and Mortar”

AI in Commerce B: AI in Retail

  1. Personalization
  2. Search
  3. Image Processing to Speed up Shopping
  4. Walmart

AI in Commerce C: The Revolution that is Amazon

  1. Retail Revolution
  2. Saves Time, Effort and Novelity with Modernized Retail
  3. Looking ahead of Retail evolution

AI in Commerce D: DLMalls e-commerce

  1. Amazon sellers
  2. Rise of Shopify
  3. Selling Products on Amazon

AI in Commerce E: Recommender Engines, Digital media

  1. Spotify recommender engines
  2. Collaborative Filtering
  3. Audio Modelling
  4. DNN for Recommender engines

10 - Python Warm Up

Python Exercise on Google Colab

Python Exercise on Google Colab

Open In Colab View in Github Download Notebook

In this exercise, we will take a look at some basic Python Concepts needed for day-to-day coding.

Check the installed Python version.

! python --version
Python 3.7.6

Simple For Loop

for i in range(10):
  print(i)
0
1
2
3
4
5
6
7
8
9

List

list_items = ['a', 'b', 'c', 'd', 'e']

Retrieving an Element

list_items[2]
'c'

Append New Values

list_items.append('f')
list_items
['a', 'b', 'c', 'd', 'e', 'f']

Remove an Element

list_items.remove('a')
list_items
['b', 'c', 'd', 'e', 'f']

Dictionary

dictionary_items = {'a':1, 'b': 2, 'c': 3}

Retrieving an Item by Key

dictionary_items['b']
2

Append New Item with Key

dictionary_items['c'] = 4
dictionary_items
{'a': 1, 'b': 2, 'c': 4}

Delete an Item with Key

del dictionary_items['a'] 
dictionary_items
{'b': 2, 'c': 4}

Comparators

x = 10
y = 20 
z = 30
x > y 
False
x < z
True
z == x
False
if x < z:
  print("This is True")
This is True
if x > z:
  print("This is True")
else:
  print("This is False")  
This is False

Arithmetic

k = x * y * z
k
6000
j = x + y + z
j
60
m = x -y 
m
-10
n = x / z
n
0.3333333333333333

Numpy

Create a Random Numpy Array

import numpy as np
a = np.random.rand(100)
a.shape
(100,)

Reshape Numpy Array

b = a.reshape(10,10)
b.shape
(10, 10)

Manipulate Array Elements

c = b * 10
c[0]
array([3.33575458, 7.39029235, 5.54086921, 9.88592471, 4.9246252 ,
       1.76107178, 3.5817523 , 3.74828708, 3.57490794, 6.55752319])
c = np.mean(b,axis=1)
c.shape
10
print(c)
[0.60673061 0.4223565  0.42687517 0.6260857  0.60814217 0.66445627 
  0.54888432 0.68262262 0.42523459 0.61504903]

11 - Distributed Training for MNIST

Distributed Training for MNIST
Open In Colab View in Github Download Notebook

In this lesson we discuss in how to create a simple IPython Notebook to solve an image classification problem with Multi Layer Perceptron with LSTM.

Pre-requisites

Install the following Python packages

  1. cloudmesh-installer
  2. cloudmesh-common
pip3 install cloudmesh-installer
pip3 install cloudmesh-common

Sample MLP + LSTM with Tensorflow Keras

Import Libraries

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, SimpleRNN, InputLayer, LSTM, Dropout
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras.datasets import mnist
from cloudmesh.common.StopWatch import StopWatch

Download Data and Pre-Process

StopWatch.start("data-load")
(x_train, y_train), (x_test, y_test) = mnist.load_data()
StopWatch.stop("data-load")


StopWatch.start("data-pre-process")
num_labels = len(np.unique(y_train))


y_train = to_categorical(y_train)
y_test = to_categorical(y_test)


image_size = x_train.shape[1]
x_train = np.reshape(x_train,[-1, image_size, image_size])
x_test = np.reshape(x_test,[-1, image_size, image_size])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
StopWatch.stop("data-pre-process")

input_shape = (image_size, image_size)
batch_size = 128
units = 256
dropout = 0.2

Define Model

Here we use the Tensorflow distributed training components to train the model in multiple CPUs or GPUs. In the Colab instance multiple GPUs are not supported. Hence, the training must be done in the device type ‘None’ when selecting the ‘runtime type’ from Runtime menu. To run with multiple-GPUs no code change is required. Learn more about distributed training.

StopWatch.start("compile")
strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
  model = Sequential()
  # LSTM Layers
  model.add(LSTM(units=units,                      
                      input_shape=input_shape,
                      return_sequences=True))
  model.add(LSTM(units=units, 
                      dropout=dropout,                      
                      return_sequences=True))
  model.add(LSTM(units=units, 
                      dropout=dropout,                      
                      return_sequences=False))
  # MLP Layers
  model.add(Dense(units))
  model.add(Activation('relu'))
  model.add(Dropout(dropout))
  model.add(Dense(units))
  model.add(Activation('relu'))
  model.add(Dropout(dropout))
  # Softmax_layer
  model.add(Dense(num_labels))
  model.add(Activation('softmax'))
  model.summary()
  plot_model(model, to_file='rnn-mnist.png', show_shapes=True)
  
  print("Number of devices: {}".format(strategy.num_replicas_in_sync))

  model.compile(loss='categorical_crossentropy',
                optimizer='sgd',
                metrics=['accuracy'])
StopWatch.stop("compile")
Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_6 (LSTM)                (None, 28, 256)           291840    
_________________________________________________________________
lstm_7 (LSTM)                (None, 28, 256)           525312    
_________________________________________________________________
lstm_8 (LSTM)                (None, 256)               525312    
_________________________________________________________________
dense_6 (Dense)              (None, 256)               65792     
_________________________________________________________________
activation_6 (Activation)    (None, 256)               0         
_________________________________________________________________
dropout_4 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_7 (Dense)              (None, 256)               65792     
_________________________________________________________________
activation_7 (Activation)    (None, 256)               0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_8 (Dense)              (None, 10)                2570      
_________________________________________________________________
activation_8 (Activation)    (None, 10)                0         
=================================================================
Total params: 1,476,618
Trainable params: 1,476,618
Non-trainable params: 0
_________________________________________________________________
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:GPU:0',)
Number of devices: 1
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).
INFO:tensorflow:Reduce to /job:localhost/replica:0/task:0/device:CPU:0 then broadcast to ('/job:localhost/replica:0/task:0/device:CPU:0',).

Train

StopWatch.start("train")
model.fit(x_train, y_train, epochs=30, batch_size=batch_size)
StopWatch.stop("train")
Epoch 1/30
469/469 [==============================] - 7s 16ms/step - loss: 2.0427 - accuracy: 0.2718
Epoch 2/30
469/469 [==============================] - 7s 16ms/step - loss: 1.6934 - accuracy: 0.4007
Epoch 3/30
469/469 [==============================] - 7s 16ms/step - loss: 1.2997 - accuracy: 0.5497
...
Epoch 28/30
469/469 [==============================] - 8s 17ms/step - loss: 0.1175 - accuracy: 0.9640
Epoch 29/30
469/469 [==============================] - 8s 17ms/step - loss: 0.1158 - accuracy: 0.9645
Epoch 30/30
469/469 [==============================] - 8s 17ms/step - loss: 0.1098 - accuracy: 0.9661

Test

StopWatch.start("evaluate")
loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
StopWatch.stop("evaluate")

StopWatch.benchmark()
79/79 [==============================] - 3s 9ms/step - loss: 0.0898 - accuracy: 0.9719

Test accuracy: 97.2%

+---------------------+------------------------------------------------------------------+
| Attribute           | Value                                                            |
|---------------------+------------------------------------------------------------------|
| BUG_REPORT_URL      | "https://bugs.launchpad.net/ubuntu/"                             |
| DISTRIB_CODENAME    | bionic                                                           |
| DISTRIB_DESCRIPTION | "Ubuntu 18.04.5 LTS"                                             |
| DISTRIB_ID          | Ubuntu                                                           |
| DISTRIB_RELEASE     | 18.04                                                            |
| HOME_URL            | "https://www.ubuntu.com/"                                        |
| ID                  | ubuntu                                                           |
| ID_LIKE             | debian                                                           |
| NAME                | "Ubuntu"                                                         |
| PRETTY_NAME         | "Ubuntu 18.04.5 LTS"                                             |
| PRIVACY_POLICY_URL  | "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" |
| SUPPORT_URL         | "https://help.ubuntu.com/"                                       |
| UBUNTU_CODENAME     | bionic                                                           |
| VERSION             | "18.04.5 LTS (Bionic Beaver)"                                    |
| VERSION_CODENAME    | bionic                                                           |
| VERSION_ID          | "18.04"                                                          |
| cpu_count           | 2                                                                |
| mem.active          | 2.4 GiB                                                          |
| mem.available       | 10.3 GiB                                                         |
| mem.free            | 4.5 GiB                                                          |
| mem.inactive        | 5.4 GiB                                                          |
| mem.percent         | 18.6 %                                                           |
| mem.total           | 12.7 GiB                                                         |
| mem.used            | 3.3 GiB                                                          |
| platform.version    | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| python              | 3.7.10 (default, Feb 20 2021, 21:17:23)                          |
|                     | [GCC 7.5.0]                                                      |
| python.pip          | 19.3.1                                                           |
| python.version      | 3.7.10                                                           |
| sys.platform        | linux                                                            |
| uname.machine       | x86_64                                                           |
| uname.node          | b39e0899c1f8                                                     |
| uname.processor     | x86_64                                                           |
| uname.release       | 4.19.112+                                                        |
| uname.system        | Linux                                                            |
| uname.version       | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| user                | collab                                                           |
+---------------------+------------------------------------------------------------------+

+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+
| Name             | Status   |    Time |     Sum | Start               | tag   | Node         | User   | OS    | Version                             |
|------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------|
| data-load        | failed   |   0.473 |   0.473 | 2021-03-07 11:34:03 |       | b39e0899c1f8 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| data-pre-process | failed   |   0.073 |   0.073 | 2021-03-07 11:34:03 |       | b39e0899c1f8 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| compile          | failed   |   0.876 |   7.187 | 2021-03-07 11:38:05 |       | b39e0899c1f8 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| train            | failed   | 229.341 | 257.023 | 2021-03-07 11:38:44 |       | b39e0899c1f8 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| evaluate         | failed   |   2.659 |   4.25  | 2021-03-07 11:44:54 |       | b39e0899c1f8 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+

# csv,timer,status,time,sum,start,tag,uname.node,user,uname.system,platform.version
# csv,data-load,failed,0.473,0.473,2021-03-07 11:34:03,,b39e0899c1f8,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,data-pre-process,failed,0.073,0.073,2021-03-07 11:34:03,,b39e0899c1f8,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,compile,failed,0.876,7.187,2021-03-07 11:38:05,,b39e0899c1f8,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,train,failed,229.341,257.023,2021-03-07 11:38:44,,b39e0899c1f8,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,evaluate,failed,2.659,4.25,2021-03-07 11:44:54,,b39e0899c1f8,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020

Reference:

  1. Advance Deep Learning with Keras
  2. Distributed With Tensorflow
  3. Keras with Tensorflow Distributed Training

12 - MLP + LSTM with MNIST on Google Colab

MLP + LSTM with MNIST on Google Colab
Open In Colab View in Github Download Notebook

In this lesson we discuss in how to create a simple IPython Notebook to solve an image classification problem with Multi Layer Perceptron with LSTM.

Pre-requisites

Install the following Python packages

  1. cloudmesh-installer
  2. cloudmesh-common
pip3 install cloudmesh-installer
pip3 install cloudmesh-common

Sample MLP + LSTM with Tensorflow Keras

Import Libraries

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, SimpleRNN, InputLayer, LSTM, Dropout
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras.datasets import mnist
from cloudmesh.common.StopWatch import StopWatch

Download Data and Pre-Process

StopWatch.start("data-load")
(x_train, y_train), (x_test, y_test) = mnist.load_data()
StopWatch.stop("data-load")


StopWatch.start("data-pre-process")
num_labels = len(np.unique(y_train))


y_train = to_categorical(y_train)
y_test = to_categorical(y_test)


image_size = x_train.shape[1]
x_train = np.reshape(x_train,[-1, image_size, image_size])
x_test = np.reshape(x_test,[-1, image_size, image_size])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
StopWatch.stop("data-pre-process")

input_shape = (image_size, image_size)
batch_size = 128
units = 256
dropout = 0.2

Define Model

StopWatch.start("compile")
model = Sequential()
# LSTM Layers
model.add(LSTM(units=units,                      
                     input_shape=input_shape,
                     return_sequences=True))
model.add(LSTM(units=units, 
                     dropout=dropout,                      
                     return_sequences=True))
model.add(LSTM(units=units, 
                     dropout=dropout,                      
                     return_sequences=False))
# MLP Layers
model.add(Dense(units))
model.add(Activation('relu'))
model.add(Dropout(dropout))
model.add(Dense(units))
model.add(Activation('relu'))
model.add(Dropout(dropout))
# Softmax_layer
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='rnn-mnist.png', show_shapes=True)


model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
StopWatch.stop("compile")
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm (LSTM)                  (None, 28, 256)           291840    
_________________________________________________________________
lstm_1 (LSTM)                (None, 28, 256)           525312    
_________________________________________________________________
lstm_2 (LSTM)                (None, 256)               525312    
_________________________________________________________________
dense (Dense)                (None, 256)               65792     
_________________________________________________________________
activation (Activation)      (None, 256)               0         
_________________________________________________________________
dropout (Dropout)            (None, 256)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 256)               65792     
_________________________________________________________________
activation_1 (Activation)    (None, 256)               0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                2570      
_________________________________________________________________
activation_2 (Activation)    (None, 10)                0         
=================================================================
Total params: 1,476,618
Trainable params: 1,476,618
Non-trainable params: 0

Train

StopWatch.start("train")
model.fit(x_train, y_train, epochs=30, batch_size=batch_size)
StopWatch.stop("train")
469/469 [==============================] - 378s 796ms/step - loss: 2.2689 - accuracy: 0.2075

Test

StopWatch.start("evaluate")
loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
StopWatch.stop("evaluate")

StopWatch.benchmark()
79/79 [==============================] - 1s 7ms/step - loss: 2.2275 - accuracy: 0.3120

Test accuracy: 31.2%

+---------------------+------------------------------------------------------------------+
| Attribute           | Value                                                            |
|---------------------+------------------------------------------------------------------|
| BUG_REPORT_URL      | "https://bugs.launchpad.net/ubuntu/"                             |
| DISTRIB_CODENAME    | bionic                                                           |
| DISTRIB_DESCRIPTION | "Ubuntu 18.04.5 LTS"                                             |
| DISTRIB_ID          | Ubuntu                                                           |
| DISTRIB_RELEASE     | 18.04                                                            |
| HOME_URL            | "https://www.ubuntu.com/"                                        |
| ID                  | ubuntu                                                           |
| ID_LIKE             | debian                                                           |
| NAME                | "Ubuntu"                                                         |
| PRETTY_NAME         | "Ubuntu 18.04.5 LTS"                                             |
| PRIVACY_POLICY_URL  | "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" |
| SUPPORT_URL         | "https://help.ubuntu.com/"                                       |
| UBUNTU_CODENAME     | bionic                                                           |
| VERSION             | "18.04.5 LTS (Bionic Beaver)"                                    |
| VERSION_CODENAME    | bionic                                                           |
| VERSION_ID          | "18.04"                                                          |
| cpu_count           | 2                                                                |
| mem.active          | 1.9 GiB                                                          |
| mem.available       | 10.7 GiB                                                         |
| mem.free            | 7.3 GiB                                                          |
| mem.inactive        | 3.0 GiB                                                          |
| mem.percent         | 15.6 %                                                           |
| mem.total           | 12.7 GiB                                                         |
| mem.used            | 2.3 GiB                                                          |
| platform.version    | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| python              | 3.6.9 (default, Oct  8 2020, 12:12:24)                           |
|                     | [GCC 8.4.0]                                                      |
| python.pip          | 19.3.1                                                           |
| python.version      | 3.6.9                                                            |
| sys.platform        | linux                                                            |
| uname.machine       | x86_64                                                           |
| uname.node          | 9810ccb69d08                                                     |
| uname.processor     | x86_64                                                           |
| uname.release       | 4.19.112+                                                        |
| uname.system        | Linux                                                            |
| uname.version       | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| user                | collab                                                           |
+---------------------+------------------------------------------------------------------+

+------------------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------+
| Name             | Status   |   Time |    Sum | Start               | tag   | Node         | User   | OS    | Version                             |
|------------------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------|
| data-load        | failed   |  0.61  |  0.61  | 2021-02-21 21:35:06 |       | 9810ccb69d08 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| data-pre-process | failed   |  0.076 |  0.076 | 2021-02-21 21:35:07 |       | 9810ccb69d08 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| compile          | failed   |  6.445 |  6.445 | 2021-02-21 21:35:07 |       | 9810ccb69d08 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| train            | failed   | 17.171 | 17.171 | 2021-02-21 21:35:13 |       | 9810ccb69d08 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| evaluate         | failed   |  1.442 |  1.442 | 2021-02-21 21:35:31 |       | 9810ccb69d08 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
+------------------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------+

# csv,timer,status,time,sum,start,tag,uname.node,user,uname.system,platform.version
# csv,data-load,failed,0.61,0.61,2021-02-21 21:35:06,,9810ccb69d08,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,data-pre-process,failed,0.076,0.076,2021-02-21 21:35:07,,9810ccb69d08,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,compile,failed,6.445,6.445,2021-02-21 21:35:07,,9810ccb69d08,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,train,failed,17.171,17.171,2021-02-21 21:35:13,,9810ccb69d08,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,evaluate,failed,1.442,1.442,2021-02-21 21:35:31,,9810ccb69d08,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020

Reference:

Orignal Source to Source Code

13 - MNIST Classification on Google Colab

MNIST Classification on Google Colab
Open In Colab View in Github Download Notebook

In this lesson we discuss in how to create a simple IPython Notebook to solve an image classification problem. MNIST contains a set of pictures

Import Libraries

Note: https://python-future.org/quickstart.html

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.utils import to_categorical, plot_model
from keras.datasets import mnist

Warm Up Exercise

Pre-process data

Load data

First we load the data from the inbuilt mnist dataset from Keras Here we have to split the data set into training and testing data. The training data or testing data has two components. Training features and training labels. For instance every sample in the dataset has a corresponding label. In Mnist the training sample contains image data represented in terms of an array. The training labels are from 0-9.

Here we say x_train for training data features and y_train as the training labels. Same goes for testing data.

(x_train, y_train), (x_test, y_test) = mnist.load_data()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

Identify Number of Classes

As this is a number classification problem. We need to know how many classes are there. So we’ll count the number of unique labels.

num_labels = len(np.unique(y_train))

Convert Labels To One-Hot Vector

Read more on one-hot vector.

y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

Image Reshaping

The training model is designed by considering the data as a vector. This is a model dependent modification. Here we assume the image is a squared shape image.

image_size = x_train.shape[1]
input_size = image_size * image_size

Resize and Normalize

The next step is to continue the reshaping to a fit into a vector and normalize the data. Image values are from 0 - 255, so an easy way to normalize is to divide by the maximum value.

x_train = np.reshape(x_train, [-1, input_size])
x_train = x_train.astype('float32') / 255
x_test = np.reshape(x_test, [-1, input_size])
x_test = x_test.astype('float32') / 255

Create a Keras Model

Keras is a neural network library. The summary function provides tabular summary on the model you created. And the plot_model function provides a grpah on the network you created.

# Create Model
# network parameters
batch_size = 4
hidden_units = 64

model = Sequential()
model.add(Dense(hidden_units, input_dim=input_size))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='mlp-mnist.png', show_shapes=True)
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_5 (Dense)              (None, 512)               401920    
_________________________________________________________________
dense_6 (Dense)              (None, 10)                5130      
_________________________________________________________________
activation_5 (Activation)    (None, 10)                0         
=================================================================
Total params: 407,050
Trainable params: 407,050
Non-trainable params: 0
_________________________________________________________________

images

Compile and Train

A keras model need to be compiled before it can be used to train the model. In the compile function, you can provide the optimization that you want to add, metrics you expect and the type of loss function you need to use.

Here we use adam optimizer, a famous optimizer used in neural networks.

The loss funtion we have used is the categorical_crossentropy.

Once the model is compiled, then the fit function is called upon passing the number of epochs, traing data and batch size.

The batch size determines the number of elements used per minibatch in optimizing the function.

Note: Change the number of epochs, batch size and see what happens.

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=1, batch_size=batch_size)
469/469 [==============================] - 3s 7ms/step - loss: 0.3647 - accuracy: 0.8947





<tensorflow.python.keras.callbacks.History at 0x7fe88faf4c50>

Testing

Now we can test the trained model. Use the evaluate function by passing test data and batch size and the accuracy and the loss value can be retrieved.

MNIST_V1.0|Exercise: Try to observe the network behavior by changing the number of epochs, batch size and record the best accuracy that you can gain. Here you can record what happens when you change these values. Describe your observations in 50-100 words.

loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
79/79 [==============================] - 0s 4ms/step - loss: 0.2984 - accuracy: 0.9148

Test accuracy: 91.5%

Final Note

This programme can be defined as a hello world programme in deep learning. Objective of this exercise is not to teach you the depths of deep learning. But to teach you basic concepts that may need to design a simple network to solve a problem. Before running the whole code, read all the instructions before a code section.

Homework

Solve Exercise MNIST_V1.0.

Reference:

Orignal Source to Source Code

14 - MNIST With PyTorch

MNIST With PyTorch
Open In Colab View in Github Download Notebook

In this lesson we discuss in how to create a simple IPython Notebook to solve an image classification problem with Multi Layer Perceptron with PyTorch.

Import Libraries

import numpy as np
import torch
import torchvision
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
from torch import nn
from torch import optim
from time import time
import os
from google.colab import drive

Pre-Process Data

Here we download the data using PyTorch data utils and transform the data by using a normalization function. PyTorch provides a data loader abstraction called a DataLoader where we can set the batch size, data shuffle per batch loading. Each data loader expecte a Pytorch Dataset. The DataSet abstraction and DataLoader usage can be found here

# Data transformation function 
transform = transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5,), (0.5,)),
                              ])

# DataSet
train_data_set = datasets.MNIST('drive/My Drive/mnist/data/', download=True, train=True, transform=transform)
validation_data_set = datasets.MNIST('drive/My Drive/mnist/data/', download=True, train=False, transform=transform)

# DataLoader
train_loader = torch.utils.data.DataLoader(train_data_set, batch_size=32, shuffle=True)
validation_loader = torch.utils.data.DataLoader(validation_data_set, batch_size=32, shuffle=True)

Define Network

Here we select the matching input size compared to the network definition. Here data reshaping or layer reshaping must be done to match input data shape with the network input shape. Also we define a set of hidden unit sizes along with the output layers size. The output_size must match with the number of labels associated with the classification problem. The hidden units can be chosesn depending on the problem. nn.Sequential is one way to create the network. Here we stack a set of linear layers along with a softmax layer for the classification as the output layer.

input_size = 784
hidden_sizes = [128, 128, 64, 64]
output_size = 10

model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], hidden_sizes[2]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[2], hidden_sizes[3]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[3], output_size),
                      nn.LogSoftmax(dim=1))

                      
print(model)
Sequential(
  (0): Linear(in_features=784, out_features=128, bias=True)
  (1): ReLU()
  (2): Linear(in_features=128, out_features=128, bias=True)
  (3): ReLU()
  (4): Linear(in_features=128, out_features=64, bias=True)
  (5): ReLU()
  (6): Linear(in_features=64, out_features=64, bias=True)
  (7): ReLU()
  (8): Linear(in_features=64, out_features=10, bias=True)
  (9): LogSoftmax(dim=1)
)

Define Loss Function and Optimizer

Read more about Loss Functions and Optimizers supported by PyTorch.

criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.003, momentum=0.9)

Train

epochs = 5

for epoch in range(epochs):
    loss_per_epoch = 0
    for images, labels in train_loader:
        images = images.view(images.shape[0], -1)
    
        # Gradients cleared per batch
        optimizer.zero_grad()
        
        # Pass input to the model
        output = model(images)
        # Calculate loss after training compared to labels
        loss = criterion(output, labels)
        
        # backpropagation 
        loss.backward()
        
        # optimizer step to update the weights
        optimizer.step()
        
        loss_per_epoch += loss.item()
    average_loss = loss_per_epoch / len(train_loader)
    print("Epoch {} - Training loss: {}".format(epoch, average_loss))
Epoch 0 - Training loss: 1.3052690227402808
Epoch 1 - Training loss: 0.33809808635317695
Epoch 2 - Training loss: 0.22927882223685922
Epoch 3 - Training loss: 0.16807103878669521
Epoch 4 - Training loss: 0.1369301250545995

Model Evaluation

Similar to training data loader, we use the validation loader to load batch by batch and run the feed-forward network to get the expected prediction and compared to the label associated with the data point.

correct_predictions, all_count = 0, 0
# enumerate data from the data validation loader (loads a batch at a time)
for batch_id, (images,labels) in enumerate(validation_loader):
  for i in range(len(labels)):
    img = images[i].view(1, 784)
    # at prediction stage, only feed-forward calculation is required. 
    with torch.no_grad():
        logps = model(img)

    # Output layer of the network uses a LogSoftMax layer
    # Hence the probability must be calculated with the exponential values. 
    # The final layer returns an array of probabilities for each label
    # Pick the maximum probability and the corresponding index
    # The corresponding index is the predicted label 
    ps = torch.exp(logps)
    probab = list(ps.numpy()[0])
    pred_label = probab.index(max(probab))
    true_label = labels.numpy()[i]
    if(true_label == pred_label):
      correct_predictions += 1
    all_count += 1

print(f"Model Accuracy {(correct_predictions/all_count) * 100} %")
Model Accuracy 95.95 %

Reference:

  1. Torch NN Sequential
  2. Handwritten Digit Recognition Using PyTorch — Intro To Neural Networks
  3. MNIST Handwritten Digit Recognition in PyTorch

15 - MNIST-AutoEncoder Classification on Google Colab

MNIST with AutoEncoder: Classification on Google Colab
Open In Colab View in Github Download Notebook

Prerequisites

Install the following packages

! pip3 install cloudmesh-installer
! pip3 install cloudmesh-common

Import Libraries

import tensorflow as tf
from keras.layers import Dense, Input
from keras.layers import Conv2D, Flatten
from keras.layers import Reshape, Conv2DTranspose
from keras.models import Model
from keras.datasets import mnist
from keras.utils import plot_model
from keras import backend as K

import numpy as np
import matplotlib.pyplot as plt

Download Data and Pre-Process

(x_train, y_train), (x_test, y_test) = mnist.load_data()

image_size = x_train.shape[1]
x_train = np.reshape(x_train, [-1, image_size, image_size, 1])
x_test = np.reshape(x_test, [-1, image_size, image_size, 1])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

input_shape = (image_size, image_size, 1)
batch_size = 32
kernel_size = 3
latent_dim = 16
hidden_units = [32, 64]
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

Define Model

inputs = Input(shape=input_shape, name='encoder_input')
x = inputs
x = Dense(hidden_units[0], activation='relu')(x)
x = Dense(hidden_units[1], activation='relu')(x)

shape = K.int_shape(x)

# generate latent vector
x = Flatten()(x)
latent = Dense(latent_dim, name='latent_vector')(x)

# instantiate encoder model
encoder = Model(inputs,
                latent,
                name='encoder')
encoder.summary()
plot_model(encoder,
           to_file='encoder.png',
           show_shapes=True)


latent_inputs = Input(shape=(latent_dim,), name='decoder_input')
x = Dense(shape[1] * shape[2] * shape[3])(latent_inputs)
x = Reshape((shape[1], shape[2], shape[3]))(x)
x = Dense(hidden_units[0], activation='relu')(x)
x = Dense(hidden_units[1], activation='relu')(x)

outputs = Dense(1, activation='relu')(x)

decoder = Model(latent_inputs, outputs, name='decoder')
decoder.summary()
plot_model(decoder, to_file='decoder.png', show_shapes=True)

autoencoder = Model(inputs,
                    decoder(encoder(inputs)),
                    name='autoencoder')
autoencoder.summary()
plot_model(autoencoder,
           to_file='autoencoder.png',
           show_shapes=True)

autoencoder.compile(loss='mse', optimizer='adam')
Model: "encoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
encoder_input (InputLayer)   [(None, 28, 28, 1)]       0         
_________________________________________________________________
dense_2 (Dense)              (None, 28, 28, 32)        64        
_________________________________________________________________
dense_3 (Dense)              (None, 28, 28, 64)        2112      
_________________________________________________________________
flatten_1 (Flatten)          (None, 50176)             0         
_________________________________________________________________
latent_vector (Dense)        (None, 16)                802832    
=================================================================
Total params: 805,008
Trainable params: 805,008
Non-trainable params: 0
_________________________________________________________________
Model: "decoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
decoder_input (InputLayer)   [(None, 16)]              0         
_________________________________________________________________
dense_4 (Dense)              (None, 50176)             852992    
_________________________________________________________________
reshape (Reshape)            (None, 28, 28, 64)        0         
_________________________________________________________________
dense_5 (Dense)              (None, 28, 28, 32)        2080      
_________________________________________________________________
dense_6 (Dense)              (None, 28, 28, 64)        2112      
_________________________________________________________________
dense_7 (Dense)              (None, 28, 28, 1)         65        
=================================================================
Total params: 857,249
Trainable params: 857,249
Non-trainable params: 0
_________________________________________________________________
Model: "autoencoder"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
encoder_input (InputLayer)   [(None, 28, 28, 1)]       0         
_________________________________________________________________
encoder (Functional)         (None, 16)                805008    
_________________________________________________________________
decoder (Functional)         (None, 28, 28, 1)         857249    
=================================================================
Total params: 1,662,257
Trainable params: 1,662,257
Non-trainable params: 0

Train

autoencoder.fit(x_train,
                x_train,
                validation_data=(x_test, x_test),
                epochs=1,
                batch_size=batch_size)
1875/1875 [==============================] - 112s 60ms/step - loss: 0.0268 - val_loss: 0.0131

<tensorflow.python.keras.callbacks.History at 0x7f3ecb2e0be0>

Test

x_decoded = autoencoder.predict(x_test)
79/79 [==============================] - 7s 80ms/step - loss: 0.2581 - accuracy: 0.9181

Test accuracy: 91.8%

Visualize

imgs = np.concatenate([x_test[:8], x_decoded[:8]])
imgs = imgs.reshape((4, 4, image_size, image_size))
imgs = np.vstack([np.hstack(i) for i in imgs])
plt.figure()
plt.axis('off')
plt.title('Input: 1st 2 rows, Decoded: last 2 rows')
plt.imshow(imgs, interpolation='none', cmap='gray')
plt.savefig('input_and_decoded.png')
plt.show()

16 - MNIST-CNN Classification on Google Colab

MNIST with Convolutional Neural Networks: Classification on Google Colab
Open In Colab View in Github Download Notebook

Prerequisites

Install the following packages

! pip3 install cloudmesh-installer
! pip3 install cloudmesh-common

Import Libraries

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout
from keras.layers import Conv2D, MaxPooling2D, Flatten, AveragePooling2D
from keras.utils import to_categorical, plot_model
from keras.datasets import mnist

Download Data and Pre-Process

(x_train, y_train), (x_test, y_test) = mnist.load_data()

num_labels = len(np.unique(y_train))

y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

image_size = x_train.shape[1]
x_train = np.reshape(x_train,[-1, image_size, image_size, 1])
x_test = np.reshape(x_test,[-1, image_size, image_size, 1])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255

input_shape = (image_size, image_size, 1)
print(input_shape)
batch_size = 128
kernel_size = 3
pool_size = 2
filters = 64
dropout = 0.2
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
(28, 28, 1)

Define Model

model = Sequential()
model.add(Conv2D(filters=filters,
                 kernel_size=kernel_size,
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))
model.add(MaxPooling2D(pool_size))
model.add(Conv2D(filters=filters,
                 kernel_size=kernel_size,
                 activation='relu',
                 input_shape=input_shape,
                 padding='same'))
model.add(MaxPooling2D(pool_size))
model.add(Conv2D(filters=filters,
                 kernel_size=kernel_size,
                 activation='relu',
                 padding='same'))
model.add(MaxPooling2D(pool_size))
model.add(Conv2D(filters=filters,
                 kernel_size=kernel_size,
                 activation='relu'))
model.add(Flatten())
model.add(Dropout(dropout))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='cnn-mnist.png', show_shapes=True)
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_4 (Conv2D)            (None, 28, 28, 64)        640       
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 14, 14, 64)        0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 14, 14, 64)        36928     
_________________________________________________________________
max_pooling2d_4 (MaxPooling2 (None, 7, 7, 64)          0         
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 7, 7, 64)          36928     
_________________________________________________________________
max_pooling2d_5 (MaxPooling2 (None, 3, 3, 64)          0         
_________________________________________________________________
conv2d_7 (Conv2D)            (None, 1, 1, 64)          36928     
_________________________________________________________________
flatten_1 (Flatten)          (None, 64)                0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 64)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 10)                650       
_________________________________________________________________
activation_1 (Activation)    (None, 10)                0         
=================================================================
Total params: 112,074
Trainable params: 112,074
Non-trainable params: 0
_________________________________________________________________

Train

model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
# train the network
model.fit(x_train, y_train, epochs=10, batch_size=batch_size)
469/469 [==============================] - 125s 266ms/step - loss: 0.6794 - accuracy: 0.7783

<tensorflow.python.keras.callbacks.History at 0x7f35d4b104e0>

Test

loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
79/79 [==============================] - 6s 68ms/step - loss: 0.0608 - accuracy: 0.9813

Test accuracy: 98.1%

17 - MNIST-LSTM Classification on Google Colab

MNIST-LSTM Classification on Google Colab
Open In Colab View in Github Download Notebook

Pre-requisites

Install the following Python packages

  1. cloudmesh-installer
  2. cloudmesh-common
pip3 install cloudmesh-installer
pip3 install cloudmesh-common

Sample LSTM with Tensorflow Keras

Import Libraries

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, SimpleRNN, InputLayer, LSTM
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras.datasets import mnist
from cloudmesh.common.StopWatch import StopWatch

Download Data and Pre-Process

StopWatch.start("data-load")
(x_train, y_train), (x_test, y_test) = mnist.load_data()
StopWatch.stop("data-load")


StopWatch.start("data-pre-process")
num_labels = len(np.unique(y_train))


y_train = to_categorical(y_train)
y_test = to_categorical(y_test)


image_size = x_train.shape[1]
x_train = np.reshape(x_train,[-1, image_size, image_size])
x_test = np.reshape(x_test,[-1, image_size, image_size])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
StopWatch.stop("data-pre-process")

input_shape = (image_size, image_size)
batch_size = 128
units = 256
dropout = 0.2

Define Model

StopWatch.start("compile")
model = Sequential()
model.add(LSTM(units=units,                      
                     input_shape=input_shape,
                     return_sequences=True))
model.add(LSTM(units=units, 
                     dropout=dropout,                      
                     return_sequences=True))
model.add(LSTM(units=units, 
                     dropout=dropout,                      
                     return_sequences=False))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='rnn-mnist.png', show_shapes=True)


model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
StopWatch.stop("compile")
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
lstm_3 (LSTM)                (None, 28, 256)           291840    
_________________________________________________________________
lstm_4 (LSTM)                (None, 28, 256)           525312    
_________________________________________________________________
lstm_5 (LSTM)                (None, 256)               525312    
_________________________________________________________________
dense_1 (Dense)              (None, 10)                2570      
_________________________________________________________________
activation_1 (Activation)    (None, 10)                0         
=================================================================
Total params: 1,345,034
Trainable params: 1,345,034
Non-trainable params: 0

Train

StopWatch.start("train")
model.fit(x_train, y_train, epochs=1, batch_size=batch_size)
StopWatch.stop("train")
469/469 [==============================] - 378s 796ms/step - loss: 2.2689 - accuracy: 0.2075

Test

StopWatch.start("evaluate")
loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
StopWatch.stop("evaluate")

StopWatch.benchmark()
79/79 [==============================] - 22s 260ms/step - loss: 1.9646 - accuracy: 0.3505

Test accuracy: 35.0%

+---------------------+------------------------------------------------------------------+
| Attribute           | Value                                                            |
|---------------------+------------------------------------------------------------------|
| BUG_REPORT_URL      | "https://bugs.launchpad.net/ubuntu/"                             |
| DISTRIB_CODENAME    | bionic                                                           |
| DISTRIB_DESCRIPTION | "Ubuntu 18.04.5 LTS"                                             |
| DISTRIB_ID          | Ubuntu                                                           |
| DISTRIB_RELEASE     | 18.04                                                            |
| HOME_URL            | "https://www.ubuntu.com/"                                        |
| ID                  | ubuntu                                                           |
| ID_LIKE             | debian                                                           |
| NAME                | "Ubuntu"                                                         |
| PRETTY_NAME         | "Ubuntu 18.04.5 LTS"                                             |
| PRIVACY_POLICY_URL  | "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" |
| SUPPORT_URL         | "https://help.ubuntu.com/"                                       |
| UBUNTU_CODENAME     | bionic                                                           |
| VERSION             | "18.04.5 LTS (Bionic Beaver)"                                    |
| VERSION_CODENAME    | bionic                                                           |
| VERSION_ID          | "18.04"                                                          |
| cpu_count           | 2                                                                |
| mem.active          | 1.5 GiB                                                          |
| mem.available       | 11.4 GiB                                                         |
| mem.free            | 9.3 GiB                                                          |
| mem.inactive        | 1.7 GiB                                                          |
| mem.percent         | 10.4 %                                                           |
| mem.total           | 12.7 GiB                                                         |
| mem.used            | 1.3 GiB                                                          |
| platform.version    | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| python              | 3.6.9 (default, Oct  8 2020, 12:12:24)                           |
|                     | [GCC 8.4.0]                                                      |
| python.pip          | 19.3.1                                                           |
| python.version      | 3.6.9                                                            |
| sys.platform        | linux                                                            |
| uname.machine       | x86_64                                                           |
| uname.node          | 351ef0f61c92                                                     |
| uname.processor     | x86_64                                                           |
| uname.release       | 4.19.112+                                                        |
| uname.system        | Linux                                                            |
| uname.version       | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| user                | collab                                                           |
+---------------------+------------------------------------------------------------------+

+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+
| Name             | Status   |    Time |     Sum | Start               | tag   | Node         | User   | OS    | Version                             |
|------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------|
| data-load        | failed   |   0.354 |   0.967 | 2021-02-18 15:27:21 |       | 351ef0f61c92 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| data-pre-process | failed   |   0.098 |   0.198 | 2021-02-18 15:27:21 |       | 351ef0f61c92 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| compile          | failed   |   0.932 |   2.352 | 2021-02-18 15:27:23 |       | 351ef0f61c92 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| train            | failed   | 377.842 | 377.842 | 2021-02-18 15:27:26 |       | 351ef0f61c92 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| evaluate         | failed   |  21.689 |  21.689 | 2021-02-18 15:33:44 |       | 351ef0f61c92 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+

# csv,timer,status,time,sum,start,tag,uname.node,user,uname.system,platform.version
# csv,data-load,failed,0.354,0.967,2021-02-18 15:27:21,,351ef0f61c92,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,data-pre-process,failed,0.098,0.198,2021-02-18 15:27:21,,351ef0f61c92,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,compile,failed,0.932,2.352,2021-02-18 15:27:23,,351ef0f61c92,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,train,failed,377.842,377.842,2021-02-18 15:27:26,,351ef0f61c92,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,evaluate,failed,21.689,21.689,2021-02-18 15:33:44,,351ef0f61c92,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020

Reference:

Orignal Source to Source Code

18 - MNIST-MLP Classification on Google Colab

MNIST-MLP Classification on Google Colab
Open In Colab View in Github Download Notebook

In this lesson we discuss in how to create a simple IPython Notebook to solve an image classification problem with Multi Layer Perceptron.

Pre-requisites

Install the following Python packages

  1. cloudmesh-installer
  2. cloudmesh-common
pip3 install cloudmesh-installer
pip3 install cloudmesh-common

Sample MLP with Tensorflow Keras

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import time 

import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.utils import to_categorical, plot_model
from keras.datasets import mnist
#import pydotplus
from keras.utils.vis_utils import model_to_dot
#from keras.utils.vis_utils import pydot


from cloudmesh.common.StopWatch import StopWatch

StopWatch.start("data-load")
(x_train, y_train), (x_test, y_test) = mnist.load_data()
StopWatch.stop("data-load")

num_labels = len(np.unique(y_train))


y_train = to_categorical(y_train)
y_test = to_categorical(y_test)


image_size = x_train.shape[1]
input_size = image_size * image_size


x_train = np.reshape(x_train, [-1, input_size])
x_train = x_train.astype('float32') / 255
x_test = np.reshape(x_test, [-1, input_size])
x_test = x_test.astype('float32') / 255

batch_size = 128
hidden_units = 512
dropout = 0.45

model = Sequential()
model.add(Dense(hidden_units, input_dim=input_size))
model.add(Activation('relu'))
model.add(Dropout(dropout))
model.add(Dense(hidden_units))
model.add(Activation('relu'))
model.add(Dropout(dropout))
model.add(Dense(hidden_units))
model.add(Activation('relu'))
model.add(Dropout(dropout))
model.add(Dense(hidden_units))
model.add(Activation('relu'))
model.add(Dropout(dropout))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='mlp-mnist.png', show_shapes=True)

StopWatch.start("compile")
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
StopWatch.stop("compile")
StopWatch.start("train")
model.fit(x_train, y_train, epochs=5, batch_size=batch_size)
StopWatch.stop("train")

StopWatch.start("test")
loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
StopWatch.stop("test")

StopWatch.benchmark()
Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 512)               401920    
_________________________________________________________________
activation (Activation)      (None, 512)               0         
_________________________________________________________________
dropout (Dropout)            (None, 512)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 512)               262656    
_________________________________________________________________
activation_1 (Activation)    (None, 512)               0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 512)               262656    
_________________________________________________________________
activation_2 (Activation)    (None, 512)               0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 512)               262656    
_________________________________________________________________
activation_3 (Activation)    (None, 512)               0         
_________________________________________________________________
dropout_3 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_4 (Dense)              (None, 10)                5130      
_________________________________________________________________
activation_4 (Activation)    (None, 10)                0         
=================================================================
Total params: 1,195,018
Trainable params: 1,195,018
Non-trainable params: 0
_________________________________________________________________
Epoch 1/5
469/469 [==============================] - 14s 29ms/step - loss: 0.7886 - accuracy: 0.7334
Epoch 2/5
469/469 [==============================] - 14s 29ms/step - loss: 0.1981 - accuracy: 0.9433
Epoch 3/5
469/469 [==============================] - 14s 29ms/step - loss: 0.1546 - accuracy: 0.9572
Epoch 4/5
469/469 [==============================] - 14s 29ms/step - loss: 0.1302 - accuracy: 0.9641
Epoch 5/5
469/469 [==============================] - 14s 29ms/step - loss: 0.1168 - accuracy: 0.9663
79/79 [==============================] - 1s 9ms/step - loss: 0.0785 - accuracy: 0.9765

Test accuracy: 97.6%

+---------------------+------------------------------------------------------------------+
| Attribute           | Value                                                            |
|---------------------+------------------------------------------------------------------|
| BUG_REPORT_URL      | "https://bugs.launchpad.net/ubuntu/"                             |
| DISTRIB_CODENAME    | bionic                                                           |
| DISTRIB_DESCRIPTION | "Ubuntu 18.04.5 LTS"                                             |
| DISTRIB_ID          | Ubuntu                                                           |
| DISTRIB_RELEASE     | 18.04                                                            |
| HOME_URL            | "https://www.ubuntu.com/"                                        |
| ID                  | ubuntu                                                           |
| ID_LIKE             | debian                                                           |
| NAME                | "Ubuntu"                                                         |
| PRETTY_NAME         | "Ubuntu 18.04.5 LTS"                                             |
| PRIVACY_POLICY_URL  | "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" |
| SUPPORT_URL         | "https://help.ubuntu.com/"                                       |
| UBUNTU_CODENAME     | bionic                                                           |
| VERSION             | "18.04.5 LTS (Bionic Beaver)"                                    |
| VERSION_CODENAME    | bionic                                                           |
| VERSION_ID          | "18.04"                                                          |
| cpu_count           | 2                                                                |
| mem.active          | 1.2 GiB                                                          |
| mem.available       | 11.6 GiB                                                         |
| mem.free            | 9.8 GiB                                                          |
| mem.inactive        | 1.4 GiB                                                          |
| mem.percent         | 8.4 %                                                            |
| mem.total           | 12.7 GiB                                                         |
| mem.used            | 913.7 MiB                                                        |
| platform.version    | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| python              | 3.6.9 (default, Oct  8 2020, 12:12:24)                           |
|                     | [GCC 8.4.0]                                                      |
| python.pip          | 19.3.1                                                           |
| python.version      | 3.6.9                                                            |
| sys.platform        | linux                                                            |
| uname.machine       | x86_64                                                           |
| uname.node          | 6609095905d1                                                     |
| uname.processor     | x86_64                                                           |
| uname.release       | 4.19.112+                                                        |
| uname.system        | Linux                                                            |
| uname.version       | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| user                | collab                                                           |
+---------------------+------------------------------------------------------------------+

+-----------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------+
| Name      | Status   |   Time |    Sum | Start               | tag   | Node         | User   | OS    | Version                             |
|-----------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------|
| data-load | failed   |  0.549 |  0.549 | 2021-02-15 15:24:00 |       | 6609095905d1 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| compile   | failed   |  0.023 |  0.023 | 2021-02-15 15:24:01 |       | 6609095905d1 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| train     | failed   | 69.1   | 69.1   | 2021-02-15 15:24:01 |       | 6609095905d1 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| test      | failed   |  0.907 |  0.907 | 2021-02-15 15:25:10 |       | 6609095905d1 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
+-----------+----------+--------+--------+---------------------+-------+--------------+--------+-------+-------------------------------------+

# csv,timer,status,time,sum,start,tag,uname.node,user,uname.system,platform.version
# csv,data-load,failed,0.549,0.549,2021-02-15 15:24:00,,6609095905d1,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,compile,failed,0.023,0.023,2021-02-15 15:24:01,,6609095905d1,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,train,failed,69.1,69.1,2021-02-15 15:24:01,,6609095905d1,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,test,failed,0.907,0.907,2021-02-15 15:25:10,,6609095905d1,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020

Reference:

Orignal Source to Source Code

19 - MNIST-RMM Classification on Google Colab

MNIST with Recurrent Neural Networks: Classification on Google Colab
Open In Colab View in Github Download Notebook

Prerequisites

Install the following packages

! pip3 install cloudmesh-installer
! pip3 install cloudmesh-common

Import Libraries

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Activation, SimpleRNN
from tensorflow.keras.utils import to_categorical, plot_model
from tensorflow.keras.datasets import mnist
from cloudmesh.common.StopWatch import StopWatch

Download Data and Pre-Process

StopWatch.start("data-load")
(x_train, y_train), (x_test, y_test) = mnist.load_data()
StopWatch.stop("data-load")


StopWatch.start("data-pre-process")
num_labels = len(np.unique(y_train))


y_train = to_categorical(y_train)
y_test = to_categorical(y_test)


image_size = x_train.shape[1]
x_train = np.reshape(x_train,[-1, image_size, image_size])
x_test = np.reshape(x_test,[-1, image_size, image_size])
x_train = x_train.astype('float32') / 255
x_test = x_test.astype('float32') / 255
StopWatch.stop("data-pre-process")

input_shape = (image_size, image_size)
batch_size = 128
units = 256
dropout = 0.2

Define Model

StopWatch.start("compile")
model = Sequential()
model.add(SimpleRNN(units=units,
                    dropout=dropout,
                    input_shape=input_shape, return_sequences=True))
model.add(SimpleRNN(units=units,
                    dropout=dropout,
                    return_sequences=True))
model.add(SimpleRNN(units=units,
                    dropout=dropout,
                    return_sequences=False))
model.add(Dense(num_labels))
model.add(Activation('softmax'))
model.summary()
plot_model(model, to_file='rnn-mnist.png', show_shapes=True)


model.compile(loss='categorical_crossentropy',
              optimizer='sgd',
              metrics=['accuracy'])
StopWatch.stop("compile")
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
simple_rnn (SimpleRNN)       (None, 28, 256)           72960     
_________________________________________________________________
simple_rnn_1 (SimpleRNN)     (None, 28, 256)           131328    
_________________________________________________________________
simple_rnn_2 (SimpleRNN)     (None, 256)               131328    
_________________________________________________________________
dense (Dense)                (None, 10)                2570      
_________________________________________________________________
activation (Activation)      (None, 10)                0         
=================================================================
Total params: 338,186
Trainable params: 338,186
Non-trainable params: 0

Train

StopWatch.start("train")
model.fit(x_train, y_train, epochs=1, batch_size=batch_size)
StopWatch.stop("train")
469/469 [==============================] - 125s 266ms/step - loss: 0.6794 - accuracy: 0.7783

<tensorflow.python.keras.callbacks.History at 0x7f35d4b104e0>

Test

StopWatch.start("evaluate")
loss, acc = model.evaluate(x_test, y_test, batch_size=batch_size)
print("\nTest accuracy: %.1f%%" % (100.0 * acc))
StopWatch.stop("evaluate")

StopWatch.benchmark()
79/79 [==============================] - 7s 80ms/step - loss: 0.2581 - accuracy: 0.9181

Test accuracy: 91.8%

+---------------------+------------------------------------------------------------------+
| Attribute           | Value                                                            |
|---------------------+------------------------------------------------------------------|
| BUG_REPORT_URL      | "https://bugs.launchpad.net/ubuntu/"                             |
| DISTRIB_CODENAME    | bionic                                                           |
| DISTRIB_DESCRIPTION | "Ubuntu 18.04.5 LTS"                                             |
| DISTRIB_ID          | Ubuntu                                                           |
| DISTRIB_RELEASE     | 18.04                                                            |
| HOME_URL            | "https://www.ubuntu.com/"                                        |
| ID                  | ubuntu                                                           |
| ID_LIKE             | debian                                                           |
| NAME                | "Ubuntu"                                                         |
| PRETTY_NAME         | "Ubuntu 18.04.5 LTS"                                             |
| PRIVACY_POLICY_URL  | "https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" |
| SUPPORT_URL         | "https://help.ubuntu.com/"                                       |
| UBUNTU_CODENAME     | bionic                                                           |
| VERSION             | "18.04.5 LTS (Bionic Beaver)"                                    |
| VERSION_CODENAME    | bionic                                                           |
| VERSION_ID          | "18.04"                                                          |
| cpu_count           | 2                                                                |
| mem.active          | 1.3 GiB                                                          |
| mem.available       | 11.6 GiB                                                         |
| mem.free            | 9.7 GiB                                                          |
| mem.inactive        | 1.5 GiB                                                          |
| mem.percent         | 8.5 %                                                            |
| mem.total           | 12.7 GiB                                                         |
| mem.used            | 978.6 MiB                                                        |
| platform.version    | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| python              | 3.6.9 (default, Oct  8 2020, 12:12:24)                           |
|                     | [GCC 8.4.0]                                                      |
| python.pip          | 19.3.1                                                           |
| python.version      | 3.6.9                                                            |
| sys.platform        | linux                                                            |
| uname.machine       | x86_64                                                           |
| uname.node          | 8f16b3b1f784                                                     |
| uname.processor     | x86_64                                                           |
| uname.release       | 4.19.112+                                                        |
| uname.system        | Linux                                                            |
| uname.version       | #1 SMP Thu Jul 23 08:00:38 PDT 2020                              |
| user                | collab                                                           |
+---------------------+------------------------------------------------------------------+

+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+
| Name             | Status   |    Time |     Sum | Start               | tag   | Node         | User   | OS    | Version                             |
|------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------|
| data-load        | failed   |   0.36  |   0.36  | 2021-02-18 15:16:12 |       | 8f16b3b1f784 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| data-pre-process | failed   |   0.086 |   0.086 | 2021-02-18 15:16:12 |       | 8f16b3b1f784 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| compile          | failed   |   0.51  |   0.51  | 2021-02-18 15:16:12 |       | 8f16b3b1f784 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| train            | failed   | 126.612 | 126.612 | 2021-02-18 15:16:13 |       | 8f16b3b1f784 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
| evaluate         | failed   |   6.798 |   6.798 | 2021-02-18 15:18:19 |       | 8f16b3b1f784 | collab | Linux | #1 SMP Thu Jul 23 08:00:38 PDT 2020 |
+------------------+----------+---------+---------+---------------------+-------+--------------+--------+-------+-------------------------------------+

# csv,timer,status,time,sum,start,tag,uname.node,user,uname.system,platform.version
# csv,data-load,failed,0.36,0.36,2021-02-18 15:16:12,,8f16b3b1f784,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,data-pre-process,failed,0.086,0.086,2021-02-18 15:16:12,,8f16b3b1f784,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,compile,failed,0.51,0.51,2021-02-18 15:16:12,,8f16b3b1f784,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,train,failed,126.612,126.612,2021-02-18 15:16:13,,8f16b3b1f784,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020
# csv,evaluate,failed,6.798,6.798,2021-02-18 15:18:19,,8f16b3b1f784,collab,Linux,#1 SMP Thu Jul 23 08:00:38 PDT 2020


Reference:

Orignal Source to Source Code