Deep Learning Explained

Melanie Swan
Philosophy Department, Purdue University
melanie@BlockchainStudies.org
Deep Learning Explained
The future of Smart Networks
Boulder Futurists: Solid State Depot Hackspace
Boulder CO, August 12, 2017
Slides: http://slideshare.net/LaBlogga
Image credit: Nvidia

12 Aug 2017
Deep Learning 1
Melanie Swan, Technology Theorist
 Philosophy and Economic Theory, Purdue
University, Indiana, USA
 Founder, Institute for Blockchain Studies
 Singularity University Instructor; Institute for Ethics and
Emerging Technology Affiliate Scholar; EDGE
Essayist; FQXi Advisor
Traditional Markets Background
Economics and Financial
Theory Leadership
New Economies research group
Source: http://www.melanieswan.com, http://blockchainstudies.org/NSNE.pdf, http://blockchainstudies.org/Metaphilosophy_CFP.pdf
https://www.facebook.com/groups/NewEconomies

12 Aug 2017
Deep Learning
Agenda
 Deep Learning
 Definition
 Technical details
 Applications
 Deep Qualia: Deep Learning and the Brain
 Smart Network Convergence Theory
 Conclusion
2
Image Source: http://www.opennn.net

12 Aug 2017
Deep Learning
Deep Learning vocabulary
What do these terms mean?
 Deep Learning, Machine Learning, Artificial Intelligence
 Perceptron, Artificial Neuron, Logit
 Deep Belief Net, Artificial Neural Net, Boltzmann Machine
 Google DeepDream, Google Brain, Google DeepMind
 Supervised and Unsupervised Learning
 Convolutional Neural Nets
 Recurrent NN & LSTM (Long Short Term Memory)
 Activation Function ReLU (Rectified Linear Unit)
 Deep Learning libraries and frameworks
 TensorFlow, Caffe, Theano, Torch, DL4J
 Backpropagation, gradient descent, loss function
3

12 Aug 2017
Deep Learning 4
Conceptual Definition:
Deep learning is a computer program that can
identify what something is
Technical Definition:
Deep learning is a class of machine learning
algorithms in the form of a neural network that
uses a cascade of layers (tiers) of processing
units to extract features from data and make
predictive guesses about new data
Source: Extending Jann LeCun, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-
on-deep-learning

12 Aug 2017
Deep Learning
Deep Learning Theory
 System is “dumb” (i.e. mechanical)
 “Learns” with big data (lots of input examples) and trial-and-error
guesses to adjust weights and bias to establish key features
 Creates a predictive system to identity new examples
 Same AI argument: big enough data is what makes a
difference (“simple” algorithms run over large data sets)
5
Input: Big Data (e.g.;
many examples)
Method: Trial-and-error
guesses to adjust node weights
Output: system identifies
new examples

12 Aug 2017
Deep Learning
Sample task: is that a Car?
 Create an image recognition system that determines
which features are relevant (at increasingly higher levels
of abstraction) and correctly identifies new examples
6
Source: Jann LeCun, http://www.pamitc.org/cvpr15/files/lecun-20150610-cvpr-keynote.pdf

12 Aug 2017
Deep Learning
Broader Computer Science Context
7
Source: Machine Learning Guide, 9. Deep Learning
 Within the Computer Science discipline, in the field of
Artificial Intelligence, Deep Learning is a class of
Machine Learning algorithms, that are in the form of a
Neural Network

12 Aug 2017
Deep Learning
Statistical Mechanics
Deep Learning is inspired by Physics
8
 Sigmoid function suggested as a model for neurons,
per statistical mechanical behavior (Jack Cowan)
 Stationary solutions for dynamic models (asymmetric
weights create an oscillator to model neuron signaling)
 Hopfield Neural Network: content-addressable
memory system with binary threshold nodes,
converges to a local minimum (John Hopfield)
 Can use an Ising model (of ferromagnetism) for neurons
 Restricted Boltzmann Machine (Geoffrey Hinton)
 Studied in theoretical physics, condensed matter field
theory; Statistical Mechanics concepts: Renormalization,
Boltzmann Distribution, Free Energies, Gibbs Sampling
Source: https://www.quora.com/Is-deep-learning-related-to-statistical-physics-particularly-network-science

12 Aug 2017
Deep Learning
What is a Neural Net?
9
 Motivation: create an Artificial Neural Network to solve
problems the same way a human brain would

12 Aug 2017
Deep Learning
What is a Neural Net?
10
 Structure: input-processing-output
 Mimic neuronal signal firing structure of brain with
computational processing units
Source: https://www.slideshare.net/ThomasDaSilvaPaula/an-introduction-to-machine-learning-and-a-little-bit-of-deep-learning,
http://cs231n.github.io/convolutional-networks/

12 Aug 2017
Deep Learning
What is an Artificial Neural Network?
 Collection of connected units called artificial
neurons (analogous to axons in a biological brain)
 Organized in layers of signaling cascades
 Each neuron transmits a signal to another neuron
 Neurons may have state
 Represented by a number between 0 and 1
 Variable parameters
 Neurons may have a weight that varies as learning
proceeds, which can increase or decrease the strength of
the signal that it sends downstream
 Neurons may have a threshold (bias) such that only if the
aggregate signal is below (or above) that level is the
downstream signal sent
11

12 Aug 2017
Deep Learning
Why is it called Deep Learning?
 Deep: Hidden layers (cascading tiers) of processing
 “Deep” networks (3+ layers) versus “shallow” (1-2 layers)
 Learning: Algorithms “learn” from data by modeling
features and updating probability weights assigned to
feature nodes in testing how relevant specific features
are in determining the general type of item
12
Deep: Hidden processing layers Learning: Updating probability
weights re: feature importance

12 Aug 2017
Deep Learning
Supervised and Unsupervised Learning
 Supervised (classify
labeled data)
 Unsupervised (find
patterns in unlabeled
data)
13
Source: https://www.slideshare.net/ThomasDaSilvaPaula/an-introduction-to-machine-learning-and-a-little-bit-of-deep-learning

12 Aug 2017
Deep Learning
Early success in Supervised Learning (2011)
 YouTube: user-classified data
perfect for Supervised Learning
14
Source: Google Brain: Le, QV, Dean, Jeff, Ng, Andrew, et al. 2012. Building high-level features using large scale unsupervised
learning. https://arxiv.org/abs/1112.6209

12 Aug 2017
Deep Learning
2 main kinds of Deep Learning neural nets
15
Source: Yann LeCun, CVPR 2015 keynote (Computer Vision ), "What's wrong with Deep Learning" http://t.co/nPFlPZzMEJ
 Convolutional Neural Nets
 Image recognition
 Convolve: roll up to higher
levels of abstraction in feature
sets
 Recurrent Neural Nets
 Speech, text, audio recognition
 Recur: iterate over sequential
inputs with a memory function
 LSTM (Long Short-Term
Memory) remembers
sequences and avoids
gradient vanishing

12 Aug 2017
Deep Learning
Image Recognition and Computer Vision
16
Source: Quoc Le, https://arxiv.org/abs/1112.6209; Yann LeCun, NIPS 2016,
https://drive.google.com/file/d/0BxKBnD5y2M8NREZod0tVdW5FLTQ/view
Marv Minsky, 1966
“summer project”
Jeff Hawkins, 2004, Hierarchical
Temporal Memory (HTM)
Quoc Le, 2011, Google
Brain cat recognition
Convolutional net for autonomous driving, http://cs231n.github.io/convolutional-networks/
History
Current state of
the art - 2017

12 Aug 2017
Deep Learning
Progression in AI Deep Learning machines
17
Single-purpose AI:
Hard-coded rules
Multi-purpose AI:
Algorithm detects rules,
reusable template
Question-answering AI:
Natural-language processing
Deep Learning prototypeHard-coded AI machine Deep Learning machine
Deep Blue, 1997 Watson, 2011 AlphaGo, 2016

12 Aug 2017
Deep Learning
Why do we need Deep Learning?
18
 A contemporary data science method to keep up
with the growth in data, older learning algorithms no
longer performing
Source: http://blog.algorithmia.com/introduction-to-deep-learning-2016

12 Aug 2017
Deep Learning
Agenda
 Deep Learning
 Definition
 Applications
 Conclusion
19

12 Aug 2017
Deep Learning
3 Key Technical Principles of Deep Learning
20
Reduce combinatoric
dimensionality
Core processing unit
(input-processing-output)
Levers: weights and bias
Squash values into
Sigmoidal S-curve
-Binary values (Y/N, 0/1)
-Probability values (0 to 1)
-Tanh values 9(-1) to 1)
Loss FunctionPerceptron StructureSigmoid Function
“Dumb” system learns by
adjusting parameters and
checking against outcome
Loss function
optimizes efficiency
of solution
Non-linear formulation
as a logistic regression
problem means
greater mathematical
manipulation
What
Why

12 Aug 2017
Deep Learning
Linear Regression
21
House price vs. Size (square feet)
y=mx+b
House price
Size (square feet)
Source: https://www.statcrunch.com/5.0/viewreport.php?reportid=5647

12 Aug 2017
Deep Learning
Logistic Regression
22
Source: http://www.simafore.com/blog/bid/99443/Understand-3-critical-steps-in-developing-logistic-regression-models

12 Aug 2017
Deep Learning
Logistic Regression
23
 Higher-order mathematical
formulation
 Sigmoid function
 S-shaped and bounded
 Maps the whole real axis into a finite
interval (0-1)
 Non-linear
 Can fit probability
 Can apply optimization techniques
 Deep Learning classification
predictions are in the form of a
probability value
Source: https://www.quora.com/Logistic-Regression-Why-sigmoid-function
Sigmoid Function
Unit Step Function

12 Aug 2017
Deep Learning
Sigmoid function: Taleb
24
Source: http://www.fooledbyrandomness.com/medicine.pdf
 Thesis: if can map a phenomenon onto
a sigmoid curve (“convexify” it), then
can control its risk
 Antifragility = convexity = risk-manageable
 Fragility = concavity
 Non-linearity of dose-response in
medicine, and therefore suggested
treatment optimality

12 Aug 2017
Deep Learning
Regression
 Logistic regression
 Predict binary outcomes:
 Perceptron (0 or 1)
 Predict probabilities:
 Sigmoid Neuron (values 0-1)
 Tanh Hyperbolic Tangent
Neuron (values (-1)-1)
25
Logistic Regression (Sigmoid function)
(0-1) or Tanh ((-1)-1)
Linear Regression
 Linear regression
 Predict continuous set
of values (house prices)

12 Aug 2017
Deep Learning
Deep Learning Architecture
26
Source: Michael A. Nielsen, Neural Networks and Deep Learning

12 Aug 2017
Deep Learning
Processing Unit, Perceptron, Neuron
27
Source: http://deeplearning.stanford.edu/tutorial
1. Input 2. Hidden layers 3. Output
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
 Unit (processing unit, logistic regression
unit), perceptron (“multilayer perceptron”),
artificial neuron

12 Aug 2017
Deep Learning
Example: Image recognition
1. Obtain training data set
2. Digitize pixels (convert images to numbers)
 Divide image into 28x28 grid, assign a value (0-255) to each
square based on brightness
3. Read into vector (array; list of numbers)
 28x28 = 784 elements per image)
28
Source: Quoc V. Le, A Tutorial on Deep Learning, Part 1: Nonlinear Classifiers and The Backpropagation Algorithm, 2015, Google
Brain, https://cs.stanford.edu/~quocle/tutorial1.pdf

12 Aug 2017
Deep Learning
4. Load spreadsheet of vectors into deep learning system
 Each row of spreadsheet (784-element array) is an input
29
Source: http://deeplearning.stanford.edu/tutorial; MNIST dataset: http://yann.lecun.com/exdb/mnist
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Vector data
784-element array

12 Aug 2017
Deep Learning
What happens in the Hidden Layers?
30
 First layer learns primitive features (line, edge, tiniest
unit of sound) by finding combinations of the input vector
data that occur more frequently than by chance
 Logistic regression performed and encoded at each processing
node (Y/N (0,1)), does this example have this feature?
 Feeds these basic features to next layer, which trains
itself to recognize slightly more complicated features
(corner, combination of speech sounds)
 Feeds features to new layers until recognizes full objects

12 Aug 2017
Deep Learning
Image Recognition
Higher Abstractions of Feature Recognition
31
Edges Object Parts
(combinations of edges)
Object Models

12 Aug 2017
Deep Learning
Speech, Text, Audio Recognition
Sequence-to-sequence Recognition + LSTM
32
Source: Andrew Ng

12 Aug 2017
Deep Learning
Example: NVIDIA Facial Recognition
33
Source: Nvidia
 First hidden layer extracts all possible low-level features
from data (lines, edges, contours); next layers abstract
into more complex features of possible relevance

12 Aug 2017
Deep Learning
Deep Learning
34
Source: Quoc V. Le et al, Building high-level features using large scale unsupervised learning, 2011, https://arxiv.org/abs/1112.6209

12 Aug 2017
Deep Learning
35
(0,1)

12 Aug 2017
Deep Learning
Mathematical methods update weights
36
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Source: http://deeplearning.stanford.edu/tutorial; MNIST dataset: http://yann.lecun.com/exdb/mnist
 Linear algebra: matrix multiplications of input vectors
 Statistics: logistic regression units (Y/N (0,1)), probability
weighting and updating, inference for outcome prediction
 Calculus: optimization (minimization), gradient descent in
back-propagation to avoid local minima with saddle points
Feed-forward pass (0,1)
0.5
Backward pass to update probabilities
.5.5
.5.5.5
0
01
.75
.25
Inference
Guess
Actual

12 Aug 2017
Deep Learning
More complicated in actual use
 Convolutional neural net scale-up for
number recognition
 Example data: MNIST dataset
 http://yann.lecun.com/exdb/mnist
37
Source: http://www.kdnuggets.com/2016/04/deep-learning-vs-svm-random-forest.html

12 Aug 2017
Deep Learning
Structure of a Node: Computation Graph
38
Edge
(input value)
Architecture
Node
(operation)
Edge
(input value)
Edge
(output value)
Example 1
3
4
Add
??
Example 2
3
4
Mult
??

12 Aug 2017
Deep Learning
Neural net unit: perceptron, neuron, node
39
Source: http://neuralnetworksanddeeplearning.com/chap1.html
(0,1)
(0,1)
(0,1)
(0,1)
Oper-
ation
 Sigmoid function means all inputs and outputs in the
system are (0,1)

12 Aug 2017
Deep Learning
Other parameters: weights and bias
40
Values have
Weights
Operation node
has Bias
W1 = (-2)
B=3
W2 = (-2)
 Weight and bias are variable parameters that
get adjusted as the system iterates and learns
Values have
Weights
Operation node
has Bias
W1 = (-2)
B=3
W2 = (-2)
= 0
(-2)*0 + (-2)*0 + 3 = 3
= 0
Output
= 0
0,0
0,1
1,0
1,1 (-2)*1 + (-2)*1 + 3 = (-1)
(-2)*0 + (-2)*1 + 3 = 1
(-2)*1 + (-2)*0 + 3 = 1
W1*X1 + W2*X2 + Bias = n Output (0,1)Input (0,1) X1, X2
Weight and Bias are
“randomly” assigned at the
beginning: (here (-2) and 3)
Mimics NAND gate
1
1
1
0

12 Aug 2017
Deep Learning
Actual: same structure, more complicated
41

12 Aug 2017
Deep Learning
Neural net: massive scale-up of nodes
42

12 Aug 2017
Deep Learning
Same Structure
43

12 Aug 2017
Deep Learning
How does the neural net actually learn?
 Vary the weights
and biases to see if
a better outcome is
obtained
 Repeat until the net
correctly classifies
the data
44
 Structural system based on cascading layers of
neurons with variable parameters: weight and bias

12 Aug 2017
Deep Learning
Backpropagation
 Problem: Inefficient to test the combinatorial
explosion of all possible parameter variations
 Solution: Backpropagation (1986 Nature paper)
 Backpropagation is an optimization method used to
calculate the error contribution of each neuron after
a batch of data is processed
45

12 Aug 2017
Deep Learning
Backpropagation of error
 Calculate the total error
 Calculate the contribution to the error at each step
going backwards
 Variety of Error Calculation methods: Mean Square Error
(MSE), sum of squared errors of prediction (SSE), Cross-
Entropy (Softmax), Softplus
46

12 Aug 2017
Deep Learning
Backpropagation
 Heart of Deep Learning
 Backpropagation: algorithm dynamically calculates
the gradient (derivative) of the loss function with
respect to the weights in a network to find the
minimum and optimize the function from there
 Algorithms optimize the performance of the network by
adjusting the weights, e.g.; in the gradient descent algorithm
 Error and gradient are computed for each node
 Intermediate errors transmitted backwards through the
network (backpropagation)
 Objective: optimize the weights so that the neural
network can learn how to correctly map arbitrary
inputs to outputs
47
Source: http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4,
https://mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/

12 Aug 2017
Deep Learning
Gradient Descent
 Gradient: derivative to find the minimum of a function
 Gradient descent: optimization algorithm to find the
biggest errors (minima) most quickly
 Error = MSE, log loss, cross-entropy; e.g.; least correct
predictions to correctly identify data
48
Source: http://briandolhansky.com/blog/2013/9/27/artificial-neural-networks-backpropagation-part-4

12 Aug 2017
Deep Learning
 Optimization Technique
 Mathematical tool used in statistics, finance, decision theory,
biological modeling, computational neuroscience
 State as non-linear equation to optimize
 Minimize loss or cost
 Maximize reward, utility, profit, or fitness
 Loss function links instance of an event to its cost
 Accident (event) means $1,000 damage on average (cost)
 5 cm height (event) confers 5% fitness advantage (reward)
 Deep learning: system feedback loop
 Use penalty cost for incorrect classifications to train system
 CNN (classification): cross-entropy; RNN (regression): MSE
Loss Function
49
Laplace

12 Aug 2017
Deep Learning
Overfitting
 Regularization
 Introduce additional information
such as a lambda parameter in the
cost function (to update the theta
parameters in the gradient descent
algorithm)
 Dropout: prevent complex
adaptations on training data by
dropping out units (both hidden and
visible)
 Test new datasets
50

12 Aug 2017
Deep Learning
Research Topics
 Layer depth vs. height (1x9, 3x3, etc.); L1/2 slow-downs
 Backpropagation, gradient descent, loss function
 Saddle-free optimization, vanishing gradients
 Composition of non-linearities
 Non-parametric manifold learning, auto-encoders
 Activation maximization
 Synthesizing preferred inputs for neurons
51
Source: http://cs231n.github.io/convolutional-networks, https://arxiv.org/abs/1605.09304,
https://www.iro.umontreal.ca/~bengioy/talks/LondonParisMeetup_15April2015.pdf

12 Aug 2017
Deep Learning
Advanced
Deep Learning Architectures
52
Source: http://prog3.com/sbdm/blog/zouxy09/article/details/8781396
 Deep Belief Network
 Connections between layers not units
 Establish weighting guesses for
processing units before run deep
learning system
 Used to pre-train systems to assign
initial probability weights (more efficient)
 Deep Boltzmann Machine
 Stochastic recurrent neural network
 Runs learning on internal
representations
 Represent and solve combinatoric
problems
Deep
Boltzmann
Machine
Deep
Belief
Network

12 Aug 2017
Deep Learning
Convolutional net: Image Enhancement
 Google DeepDream: Convolutional neural network
enhances (potential) patterns in images; deliberately
over-processing images
53
Source: Georges Seurat, Un dimanche après-midi à l'Île de la Grande Jatte, 1884-1886;
http://web.cs.hacettepe.edu.tr/~aykut/classes/spring2016/bil722; Google DeepDream uses algorithmic pareidolia (seeing an image
when none is present) to create a dream-like hallucinogenic appearance

12 Aug 2017
Deep Learning
Hardware and Software Tools
54

12 Aug 2017
Deep Learning
Deep Learning frameworks and libraries
55
Source: http://www.infoworld.com/article/3163525/analytics/review-the-best-frameworks-for-machine-learning-and-deep-
learning.html#tk.ifw-ifwsb

12 Aug 2017
Deep Learning
What is TensorFlow?
56
Source: https://www.youtube.com/watch?v=uHaKOFPpphU
Python code invoking TensorFlowTensorBoard (TensorFlow) visualization
Computation graph Design in TensorFlow
 Google’s open-source machine learning library
 “Tensor” = multidimensional arrays used in NN operations

12 Aug 2017
Deep Learning
Hardware
 Advances in chip design
 GPU chips (graphics processing unit):
3D graphics cards designed to do fast
matrix multiplication
 Google TPU chip (tensor processing
unit): custom ASICs for machine
learning, used in AlphaGo
 TPUs process matrix
multiplications without storing
intermediate values in memory
 NVIDIA DGX-1 integrated deep
learning system
 Eight Tesla P100 GPU accelerators
57
Google TPU chip (Tensor
Processing Unit), 2016
Source: http://www.techradar.com/news/computing-components/processors/google-s-tensor-processing-unit-explained-this-is-what-
the-future-of-computing-looks-like-1326915
NVIDIA DGX-1
Deep Learning System

12 Aug 2017
Deep Learning
USB and Browser-based Machine Learning
 Intel: Movidius Visual Processing
Unit (VPU): USB ML for IOT
 Security cameras, industrial
equipment, robots, drones
 Apple: ML acquisition Turi (Dato)
 Browser-based Deep Learning
 ConvNetJS; TensorFire
 Javascript library to run Deep
Learning (Neural Networks) in a
browser
 Smart Network in a browser
 JavaScript Deep Learning
 Blockchain EtherWallets
58
Source: http://cs.stanford.edu/people/karpathy/convnetjs/, http://www.infoworld.com/article/3212884/machine-learning/machine-learning-
comes-to-your-browser-via-javascript.html

12 Aug 2017
Deep Learning
How big are Deep Learning neural nets?
 Google Deep Brain cat recognition, 2011
 1 billion connections, 10 million images (200x200
pixel), 1,000 machines (16,000 cores), 3 days, each
instantiation of the network spanned 170 servers, and
20,000 object categories
 State of the art, 2016-2017
 NVIDIA facial recognition, 100 million images, 10
layers, 1 bn parameters, 30 exaflops, 30 GPU days
 Google, 11.2-billion parameter system
 Lawrence Livermore Lab, 15-billion parameter system
 Digital Reasoning, cognitive computing (Nashville TN),
160 billion parameters, trained on three multi-core
computers overnight
59
Source: https://futurism.com/biggest-neural-network-ever-pushes-ai-deep-learning, Digital Reasoning paper:
https://arxiv.org/pdf/1506.02338v3.pdf

12 Aug 2017
Deep Learning
Agenda
 Deep Learning
 Definition
 Applications
 Conclusion
60

12 Aug 2017
Deep Learning
Applications: Cats to Cancer to Cognition
61
Computational imaging: Machine learning for 3D microscopy
https://www.nature.com/nature/journal/v523/n7561/full/523416a.html

12 Aug 2017
Deep Learning
Tumor Image Recognition
62
Source: https://www.nature.com/articles/srep24454
 Computer-Aided
Diagnosis with
Deep Learning
Architecture
 Breast tissue
lesions in images
and pulmonary
nodules in CT
Scans

12 Aug 2017
Deep Learning
Melanoma Image Recognition
63
Source: http://www.nature.com/nature/journal/v542/n7639/full/nature21056.html

12 Aug 2017
Deep Learning
DIY Image Recognition: use Contrast
64
Source: https://developer.clarifai.com/modelshttps://developer.clarifai.com/models
How many orange pixels?
Apple or Orange? Melanoma risk or healthy skin?
Degree of contrast in photo colors?

12 Aug 2017
Deep Learning
Deep Learning and Genomics
 Large classes of hypothesized but unknown correlations
 Genotype-phenotype disease linkage unknown
 Computer-identifiable patterns in genomic data
 CNN: genome symmetries; RNN: textual analysis
65
Source: http://ieeexplore.ieee.org/document/7347331

12 Aug 2017
Deep Learning
Deep Learning and the Brain
66

12 Aug 2017
Deep Learning
 Deep learning neural networks are inspired by the
structure of the cerebral cortex
 The processing unit, perceptron, artificial neuron is the
mathematical representation of a biological neuron
 In the cerebral cortex, there can be several layers of
interconnected perceptrons
67
Deep Qualia machine? General purpose AI
Mutual inspiration of neurological and computing research

12 Aug 2017
Deep Learning
Deep Qualia machine?
 Visual cortex is hierarchical with intermediate layers
 The ventral (recognition) pathway in the visual cortex has multiple
stages: Retina - LGN - V1 - V2 - V4 - PIT – AIT
 Human brain simulation projects
 Swiss Blue Brain project, European Human Brain Project
68

12 Aug 2017
Deep Learning
Social Impact of Deep Learning
 WHO estimates 400 million people without
access to essential health services
 6% in extreme poverty due to healthcare costs
 Next leapfrog technology: Deep Learning
 Last-mile build out of brick-and-mortar clinics
does not make sense in era of digital medicine
 Medical diagnosis via image recognition, natural
language processing symptoms description
 Convergence Solution: Digital Health Wallet
 Deep Learning medical diagnosis + Blockchain-
based EMRs (electronic medical records)
 Empowerment Effect: Deep learning = “tool I
use,” not hierarchically “doctor-administered”
69
Source: http://www.who.int/mediacentre/news/releases/2015/uhc-report/en/
Digital Health Wallet:
Deep Learning diagnosis
Blockchain-based EMRs

12 Aug 2017
Deep Learning
Agenda
 Deep Learning
 Definition
 Applications
 Conclusion
70

12 Aug 2017
Deep Learning 71
Better horse AND new car
New Technology

12 Aug 2017
Deep Learning 72
Smart networks are computing networks with
intelligence built in such that identification
and transfer is performed by the network
itself through protocols that automatically
identify (deep learning), and validate,
confirm, and route transactions (blockchain)
within the network
Smart Network Convergence Theory

12 Aug 2017
Deep Learning
Smart Network Convergence Theory
 Network intelligence “baked in” to smart networks
 Deep Learning algorithms for predictive identification
 Blockchains to transfer value, confirm authenticity
73
Source: Expanded from Mark Sigal, http://radar.oreilly.com/2011/10/post-pc-revolution.html
Two Fundamental Eras of Network Computing

12 Aug 2017
Deep Learning 74
Blockchain is the tamper-resistant
distributed ledger software underlying
cryptocurrencies such as Bitcoin, for the
secure transfer of money, assets, and
information via the Internet without a third-
party intermediary
Source: http://www.amazon.com/Bitcoin-Blueprint-New-World-Currency/dp/1491920491

12 Aug 2017
Deep Learning
Blockchain Deep Learning nets
 Provide increasingly sophisticated automated network
computational infrastructure
 Make predictive guesses of reality states of the world
 Predictive inference (deep learning) and cryptographic nonce-
guesses (blockchain)
 Instantiate decentralization
 Hierarchical models do not scale
75

12 Aug 2017
Deep Learning
Next Phase
 Put Deep Learning systems on the Internet
 Deep Learning Blockchain Networks
 Combine Deep Learning and Blockchain Technology
 Blockchain offers secure audit ledger of activity
 Advanced computational infrastructure to tackle
larger-scale problems
 Genomic disease, protein modeling, energy storage,
global financial risk assessment, voting, astronomical data
76

12 Aug 2017
Deep Learning
Example: Autonomous Driving
 Requires the smart network functionality
of deep learning and blockchain
 Deep Learning: identify what things are
 Convolutional neural nets core element of
machine vision system
 Blockchain: secure automation
technology
 Track arbitrarily-many fleet units
 Legal accountability
 Software upgrades
 Remuneration
77

12 Aug 2017
Deep Learning
The Very Small
Blockchain Deep Learning nets in Cells
 Medical nanorobotics for cell repair
 Deep Learning: identify what things are
(diagnosis)
 Blockchain: secure automation technology
 Bio-cryptoeconomics: secure automation
of medical nanorobotics for cell repair
 Medical nanorobotics as coming-onboard
repair platform for the human body
 High number of agents and “transactions”
 Identification and automation is obvious
78
Sources: Swan, M. Blockchain Thinking: The Brain as a DAC (Decentralized Autonomous Corporation). Technology and Society Magazine,
IEEE 2015; 34(4): 41-52 , https://www.slideshare.net/lablogga/biocryptoeconomy-smart-contract-blockchainbased-bionano-repair-dacs

12 Aug 2017
Deep Learning
The Very Large
Blockchain Deep Learning nets in Space
 Automated space
construction bots/agents
 Deep Learning: identify
what things are
(classification)
 Blockchain: secure
automation technology
 Applications: asteroid
mining, terraforming,
radiation-monitoring,
space-based solar power,
debris tracking net
79

12 Aug 2017
Deep Learning
Agenda
 Deep Learning
 Definition
 Applications
 Conclusion
80

12 Aug 2017
Deep Learning
Our human future
81
 Are we doomed?

12 Aug 2017
Deep Learning
Human-machine collaboration
82
 Team-members excel at different things
 Differently-abled agents in society
Source: Swan, M. (2017). Is Technological Unemployment Real? In: Surviving the Machine Age.
http://www.springer.com/us/book/9783319511641

12 Aug 2017
Deep Learning 83
Conceptual Definition:
Deep learning is a computer program that can
identify what something is
Technical Definition:
Deep learning is a class of machine learning
algorithms in the form of a neural network that
uses a cascade of layers (tiers) of processing
units to extract features from data and make
predictive guesses about new data
Source: Extending Jann LeCun, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/facebook-ai-director-yann-lecun-
on-deep-learning

12 Aug 2017
Deep Learning
Deep Learning Theory
 System is “dumb” (i.e. mechanical)
 “Learns” with big data (lots of input examples) and trial-and-error
guesses to adjust weights and bias to establish key features
 Creates a predictive system to identity new examples
 Same AI argument: big enough data is what makes a
difference (“simple” algorithms run over large data sets)
84
Input: Big Data (e.g.;
many examples)
Method: Trial-and-error
guesses to adjust node weights
Output: system identifies
new examples

12 Aug 2017
Deep Learning
3 Key Technical Principles of Deep Learning
85
Reduce combinatoric
dimensionality
Core processing unit
(input-processing-output)
Levers: weights and bias
Squash values into
probability function
(Sigmoid (0-1);
Tanh ((-1)-1))
Loss FunctionPerceptron StructureSigmoid Function
“Dumb” system learns by
adjusting parameters and
checking against outcome
Loss function
optimizes efficiency
of solution
Formulate as a logistic
regression problem for
greater mathematical
manipulation
What
Why

12 Aug 2017
Deep Learning
Conclusion
 Next-generation global infrastructure:
Deep Learning Blockchain Networks
merging deep learning systems and
blockchain technology
 Smart Network Convergence Theory:
pushing more complexity and
automation through Internet pipes
 Blockchain Deep Learning nets: Ability to
identify what something is (machine
learning) and securely verify and transact it
(blockchain)
86

12 Aug 2017
Deep Learning
 Neural Networks and Deep Learning, Michael Nielsen,
http://neuralnetworksanddeeplearning.com/
 Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron
Courville, http://www.deeplearningbook.org/Machine learning and deep neural nets
 Machine Learning Guide podcast, Tyler Renelle,
http://ocdevel.com/podcasts/machine-learning
 notMNIST dataset http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html
 Metacademy; Fast.ai; Keras.io
Resources
87
Distill (visual ML journal)
http://distill.pubSource: http://cs231n.stanford.edu
https://www.deeplearning.ai/

Melanie Swan
Philosophy Department, Purdue University
melanie@BlockchainStudies.org
Deep Learning Explained
The future of Smart Networks
Boulder Futurists: Solid State Depot Hackspace
Boulder CO, August 12, 2017
Slides: http://slideshare.net/LaBlogga
Image credit: Nvidia
Thank You! Questions?

12 Aug 2017
Deep Learning
Deep Learning Taxonomy
89
Source: Machine Learning Guide, 9. Deep Learning;
AI (artificial intelligence)
Machine learning Other methods
Supervised learning
(labeled data:
classification)
Unsupervised learning
(unlabeled data: pattern
recognition)
Reinforcement learning
Shallow learning
(1-2 layers)
Deep learning
(5-20 layers)
Recurrent nets (text, speech)
Convolutional nets (images)
Neural Nets (NN) Other methods
Bayesian inference
Support Vector Machines
Decision trees
K-means clustering
K-nearest neighbor

12 Aug 2017
Deep Learning
Kinds of Deep Learning Systems
What Deep Learning net to choose?
90
 Supervised algorithms (classify labeled data)
 Image (object) recognition
 Convolutional net (image processing), deep belief
network, recursive neural tensor network
 Text analysis (name recognition, sentiment
analysis)
 Recurrent net (iteration; character level text),
recursive neural tensor network
 Speech recognition
 Recurrent net
 Unsupervised algorithms (find patterns in
unlabeled data)
 Boltzmann machine or autoencoder

Deep Learning Explained

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Deep Learning Explained (20)

More from Melanie Swan (20)

Recently uploaded (20)

Deep Learning Explained