Advance deep learning

Takeoff Edu Group
https://takeoffprojects.com/advanced-deep-learning-projects
Takeoff Edu Group
Deep Learning Techniques: An Overview
Abstract:
Advance Deep Learning is a class of machine learning which performs much better on unstructured data. Deep learning techniques are
outper- forming current machine learning techniques. It enables computational models to learn features progressively from data at
multiple levels. The popularity of deep learning amplified as the amount of data available increased as wellas theadvancement of
hardwarethat provides powerful computers. This article comprises of the evolution of deep learning, vari- ousapproachestodeep
learning,architecturesofdeep learning,methods, and applications.
Keywords: Deep Learning (DL), Recurrent Neural Network (RNN), Deep Belief Networks (DBN), Convolutional Neural Networks(CNN),
Generative Adversarial Networks(GAN)
Introduction
Deep learning techniques which implement deep neural networks became pop- ular due to the increase of high-performance
computing facility. Deep learning achieves higher power and flexibility due to its ability to process a large number offeatures when itdeals
with unstructured data. Deep learningalgorithmpasses the data through several layers; each layer is capable of extracting features pro-
gressively and passes ittothenextlayer.Initiallayers extractlow-levelfeatures, and succeeding layers combines features to form a
complete representation. Sec- tion 2 gives an overview of the evolution of deep learning models. Section 3 provides a brief idea about
the different learning approaches, such as supervised learning, unsupervised learning, and hybrid learning. Supervised learning uses
labeled data to train the neural network. In supervised learning, the network uses unlabeled data and learns the recurring patterns.
Hybrid learning combines supervised and unsupervised methods to get a better result. Deep learning can be implemented using
different architectures such as architectures like Unsuper- vised Pre-trained Networks, Convolutional Neural Networks, Recurrent Neural
Networks, and Recursive Neural Networks, which are described in section 4. Section 5 introduces various training methods and
optimization techniques that help in achieving better results. Section 6 describes the frameworks which allow us to develop tools that
offer a better programming environment. Despite the various challenges indeep learning applications, many excitingapplications that
may rule the world are briefed in Section 7.

Takeoff Edu Group
Takeoff Edu Group
Evolution of Deep Learning
First Generation of Artificial Neural networks(ANN) was composed of per- ceptrons in neural layers, which were limited in
computations. The second- generation calculated the error rate and backpropagated the error. Restricted Boltzmann machine
overcame the limitation of backpropagation, which made the learningeasier. Then other networks areevolved eventually [15,24].
Figure.1 illustrates a timeline showing the evolution of deep models along with the traditional model. The performance of classifiers
using deep learning improves on a large scale with an increased quantity of data when compared to traditional learningmethods.
Figure.2depictstheperformanceoftraditionalmachinelearn- ing algorithms and deep learning algorithms [6]. The performance of
traditional machine learning algorithms becomes stable when it reaches the threshold of training data whereas the deep learning
upturns it’s performance with increased amountofdata.Nowadaysdeeplearningisusedinalotmanyapplicationssuch as Google’s voice
and image recognition, Netflix and Amazon’s recommendation engines, Apple’s Siri, automatic email and text replies, chatbots etc.
Fig. 1: Evolution of Deep Models

Takeoff Edu Group
Takeoff Edu Group
Fig. 2: Why Deep Learning ?
Deep Learning Approaches
Deep neural networks are successful in Supervised learning, Unsupervised learning, Reinforcement learning, as well as hybrid learning.
Supervised Learning
In supervised learning, the input variables represented as X are mapped to output variables represented as Y by using an algorithm to
learn the mapping function f.
Y = f (X)(1)
The aim of the learning algorithm is to approximate the mapping function to predict the output (Y) for a new input (X). The error from the
predictions made duringtrainingcan be usedtocorrecttheoutput. Learningcan be stopped when all the inputs are trained to get the
targeted output [11]. Regression for solving regression problems [18], Support Vector machines used for classification [21]], Random
forest for classification as well as regression problems [20].
Unsupervised Learning
Inunsupervised learning,wehavetheinput dataonly andnocorresponding output to map. This learning aims to learn about data by
modeling the distribution in data. Algorithms can be able to discover the exciting structure present in the data. Clustering problems and
association problems use Unsupervised learning. The unsupervised learning algorithms such as K-means algorithm is used in clustering
problems [9], Apriori algorithm is used in association problems[10]

Takeoff Edu Group
Takeoff Edu Group
Reinforcement Learning
Reinforcement learning uses a system of reward and punishment to train the algorithm. In this, the algorithm or an agent learns from
its environment. The agent gets rewards for correct performance and penalty for incorrect performance. For example, consider the
case of a self-driving car, the agent gets a reward for driving safelyto destination and penalty for goingoff-road. Similarly, in the case of a
program for playing chess, the reward state may be winning the game and the penalty for being checkmated. The agent tries to
maximize the reward and minimizethepenalty.Inreinforcement learning, thealgorithmis not told how to perform the learning; however,
it works through the problem onits own [16].
Hybrid Learning
Hybrid learning refers to architectures that make use of generative (unsupervised) as well as discriminative (supervised) components.
The combination of different architectures can be used todesignahybrid deep neuralnetwork.They are used for action recognition of
humans using action bank features and are expected to produce much better results [3].
Fundamental deep learning architectures
DeeplearningarchitecturesperformbetterthansimpleANN,eventhoughtrain- ing time of deep structures are higher than ANN.
However, training time can be reduced using methods such as transfer learning, GPU computing. One of the factors which decide the
success of neural networks lies in the careful design of network architecture. Some of the relevant deep learning architectures are
discussed below.
Unsupervised Pre-trained Networks
In unsupervised pre-training, a model is trained unsupervised, and then the model used for prediction. Some unsupervised pre-training
architectures are discussed below [4].
Autoencoders : are used for the reduction of the dimension of data, novelty detectionproblems,aswellasinanomalydetection
problems.Inanautoencoder, the first layer is built as an encoding layer and transpose of that as a decoder. Then train it to recreate the
input using the unsupervised method. Aftertrain- ing, fix the weights of that layer. Then move to the subsequent layer until we pre-train
all the layers of deep net. Then go back to the original problem that we want to solve with deep net (Classification/Regression) and
optimize it with Stochastic gradient descent by starting from weights learned using pre-training.

Takeoff Edu Group
Takeoff Edu Group
Autoencodernetworkconsistsoftwoparts[7].Theinputistranslatedtoalatent space representation by the encoder, which can be
denoted as:
h = f (x)(2)
The input is reconstructed from the latent space representation by the decoder, which can be denoted as:
r = g(h)(3)
In essence, autoencoders can be described as in equation (4). r is the decoded output which will be similar to input x :
g(f (x))= r (4)
Deep Belief Networks: The first step for training the deep belief network is to learn features using the first layer. Then use the
activation of trained features in the next layer. Continue this until the final layer. Restricted Boltzmann Machines (RBM) is used to train
layers of the Deep Belief Networks (DBNs), and the feed-forward network is used for fine-tuning. DBN learns hidden pattern globally,
unlike other deep nets where each layer learns complex patterns progressively [19].
GenerativeAdversarialNetworks:GenerativeAdversarialNetworks(GAN) were presented by Ian Goodfellow.It comprises of Generator
network and Dis- criminator network. Generator generates the content while the discriminator validates thegenerated content.
Generator createsnatural-lookingimages,while the discriminator decides whether the image looks natural. GAN is considered as a
minimax two-player algorithm. GANs uses convolutional and feed-forward Neural Nets [5].
Convolutional Neural Networks
Convolutional Neural Networks (CNN) are used mainly for images. It assigns weights and biases to various objects in the image and
differentiates one from theother.Itrequireslesspreprocessingrelated tootherclassificationalgorithms. CNN uses relevant filters to
capture the spatial and temporal dependencies in an image [12,25]. The different CNN architectures include LeNet, AlexNet, VG- GNet,
GoogleNet, ResNet, ZFNet. CNN’s are mainly used in applications such as Object Detection, Semantic Segmentation, Captioning.
Recurrent Neural Networks
Inrecurrentneuralnetworks(RNN),outputsfromtheprecedingstatesarefedas inputtothecurrentstate.ThehiddenlayersinRNNcan
rememberinformation. Thehidden stateisupdatedbased ontheoutput generatedinthe previousstate. RNN can be used for time series
prediction because it can remember previous inputs also, which is called Long-Short Term Memory [2].

Takeoff Edu Group
Takeoff Edu Group
Deep learning methods
Some of the powerful techniques that can be applied todeep learning algorithms toreducethetrainingtimeandtooptimizethemodelare
discussedinthefollow- ingsection.ThemeritsanddemeritsofeachmethodarecomprisedintheTable1
Back propagation : While solving an optimization problem using agradient- based method, backpropagation can be used to calculate
the gradient of the function for each iteration [18].
StochasticGradientDescent:Usingtheconvexfunctioningradientdescent algorithms ensures finding an optimal minimum without
getting trapped in a local minimum. Depending upon the values of the function and learning rate or step size,itmayarriveatthe
optimumvalueindifferentpathsandmanners[14].
Learning Rate Decay :Adjusting the learning rate increases the performance and reduces the training time of stochastic gradient descent
algorithms. The widely used technique is to reduce the learning rate gradually, in which wecan make large changes at the beginning and
then reduce the learning rate gradually inthetrainingprocess.This allows fine-tuningthe weights inthelaterstages [7].
Dropout:Theoverfittingproblemindeepneuralnetworkscanbeaddressedus- ing the drop out technique. This method is applied by
randomly dropping units and their connections during training [9]. Dropout offers an effective regularization method to reduce
overfitting and improve generalization error.Dropout gives an improved performance on supervised learning tasks in computer vision,
computational biology, document classification, speech recognition [1].
Max-Pooling: In max-pooling a filter is predefined, and this filter is then applied across the nonoverlapping sub-regions of the input
taking the max of the values contained in the window as the output. Dimensionality, as well as the computational cost to learn several
parameters, can be reduced using max- pooling [23].
Batch Normalization: Batch normalization reduces covariate shift, thereby accelerating deep neural network. It normalizes the inputs to
a layer, for each mini-batch, when the weights are updated during the training. Normalization stabilizes learning and reduces the
training epochs. The stability of a neural network can be increased by normalizing the output from the previous activation layer [8].
Skip-gram : Word embedding algorithms can be modeled using Skip-gram. In the skip-gram model, two vocabulary terms share a similar
context; then those terms are identical. For example, the sentences ”cats are mammals” and ”dogs are mammals” are meaningful
sentences which shares the same meaning ”are mammals.” Skip-gram can be implemented by considering a context win-

Takeoff Edu Group
Takeoff Edu Group
dow containing n terms and train the neural network by skipping one of this term and then use the model to predict skipped term [13].
Transfer learning: In transfer learning, a model trained on a particular task is exploited on another related task. The knowledge
obtained while solving a particular problem can be transferred to another network, which is tobe trained on a related problem. This
allows for rapid progress and enhancedperformance while solving the second problem [17].
Table 1: Comparison of Deep learning methods
Method Description Merits Demerits
Back
propagation
Used in Optimization
problem
For calculation
of gradient
Sensitive to noisy data
Stochastic
Gradient
Descent
To find optimal
minimum in
optimization
problems
Avoids trapping in local
minimum
Longer convergence
time, computationally
expensive
Learning
Rate Decay
Reduce learning
rate gradually
Increases performance,
Reduces training time
Computationally
expensive
Dropout Dropsout units/
connection during
training
Avoids overfitting Increases number
of iterations required
to converge
Max-Pooling Applies a max filter Reduces dimension
and computational
cost
Considers only the
maximum element
which may lead to
unacceptable result in
some cases
Batch
Normalization
Batch-wise
normalization
of input to a layer
Reduces covariant shift,
Increases stability of the
network,
Network trainsfaster,
Allows higher
learning rates
Computational
overhead during
training
Skip-gram Used in word
embedding
algorithms
Can work on any raw text,
Requires less memory
Softmax function is
computationally
expensive, Training
Time is high
Transfer
learning
Knowledge of
first model is
transferred to
second problem
Enhances performance,
Rapid progress in
training of second
problem
Works with similar
problems only

Takeoff Edu Group
Takeoff Edu Group
Deep learning frameworks
A deep learning framework helps in modeling a network more rapidly without going into details of underlying algorithms. Each
framework is built for different purposesdifferently.Somedeeplearningframeworksarediscussedbelowandare summarized in Table 2.
TensorFlow TensorFlow, developed by Google brain, supports languages such as Python,C++andR.Itenablesustodeployourdeep
learningmodelsinCPUs as well as GPUs [22].
Keras Keras is an API, written in Python and run on top of TensorFlow. It enables fast experimentation. It supports both CNNs and RNNs
and runs on CPUs and GPUs [22].
PyTorchPyTorch can be used for building deep neural networks as well as ex- ecuting tensor computations. PyTorch is a Python-based
package thatprovides Tensor computations. PyTorch delivers a framework to create computational graphs [22].
CaffeYangqing Jia developed Caffe, and it is open source as well. Caffe stands out from other frameworks in its speed of processing as
well as learning from images. Caffe Model Zoo framework facilitates us to access pre-trained models, which enable us to solve various
problems effortlessly [22].
Deeplearning4j Deeplearnig4j is implemented in Java, and hence, it is more efficientwhencomparedtoPython.TheND4Jtensorlibrary
usedbyDeeplearn- ing4j provides the capability to work with multi-dimensional arrays or tensors. This framework supports CPUs and
GPUs. Deeplearnig4j works with images, csv as well as plaintext [22].
Table 2: Comparison of Deep Learning Frameworks
Deep Learning
Framework
Release
Year
Language
written in
CUDA
supported
Pre-trained
models
TensorFlow 2015 C++, Python Yes Yes
Keras 2015 Python Yes Yes
PyTorch 2016 Python, C Yes Yes
Caffe 2013 C++ Yes Yes
Deeplearning4j 2014 C++, Java Yes Yes

Takeoff Edu Group
Takeoff Edu Group
Applications of Deep Learning
Deep learning networks can be used in a variety of applications such as self- driving cars, Natural Language Processing, Google’s Virtual
Assistant, Visual Recognition, Frauddetection, healthcare, detecting developmental delay in chil- dren, adding sound to silent movies,
automatic machine translation, text to image translation, image to image synthesis, automatic image recognition, Image colorization,
earthquake prediction, market-rate forecasting, news aggregation and fraud news detection.
Conclusion
Deep learning is continuously evolving faster; still, there are a number ofprob- lems to deal with and can be solved using deep learning.
Even though a full understanding of the working of deep learning is still a mystery, we can make machines smarter using Deep learning,
sometimes even smarter than human. Now the aim is to develop deep learning models that work with mobile to make theapplications
smarterandmoreintelligent.Letdeeplearningbemoredevoted to the betterment of humanity and thus making our domain a better
place to live.
References
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE
transactions on pattern analysis and machine intelligence, 40(12):2897–2905, 2018. doi: 10.1109/TPAMI.2017. 2784440.
Filippo MariaBianchi,EnricoMaiorino, MichaelCKampffmeyer, Antonello Rizzi, and Robert Jenssen. An overview and comparative
analysis of recurrent neural networks for short term load forecasting. arXiv preprint arXiv:1705.04378,2017.
Li Deng, Dong Yu, et al. Deep learning: methods and applications. Founda-
tions and TrendsⓍ
R
978-981-13-3459-7 3.
in Signal Processing, 7(3–4):197–387, 2014. doi: 10.1007/
Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pas- cal Vincent, and Samy Bengio. Why does unsupervised
pre-training help deep learning? Journal of Machine Learning Research, 11(Feb):625–660, 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets. In Advances in neural information processing systems, pages 2672–2680,2014.
Palash Goyal, Sumit Pandey, and Karan Jain. Introduction to natural language processing and deep learning. In Deep Learning for Natural
Language Processing, pages 1–74. Springer, 2018. doi: 10.1007/978-1-4842-3685-7 1.
Nathan Hubens. Deep inside: Autoencoders - towards data science, Apr 2018. 8.Sergey Ioffe and Christian Szegedy. Batch
normalization: Accelerating
deep network training by reducing internal covariate shift.arXiv preprint arXiv:1502.03167, 2015.

Takeoff Edu Group
Takeoff Edu Group
AnilK Jain. Data clustering: 50 years beyond k-means. Pattern recognition letters, 31(8):651–666, 2010. doi:
10.1016/j.patrec.2009.09.011.
Sotiris Kotsiantis and Dimitris Kanellopoulos. Association rules mining: A recent overview. GESTS International TransactionsonComputer
Science andEngineer- ing, 32(1):71–82, 2006.
Sotiris B Kotsiantis, I Zaharakis, and P Pintelas. Supervised machine learning: A review of classification techniques. Emerging artificial
intelligence applications in computer engineering, 160:3–24, 2007.
Quoc V Le et al. A tutorial on deep learning part 2: Autoencoders, convolutional neural networks and recurrent neural networks. Google
Brain, pages1–20, 2015. 13.Chaochun Liu, Yaliang Li, Hongliang Fei, and Ping Li. Deep skip-gram networks
for text classification. In Proceedings of the 2019 SIAM International Conference on Data Mining, pages 145–153. SIAM, 2019.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernetworks. arXiv preprint
arXiv:1802.09419, 2018.
Risto Miikkulainen, Jason Liang, Elliot Meyerson, Aditya Rawal, Daniel Fink, Olivier Francon, Bala Raju, Hormoz Shahrzad, Arshak
Navruzyan, Nigel Duffy, et al. Evolving deep neural networks. In Artificial Intelligence in the Age of Neural Networks and Brain
Computing, pages 293–312. Elsevier, 2019. doi: 10. 1016/B978-0-12-815480-9.00015-3.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timo- thy Lillicrap, Tim Harley, David Silver, and Koray
Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–
1937, 2016.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345–1359,
2009. doi: 10.1109/TKDE.2009.191.
Abhishek Panigrahi, Yueru Chen, and C-C Jay Kuo. Analysis on gradient propagation in batch normalized residual networks. arXiv
preprint arXiv:1812.00342, 2018.
RuslanSalakhutdinovandGeoffreyHinton. Semantichashing. InternationalJour- nal of Approximate Reasoning, 50(7):969–978, 2009.
doi: 10.1016/j.ijar.2008.11. 006.
Bernhard Scholkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond.
MIT press, 2001.
George AF Seber and Alan J Lee. Linear regression analysis, volume 329. John Wiley & Sons, 2012.
Pulkit Sharma. Top 5 deep learning frameworks, their applications, and compar- isons!, May 2019.
Toshihiro Takahashi. Statistical max pooling with deep learning, July 3 2018. US Patent 10,013,644.
Bhiksha Wang, HaohanandRaj. On the origin of deep learning. arXiv preprint arXiv:1702.07800, 2017.
Rikiya Yamashita, Mizuho Nishio, Richard Kinh Gian Do, and Kaori Togashi. Convolutional neural networks: an overview and application
in radiology. Insights into imaging, 9(4):611–629, 2018. doi: 10.1007/s13244-018-0639-9.

Advance deep learning

More Related Content

What's hot (20)

Similar to Advance deep learning (20)

Recently uploaded (20)

Advance deep learning