Chapter 3 Deep Learning architectures.pdf
- 1. 31 Pr. Meftah Boudjelal Chapter 3 Deep Learning Architectures
- 2. 32 Pr. Meftah Boudjelal 1. Introduction The evolution of deep learning has led to the development of diverse architectures, each tailored to address specific types of challenges across various domains. Whether it’s through CNNs for image analysis, RNNs for sequence understanding, GANs for content generation, autoencoders for data representation, or transformers for language tasks, these classes exemplify the versatility and power of deep learning in transforming technology and enhancing our capabilities in multiple fields. 2. Convolutional Neural Networks (CNNs) Convolutional Neural Networks (CNNs) are primarily used in image processing and computer vision tasks. They are designed to automatically and adaptively learn spatial hierarchies of features from images. The key component of CNNs is the convolutional layer, which applies various filters to the input data to extract important features such as edges, shapes, and textures. Pooling layers are typically used to reduce the spatial dimensions of the feature maps, thereby decreasing the computational load and helping to prevent overfitting. CNNs have revolutionized fields such as image classification, object detection, and segmentation. Popular examples include ResNet, VGG, and AlexNet, which have achieved state-of- the-art results in computer vision tasks. Imagine you have a large photograph, and you want to find specific patterns or features in it, like edges, shapes, or textures. You don’t look at the entire photo all at once—instead, you use a magnifying glass to scan small sections of the photo one at a time. 2.1. The Magnifying Glass = Convolutional Filter (Kernel) • The magnifying glass represents the convolutional filter (or kernel). It’s a small window that focuses on a tiny part of the image at a time. • The filter is designed to detect specific patterns, like edges, curves, or textures, depending on its configuration. 2.2. Scanning the Photo = Sliding the Filter Across the Image • You move the magnifying glass across the photograph, step by step, to examine every small section. This is similar to how the convolutional filter slides (or convolves) across the image. • At each step, the filter checks if the pattern it’s looking for (e.g., a vertical edge) exists in that section of the image.
- 3. 33 Pr. Meftah Boudjelal 2.3. Detecting Patterns = Feature Maps • When the magnifying glass finds a pattern it’s looking for (e.g., an edge), it highlights that area. In CNNs, this is represented by a feature map. • The feature map is a grid of values that shows where the filter detected the pattern. High values indicate a strong match, while low values mean the pattern wasn’t found. 2.4. Different Magnifying Glasses = Multiple Filters • You don’t just use one magnifying glass—you use many, each designed to detect a different pattern (e.g., one for edges, one for textures, etc.). • Similarly, a convolutional layer uses multiple filters, each learning to detect a different feature. This creates multiple feature maps, each capturing a unique aspect of the image. 2.5. Zooming Out = Hierarchical Feature Extraction • After scanning the photo with the magnifying glass, you might step back and look at the bigger picture. You notice that the small patterns (edges, textures) combine to form larger structures (shapes, objects). • In CNNs, this is done through multiple convolutional layers. Early layers detect simple features (edges, corners), while deeper layers combine these to detect more complex patterns (eyes, wheels, etc.). 2.6. Adjusting the Magnifying Glass = Learning Filters • At first, you might not know exactly what patterns to look for, so you adjust the magnifying glass (e.g., change its focus or shape) until it works well. • In CNNs, the filters are learned during training. The network adjusts the weights of the filters to better detect the most useful patterns for the task (e.g., recognizing cats vs. dogs). 2.7. Real-World Example: If you’re analyzing a photo of a cat: • The first convolutional layer might detect edges and textures (e.g., fur). • The next layer combines these to detect parts like ears or eyes. • Deeper layers assemble these parts to recognize the entire cat. 3. Transformer Networks Transformer networks have emerged as one of the most significant advancements in deep learning, particularly in NLP. Unlike RNNs, transformers rely on self-attention mechanisms to process input data in parallel rather than sequentially and have become the backbone of modern NLP models. This allows them to capture long-range
- 4. 34 Pr. Meftah Boudjelal dependencies more effectively and efficiently, making them suitable for handling large-scale datasets. Transformers consist of encoders and decoders, where the encoder processes the input text and transforms it into a context-rich representation, while the decoder generates the output—often used in tasks like machine translation and text summarization. The introduction of models like BERT, GPT, and T5 has further popularized transformers, enabling breakthroughs in understanding and generating human language. Imagine you have a large photograph spread out on a table, and you want to understand what’s happening in the picture. You could look at it in two ways: by examining small patches of the image one at a time or by stepping back and considering how all the patches relate to each other. Vision Transformers (ViTs) take the latter approach, and here’s how it works, using a metaphor: 3.1. Step 1: Breaking the Photo into Smaller Pieces Instead of looking at the entire photograph all at once, you decide to cut it into smaller, equally sized squares (patches). Each square represents a specific part of the image, like a corner of a building, a patch of sky, or a piece of a person’s face. In Vision Transformers, this is exactly what happens: the image is divided into smaller patches, which are then flattened into sequences of pixels. These patches are like puzzle pieces that, when combined, make up the whole picture. 3.2. Step 2: Scanning Each Piece for Patterns Now, you take each square and examine it closely to identify patterns or features. For example, you might notice that one patch contains horizontal lines (maybe a fence), while another has curves (perhaps part of a tree). In Vision Transformers, each patch is processed independently at first, and the model extracts basic features like edges, textures, or colors. This is similar to how Convolutional Neural Networks (CNNs) work, but ViTs don’t use convolutions—they treat each patch as a separate input. 3.3. Step 3: Connecting the Dots Between Pieces Here’s where Vision Transformers shine. Instead of just looking at each patch in isolation, you start thinking about how the patches relate to each other. For example, you might notice that the patch with horizontal lines (the fence) is next to a patch with green textures (the grass), and together they form a scene of a backyard. In ViTs, this is done using self-attention mechanisms. The model assigns importance (or "attention") to each patch based on how relevant it is to the others. This allows the model to understand the global context of the image, not just the local patterns.
- 5. 35 Pr. Meftah Boudjelal 3.4. Step 4: Building a Bigger Picture As you continue to analyze the patches and their relationships, you start to piece together the entire photograph. You might realize that the fence, grass, and tree patches are part of a park scene, while other patches form a person walking a dog. Similarly, Vision Transformers combine the information from all the patches to build a comprehensive understanding of the image. The model uses multiple layers of attention and transformation to refine its understanding, gradually recognizing more complex patterns like objects, shapes, and scenes. 3.5. Step 5: Making Sense of the Whole Scene Finally, you step back and see the photograph as a complete picture. You’ve not only identified individual elements (like the fence, tree, and person) but also understood how they fit together to tell a story. In Vision Transformers, this final step involves classifying the image (e.g., "a park scene with a person and a dog") or performing other tasks like object detection or segmentation. The model’s ability to consider both local details and global relationships makes it highly effective for understanding complex visual data. 3.6. Why This Metaphor Works • Patches as Puzzle Pieces: Just like cutting a photo into pieces, ViTs divide the image into patches to process them individually. • Attention as Contextual Understanding: The self-attention mechanism is like connecting the dots between patches to see the bigger picture. • Hierarchical Learning: As you move from examining individual patches to understanding the whole scene, ViTs similarly build a hierarchical understanding of the image. 3.7. Main Differences Between CNNs and ViTs Aspect CNNs Vision Transformers (ViTs) Focus Local details first, then global Global context first, then local Mechanism Convolutional filters Self-attention mechanisms Strengths Captures local patterns efficiently Captures long-range dependencies Data Efficiency Works well with smaller datasets Requires large datasets to train Computational Cost Lower computational cost Higher computational cost Use Cases Object detection, classification Complex scene understanding, global context tasks
- 6. 36 Pr. Meftah Boudjelal 4. Recurrent Neural Networks (RNNs) Recurrent Neural Networks (RNNs) are designed for processing sequences of data. RNNs maintain a hidden state that captures information about previous inputs in the sequence, allowing them to learn temporal dependencies. This makes RNNs particularly effective for tasks involving sequential data, such as natural language processing (NLP), speech recognition, and time series forecasting. A notable variant of RNNs is the Long Short-Term Memory (LSTM) network and Gated Recurrent Units (GRUs). They introduce mechanisms to better handle long-term dependencies by addressing issues like vanishing gradients. These models are widely used in natural language processing (NLP), speech recognition, and time series forecasting. Let’s use a metaphor about memory to explain the differences between Recurrent Neural Networks (RNNs), Long Short-Term Memory networks (LSTMs), and Gated Recurrent Units (GRUs). Imagine you’re trying to remember a story someone is telling you, but you have different types of memory systems to help you retain the information. 4.1. RNN: Short-Term Memory with Limited Capacity • Metaphor: Think of an RNN as someone with short-term memory who can only remember the most recent part of the story. As the story progresses, they keep forgetting earlier details because their memory is limited. • How It Works: RNNs process sequences (like sentences or time series) one step at a time, passing information from one step to the next. However, they struggle to remember information from many steps ago because their memory "fades" over time (this is called the vanishing gradient problem). • Limitation: Just like someone with short-term memory, RNNs are bad at remembering long-term dependencies (e.g., the beginning of a long story). 4.2. LSTM: Long-Term Memory with a Filing System • Metaphor: An LSTM is like someone with a filing system for their memory. They can decide what to remember for the long term, what to forget, and what to focus on right now. This filing system helps them keep track of important details, even if they were mentioned much earlier in the story. • How It Works: LSTMs use gates to control the flow of information: o Input Gate: Decides what new information to store in memory. o Forget Gate: Decides what old information to throw away. o Output Gate: Decides what information to use for the current step. • Advantage: LSTMs can remember long-term dependencies, making them great for tasks like translating long sentences or predicting trends over time.
- 7. 37 Pr. Meftah Boudjelal 4.3. GRU: A Simplified Filing System • Metaphor: A GRU is like someone with a simplified filing system. They still have a way to remember important details, but they don’t bother with as many folders or rules as the LSTM. They’re faster and more efficient but might not be as precise with very complex stories. • How It Works: GRUs also use gates, but they combine some of the LSTM’s components to make the system simpler: • Update Gate: Decides how much of the old memory to keep and how much new information to add. • Reset Gate: Decides how much of the past information to ignore when making new predictions. • Advantage: GRUs are faster to train and use fewer resources than LSTMs, while still being able to handle long-term dependencies reasonably well. 4.4. Main differences in Technical Terms Feature RNN LSTM GRU Memory Mechanism Simple memory cell Complex memory cell with gates Simplified memory cell with gates Gates None Input, Forget, Output gates Update, Reset gates Long-Term Dependencies Struggles with long- term memory Excels at long-term memory Handles long-term memory well Training Speed Fast Slower (more parameters) Faster (fewer parameters) Use Cases Short sequences Long sequences, complex tasks Long sequences, efficient tasks 4.5. Real-World Example: Imagine you’re translating a sentence: • An RNN might forget the subject of the sentence by the time it reaches the end. • An LSTM will remember the subject and use it to correctly translate the entire sentence. • A GRU will also remember the subject but might do so more efficiently, though it might occasionally make small mistakes on very complex sentences. 5. Generative Adversarial Networks (GANs) Generative Adversarial Networks (GANs) are used for generative modeling. They consist of two neural networks: a generator and a discriminator, which are trained simultaneously in a game-theoretic framework. The generator creates synthetic data
- 8. 38 Pr. Meftah Boudjelal from random noise, while the discriminator evaluates whether given data is real or fake. This adversarial process leads to the generation of high-quality, realistic data. GANs have shown remarkable success in a variety of applications, including image synthesis, video generation, and even music generation, leading to new possibilities in creative fields. Variants of GANs, such as StyleGAN and CycleGAN, have further expanded their applicability in art generation and style transfer. To understand how GANs work, let’s use an analogy about creating a painting. Imagine two artists: a forger (the generator) and an art critic (the discriminator). Together, they engage in a creative duel to produce a masterpiece that looks indistinguishable from a real painting. 5.1. Step 1: The Forger’s First Attempt (Generator’s Initial Output) The forger starts by creating a rough sketch of a painting. At this stage, the painting is clearly fake—it lacks detail, depth, and realism. Similarly, the generator in a GAN begins by producing a random output, such as a blurry image or a nonsensical pattern. This output is generated from random noise (like a blank canvas) and is far from resembling real data. 5.2. Step 2: The Art Critic’s Evaluation (Discriminator’s Judgment) The art critic examines the forger’s painting and compares it to real masterpieces. The critic can easily tell that the painting is fake because it lacks the sophistication and detail of genuine art. In a GAN, the discriminator plays the role of the critic. It takes the generator’s output and compares it to real data (e.g., actual images from a dataset). The discriminator then assigns a probability score indicating how likely it is that the input is real. 5.3. Step 3: Learning from Feedback (Generator Improves) The forger receives feedback from the critic and uses it to improve their painting. They might add more details, refine the colors, or adjust the composition to make the painting look more realistic. Similarly, the generator in a GAN uses the feedback from the discriminator to improve its output. It adjusts its parameters (weights) through backpropagation to create data that is more likely to fool the discriminator. 5.4. Step 4: The Critic Gets Better Too (Discriminator Improves) As the forger improves, the art critic also becomes more discerning. They learn to spot even the subtlest flaws in the forger’s work. In a GAN, the discriminator also improves over time. It learns to better distinguish between real and fake data by updating its own parameters. This creates a dynamic where both the generator and discriminator are constantly evolving, pushing each other to get better.
- 9. 39 Pr. Meftah Boudjelal 5.5. Step 5: The Creative Duel Continues (Training Process) The forger and critic continue this back-and-forth process. Each time the forger creates a new painting, the critic evaluates it, and the forger uses the feedback to refine their technique. Over time, the forger’s paintings become so realistic that even the critic struggles to tell them apart from genuine masterpieces. In a GAN, this iterative process continues until the generator produces data that is virtually indistinguishable from real data. 5.6. Step 6: The Final Masterpiece (Generating New Data) Eventually, the forger creates a painting that is so convincing that it could hang in a gallery alongside real masterpieces. Similarly, once the GAN is fully trained, the generator can produce new data that looks authentic. For example, if the GAN was trained on images of faces, it can generate realistic-looking faces of people who don’t exist. If it was trained on paintings, it can create new artworks in the style of the training data. 5.7. Why This Analogy Works • Generator as the Forger: The generator is like an artist trying to create something realistic. • Discriminator as the Critic: The discriminator is like an expert who evaluates the quality of the work. • Adversarial Process as a Creative Duel: The competition between the generator and discriminator drives both to improve, just as the forger and critic push each other to excel. 6. Autoencoders Autoencoders are unsupervised neural networks used primarily for dimensionality reduction and feature learning. An autoencoder consists of two main parts: an encoder that compresses the input data into a lower-dimensional representation and a decoder that attempts to reconstruct the original data from this compressed form. By training the model to minimize the reconstruction error, autoencoders can learn efficient representations and capture the underlying structure of the data. They are particularly effective for tasks like anomaly detection, denoising images, and data compression. Variational Autoencoders (VAEs) extend the basic concept of autoencoders by incorporating probabilistic modeling, making them useful for generative tasks as well. To understand how they work, let’s use an analogy about creating variations of a song. Imagine you’re a musician who wants to create new versions of a song while keeping its core melody intact.
- 10. 40 Pr. Meftah Boudjelal 6.1. Step 1: Learning the Core Melody (Encoding) First, you listen to the original song repeatedly to understand its core melody, rhythm, and structure. This is like the encoding phase of an autoencoder. The encoder part of the network takes the input data (e.g., an image, a song, or any other type of data) and compresses it into a lower-dimensional representation called the latent space. This latent space captures the most important features of the data, just as you’ve distilled the song into its core melody. 6.2. Step 2: Storing the Essence (Latent Space) Once you’ve identified the core melody, you store it in your mind as a simplified version of the song. Similarly, the autoencoder stores the compressed representation of the data in the latent space. This latent space acts as a "summary" of the data, containing only the most essential information needed to reconstruct it. For example, if the input is an image of a face, the latent space might capture features like the shape of the eyes, nose, and mouth, but not the fine details like individual pixels. 6.3. Step 3: Creating Variations (Sampling from the Latent Space) Now, as a musician, you start experimenting with the core melody to create new variations. You might change the tempo, add new instruments, or tweak the harmonies. In autoencoders, this is analogous to sampling from the latent space. By slightly modifying the values in the latent space, you can generate new data points that are similar to the original but have unique variations. For example, if the latent space represents faces, tweaking the values might result in a new face with slightly different features, like wider eyes or a different nose shape. 6.4. Step 4: Reconstructing the Song (Decoding) Once you’ve created a new variation of the melody, you play it back to see how it sounds. This is like the decoding phase of an autoencoder. The decoder part of the network takes the modified latent representation and reconstructs it into a full-fledged data point. For instance, if the input was an image, the decoder generates a new image that resembles the original but has unique variations based on the changes made in the latent space. 6.5. Step 5: Refining the Output (Training the Autoencoder) As a musician, you might not get the perfect variation on the first try. You’ll likely refine your new version by listening to it, comparing it to the original, and making adjustments. Similarly, autoencoders are trained by comparing the reconstructed output to the original input and minimizing the difference between them (using a loss function like Mean Squared Error). Over time, the autoencoder gets better at generating realistic variations because it learns to capture the essential features of the data more accurately.
- 11. 41 Pr. Meftah Boudjelal 6.6. Types of Autoencoders for Data Generation While standard autoencoders can reconstruct data, some specialized variants are better suited for generating new data: o Variational Autoencoders (VAEs): These introduce randomness into the latent space, allowing for smoother and more diverse variations. In our song analogy, this would be like adding improvisation to the melody, creating more unique and creative versions. o Denoising Autoencoders: These are trained to reconstruct clean data from noisy inputs, making them robust for generating high-quality outputs even when the input is imperfect. 6.7. Why This Analogy Works • Encoding as Learning the Core: Just as you distill a song into its core melody, the encoder extracts the essential features of the data. • Latent Space as a Creative Playground: The latent space is like a musician’s mind, where variations are imagined and experimented with. • Decoding as Reconstructing the Song: The decoder brings the new variations to life, just as a musician plays back a new version of the song.