Compressing and Sparsifying LLM in GenAI Applications

Editor's Notes

• #9: In language translation, think of these as punction experts, article experts
• #10: Learning Process: The routing mechanism is not static; it learns and adapts through training. As the MoE model is exposed to more data, the routing mechanism refines its criteria for selecting experts for different types of input tokens. Expert Specialization: Over time, this learning process leads to the specialization of experts. Each expert becomes more adept at handling specific types of input tokens, improving the overall efficiency and effectiveness of the model. Reduced Computation: At inference time, the routing mechanism's learned behavior allows the model to use only a fraction of its total computational resources for each input token. Instead of engaging all experts, only the most relevant expert(s) for a given input token are activated. Scalability: This approach significantly reduces the computational load, making the MoE model more scalable and efficient. The computational cost for processing an input token becomes roughly proportional to 1 divided by the number of experts, assuming each expert handles an equal share of the workload over time.
• #11: This graph compares the speedup factor of (MoE) over dense training in terms of zetta floating-point operations per second (ZFLOPs) across three different scenarios X-Axis: Dense Training ZFLOPs - The x-axis represents the computational power used for training language models, measured in zetta floating-point operations per second (ZFLOPs). The scale is linear, ranging from 1 to 20 ZFLOPs. Y-Axis: MoE Speedup Factor - The y-axis quantifies the speedup factor of Mixture of Experts (MoE) models compared to traditional dense training models. The scale is linear, ranging from 1 to above 15, indicating how many times faster the MoE model is than its dense counterpart. Data Series: - Blue Line (In-domain LM): This line shows the speedup factor for MoE models when used for in-domain language modeling tasks. The line has an upward trend, indicating that as the computational power increases, the MoE models become significantly faster than dense models, particularly after 10 ZFLOPs, reaching a peak speedup of over 15 times faster at around 15 ZFLOPs before declining slightly. - Red Line (Out-of-domain LM): This line represents the speedup factor for MoE models in out-of-domain language modeling tasks. The speedup factor increases more gradually compared to the in-domain tasks, peaking at around 6 times faster at approximately 12 ZFLOPs and then declining. - Yellow Line (Zero-shot priming): This line shows the speedup factor for MoE models when applied to zero-shot priming tasks. The speedup factor remains relatively constant and much lower than the other two scenarios, hovering around 3 to 4 times faster than dense models across all ZFLOPs. Key Observations: - In-domain language modeling benefits the most from MoE models, especially as the computational power increases beyond 10 ZFLOPs. - The advantage of MoE models in out-of-domain language modeling grows at a slower rate and peaks at a lower value than in-domain modeling, indicating that MoE models may be more specialized. - Zero-shot priming shows the least benefit from MoE models, suggesting that MoE may not be as effective for tasks requiring generalization without prior examples. Interpretation: - MoE models can greatly accelerate language modeling tasks by leveraging increased computational resources, particularly for in-domain tasks. - There's a point of diminishing returns for both in-domain and out-of-domain LM after certain computational thresholds, indicating an optimal ZFLOPs range for MoE efficiency. - Zero-shot priming's consistent speedup factor across different computational powers might suggest that for such tasks, MoE models do not benefit as much from additional computational resources as they do for specific language modeling tasks.
• #12: The speed-up factor is calculated using a function c(t), which is an exponential function. Components of the Speed-Up Function: c(t): This represents the computed cost function at a given performance level t. r: This is a ratio representing how far the performance level t is between the lower and higher reference performances t_lo and t_hi. Interpolation: The speed-up factor interpolates the computational cost between two known data points: t_lo and t_hi, which are the closest performances to t from the available models that are lower and higher than t, respectively. C_lo(t) and C_hi(t) are the corresponding training costs for t_lo and t_hi in ZFLOPs. Speed-Up Factor Formula: The speed-up factor is given by the ratio of the computational cost of the dense model at performance t (C_dense(t)) to the computational cost of the MoE model at the same performance (C_moe(t))
• #15: evaluate models on their ability to predict the next token in a sequence as measured by perplexity. Similar to training, they concatenate all documents in a given dataset using empty lines as separators, split the resulting sequence into non- overlapping blocks of 2048 tokens, and score each block independently, evaluate and report perplexity in both in- domain and out-of-domain settings. In-domain, we sample a held-out subset of the combined pretraining data. For out-of-domain they use data from The Pile a public dataset that combines data from 22 diverse sources
• #16: Overall, these graphs are suggesting that MoE models are not only more parameter-efficient but also scale better with computational resources compared to dense models, both when dealing with familiar data and when generalizing to new, unseen data.
• #18: Why are we scaling?
• #22: This equation denotes that for a given input token xs​, a gating mechanism determines which experts should be activated. The gating function, represented as GATE, outputs a vector Gs,E​ that contains a gating score for each expert EE. These scores determine the weight or importance of each expert's contribution for the given input. This represents the computation within each expert. Each expert e is a feed-forward neural network, which first applies an input projection (multiplying the input token xsxs​ by a weight matrix wiewie​), then applies a ReLU activation function, and finally applies an output projection woe​. The ReLU activation function introduces non-linearity into the model, allowing it to learn complex patterns. The final output ys​ for the input token xs​ is computed as the weighted sum of the outputs from all experts. The weights for this sum come from the gating scores Gs,EGs,E​, ensuring that only the outputs of the relevant experts, as determined by the gating function, are used in computing the final output
• #23: The auxiliary loss lauxlaux​ is defined as: laux=1E∑e=1EceS⋅melaux​=E1​∑e=1E​Sce​​⋅me​ Here’s the intuition behind the formula: We want to minimize the variance in how much each expert is used, which would be represented by the squared term (ceS)2(Sce​​)2. However, cece​ isn’t differentiable because it comes from a discrete counting process. Instead, we use meme​ as a stand-in for ceSSce​​ because it is differentiable. It represents the average gate activation for expert ee, which we can backpropagate through. The loss function penalizes the model when the usage of experts deviates from the mean gate value meme​ weighted by the proportion of tokens ceSSce​​. This encourages the training process to steer gate activations toward a uniform distribution.
• #37: Introducing ColT5
• #53: In COLT5, applying conditional computation in the decoder is complex because the model's routing method is not causal, generation of an output token at a certain position should not be influenced by tokens that come after it, which is necessary for tasks like language modeling where the model generates text one token at a time.
• #55: So by now we all are aware of the difficulties of deploying large language models on a bigger scale, we have ofcourse the high compute requirements, the time it takes for training and the environmental impact of these high-energy computations. Now as harrum discussed sparsifying LLMs is one way of helping with that problem where we are reducing the number of parameters or computations needed for a given task. Another way to save memory and computation power is to quantize the parameters and variables of a model.
• #56: I am going to go over the basics of quantization just so understanding the papers is easier, quantization is a process by which we reduce the precision or significant digits of a value to represent it in a smaller number of bits. Basically, a more memory efficient representation of a number. This most of the times comes with a loss in accuracy.
• #57: There are different techniques of quantization, basic techniques are uniform and non-uniform quantization where we will divide a distribution into intervals and quantize those intervals with a single value. Dynamic quantization is when we reduce the number of bits a model’s parameters will take during runtime, block-wise quantization is when we divide the data into chunks and quantize those chuncks independently.
• #58: Another cool quantization is mixed-precision where we will choose the quantization based on the magnitude, smaller magnitude values may be reduced to a smaller bit representation like float 16 to int 8 while large magnitude values may be left as is. This way we do compress most of the values while keeping the outlier values full-precision, this gives a good balance between efficiency and precision. Now moving to the papers under study, 
• #62: LLM.int8() maintains full 16-bit performance as we scale from 125M to 175B parameters. On the other hand, the baseline, 8-bit absmax vector-wise quantization, scales poorly and degenerates into random performance.
• #67: in matrix multiplication, where each row of one matrix and each column of another matrix are treated as independent vectors. By using separate scaling constants for each vector, vector-wise quantization improves the precision of the quantization, as it adjusts to the specific distribution of each vector.
• #68: The equation provided is essentially saying that you can perform matrix multiplication by breaking down the process into smaller, more manageable parts (the inner products of rows and columns), each scaled appropriately. Then, to get back to the original scale, you adjust these results using the outer product of the scaling constants.
• #69: in matrix multiplication, where each row of one matrix and each column of another matrix are treated as independent vectors. By using separate scaling constants for each vector, vector-wise quantization improves the precision of the quantization, as it adjusts to the specific distribution of each vector.
• #70: The equation provided is essentially saying that you can perform matrix multiplication by breaking down the process into smaller, more manageable parts (the inner products of rows and columns), each scaled appropriately. Then, to get back to the original scale, you adjust these results using the outer product of the scaling constants.
• #71: The equation provided is essentially saying that you can perform matrix multiplication by breaking down the process into smaller, more manageable parts (the inner products of rows and columns), each scaled appropriately. Then, to get back to the original scale, you adjust these results using the outer product of the scaling constants.
• #72: absmax, row-wise, and zeropoint quantization fail as we scale, where models after 2.7B parameters perform worse than smaller models. Zeropoint quantization fails instead beyond 6.7B parameters. Zeropoint quantization shows an advantage due to asymmetric quantization but is no longer advantageous when used with mixed-precision decomposition Our method, LLM.int8(), is the only method that preserves perplexity. As such, LLM.int8() is the only method with a favorable scaling trend.
• #78: This approach reduces the memory usage of the optimizer states, allowing larger models to be trained within the same memory constraints without losing the performance benefits that come from using stateful optimizers.
• #81: The second point is about adjusting the quantization technique used in optimizers like Adam, which generally involves keeping track of the "second state" – a running average of the square of gradients. Since these values are always positive, the sign bit typically used to indicate negative numbers in binary representation is not needed. The authors utilize this redundant sign bit to increase the precision of the quantization, thus allowing the optimizer to handle a broader range of values that the second state might encounter during training. This modification helps to maintain the fidelity of the optimizer's state representation, especially important for large-scale language models that can have significant variations in the scale of gradient information.
• #82: Traditional embedding layers can become unstable with lower-precision formats like 8-bit integers because the reduced precision can lead to large errors or instabilities in the training process. However, the stable embedding layer described here allows for the use of 32-bit optimizer states even when the embeddings are quantized to 8-bit. This means that the calculations for updating the weights during training (handled by the optimizer) can still use high precision even though the embeddings themselves use low precision.
• #83: Median performance on diverse NLP and computer vision tasks: GLUE, object classification with (Moco v2) and without pretraining (CLS), machine translation (MT), and large-scale language modeling (LM). While 32-bit Adafactor is competitive with 8-bit Adam, it uses almost twice as much memory and trains slower. 8-bit Optimizers match or exceed replicated 32-bit performance on all tasks. We observe no instabilities for 8-bit optimizers. Time is total GPU time on V100 GPUs, except for RoBERTa and GPT3 pretraining, which were done on A100 GPUs.
• #86: Low-rank Adapter (LoRA) finetuning [28] is a method that reduces memory requirements by using a small set of trainable parameters, often termed adapters, while not updating the full model parameters which remain fixed. Gradients during stochastic gradient descent are passed through the fixed pretrained model weights to the adapter, which is updated to optimize the loss function.
• #88: This quantization approach is different because it balances the data across quantization bins more evenly. Traditional methods may not distribute data as uniformly, especially when there are outliers or a wide range of values. This method, by focusing on equalizing the number of values in each bin and optimizing for a normal distribution, aims to minimize information loss during the quantization process. It's particularly suited to neural network weights that usually follow a normal distribution around zero.
• #91: Mean zero-shot accuracy over Winogrande, HellaSwag, PiQA, Arc-Easy, and ArcChallenge using LLaMA models with different 4-bit data types. The NormalFloat data type significantly improves the bit-for-bit accuracy gains compared to regular 4-bit Floats. While Double Quantization (DQ) only leads to minor gains, it allows for a more fine-grained control over the memory footprint to fit models of certain size (33B/65B) into certain GPUs (24/48GB).
• #92: comparison with full 16-bit finetuning of RoBERTA and T5 models sized 125M to 3B parameters on GLUE and the Super-NaturalInstructions dataset. Results are shown in Table 3. In both datasets, we observe that 16-bit, 8-bit, and 4-bit adapter methods replicate the performance of the fully finetuned 16-bit baseline. This suggests that the performance lost due to the imprecise quantization can be fully recovered through adapter finetuning after quantization.
• #93: possible to finetune a quantized 4-bit model without any performance degradation
• #97: The novelty of BitLinear lies in its ability to maintain training stability and model accuracy even with the extreme quantization of weights to 1 bit, which is not commonly found in other methods that typically use higher precision weights.
• #98: The novelty of BitLinear lies in its ability to maintain training stability and model accuracy even with the extreme quantization of weights to 1 bit, which is not commonly found in other methods that typically use higher precision weights.
• #99: The novelty of BitLinear lies in its ability to maintain training stability and model accuracy even with the extreme quantization of weights to 1 bit, which is not commonly found in other methods that typically use higher precision weights.
• #100: The novelty of BitLinear lies in its ability to maintain training stability and model accuracy even with the extreme quantization of weights to 1 bit, which is not commonly found in other methods that typically use higher precision weights.
• #101: For the full-precision computation, the variance of the output Var(y) is at the scale of 1 with the standard initialization methods (e.g., Kaiming initialization or Xavier initialization), which has a great benefit to the training stability. To preserve the variance after quantization, we introduce a LayerNorm [BKH16] function before the activation quantization. In this way, the variance of the output y is then estimated as Var(y) ≈ E[LN(xe) 2 ] = 1,
• #102: For a weight matrix W ∈ R^(n×m), we divide it into G groups along the partition dimension, and each group has a size of (n/G) × m. We then estimate the parameters for each group independently: The formula is indicating that for each group g of weights in the matrix W, you calculate the scaling (αg​) and normalization (βg​) factors based on the values within that specific group. This allows for localized calculations within each group, facilitating model parallelism without the need for extensive synchronization across different parts of the network.
• #103: It estimates the gradients of a function. Specifically, it ignores the derivative of the threshold function and passes on the incoming gradient as if the function was an identity function.  When we binarize the weights their gradient becomes 0, so during back propagation we just bypass the binarization function and directly passed the incoming gradient to the next layer. One challenge for the optimization is that a small update on the latent weights often makes no difference in the 1-bit weights. This results in a biased gradient and update which are estimated based on the 1-bit weights. Specially at the beginning of the training.
• #104: : BitNet trains 1-bit Transformers from scratch, obtaining competitive results in an energyefficient way. BitNet significantly outperforms state-of-the-art quantization methods. As the model size scales up, the cost savings become more significant while achieving competitive performance with the models trained with FP16.
• #105: : BitNet trains 1-bit Transformers from scratch, obtaining competitive results in an energyefficient way. BitNet significantly outperforms state-of-the-art quantization methods. As the model size scales up, the cost savings become more significant while achieving competitive performance with the models trained with FP16.
• #106: It proves that BitNet has much higher scaling efficiency. Given a fixed computation budget, BitNet achieves a significantly better loss. Meanwhile, the inference cost is much smaller to get the same performance as the FP16 models
• #107: It proves that BitNet has much higher scaling efficiency. Given a fixed computation budget, BitNet achieves a significantly better loss. Meanwhile, the inference cost is much smaller to get the same performance as the FP16 models
• #108: : BitNet trains 1-bit Transformers from scratch, obtaining competitive results in an energyefficient way. BitNet significantly outperforms state-of-the-art quantization methods. As the model size scales up, the cost savings become more significant while achieving competitive performance with the models trained with FP16.
• #109: It proves that BitNet has much higher scaling efficiency. Given a fixed computation budget, BitNet achieves a significantly better loss. Meanwhile, the inference cost is much smaller to get the same performance as the FP16 models
• #110: It proves that BitNet has much higher scaling efficiency. Given a fixed computation budget, BitNet achieves a significantly better loss. Meanwhile, the inference cost is much smaller to get the same performance as the FP16 models
• #111: It proves that BitNet has much higher scaling efficiency. Given a fixed computation budget, BitNet achieves a significantly better loss. Meanwhile, the inference cost is much smaller to get the same performance as the FP16 models