17 large scale machine learning
- 1. 17. Large Scale Machine Learning -- Machine learning algos work better if the large amount of data is fed to it. So the algos have to be efficient to works with such large datasets Accuracy of prediction grows as the amount of data increases, for each type of algorithm used:
- 2. In a Single step of gradient descent, it will have to compute sum of 100 million terms Sanity check: use just a small set of examples and check if increase the no of examples will benefit the model: Draw the learning curves: If this type of curve occurs, it means that it’s a high variance problem : and increase no. of examples is likely to help. But mainly it will require the addition of new extra features, since its high variance problem.
- 3. If this type of curve occurs, it means that it’s a high bias problem : then increase the no. of examples is not very likely to benefit the model. STOCHASTIC GRADIENT DESCCENT: to make it less expensive than simple gradient descent What gradient descent does:
- 4. What will happen is: Contours: ➢Its called Batch Gradient Descend.
- 5. What it does: ➢Each time the inner for loop works: it fits the parameters a little better for that particular example The simple gradient descent converges to the optimum in a reasonably straight-line path But SGD does not follow linear path, it just moves into a random direction, but in the end, it converges to a region near optimum.
- 6. It does never reach the optimum, it continuously wanders around the optimum, but it’s a fine fit as the region near optimum is OK to converge The outer loop may be made to run 1 to 10 times, based on the size of m. If m is very-very large, running the outer loop just once will give a reasonably good hypothesis, while in case of batch gradient descent, looping over all examples once will only move the hypothesis one step closer. MINI BATCH GRADIENT DESCENT:
- 7. Using b - examples at a time benefits the vectorization for parallel computation SGD CONVERGENCE:
- 8. If the plot looks like this: Red : with smaller α; Blue : with larger α The curve is noisy as the cost is only averaged over 1000 examples Using smaller learning rate, the algo will initially learn slowly but it will eventually give a slightly better estimate: because the SGD doesn’t converge, but the parameters will oscillate around the optimum, so slower learning rate means the oscillations are also smaller.
- 9. If we average over 5000 examples, the curve will smoothen, b’coz we only plot one data point out of every 5000 examlpes If the plot looks like this: Blue – no of examples to average over is very low Red – incr in no examples to average over has given a smooth curve and the cost is decr Pink – if the curve looks like this after incr no of ex: there is something wrong with the algo and the algo is not learning. We have to change some parameters of the algo.
- 10. If the plot looks like this: the algo is diverging How to converge the SGD:
- 11. ONLINE LEARNING: when there is continuous input data like on websites which track users’ actions ➢It can adapt to changing user preferences, even if the entire pool of users changes ➢After updating the parameters with current example, we can just throw away the data. ≫ if the data is continuous, its good to use this algo ≫ if the data is small, then its better to use logistic regression.
- 12. CLICK THROUGH RATE PREDICTION PROBLEM – online learning Lets say we have 100 phones in the shop and we want to return the top 10 results to user: Each time the user clicks a link, we will collect the (x, y) – pair to learn through and show them the products they are most likely to click on
- 13. MAP REDUCE: When an ML problem is too big for a single machine, we can use map-reduce: ➢We split the training set and sent each part to different computers ➢Computers process data and sent them to centralized controller ➢Centralized controller combines the results and produce the model When to use map-reduce: Ask yourself: can the algorithm be expressed as a sum of functions over training set? ▪ If yes: use map-reduce
- 14. DATA PARALLELISM: We can also use map reduce on machines having multiple cores: ➢This way we don’t have to care about network latencies Some linear algebra libraries automatically take care of it.