![Adaptive learning rate
• Difficulty using SGD is finding a good learning rate
• An adaptive learning rate will
– ADAGRAD is an adaptive method
• Simple learning rate in example code
– alpha / (sqrt(n) + 1)
• Where N is the number of times a specific feature has been
encountered
– w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.)
• Full weight update will shrink the change by the learning rate
of the specific feature](https://image.slidesharecdn.com/modernclassificationtechniques-150121063222-conversion-gate01/85/Modern-classification-techniques-12-320.jpg)
Modern classification techniques
- 1. Modern Classification Techniques Mark Landry Austin Machine Learning Meetup 1/19/2015
- 2. Overview • Problem & Data – Click-through rate prediction for online auctions – 40 million rows – Sparse: gather characteristics – Down-sampled • Methods – Logistic regression – Sparse feature handling – Hash trick – Online learning – Online gradient descent – Adaptive learning rate – Regularization (L1 & L2) • Solution characteristics – Fast: 20 minutes – Efficient: ~4GB RAM – Robust: Easy to extend – Accurate: competitive with factorization machines, particularly when extended to key interactions
- 3. Two Data Sets • Primary use case: click logs – 40 million rows – 20 columns – Values appear in dense fashion, but a sparse feature space • For highly informative features types (URL/site) 70% of features have 3 or fewer instances – Note: negatives have been down-sampled • Extended to separate use case: clinical + genomic – 4k rows – 1300 columns – Mix of dense and sparse features
- 4. Methods and objectives • Logistic regression: accuracy/base algorithm • Stochastic gradient descent: optimization • Adaptive learning rate: accuracy, speed • Regularization (L1 & L2): generalized solution • Online learning: speed • Sparse feature handling: memory efficiency • Hash trick: memory efficiency, robustness
- 5. Implementation Infrastructure • From scratch: no machine learning libraries • Maintain vectors for – Features (1/0) – Weights – Feature Counts • Each vector will use the same index scheme • Hash trick means we can immediately find the index of any feature and we bound the vector size (more later)
- 6. Logistic Regression • Natural fit for probability problems (0/1) – 1 / (1 + exp(sum(weight*feature))) – Solves based on log odds – Higher calibration than many other algorithms (particularly decision trees), which is useful for Real Time Bid problem
- 7. Sparse Features • All values experience receive a column where the absence/presence • So 1 / (1 + exp(sum(weight*feature))) resolves to 1 / (1 + exp(sum(weight))) for only the features in each instance
- 8. Hash Trick • Hash trick allows for quick access into parallel arrays that hold key information to your model • Example: use native python hash(‘string’) to cast into a large integer • Bound the parameter space by using modulo – E.g. abs(hash(‘string’)) % (2 ** 20) – The size of that integer is a parameter, and it allows you to set it as large as your system can handle – Why set it larger? Hash collisions – Keep features separate: abs(hash(feature-name + ‘string’)) % (2 ** 20) • Any hash function can have a collision. The particular function used is fast, but much more likely to encounter a collision than a murmur hash or something more elaborate. • So a speed/accuracy tradeoff dictates what function to use. The larger the bits, the lower the hash collisions.
- 9. Online Learning • Learn one record at a time – A prediction is always available at any point, and the best possible given the data the algorithm has seen – Do not have to retrain to take in more data • Though you may still want to • Depending on learning rate used, may desire to iterate through data set more than once • Fast: VW approaches speed of network interface
- 10. OGD/SGD: online gradient descent Gradient descent Optimization algorithms are required to minimize the loss in logistic regression Gradient descent, and many variants, are a popular choice, especially with large –scale data. Visualization (in R) library(animation) par(mar = c(4, 4, 2, 0.1)) grad.desc() ani.options(nmax = 50) par(mar = c(4, 4, 2, 0.1)) f2 = function(x, y) sin(1/2 * x^2 - 1/4 * y^2 + 3) * cos(2 * x + 1 - exp(y)) grad.desc(f2, c(-2, -2, 2, 2), c(-1, 0.5), gamma = 0.3, tol = 1e-04) ani.options(nmax = 70) par(mar = c(4, 4, 2, 0.1)) f2 = function(x, y) sin(1/2 * x^2 - 1/4 * y^2 + 3) * cos(2 * x + 1 - exp(y)) grad.desc(f2, c(-2, -2, 2, 2), c(-1, 0.5), gamma = 0.1, tol = 1e-04) # interesting comparison: https://imgur.com/a/Hqolp
- 11. Other common optimization algorithms ADAGRAD Still slightly sensitive to choice of n ADADELTANewton’s Method Quasi-Newton Momentum
- 12. Adaptive learning rate • Difficulty using SGD is finding a good learning rate • An adaptive learning rate will – ADAGRAD is an adaptive method • Simple learning rate in example code – alpha / (sqrt(n) + 1) • Where N is the number of times a specific feature has been encountered – w[i] -= (p - y) * alpha / (sqrt(n[i]) + 1.) • Full weight update will shrink the change by the learning rate of the specific feature
- 13. Regularization (L1 & L2) • Regularization attempts to ensure robustness of a solution • Enforces a penalty term on the coefficients of a model, guiding toward a simpler solution • L1: guides parameter values to be 0 • L2: guides parameters to be close to 0, but not 0 • In practice, these ensure large coefficients are not applied to rare features
- 14. Related Tools • Vowpal Wabbit – Implements all of these features, plus far more – Command line tool – svmLite-like data format – Source code available on Github with fairly open license • Straight Python implementation (see code references slide) • glmnet, for R: L1/L2 regression, sparse • Scikit-learn, python ML library: ridge, elastic net (l1+l2), SGD (can specify logistic regression) • H2O, Java tool; many techniques used, particularly in deep learning • Many of these techniques are used in neural networks, particularly deep learning
- 15. Code References • Introductory version: online logistic regression, hash trick, adaptive learning rate – Kaggle forum post • Data set is available on that competition’s data page • But you can easily adapt the code to work for your data set by changing the train and test file names (lines 25-26) and the names of the id and output columns (104-107, 129-130) – Direct link to python code from forum post – Github version of the same python code • Latest version: adds FTRL-proximal (including SGD, L1/L2 regularization), epochs, and automatic interaction handling – Kaggle forum post – Direct link to python code from forum post (version 3) – Github version of the same python code
- 16. Additional References • Overall process – Google paper, FTRL proximal and practical observations – Facebook paper, includes logistic regression and trees, feature handling, down-sampling • Follow The Regularized Leader Proximal (Google) • Optimization – Stochastic gradient descent: examples and guidance (Microsoft) – ADADELTA and discussion of additional optimization algorithms (Google/NYU intern) – Comparison Visualization • Hash trick: – The Wikipedia page offers a decent introduction – general description and list of references, from VW author