Nuit Blanche: autoML

Showing posts with label autoML. Show all posts

Friday, June 28, 2019

Improving Neural Architecture Search Image Classifiers via Ensemble Learning - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Improving Neural Architecture Search Image Classifiers via Ensemble Learning by Vladimir Macko, Charles Weill, Hanna Mazzawi and Javier Gonzalvo‎

AdaNAS is an algorithm for learning an ensemble that improves the performance of neural architecture search models while having a similar parameter count as single large model. Our experiments demonstrate that these ensembles improve accuracy upon a single neural network of the same size. Our models achieve comparable results with the state-of-the-art on CIFAR-10 and set a new state-of-the-art on CIFAR-100.

An implementation is ehre: https://github.com/tensorflow/adanet/tree/master/research/improve_nas

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Other links:
Paris Machine Learning: Meetup.com||@Archives||LinkedIn||Facebook|| @ParisMLGroup< br/> About LightOn: Newsletter ||@LightOnIO|| on LinkedIn || on CrunchBase || our Blog
About myself: LightOn || Google Scholar || LinkedIn ||@IgorCarron ||Homepage||ArXiv

Wednesday, June 26, 2019

NAS-Bench-101: Towards Reproducible Neural Architecture Search - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

NAS-Bench-101: Towards Reproducible Neural Architecture Search by Chris Ying, Aaron Klein, Esteban Real, Eric Christiansen, Kevin Murphy, Frank Hutter

Recent advances in neural architecture search (NAS) demand tremendous computational resources, which makes it difficult to reproduce experiments and imposes a barrier-to-entry to researchers without access to large-scale computation. We aim to ameliorate these problems by introducing NAS-Bench-101, the first public architecture dataset for NAS research. To build NAS-Bench-101, we carefully constructed a compact, yet expressive, search space, exploiting graph isomorphisms to identify 423k unique convolutional architectures. We trained and evaluated all of these architectures multiple times on CIFAR-10 and compiled the results into a large dataset of over 5 million trained models. This allows researchers to evaluate the quality of a diverse range of models in milliseconds by querying the pre-computed dataset. We demonstrate its utility by analyzing the dataset as a whole and by benchmarking a range of architecture optimization algorithms.

Data and code for NAS-Bench-101 is here: https://github.com/google-research/nasbench

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Monday, June 24, 2019

Meta-learning of textual representations - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Meta-learning of textual representations by Jorge Madrid and Hugo Escalante

Recent progress in AutoML has lead to state-of-the-art methods (e.g., AutoSKLearn) that can be readily used by non-experts to approach any supervised learning problem. Whereas these methods are quite effective, they are still limited in the sense that they work for tabular (matrix formatted) data only. This paper describes one step forward in trying to automate the design of supervised learning methods in the context of text mining. We introduce a meta learning methodology for automatically obtaining a representation for text mining tasks starting from raw text. We report experiments considering 60 different textual representations and more than 80 text mining datasets associated to a wide variety of tasks. Experimental results show the proposed methodology is a promising solution to obtain highly effective off the shell text classification pipelines.

An implementation is here: https://github.com/jorgegus/autotext

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

One-Shot Neural Architecture Search via Compressive Sensing

** Nuit Blanche is now on Twitter: @NuitBlog **

Using sparse recovery in network architecture search, there has to be a meta-thread somewhere here !

One-Shot Neural Architecture Search via Compressive Sensing by Minsu Cho, Mohammadreza Soltani, Chinmay Hegde

Neural architecture search (NAS), or automated design of neural network models, remains a very challenging meta-learning problem. Several recent works (called "one-shot" approaches) have focused on dramatically reducing NAS running time by leveraging proxy models that still provide architectures with competitive performance. In our work, we propose a new meta-learning algorithm that we call CoNAS, or Compressive sensing-based Neural Architecture Search. Our approach merges ideas from one-shot approaches with iterative techniques for learning low-degree sparse Boolean polynomial functions. We validate our approach on several standard test datasets, discover novel architectures hitherto unreported, and achieve competitive (or better) results in both performance and search time compared to existing NAS approaches. Further, we support our algorithm with a theoretical analysis, providing upper bounds on the number of measurements needed to perform reliable meta-learning; to our knowledge, these analysis tools are novel to the NAS literature and may be of independent interest.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Liked this entry ? subscribe to Nuit Blanche's feed, there's more where that came from. You can also subscribe to Nuit Blanche by Email.

Friday, June 21, 2019

An Open Source AutoML Benchmark - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

An Open Source AutoML Benchmark by Erin Ledell, Pieter Gijsbers, Joaquin Vanschoren, Janek Thomas, Bernd Bischl and Sebastien Poirier

In recent years, an active field of research has developed around automated machine learning (AutoML). Unfortunately, comparing different AutoML systems is hard and often done incorrectly. We introduce an open, ongoing, and extensible benchmark framework which follows best practices and avoids common mistakes. The framework is open-source, uses public datasets and has a website with up-to-date results. We use the framework to conduct a thorough comparison of 4 AutoML systems across 39 datasets and analyze the results.

An implementation of the benchmark is here: https://github.com/openml/automlbenchmark/

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings by Matilde Gargiani, Aaron Klein, Stefan Falkner and Frank Hutter

We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves. We study instantiations of this framework based on random forests and Bayesian recurrent neural networks. Our experiments show that these models yield better predictions than state-of-the-art models from the hyperparameter optimization literature when extrapolating the performance of neural networks trained with different hyperparameter settings.

An implementation is here: https://github.com/gmatilde/vdrnn

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Thursday, June 20, 2019

Improving Automated Variational Inference with Normalizing Flows - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Improving Automated Variational Inference with Normalizing Flows by Stefan Webb, J. P. Chen, Martin Jankowiak and Noah Goodman

We describe a framework for performing automatic Bayesian inference in probabilistic programs with fixed structure. Our framework takes a probabilistic program with fixed structure as input and outputs a learnt variational distribution approximating the posterior. For this purpose, we exploit recent advances in representing distributions with neural networks. We implement our approach in the Pyro probabilistic programming language, and validate it on a diverse collection of Bayesian regression models translated from Stan, showing improved inference and predictive performance relative to the existing state-of-the-art in automated inference for this class of models.

An implementation is here: https://github.com/stefanwebb/autoguides

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Efficient Forward Architecture Search - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Efficient Forward Architecture Search by Hanzhang Hu, John Langford, Rich Caruana, Saurajit Mukherjee, Eric Horvitz and Debadeepta Dey

In this work, we propose a neural architecture search (NAS) algorithm that iteratively augments existing networks by adding shortcut connections and layers. At each iteration, we greedily select among the most cost-efficient models a parent model, and insert into it a number of candidate layers. To learn which combination of additional layers to keep, we simultaneously train their parameters and use feature selection techniques to extract the most promising candidates which are then jointly trained with the parent model. The result of this process is excellent statistical performance with relatively low computational cost. Furthermore, unlike recent studies of NAS that almost exclusively focus on the small search space of repeatable network modules (cells), this approach also allows direct search among the more general (macro) network structures to find cost-effective models when macro search starts with the same initial models as cell search does. Source code is available at https://github.com/microsoft/petridishnn.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Wednesday, June 19, 2019

Bayesian Optimization over Sets - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Bayesian Optimization over Sets by Jungtaek Kim, Michael McCourt, Saehoon Kim, Tackgeun You and Seungjin Choi

We propose a Bayesian optimization method over sets, to minimize a black-box function that can take a set as single input. Because set inputs are permutation-invariant and variable-length, traditional Gaussian process-based Bayesian optimization strategies which assume vector inputs can fall short. To address this, we develop a Bayesian optimization method with set kernel that is used to build surrogate functions. This kernel accumulates similarity over set elements to enforce permutation-invariance and permit sets of variable size, but this comes at a greater computational cost. To reduce this burden, we propose a more efficient probabilistic approximation which we prove is still positive definite and is an unbiased estimator of the true set kernel. Finally, we present several numerical experiments which demonstrate that our method outperforms other methods in various applications.

The attendant implementation is here: https://github.com/jungtaekkim/bayeso

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Graduated Optimisation of Black-Box Functions - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Graduated Optimisation of Black-Box Functions by Weijia Shao, Christian Geißler and Fikret Sivrikaya

Motivated by the problem of tuning hyperparameters in machine learning, we present a new approach for gradually and adaptively optimizing an unknown function using estimated gradients. We validate the empirical performance of the proposed idea on both low and high dimensional problems. The experimental results demonstrate the advantages of our approach for tuning high dimensional hyperparameters in machine learning.

The attendant implementation is here: https://github.com/christiangeissler/gradoptbenchmark

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Accelerating the Nelder - Mead Method with Predictive Parallel Evaluation - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Accelerating the Nelder - Mead Method with Predictive Parallel Evaluation by Yoshihiko Ozaki, Shuhei Watanabe and Masaki Onishi

The Nelder–Mead (NM) method has been recently proposed for application in hyperparameter optimization (HPO) of deep neural networks. However, the NM method is not suitable for parallelization, which is a serious drawback for its practical application in HPO. In this study, we propose a novel approach to accelerate the NM method with respect to the parallel computing resources. The numerical results indicate that the proposed method is significantly faster and more efficient when compared with the previous naive approaches with respect to the HPO tabular benchmarks.

The attendant implementaiton is here.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Tuesday, June 18, 2019

Toward Instance-aware Neural Architecture Search - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Toward Instance-aware Neural Architecture Search by An-Chieh Cheng, Chieh Hubert Lin, Da-Cheng Juan, Wei Wei and Min Sun

Recent advancements in Neural Architecture Search (NAS) have achieved significant improvements in both single and multiple objectives settings. However, current lines of research only consider searching for a single best architecture within a search space. Such an assumption restricts the model from capturing the high diversity and variety of real-world data. With this observation, we propose InstaNAS, an instance-ware NAS framework that aims to search for a distribution of architectures. Intuitively, we assume that real-world data consists of many domains (e.g., different difficulties or structural characteristics), and each domain can have one or multiple experts that have relatively more preferable performance. The controller of InstaNAS is not only responsible for sampling architectures during its search phase, but also needs to identify which down-stream expert architecture to use for each input instance during the inference phase. We demonstrate the effectiveness of InstaNAS in a multiple-objective NAS setting that considers the trade-offs between accuracy and latency. Within a search space inspired by MobileNetV2 on a series of datasets, experiments show that InstaNAS can achieve either higher accuracy with same latency or significant latency reduction without compromising accuracy against MobileNetV2.

The attendant implementation is here: https://github.com/AnjieZheng/InstaNAS

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

A simple dynamic bandit algorithm for hyper-parameter tuning - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

A simple dynamic bandit algorithm for hyper-parameter tuning by Xuedong Shang, Emilie Kaufmann and Michal Valko

Hyper-parameter tuning is a major part of modern machine learning systems. The tuning itself can be seen as a sequential resource allocation problem. As such, methods for multi-armed bandits have been already applied. In this paper, we view hyper-parameter optimization as an instance of best-arm identification in infinitely many-armed bandits. We propose D-TTTS, a new adaptive algorithm inspired by Thompson sampling, which dynamically balances between refining the estimate of the quality of hyper-parameter configurations previously explored and adding new hyper-parameter configurations to the pool of candidates. The algorithm is easy to implement and shows competitive performance compared to state-of-the-art algorithms for hyper-parameter tuning.

The attendant code is here.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Monday, June 17, 2019

Random Search and Reproducibility for Neural Architecture Search

** Nuit Blanche is now on Twitter: @NuitBlog **

Random Search and Reproducibility for Neural Architecture Search by Liam Li and Ameet Talwalkar

Neural architecture search (NAS) is a promising research direction that has the potential to replace expertdesigned networks with learned, task-specific architectures. In order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) NAS is a specialized hyperparameter optimization problem; and (ii) random search is a competitive baseline for hyperparameter optimization. Leveraging these observations, we evaluate both random search with early-stopping and a novel random search with weight-sharing algorithm on two standard NAS benchmarks—PTB and CIFAR-10. Our results show that random search with early-stopping is a competitive NAS baseline, e.g., it performs at least as well as ENAS, a leading NAS method, on both benchmarks. Additionally, random search with weight-sharing outperforms random search with early-stopping, achieving a state-of-the-art NAS result on PTB and a highly competitive result on CIFAR-10. Finally, we explore the existing reproducibility issues of published NAS results.

An implementation of the paper is at: https://github.com/liamcli/randomNAS_release

The code base requires the following additional repositories:

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn

Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks - implementation -

** Nuit Blanche is now on Twitter: @NuitBlog **

Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks by Yochai Zur, Chaim Baskin, Evgenii Zheltonozhskii, Brian Chmiel, Itay Evron, Alex M. Bronstein and Avi Mendelson

Recently, deep learning has become a de facto standard in machine learning with convolutional neural networks (CNNs) demonstrating spectacular success on a wide variety of tasks. However, CNNs are typically very demanding computationally at inference time. One of the ways to alleviate this burden on certain hardware platforms is quantization relying on the use of low-precision arithmetic representation for the weights and the activations. Another popular method is the pruning of the number of filters in each layer. While mainstream deep learning methods train the neural networks weights while keeping the network architecture fixed, the emerging neural architecture search (NAS) techniques make the latter also amenable to training. In this paper, we formulate optimal arithmetic bit length allocation and neural network pruning as a NAS problem, searching for the configurations satisfying a computational complexity budget while maximizing the accuracy. We use a differentiable search method based on the continuous relaxation of the search space proposed by Liu et al. (2019a). We show, by grid search, that heterogeneous quantized networks suffer from a high variance which renders the benefit of the search questionable. For pruning, improvement over homogeneous cases is possible, but it is still challenging to find those configurations with the proposed method. The code is publicly available at https://github.com/yochaiz/Slimmable and https://github.com/yochaiz/darts-UNIQ.

Follow @NuitBlog or join the CompressiveSensing Reddit, the Facebook page, the Compressive Sensing group on LinkedIn or the Advanced Matrix Factorization group on LinkedIn