VSSML17 Review. Summary Day 2 Sessions

BigML, Inc 2
Day 2 – Morning sessions

BigML, Inc 3
Basic transformations
Expectations
Poul Petersen
Reality
$
ML-ready data needs work!!!
Any data is always ML-ready
What does ML-ready mean?
● Machine Learning algorithms consume instances of the question that
you want to model. Each row must describe one of the instances and
each column a property of the instance
● Fields can be:
– already present in your data
– derived from your data
– generated using other fields

BigML, Inc 4
●
Define your goal and select the right model for the problem you want
to solve: Classification, regression, cluster analysis, anomaly
detection, association discovery
●
Perform cleansing, denormalizing, aggregating, pivoting, and
other data wrangling tasks to generate a collection of instances
relevant to the problem at hand. Finally use a very common format as
output format: CSV
●
Choose the right format to store each type of feature into a field
●
Feature engineering: Using domain knowledge and Machine
Learning expertise, generate explicit features that help to better
represent the instances (Flatline)
ML-ready steps

BigML, Inc 5
Cleansing: Homogenize missing values and different types in the
same feature, fix input errors, correct semantic issues, etc.
Denormalizing: Data is usually normalized in relational databases,
ML-Ready datasets need the information de-normalized in a single
file/dataset.
Aggregation: When data is stored as individual transactions, as in log
files, an aggregation to get the entity might be needed
Pivoting: Different values of a feature are pivoted to new columns in
the result dataset
Regular time windows: Create new features using values over
different periods of time
Preprocessing data

BigML, Inc 6
●
Define a clear idea of the goal.
●
Understand what ML tasks will achieve the goal.
●
Understand the data structure to perform those ML tasks.
●
Find out what kind of data you have and make it ML-Ready
– where is it, how is it stored?
– what are the features?
– can you access it programmatically?
●
Feature Engineering: transform the data you have into the
data you actually need.
●
Evaluate: Try it on a small scale
●
Accept that you might have to start over….
●
But when it works, automate it!!!
Preparation tasks

BigML, Inc 7
Feature Engineering
Adding some domain knowledge to your data by creating new
predicates from the existing features to help ML algorithms
What do ML algorithms know about your fields?
●
Numeric: contain sequences of numbers (no idea about odd/even, prime, etc.)
●
Date-time: contain a timestamp (no idea about weekends, special holidays or
seasons)
●
Categorical: contain an enumeration of values (no relations between them)
●
Text/Items: contain terms (no relations between them)
Features can be useless to the algorithm if:
●
They are not correlated to the objective to be predicted
●
Their values change their meaning when combined with other features
For ML Algorithms to work there must be some kind of statistical relation between
some of the features and the objective. Sometimes, you must transform the
available features to find such relations

BigML, Inc 8
Feature Engineering
When do you need Feature Engineering?
●
When the relationship between the feature and the
objective is mathematically unsatisfying
●
When the relationship of a function of two or more features
with the objective is far more relevant than the one of the
original features
●
When there is missing data
●
When the data is time-series, especially when the previous
time period’s objective is known
●
When the data can’t be used for machine learning in the
obvious way (e.g., timestamps, text data)

BigML, Inc 9
Feature Engineering
For numeric features:
– Discretization: percentiles, within percentiles, groups
– Replacement of missings
– Normalization
– Exponentiation, logarithms, etc.
– Casting to categorical, integer or real
– Statistics
– Shocks (speed of change compared to stdev)
For text features:
– Mispellings
– Length
– Number of subordinate sentences
– Language
– Levenshtein distance

BigML, Inc 10
Feature Engineering
Date-time features
●
Cannot be used “as is” in a model. It's a collection of features. BigML is able to
decompose them automatically when they are provided in the most usual
formats. With Flatline, you can decompose them all.
●
Date-time predicates that the computer does not know (some of them, domain
dependent): Working hours? Daylight? Is rush hour?...
Text features
●
Bag of words: a new feature is associated to each word in the document (built-
in in BigML)
●
Tokenization: how do we select tokens? Do we want n-grams? What about
numbers?
●
Stemming: grouping forms of the same word in a unique term
●
Length
●
Text predicates: Dollar amounts? Dates? Salutations? Please and Thank you?

BigML, Inc 11
Feature Engineering
Time-series transformations
●
Better objective (percent change instead of absolute
values)
●
Deltas from previous reference time points
●
Deltas from moving average (time windows)
●
Recent Volatility...
Problem: Exponential explosion of possible transformations

BigML, Inc.
12
● Regressions are typically used to
relate two numeric variables
● But using the proper function we
can relate discrete variables too
Ensembles and Logistic Regressions
How comes we use a regression to classify?
Logistic Regression is a classification ML Algorithm

BigML, Inc.
13
● We should use feature engineering to transform raw
features in linearly related predictors, if needed.
● The ML algorithm searches for the coefficients to
solve the problem
by transforming it into a linear regression problem
In general, the algorithm will find a coefficient per
feature plus a bias coefficient and a missing
coefficient
Assumption: The output is linearly related to the
predictors.

BigML, Inc.
14
Default numeric: Replaces missing numeric values.
Missing numeric: Adds a field for missing numerics.
Bias: Allows an intercept term. Important if P(x=0) != 0
Regularization
L1: prefers zeroing individual coefficients
L2 (default): prefers pushing all coefficients towards zero
Strength “C”: Higher values reduce regularization.
EPS: The minimum error between steps to stop.
Auto-scaling: Ensures that all features contribute equally.
Recommended, unless there is a specific need to not auto-
scale.
Configuration parameters

BigML, Inc.
15
• Multi-class LR: Each class has its own LR computed as
a binary problem (one-vs-the-rest). A set of coefficients is
computed for each class.
• Non-numeric predictors: As LR works for numeric
predictors, the algorithm needs to do some encoding of
the non-numeric features to be able to use them. These
are the field-encodings.
– Categorical: one-shot, dummy encoding, contrast
encoding
– Text and Items: frequencies of terms
● Curvilinear LR: adding quadratic features as new features
Extending the domain for the algorithm

BigML, Inc.
16
Logistic Regressions versus Decision Trees
● Expects a "smooth" linear
relationship with predictors
● LR is concerned with probability
of a discrete outcome.
● Lots of parameters to get
wrong: regularization, scaling,
codings
● Slightly less prone to over-fitting
● Because it fits a shape, might
work better when less data
available if it fulfills the expected
linear relationship.
● Adapts well to ragged non-
linear relationships
● No concern:
classification, regression,
multi-class all fine.
● Virtually parameter free
● Slightly more prone to over-
fitting
● Prefers surfaces parallel to
parameter axes, but given
enough data will discover
any shape.

BigML, Inc.
17
Compared to the other classifiers
● Shares the massive predictional power of decision
trees and ensembles
● Some smooth, multivariate functions are not a
problem (like in LR)
● Can improve some of their cons
But...
● Need massive data to learn every coefficient in a
massive parameter space
The goal is again predicting a classification
Time series and Deepnets
Deepnets are also a classifier (supervised learning)

BigML, Inc.
18
● Low efficiency: The right structure for given data is
not easily found, and most structures are bad
● Difficult interpretability: Nothing like the
interpretability of trees.
● Small data
● Problems that need quick iteration
● Problems easy or not so performance demanding
Deepnets cons
When it’s not so useful?

BigML, Inc.
19
•
● The time series model solves a forecast problem
● The training data must be a temporal identical
distributed sequence of data (so order in rows is
important!)
● The goal is predicting numeric properties in the
future based on past behaviour.
Time series are supervised learning models able to
extrapolate to the future the patterns learnt from data in the
past

BigML, Inc.
20
The resulting family of models use exponential smoothing
to fit the past training data and generate the different
components of the solution:
●
Trend: the slope between two consecutive points in time
●
Seasonality: periodically recurrent pattern of variation
●
Error: variations that cannot be described by trend or
seasonality
Each of those can contribute in an additive or multiplicative
way to the particular model.

BigML, Inc.
21
Each additive or multiplicative combination of these
components generates a different model. Which is the
best? There are some error metrics:
● AIC: Akaike Information Criterion
●
AICc: Corrected Akaike Information Criterion
● BIC: Schwarz Bayesian Information Criterion
● R-squared
And finally, they can be evaluated: Watch out! You need
linear train/test splits to maintain the sequence order

BigML, Inc.
22
● Forecast of one or many numeric features for a
user-given horizon using all possible ETS models
● The error intervals associated to these forecasts
Time series outputs

BigML, Inc 23
Day 2 – Evening sessions

BigML, Inc 24
REST API, bindings and basic workflows
jao (José Antonio Ortega)
Academics Real world
How do Machine Learning Workflows look like?
We need high-level tools to face the real world workflows by growing in:
● Automation
● Abstraction

BigML, Inc 25
The foundations
●
REST API first applications: Standards in software development.
First level of abstraction
Client side tools
●
Web UI: Sitting on top of the REST API. Human-friendly access and
visualizations for all the Machine Learning resources. Workflows must
be defined and executed step by step. Second level of abstraction.
●
Bindings: Sitting on top of the REST API. Fine-grained accessors for
the REST API calls. Workflows must be defined and executed step by
step. Second level of abstraction.
●
BigMLer: Relying on the bindings. High-level syntax. Entire workflows
can be created in only one command line. Third level of abstraction.

BigML, Inc 26
.
BigMLer automation
●
Basic 1-click workflows in one command line
●
Rich parameterized workflows: feature selection, cross-validation, etc.
●
Models are downloaded to your laptop, tablet, cell phone, etc. once
and can be used offline to create predictions
Still..
Great for local predictions

BigML, Inc 27
.
Problems of client-side solutions
●
Complexity Lots of details outside the problem
domain
●
Reuse No inter-language compatibility
●
Scalability Client-side workflows hard to optimize
●
Extensibility BigMLer hides complexity at the cost of
flexibility
●
Not enough abstraction

BigML, Inc 28
.Solution: bringing automation and abstraction to the server-side
●
DSL for ML workflow automation
●
Framework for scalable, remote execution of ML workflows
Sophisticated server-side optimization
Out-of-the-box scalability
Client-server brittleness removed
Infrastructure for creating and sharing ML scripts and libraries
WhizzML

BigML, Inc 29
.
WhizzML's new REST API resources:
Scripts: Executable code that describes an actual
workflow, taking a list of typed inputs and producing
a list of outputs.
Executions: Given a script and a complete set of
inputs, the workflow can be executed and its outputs
generated.
Libraries: A collection of WhizzML definitions that
can be imported by other libraries or scripts.

BigML, Inc 30
Scripts
Creating scripts
●
Usable by any binding (from any language)
●
Built-in parallelization
●
BigML resources management as primitives of the language
●
Complete programming language for workflow definition
Using scripts
Web UI
Bindings
BigMLer
WhizzML

BigML, Inc 31
Advanced WhizzML workflows
Charles Parker
WhizzML offers:
● Primitives for all ML resources: (datasets, models, clusters, etc.)
● A complete programming language to compose at will these ML
resources.
● Parallelization and Scalability built-in.
This empowers the user to benefit from:
● Automated feature engineering: Best-first feature selection.
● Automated configuration choice: Randomized parameter
optimization, SMACdown.
● Complex algorithms as 1-click: Stacked generalization, Boosting.
All of them can be shared, reproduced and reused as
one more BigML resource in a language-agnostic way.

BigML, Inc 32
Selected
fields
Following iterations don't improve the score for the model
with (f5 f7), so the process stops

BigML, Inc 33
Stacked generalization

BigML, Inc 34
Process stops when you reach the expected performance
or the user-given iterations limit
Randomized parameter optimization

BigML, Inc 35

BigML, Inc 36
… …
The final model is an ensemble of models
T0
F0
T1
F1
T2
F2
F8
T8
Boosting

BigML, Inc 37
Script it once, for everybody anywhere
Publish scripts
in the gallery
Add scripts to
your menus

VSSML17 Review. Summary Day 2 Sessions

More Related Content

What's hot (20)

Similar to VSSML17 Review. Summary Day 2 Sessions (20)

More from BigML, Inc (20)

Recently uploaded (20)

VSSML17 Review. Summary Day 2 Sessions