Data Science in the Elastic Stack

Data Science
in the Elastic Stack

[1]: df = pd.read_csv(“data.csv”)
[2]: train, test = preprocess(df)
[3]: pipe = Pipeline([('transform', ct), ('lr', LR())])
[4]: pipe.fit(train, y)
[5]: plt.plot(results)
[6]: plt.save_fig(‘results.png’)
> mkdir project
> cd project
> mkdir data
> mv ~/Downloads/data/* data
> virtualenv venv
> source venv/bin/activate
> pip install pandas numpy
matplotlib sklearn nltk requests
bs4 boto3 jupyter
> jupyter notebook

du -sch data/*
du -sch data/*
98.4G total
919.8M total

data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

google: data too big for
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

jupyter notebook
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

jupyter notebook
json.dump(“export_FINAL.json”)
with open(“export_FINAL_2.csv”)
pandas
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

jupyter notebook
pandas
google: deploy model to production
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

jupyter notebook
import dask
pandas
google: deploy model to production
data actionable
results?
data
data
data
data
data
data
data
ta
data
ta
ta
data

ModelDiscover Ingest Operationalize

Methods OutcomesGoals
● work with customers and
stakeholders to understand and
identify business problems
● data audit
● project charter (README.md)
● data source definitions
● data dictionaries
● define objectives - specify key
variables and related metrics
● identify relevant data sources

Methods ArtifactsGoals
● ad-hoc exploratory data analysis
of raw data.
● development of data pipeline that
transforms raw data.
● charter updates (README.md)
● architecture
● produce high-quality datasets with
a clear relationship to the target
objectives
● provide data to an analytical
environment
● develop architecture to keep data
fresh and up to date

POST _bulkhelpers.bulk(es, data)

Methods ArtifactsGoals
● feature engineering
● modeling training
● feature sets
● A standardized way of
benchmarking model results.
● in our case, job configs :D
● identification of optimal features
for machine learning modeling
● creation of a model that best fits
the business objectives and
modelling task
● production ready model!

Logging and Metrics: Spot an unusual drop in application
requests and drill in on the troublesome server contributing to
the problem.
Security Analytics: Identify unusual network activity or user
behavior to pinpoint attackers before they do damage.
Business Analytics: Get notified if there is an unusual
increase in abandoned shopping carts in your ecommerce
site.
Application Performance Monitoring: Catch bottlenecks
and slow response times so your apps can keep running
smoothly.

Methods OutcomesGoals
● validation
● project hand-off
● operationally useful KPIs powered
by ML
● user acceptance

Data Science in the Elastic Stack

- Identify demand in real time
Data
Metrics
Goals & Objectives
- raw taxi trip information

?
Data
Metrics
Goals & Objectives
Data
Features
Success Factors
Goals & Objectives
- decrease waiting times by 50%
- aggregate rides to and from specific locations

KPIs
Data
Features
Success Factors
Goals & Objectives
average waiting time
raw taxi trip information
aggregate rides to and from specific locations
decrease waiting times by 50%
Identify demand in real time

http://www.nyc.gov/html/tlc/downloads/pdf/data_dictionary_trip_r
ecords_yellow.pdf

TLC Objectives
● identify pockets of demand as
they materialize
● identify potential failures of
payment systems
● get a sense of volume in the
coming weeks
helpers.bulk(es, data)

Data Science in the Elastic Stack

More Related Content

Similar to Data Science in the Elastic Stack (20)

Recently uploaded (20)

Data Science in the Elastic Stack