Driving Real Insights Through Data Science

The Journey to Becoming a Data-
Driven Enterprise
Pivotal Big Data Roadshow 2015

2© Copyright 2015 Pivotal. All rights reserved.
Where we’re going today…
3 Great Keynotes
• Journey to a Data-driven Enterprise
• Data Science Use Cases
• Streaming Data and Predictive Analytics
Stock Inference Demo and Architecture Overview
Intensive hands-on training sessions

Today’s Agenda
12:00 PM – 12:45 PM - Check-In & Lunch
12:45 PM – 1:00 PM – Welcome and Agenda Review
1:00 PM – 1:PM AM – How Pivotal’s Tools Help Drive Value from Data Science
1:20 PM – 2:20 PM – Accelerating the Generation of New Insights – R&D Use Case
Review and Demo
2:20 PM – 2:30 PM – Coffee Break
2:30 PM – 3:00 PM – Manufacturing Use Case Review and Demo
3:00 PM – 3:15 PM – Closing Remarks

© Copyright 2015 Pivotal. All rights reserved.
MASHING BIG DATA WITH BIG MACHINES
IS ‘BEAUTIFUL, DESIRABLE, INVESTABLE’
- IT COULD TRANSFORM GE'S BUSINESS -
AND THE ECONOMY.
“
”JEFF IMMELT, CEO, GE

THE POWER OF 1
R
X
Increasing
Freight Utilization Rail
Predictive
Maintenance Healthcare
Predictive
Diagnostics Power
Driving Outcomes That Matter
One Percent Improvement Equals
$27B
Industry Value by
Reducing System
Inefficiency
$63B
Industry Value by
Reducing Process
Inefficiency
$66B
Industry Value with
Efficiency Improvements
In Gas-fired Power
Plant Fleets
Source: General Electric

DATA-DRIVEN ENTERPRISE JOURNEY
STORE
• Structured
• Unstructured
• High Volume
• High Velocity
ANALYZE
• Predictive Analytics
• Machine Learning
• Advance Data Science
• Realtime Analytics
DEVELOP
• Advanced Analytic Pipelines
• Realtime Analytical Applications
• Global Scale Data-Driven
Applications
• Enterprise, Consumer, and
Mobile
INNOVATE
• Agile Dev Expertise
• DevOps
• Microservice
• Continuous Delivery
• Closed Loop Applications
AGILE DEVELOPMENT
BIG DATA
PREDICTIVE ANALYTICS
CLOUD NATIVE PLATFORM

0% of CIOs think
their IT infrastructure
is fully prepared for
big data (3)
30% of
companies have
deployed advanced
analytics, 11% big
data analysis (4)
44% of new applications
failed to meet performance
expectations (5)
2X
90% of companies
allocate at least 2X
more cloud capacity
than needed to ensure
performance (6)
But…
80% of
CEOs thinking
data mining and
analysis are
strategically important
(1)
4% of
companies use
analytics
effectively (2)
(1) 2015 PWC CEO Survey; (2)2013 Baine and Company - The Value of Big Data; (3) 2014 IT Infrastructure Conversation - IBM; (4) Ernest and Young - 2014 Enterprise IT Trends and
Investments; (5) 2014 Riverbed Tecnologies - The Transformers; (6) 2014 ElasticHosts CIO Study
LARGE ENTERPRISE BIG DATA TROUBLE

BIG DATA
CHASM
70%
of data
generated by
customers
80%
of data stored
3%
prepared for
analysis
0.5%
being
analyzed
<0.5%
being
operationalized
9
THE DATA DIVIDE

Software Is Eating The World
Data Is Fueling Software
SOFTWARE IS EATING THE WORLD

WE CHOSE PIVOTAL BECAUSE WE BELIEVE IT
PROVIDES A 360-DEGREE VIEW OF THE PROCESS.
FROM A DATA SCIENCE AND DATA TECHNOLOGY
PERSPECTIVE, IT MEANS DELIVERING BEST-IN-
CLASS DATA TECHNOLOGIES AND ENABLING THEM
ON THEIR PLATFORM.
“
”

ACROSS INDUSTRIES

THE NEW DATA IMPERATIVES
Converged
Data & Cloud
OpenData-Driven
Apps

THE BIG DATA PROBLEM
Fragmentation ConstraintsComplexity

• Remove Lock-in
• Leverage Ecosystem
• Co-innovate
GUIDING PRINCIPLES IN THE NEW ERA
OPEN AGILE CLOUD-READY
• Shorten innovation
cycles
• Reduce TCO
• Improve TTM
• Solve business
problems
• Avoid lock-in
• Appropriate security

JOURNEY TO A DATA-DRIVEN ENTERPRISE
Deploy analytic apps and
automate at scale
Perform advanced analytics
Discover insights
Modernize data
infrastructure

automate at scale
Discover insights
Modernize data
infrastructure
DATA-DRIVEN COMPANIES:
USE MODERN DATA INFRASTRUCTURE

MODERNIZE DATA INFRASTRUCTURE
Elastic, Scale-out
storage and processing
Flexible data types and
pipelining
ETL on demand: low operational cost
Expanded use cases
Higher quality analytics
Lowered storage/processing cost
Less fragmented ecosystem
Reduced vendor lock-in
REQUIREMENTS BENEFITS
Cloud friendly and
open-source based

Modernize data
infrastructure
automate at scale
Discover insights
STRATEGICALLY USE ADVANCED ANALYTICS

ADVANCED ANALYTICS
Leverage existing skills and tools
Rapid time to insights
Internet of Things use cases
Rapid time to insights
Solve business problems
Predictive insights: proactive execution
Machine learning and
advanced analytics
01010101010101
01001010101010
10101100101010
SQL- compliant batch
and interactive queries
Massive stream
processing
0101010101010101001010
1010101010110010101010
10101010

Modernize data
infrastructure
Discover insights
INNOVATE AT SCALE
automate at scale

ANALYTIC APPS AND AUTOMATION AT SCALE
Reduced time to action
Low ‘analytics  app-dev’ integration cost
Reduced time to insights
Flexible ingestion: low operating cost
High performance: low operating cost
Transactional safety: business critical ops
Low-latency, distributed
in-memory transactions
Resilient, scale-out
messaging and object storage
Agile analytic app-dev
with enterprise PaaSPaaS

JOURNEY TO A DATA-DRIVEN ENTERPRISE
automate at scale
Discover insights
Modernize data
infrastructure
Pivotal Data Science
helps you move from BI to
Data Science
Pivotal Labs
helps you move to an
agile development of apps
at scale
Pivotal Data Engineering
helps you move from data
administration to data engineering

PIVOTAL BIG DATA SUITE

Open sourcing all Pivotal Big Data Suite components including:
WORLD’S FIRST OPEN SOURCED BIG DATA
PORTFOLIO
BUILDING ON SUCCESS OF CLOUD FOUNDRY FOUNDATION
BUILT FOR ENTERPRISES
Pivotal GemFire
Apache Geode
Apache HAWQ
Pivotal HDB
Pivotal
Greenplum Database

BUILT FOR ENTERPRISES
Value added features: enterprise grade performance + robustness without lock-in
• Advanced Query Optimization in analytics
• WAN replication and continuous query in transactional processing
Flexible Deployment models: align to business objectives and needs
• Balance cost objectives with policy and compliance requirements
• Leverage Pivotal’s pre-integration + certification on supported configurations
Enterprise grade support: one throat to choke for the suite
• Focus on business problems – not on lifecycle management
• Expert support on Big Data Suite means reduced business risk

• Common core for Hadoop ecosystem
• Rapidly accelerated certifications, ecosystem
development and enterprise-grade quality
OpenDataPlatform.org
OPEN

AGILE
automate at scale
Discover insights
Modernize data infrastructure
Spring XD Spark Pivotal HD & Open Data Platform
Pivotal Greenplum Database
Pivotal HDB Rabbit MQ
Redis
Pivotal GemFire Pivotal BDS on PCF
Pivotal Cloud Foundry

CLOUD-READY
COMMODITY
HARDWARE
APPLIANCE HYBRID CLOUDCLOUD
IaaS IaaS
PAAS

DATA-DRIVEN ENTERRPRISE JOURNEY WITH
PIVOTAL BIG DATA SUITE
STORE
• Structured
• Unstructured
• High Volume
• High Velocity
ANALYZE
• Predictive Analytics
• Advance Data Science
• Realtime Analytics
DEVELOP
• Advanced Analytic Pipelines
• Realtime Analytical Applications
• Global Scale Data-Driven
Applications
• Enterprise, Consumer, IoT, and
Mobile
INNOVATE
• Agile Dev Expertise
• DevOps
• Microservices
• Continuous Delivery
• Closed Loop Applications
AGILE DEVELOPMENT
BIG DATA
PREDICTIVE ANALYTICS
CLOUD NATIVE PLATFORM
Spring XD
Spark
Pivotal HD &
Open Data Platform
Spring XD
Pivotal Greenplum
Database
Pivotal HDB
Spring XD
Pivotal GemFire
Rabbit MQ
Spring Cloud
Pivotal BDS on PCF
Pivotal Cloud Foundry
Pivotal LabsData ScienceData Engineering

FOR FURTHER INFO, CHECKOUT…
• Pivotal Data Product Info, Docs and Downloads @ http://pivotal.io/big-data
• Pivotal Blog @ http://blog.pivotal.io
• Pivotal Data Science Blog @ http://blog.pivotal.io/data-science-pivotal
• Pivotal Academy @ https://pivotal.biglms.com
Or reach out to your local Pivotal Account Executive…

36© Copyright 2015 Pivotal. All rights reserved. 36© Copyright 2013 Pivotal. All rights reserved.
Pivotal Data
Science Overview
and Use Cases
Pivotal Big Data Roadshow

DATA SCIENCE?
App Development
Analytics
Business Intelligence
Reporting
Visualization
Dashboards
Insights
Big Data
Machine Learning
Statistics
Mathematics
Time Series
Algorithms
Databases
Software
Modeling
Queries
Real-Time
Sensors
Predictive Models
ETL
Research
Hadoop
Distributed Computing
MapReduce
SQL
In-Memory
OLAP
Text Mining
Unstructured Data
Open Source
Decision Science
Ad Hoc Queries
Hacking
In-Database Analytics
Internet of Things
Data Cleansing
Sentiment

• ETL
• Unstructured
• Data Cleansing
• Sensors
Data Related
• Algorithms
• Mathematics
• Statistics
• Econometrics
• Predictive Modeling
• Text Mining
• Sentiment
• Map Reduce
Fields of Study &
Techniques
• Dashboards
• Insights
• Visualization
• Ad Hoc Queries
• Reporting
Business
Intelligence
• Software
• In-Database
Analysis
• Distributed
Computing
• Hadoop
• Open Source
Implementation
• Big Data
• Decision
Science
• Internet of
Things
• Real-Time
• Hacking
• In-Memory
Industry
Buzzwords

What is Data Science?
The use of statistical and machine learning techniques on
big multi-structured data in a distributed computing
environment to identify correlations and causal
relationships, classify and predict events, identify patterns
and anomalies, and infer probabilities, interest, and
sentiment.
DRIVE AUTOMATED, LOW-LATENCY ACTIONS IN RESPONSE TO EVENTS OF INTEREST

Gene Sequencing
Smart Grids
COST TO SEQUENCE
ONE GENOME
HAS FALLEN FROM
$100M IN 2001
TO $10K IN 2011
TO $1K IN 2014
READING SMART METERS
EVERY 15 MINUTES IS
3000X MORE
DATA INTENSIVE
Stock Market
Social Media
FACEBOOK UPLOADS
250 MILLION
PHOTOS EACH DAY
Billions of Data Points
Oil Exploration
Video Surveillance
OIL RIGS GENERATE
25000
DATA POINTS
PER SECOND
Medical Imaging
Mobile Sensors

What is Big Data Analytics?
Descriptive
Analytics
WHAT HAPPENED?
Diagnostic
Analytics
WHY DID IT HAPPENED?
Predictive
Analytics
WHAT WILL HAPPEN?
Prescriptive
Analytics
HOW CAN WE MAKE IT HAPPEN?
Complexity
Value of
Analytics
($)

P L A T F O R M
Data Science Toolkit
KEY TOOLS KEY LANGUAGES
SQL

Scalable, In-Database ML
• Open Source https://github.com/madlib/madlib
• Works on Greenplum DB, HAWQ and PostgreSQL
• In active development by Pivotal
• Downloads and Docs: http://madlib.net/

Functions
Supervised Learning
Regression Models
• Cox Proportional Hazards Regression
• Elastic Net Regularization
• Generalized Linear Models
• Linear Regression
• Logistic Regression
• Marginal Effects
• Multinomial Regression
• Ordinal Regression
• Robust Variance, Clustered Variance
• Support Vector Machines
Tree Methods
• Decision Tree
• Random Forest
Other Methods
• Conditional Random Field
• Naïve Bayes
Unsupervised Learning
• Association Rules (Apriori)
• Clustering (K-means)
• Topic Modeling (LDA)
Statistics
Descriptive
• Cardinality Estimators
• Correlation
• Summary
Inferential
• Hypothesis Tests
Other Statistics
• Probability Functions
Other Modules
• Conjugate Gradient
• Linear Solvers
• PMML Export
• Random Sampling
• Term Frequency for Text
Time Series
• ARIMA
Aug 2015
Data Types and Transformations
• Array Operations
• Dimensionality Reduction (PCA)
• Encoding Categorical Variables
• Matrix Operations
• Matrix Factorization (SVD, Low Rank)
• Norms and Distance Functions
• Sparse Vectors
Model Evaluation
• Cross Validation
Predictive Analytics Library

A single address for everything analytics
Analytics with Pivotal
Time-to-Insights
FORECASTING CLUSTERING
REGRESSION
CLASSIFICATION
OPTIMIZATION

Smart Systems = Sensors + Digital Brain + Actuators
Problem
Formulation
Modeling
Step
Data Step
Application
Step
Data Science for
Building Models
Sensors &
Actuators
Data Lake

Data Science
Use Cases

Financial Services

Identifying and Pricing Cross-Sell Opportunities
CUSTOMER
A global financial services provider
BUSINESS PROBLEM
Identify cross-sell opportunities between two
business arms of a financial institution.
CHALLENGES
Integration of large-scale data originating from
multiple data warehouses. Developing predictive
models to identify novel cross-sell opportunities
within the financial institution. Evaluate the
identified cross-sell opportunities by their revenue
potential.
SOLUTIONS
 Fast integration of data in Pivotal Greenplum
Database.
 Predictive models and evaluation of profitability:
– Association rule.
– Logistic regression for each product
offered.
– Estimation of revenue opportunity.
 On-demand reporting and visualization via
custom dashboards connected to
in-database models.
 Identified multi-million dollar opportunities for the
bank.

Financial Compliance
BUSINESS PROBLEM
• Ensure compliance with Dodd-Frank and Basel
Committee regulations
• Identify underlying risk and fraud while reducing the
compliance department’s overburdened
Emails Chats Trades
Transactions Policy Securities
Phone Calls Watch Lists …
Financial compliance
Data Lake
Data
integration
Data
clean up
Modeling
Classification
and ranking
Analyst user interfaces
Feedback
Analytics
Analyst feedback
Data integration: e.g., append
trade information with email
and chat communications
Data cleanup: e.g.,
identify newsletters and
spam emails
Modeling:
• Predictive modeling to flag
messages and trades
• Graph and cohort analysis
Analyst feedback
Reviewed fraud instances
included in periodic model
refreshes
SOLUTION
 A data lake platform coupled with cutting edge data
science techniques
 Flexible user interface to promote an adaptive,
continuously learning compliance framework

Telco & Mobile

Subscriber Micro-Segmentation
CUSTOMER
A major telco with cable & VOD, internet, and
phone business units
BUSINESS PROBLEM
Better understand aggregated subscriber behavior
to drive business strategy using newly available
data sources
CHALLENGES
▪ Large quantities of deep packet inspection data
and set top box data that had not been analyzed
before
▪ Needed to incorporate internet usage and TV
consumption information into pre-existing
subscriber segments
SOLUTION
▪ Generated new subscriber segments that
incorporated features based on consumption of TV
and internet services across a variety of devices
▪ Crossed new segments with existing segments to
generate new micro-segments for cross-sell/upsell
and new product development opportunities
Customized Micro-Segments

Newly Identified Behavior-Based SegmentsSubscribers
Moderates
OTT & Data Heavyweights
Portable OTT Entertainment Seekers
iPhone Heavy
Android Heavy
iPad Heavy
In-Home OTT Entertainment Seekers
In-Home Native Content Seekers
VOD Heavy
TV Heavy

Opportunities for
Data-Driven Decisions
in Pharma

Data driven drugs: From discovery to delivery
RICH DATA SOURCES
• Molecular data
• Cellular drug screens
• Animal models
• Clinical data including notes, images,
markers (e.g. genomics, lab results)
• Sensor and assay data
• Internal and partner/purchased
external data
• Contact center data
• Patient registries, public and federal
data, clinical partnerships
Clinical
Trials
Manufacturing
Marketing
Distribution and
surveillance
Drug discovery
+ development

A pipeline of sensors and opportunities for optimizing output
Internet of Things in Manufacturing
Input materials Mix Incubate Filter Centrifuge Final Product
0
5
10
15
20
25
30
0 50 100 150 200
Sensors
High-Content
Screens
TEMP
TIME
Absorbance
Elution volume
Velocity
Time
Automated raw
materials mixing

Vaccine Potency Prediction
CUSTOMER
A major pharmaceutical company
BUSINESS PROBLEM
Predict potency and antigen levels of live virus
vaccines based on manufacturing sensor data
and manual data collected throughout the
process.
CHALLENGES
 Customer’s data model was not optimal for
running analytical queries
 Manual data quality issues
 Data capture was performed with varying
consistency due to high cost associated with
manual data collection
SOLUTION
 Introduced a new data model to make data
accessible and enable analytics (including
LIMS and DeltaV)
 Built automated outlier detection/correction
methods to address manual data entry
quality issues
 Devised imputation methods to deal with
data completeness issues
 Built predictive models with high accuracy

http://blog.pivotal.io/data-science-pivotal
Check out the Pivotal Data Science Blog!

FOR FURTHER INFO…
• Or reach out to your local Pivotal Account Executive…

Pivotal Data Science Labs: Packaged Services
• Analytics
Roadmap
• Prioritized
Opportunities
• Architectural
Recommendati
ons
• Hands-on
training
• Hosted data on
Pivotal Data
stack
• Results review
&
assessment
• On-site MPP
analytics
training
• Analytics tool-
kit
• Quick insight
(2 weeks)
• Prof. services
• Data science
model building
• Ready-to-
deploy
model(s)
• Prof. services
• Data science
model building
• Ready-to-
deploy
model(s)
LAB PRIMER
(2-Week
Roadmapping)
LAB 600
(6-Week Lab)
LAB 1200
(12-Week Lab)
LAB 100
(Analytics Bundle)
DATA JAM
(Internal DS
Contest)

Data Streaming and
Predictive Analytics
Using Pivotal Big Data Suite

Converging
Trends
Innovation
New Data New
Processes
New Insights
The Journey to the Data-Driven Enterprise
Data Science
and Machine
Learning
Big Data
IoT, Mobile Apps,
Social Media

HDFS
Data Lake
Ingest Store Analytics
Hard to change
Labor intensive
Inefficient
Coding based
No real-time information
Based on expensive ETL
Migrating from a Reactive, Static and
Constrained Model…

HDFSData Lake
Expert System /
Machine Learning
In-Memory Real-
Time Data
Continuous Learning
Continuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data Sources
Real-Time Processing
Store Everything
To Pro-Active, Self-Improving, Machine
Learning Systems

New York Times Research: http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html
“
50-80% OF THE TIME ON DATA
SCIENCE PROJECTS IS SPENT ON
DATA WRANGLING
”

Data Feeds
Stream Processing
Expert Systems
Machine Learning
Historical Data
Business Value
Smart Decisions
Still…
HDFS
Data Lake

Ingest Transform Sink
SpringXD
GemFire
Data Stream Needs an Agile, Scalable and
Fast Solution
HAWQ GPDB
Data
Lake

SpringXD
Distributed
Computing
In-Memory
Real-Time Data
Spring XD Orchestrates and Automates all the
Steps on Data Stream Pipelining
Expert System /
Machine Learning
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable HAWQ GPDB
Data
Lake

INGEST / SINK PROCESS ANALYZE
• No coding required
• Dozens of built-in
connectors
• Seamless integration with
Kafka, Sqoop
• Create new connectors
easily using Spring
• Call Spark, Reactor or
RxJava
• Built-in configurable
filtering, splitting and
transformation
• Out-of-box configurable
jobs for batch processing
• Import and invoke PMML
jobs easily
• Call Python, R, Madlib and
other tools
counters and gauges
Spring XD
State of the Art Data Pipeline Automation

SpringXD
Distributed
Computing
GemFire Provides Scalable, Low-Latency
Data Access, Storage and Event Processing
Expert System /
Machine Learning
GemFire
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable HAWQ GPDB
Data
Lake

GemFire
• In-Memory Enterprise Data Grid
• Horizontally Scalable, Consistent, Highly
Available
• Event handling
• Continuous Queries
• Enterprise Data Geo Distribution
In-memory Real Time Data

SpringXD
Distributed
Computing
Pivotal Provides SQL Based
Advanced Analytics
Expert System /
Machine Learning
GemFire
Extensible
Open-Source
Fault-Tolerant
Horizontally Scalable
Data
Lake
HAWQ GPDB

HAWQ
• Massively Parallel Processing
RDBMS on HADOOP
• ANSI SQL on Hadoop
• Extremely high performance for
analytics (not like Hive)
• Stores all data directly on
HDFS
• Open-Source
Advanced SQL analytics in Hadoop
Combining SQL with Hadoop is key for analytics
SQL remains #1 choice for Data Science

SpringXD
Developers and Data Scientists Can Focus on
the Business Value of Data
GemFire
Extensible
Open-Source
Fault-Tolerant
Data
Lake
HAWQ GPDB

Data Streaming Reference Architecture
Data Feeds Transactional Apps Analytic Apps
Distributed
Computing
Real-Time Data
Expert Systems &
Machine Learning
Advanced
Analytics
HDFSData Lake

Data Streaming Reference Architecture
Data Feeds Transactional Apps Analytic Apps
HDFSData Lake
GemFire HAWQ GPDB
SpringXD

“
SO WE ARE MOVING TO A WORLD WHERE THE
MACHINES WE WORK WITH ARE NOT JUST
INTELLIGENT; THEY ARE BRILLIANT.THEY ARE
SELF-AWARE, THEY ARE PREDICTIVE, REACTIVE
AND SOCIAL. IT'S A WORLD WHERE INFORMATION
ITSELF BECOMES INTELLIGENT AND COMES TO US
AUTOMATICALLY WHEN WE NEED IT WITHOUT
HAVING TO LOOK FOR IT.
”MARCO ANNUNZIATA, GE

Demo
Powered by Pivotal Big Data Suite

It's all about DATA
Data Sources
Look for patterns
Prediction

Transform Sink
SpringXD
Extensible
Open-Source
Fault-Tolerant
Cloud-Native
Machine Learning
Enrich Filter
Split
Dashboard
Indicators
1
2
Predict
3
Real data
Simulator
/Stocks
/TechIndicators
/Predictions

Driving Real Insights Through Data Science

“
THE REAL OPPORTUNITY FOR
CHANGE...SURPASSING THE MAGNITUDE OF THE
CONSUMER INTERNET...IS THE INDUSTRIAL
INTERNET, AN OPEN, GLOBAL NETWORK THAT
CONNECTS PEOPLE, DATA AND MACHINES.
”JEFF IMMELT, CEO, GE

• Or reach out to your local Pivotal Account Executive…

BUILT FOR THE SPEED OF BUSINESS

Accelerating the Generation of
New Insights
October 27, 2015
Sarah Aerni, Data Science
Antonio Petrole, Data Engineering

Gene Sequencing
Smart Grids
COST TO SEQUENCE
ONE GENOME
HAS FALLEN FROM
$100M IN 2001
TO $10K IN 2011
TO $1K IN 2014
READING SMART METERS
EVERY 15 MINUTES IS
3000X MORE
DATA INTENSIVE
Stock Market
Social Media
FACEBOOK UPLOADS
250 MILLION
PHOTOS EACH DAY
Technology to process and store data is needed in
all industries
Oil Exploration
Video Surveillance
OIL RIGS GENERATE
25000
DATA POINTS
PER SECOND
Medical Imaging
Mobile Sensors

What is Big Data Analytics?
Descriptive
Analytics
WHAT HAPPENED?
Diagnostic
Analytics
WHY DID IT HAPPEN?
Predictive
Analytics
WHAT WILL HAPPEN?
Prescriptive
Analytics
HOW CAN WE MAKE IT HAPPEN?
Complexity
Value of
Analytics
($)

Opportunities for
Data-Driven Decisions
in Pharma

Data driven drugs: From discovery to delivery
RICH DATA SOURCES
• Molecular data
• Cellular drug screens
• Animal models
• Clinical data including notes, images,
markers (e.g. genomics, lab results)
• Sensor and assay data
• Internal and partner/purchased
external data
• Contact center data
• Patient registries, public and federal
data, clinical partnerships
Clinical
Trials
Manufacturing
Marketing
Distribution and
surveillance
Drug discovery
+ development

A pipeline of sensors and opportunities for optimizing output
Internet of Things in Manufacturing
Input materials Mix Incubate Filter Centrifuge Final Product
0
5
10
15
20
25
30
0 50 100 150 200
Sensors
High-Content
Screens
TEMP
TIME
Absorbance
Elution volume
Velocity
Time
Automated raw
materials mixing

Vaccine Potency Prediction
CUSTOMER
BUSINESS PROBLEM
Predict potency and antigen levels of live virus
vaccines based on manufacturing sensor data
and manual data collected throughout the
process.
CHALLENGES
 Customer’s data model was not optimal for
running analytical queries
 Manual data quality issues
 Data capture was performed with varying
consistency due to high cost associated with
manual data collection
SOLUTION
 Introduced a new data model to make data
accessible and enable analytics (including
LIMS and DeltaV)
 Built automated outlier detection/correction
methods to address manual data entry
quality issues
 Devised imputation methods to deal with
data completeness issues
 Built predictive models with high accuracy

Interpreting the utility of a measure obtained during manufacturing based
on model outcomes
Sample model insights
 Some features may reveal
tunable parameters to
alter potency, others may
simply be markers
 Features consistently
absent from models may
be uninformative for
predicting potency
 Opportunities to provide
real-time feedback on data
entry errors and predicted
potency outcomes
Assayed value Duration of a step
Potency
Potency
Correlation=0.45 Correlation=0.38

Need for new environments to process big data?
HDFS STORAGE AND MPP
ARCHITECTURES DISTRIBUTE STORAGE
AND PREVENT DATA MOVEMENT
VARIETY/VELOCITY
DISTRIBUTED COMPUTATION FOR
PARALLELIZATION
PETABYTES OF DATA
OPEN-SOURCE LIBRARY FOR MACHINE
LEARNING AT SCALE AND FRAMEWORK
TO ACCESS COMMON LANGUAGES
RAPIDLY EVOLVING FIELD
OF DATA SCIENCE AND
TOOLS
SQL ENGINE AND ODBC/JDBC
CONNECTIONS TO HADOOP
MANY EXISTING LIBRARIES,
TOOLS AND EXPERTISE
FLEXIBLE
SCALABLE
ENABLING
ACCESSIBLE

Multiple tools with a single, simple goal: Distributed
storage with in-place computation
Pivotal
Hadoop
Pivotal Greenplum
Database
HAWQ

Think of it as multiple PostGreSQL servers
Segments/Workers
Master
Rows are distributed across segments by a particular field (or randomly)
Pivotal
Hadoop
Pivotal Greenplum
Database
HAWQ

Identifying duplicates: counting with grouping
Opportunities for
performance improvements
– Sorting and re-sorting is
required in many
pipelines
– Single-threaded
processes create
bottlenecks in speed and
need to move data
https://www.broadinstitute.org/gatk//events/2038/GATKwh0-BP-1-Map_and_Dedup.pdf
Solution: Leverage Pivotal’s
distributed MPP environment by using
common database functions

Reference genome
Mapped reads

Duplicates
1
3
1
1
1
1
5
1
3
select locus, count(*) from reads
group by locusReference genome
Mapped reads

Reference genome
Mapped reads
Counting numbers of reads mapped to features
select exon, count(*) from
reads JOIN refseq
ON(<reads overlap exon>)
group by exon
5
17
12

Think of it as distributed file system with very large blocks of data
Schema on read allows flexibility for a variety of datasets
Compute using a variety of paradigms (e.g. MapReduce)
Pivotal
Hadoop
Pivotal Greenplum
Database
HAWQ
Name Node
Data
Node 1
Data
Node 2
Data
Node 3
Data
Node 4
1 2 3 2 3 1 1 2

• SQL compliant
• World-class query optimizer
• Interactive query
• Horizontal scalability
• Robust data management
• Common Hadoop formats
• Deep analytics
Pivotal
Hadoop
Pivotal Greenplum
Database
HAWQ
Think of it as distributed PostGreSQL (GPDB) on
Hadoop
• SQL compliant
• World-class query optimizer
• Interactive query
• Horizontal scalability
• Robust data management
• Common Hadoop formats
• Deep analytics

A single address for everything analytics
Analytics with Pivotal
Time-to-Insights
FORECASTING CLUSTERING
REGRESSION
CLASSIFICATION
OPTIMIZATION

P L A T F O R M
Data Science Toolkit
KEY TOOLS KEY LANGUAGES
SQL

Historically data was studied in silos
BRCA dataset
Treatments
Protein Assays
Imaging
VariantsPatient History &
Follow-Ups
Gene
Expression
Copy Number
Variation
miRNA

Genomics
Data Center
Researcher
Computing
ClusterUnnecessary data
movement
Network usage
Need for new environment: Data movement
Computation and storage in a single location

In-database genome-wide association study
Network
Interconnect
Master
Severs
Segment
Severs
SQL & RCOVARIATES GENOTYPES
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
SNP
1 2 M
A
A
C
C
TT
A
T
C
G
TT
A
A
G
G
T
C
TT C
G
T
C

Network
Interconnect
Master
Severs
Segment
Severs
SNP1 SNP2 SNPM
SQL & R
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
Indiv SN
P
Gen
o
1 1 AA
2 1 AT
3 1 AA
1 2 CC
2 2 CG
3 2 GG
N M TC
COVARIATES GENOTYPES

Network
Interconnect
Master
Severs
Segment
Severs
SNP1 SNP2 SNPM
Pval1 Pval2 PvalM
SQL & R
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
Indiv SN
P
Gen
o
1 1 AA
2 1 AT
3 1 AA
1 2 CC
2 2 CG
3 2 GG
N M TC

Network
Interconnect
Master
Severs
Segment
Severs
SNP1 SNP2 SNPM
Pval1 Pval2 PvalM
SQL & R
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
Indiv SN
P
Gen
o
1 1 AA
2 1 AT
3 1 AA
1 2 CC
2 2 CG
3 2 GG
N M TC
SNP P-value
1 2.34x10-21
2 0.395
3 7.15x10-17
M 0.000142
COVARIATES GENOTYPES RESULTS

Network
Interconnect
Master
Severs
Segment
Severs
SNP1 SNP2 SNPM
Pval1 Pval2 PvalM
SQL & R
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
Indiv SN
P
Gen
o
1 1 AA
2 1 AT
3 1 AA
1 2 CC
2 2 CG
3 2 GG
N M TC
SNP P-value
1 2.34x10-21
2 0.395
3 7.15x10-17
M 0.000142
• In-database computation of 1 million loci for thousands of
individuals in seconds
• Results are easily manipulated and explored

Network
Interconnect
Master
Severs
Segment
Severs
SNP1 SNP2 SNPM
Pval1 Pval2 PvalM
LOR1 LOR2 LORM
SQL & R
Indiv Covariates
1 2 10
1 F 23 18
2 M 39 41
3 M 50 23
N F 19 24
Indiv SN
P
Gen
o
1 1 AA
2 1 AT
3 1 AA
1 2 CC
2 2 CG
3 2 GG
N M TC
SNP P-value
1 2.34x10-21
2 0.395
3 7.15x10-17
M 0.000142
• In-database computation of 1 million loci for thousands of
individuals in seconds
• Results are easily manipulated and explored

Procedural Languages in Big Data Science
 HAWQ & PL/X can take advantage of “data
parallel” tasks by performing analyses in
parallel – embarrassingly parallel tasks
– Little/no effort required to break up the problem
into parallel tasks
– No dependency (or communication) between
tasks
 Examples of ‘data parallel’ problems:
– Counting words in documents
– Genome-Wide Association Study
– Studying network anomalies
 Sample Implementations by PDL
– Digital image processing
– Bayesian Inference with MCMC
– Parallel Bagged Decision Trees
Doc1 Doc2 DocM
Stem1 Stem2 StemM
SQL & R
Count1 Count2 Count
M
Network
Interconnect
Master
Severs
Segment
Severs

Finding Causal Variants in Lupus
Customer
Biotech Company
Business Problem
The customer wants to establish internal data
science capabilities: building a culture and
acquiring hardware and people to support it
Challenges
 Customer needs to establish a culture around
sharing and analyzing datasets for value
 Current in-house technology is unable to
support large-scale analysis (e.g. unable to
analyze genomics datasets)
 Need to learn new paradigms for analyzing data
at-scale
Solution
 Train customer employees on our solution stack,
provide one-on-one consulting and run a
hackathon
 Greatly reduced computation time enabling
implementation and results during a 30 hour
period
– Module previously requiring 30min took only 5sec
in using SQL in database
 Novel scientific discovery on untouched data
– Dataset previously untouched for 2 years due to
limited resources
– Mine for statistically significant associations
between ~400,000 variants for 1000 phenotypes

Processing images and building integrated
models at scale

Image Computation Framework
Hadoop
Sequence
File
Thousands of
Images
Image
Pre-Processing
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Feature Generation
HDFS
Map
reduce
Map
reduce
One sequence file

Hadoop
Sequence
File
Thousands of
Images
Image
Pre-Processing
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Feature Generation
HDFS HAWQ/GPDB
Map
reduce
Map
reduce
Join to
additional
datasets
Proteomics
Medical
History
Variants
Additional Datasets
Build Models at-Scale
SQLOne sequence file

Hadoop
Sequence
File
Thousands of
Images
One sequence file
Image
Pre-Processing
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Feature Generation
HDFS HAWQ/GPDB
Map
reduce
Map
reduce
Raw Pixels
img1 [rgb1-rgbK]
img2 [rgb1-rgbK]
imgN [rgb1-rgbK]
Map
reduce
Join to
additional
datasets
Proteomics
Medical
History
Variants
Additional Datasets
SQL

Hadoop
Sequence
File
Thousands of
Images
One sequence file
Image
Pre-Processing
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Feature Generation
HDFS HAWQ/GPDB
Map
reduce
Map
reduce
Raw Pixels
img1 [rgb1-rgbK]
img2 [rgb1-rgbK]
imgN [rgb1-rgbK]
Map
reduce PL/X
SQL
Join to
additional
datasets
Feature
Generation
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Proteomics
Medical
History
Variants
Additional Datasets
SQL

Representing an image in HAWQ
HAWQ enables rapid processing of multiple or extremely
large images in parallel without memory limitations
Source Image:
Col
Row
0 1 2
0
1
2
0 0
0 1
0 2
1 0
1 1
1 2
2 0
2 1
2 2
col
row
intsy
Structured:

Translating image processing to simple SQL
Function Distribution of pixel intensities
SQL SELECT intsy, count(*)
FROM tbl
GROUP BY intsy
Output 150, 5
215, 4
HAWQ enables rapid processing of multiple or extremely
large images in parallel without memory limitations
 No data movement required
 Simple SQL queries for data exploration
0 0
0 1
0 2
1 0
1 1
1 2
2 0
2 1
2 2
Source Image:
Col
Row
0 1 2
0
1
2
col
row
intsy
Structured:

Image Processing Pipeline
For Object Counting
Original
Image name # Cells
Tma_001.jpg 359
Tma_002.jpg 1892
Tma_003.jpg 871
… …
Smoothing
Average over
window of pixels
Thresholding
Select pixels under
intensity threshold
Cleanup
Min/max over
window of pixels
Object Detection
Connected
components
Object Counting
Select components
with size filter

Hadoop
Sequence
File
Thousands of
Images
One sequence file
Image
Pre-Processing
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Feature Generation
HDFS HAWQ/GPDB
Map
reduce
Map
reduce
Raw Pixels
img1 [rgb1-rgbK]
img2 [rgb1-rgbK]
imgN [rgb1-rgbK]
Map
reduce PL/X
SQL
Join to
additional
datasets
Feature
Generation
Features
img1 [x1-xM]
img2 [x1-xM]
imgN [x1-xM]
Proteomics
Medical
History
Variants
Additional Datasets
SQL

A Drug-Centric Data Lake to Enable Drug Discovery
Customer
Business Problem
Identifying promising drug targets leveraging
and integrating the vast datasets available
will reduce time and cost to bring a new
product to market
Challenges
• Data for drugs screens across multiple
modalities cannot be easily integrated
• Current environments cannot support the
growing data form high-content screens
• Researchers are unable to leverage the
entirety of datasets, instead work with
aggregates or summaries
Solution
• Proved that current customer models can be
ported, sped up and scaled in the Pivotal
environment
• Created richer models integrating multiple
types of data (genomics, images, etc)
• Improve models using the raw, most-granular
data
• Demonstrate how the availability of tools and
data enables scientists to interrogate models
and derive a deeper, actionable insight

http://blog.pivotal.io/data-science-pivotal
Check out the Pivotal Data Science Blog!

Driving Insights
from Data Lakes
Manufacturing
Demo
Matthew Ross & Antonio Petrole
Pivotal Data Engineering

Internet of What????

Industrial Internet of Things?

IoT Goes Mainstream
According to Gartner, Inc. (a technology research and
advisory corporation), there will be nearly 26 billion
devices on the Internet of Things by 2020.

GE Doubles Down
GE invests in IIoT cloud and creates Predix cloud built upon Pivotal Cloud Foundry.
GE estimates that connecting these industrial machines to the IoT could boost
global GDP by $10 trillion to $15 trillion in 20 years. McKinsey Global Institute
research holds that IoT in general could add $6.2 billion to the global economy by
2025.

Converging
Trends
Innovation
New Data New
Processes
New Insights
The Journey to the Data-Driven Enterprise
Data Science
and Machine
Learning
Big Data
Internet
of Things

IoT Key Workflows
Data Flow
Management
Reliable
Infrastructure
Enterprise Level
Tooling
● High Availability
● Fault Tolerant
● Scalable
● Data Overload
● Normalization
● Multiple Sources
and Destinations
● Workflow
Orchestration
● Admin Tooling
● Developer
Enablement

IoT Key Workflows
Data Flow
Management
Reliable
Infrastructure
Enterprise Level
Tooling
● Fault Tolerant
● Scalable
● Data Overload
● Normalization
and Destinations
● Workflow
Orchestration
● Admin Tooling
● Developer
Enablement

Data Flow Management-Data Overload
● The ability to stream
and process massive
amounts of data
● Must have a platform
that can handle that
much data without
losing or corrupting
any of it

Data Flow Management-Data Normalization
and Cleansing
● Organizing Fields
to fit into a
relational structure
● Adding extra fields
or removing
unneeded ones

Data Flow Management-Multiple Sources and
Destinations
● Stream data from
multiple different
sources
● Persist it so multiple
different destinations
● Process multiple
different data formats

IoT Key Workflows
Data Flow
Management
Reliable
Infrastructure
Enterprise Level
Tooling
● Fault Tolerant
● Scalable
● Data Overload
● Normalization
and Destinations
● Workflow
Orchestration
● Admin Tooling
● Developer
Enablement

Reliable Infrastructure-High Availability and
Fault Tolerance
● Resources are
available under any
conditions
● System stays up
even if some
resources go down

Reliable Infrastructure-Scalability
● Must be able to
handle more traffic
demand at any time
● Easily process big
data workloads

IoT Key Workflows
Data Flow
Management
Reliable
Infrastructure
Enterprise Level
Tooling
● Fault Tolerant
● Scalable
● Data Overload
● Normalization
and Destinations
● Workflow
Orchestration
● Admin Tooling
● Developer
Enablement

Enterprise Level Tooling-Workflows and Tooling
● Manage complex
flows of data
● Provide rich User
Interface
Applications

Enterprise Level Tooling-Developer Enablement
● Full Featured APIs
● Extreme module
customization if
needed

Are you making the most out of your data?

Bringing it all together

Reporting is nice, but being
able to take action is what
drives the value of a platform

Predictive
Analytics
Proactive
Monitoring
Reactive
Maintenance

ETL vs Streaming
● Data is loaded in large
batches
● Typically happens once a
day
● Analysis can only be done
once data is transformed
and persisted to data
warehouse

ETL vs Streaming
● Continuous, data streams
are “listening” for data
being emitted from
sensors
● Data can be analyzed in
stream
● Can be integrated into
data driven applications

Reactive Maintenance
● Alert is sent out to someone
on factory floor
● Worker must receive alert, and
be able to react
● Equipment is down for
however long it takes for
worker to receive notification
and complete repairs.

Proactive Maintenance Workflow
● Manager has Dashboard with
all gauges of the system
● Historical Log of Recent Alerts
are visible on the Dashboard
● Has the ability to dispatch a
worker to investigate a
specific line or robot

Predictive Analytics Workflow
● Run Machine Learning and
Data Science models
● Incorporate Business
Intelligence Tools
● Persist data to Data Lake and
run advanced queries

Data Streaming Needs an Agile, Scalable and Fast
Solution
Data Lake
Data Ingestion
Business Intelligence
Real Time
Analytics
Mobile
App

SpringXD
Spring XD Orchestrates and Automates all the Steps
on Data Stream Pipelining
HDB PHD
Data
Lake
Extensible
Open-Source
Fault-Tolerant

INGEST / SINK PROCESS ANALYZE
• No coding required
• Dozens of built-in
connectors
• Seamless integration with
Kafka, Sqoop
• Create new connectors
easily using Spring
• Call Spark, Reactor or
RxJava
filtering, splitting and
transformation
• Out-of-box configurable
jobs for batch processing
• Import and invoke PMML
jobs easily
• Call Python, R, Madlib and
other tools
counters and gauges
Spring XD
State of the Art Data Pipeline Automation

Pivotal HDB
Hadoop Native SQL
• Exceptional Hadoop Native
SQL Performance
• No compatibility risks to
SQL developers or SQL BI
tools and applications
• Support query roll-ups,
dynamic partitions and joins
• Massive MPP scalability to
petabytes
• On premise or on the cloud
• Scale your cluster out, not
up
• World class parallel loading
and unloading
• Fast performance for
complex and advanced data
analytics
• Integrated with MADLib for
advanced machine learning
• Powerful Cost-based Query
Optimizer

Driving Real Insights Through Data Science

More Related Content

What's hot (20)

Similar to Driving Real Insights Through Data Science (20)

More from VMware Tanzu (20)

Recently uploaded (20)

Driving Real Insights Through Data Science