Data and its Role in Your Digital Transformation

The Role of Data in Your
Digital Transformation
Les Klein
Field CTO Data
Pivotal
@LesKlein #PivotalForum #DigitalTransformation #Istanbul #BigData #Analytics

Great software companies leverage Big Data
to fundamentally change the consumer
experience and pioneer entirely new business
models

4© 2016 Pivotal Software, Inc. All rights reserved.
$4BN
Financial Services
$26BN
Hospitality
$50BN
Transportation
$54BN
Entertainment
$30BN
Automotive
$3.2BN
Industrial Products
CLOUD NATIVE SOFTWARE IS CHANGING INDUSTRIES
Data is Fueling Software

5© Copyright 2015 Pivotal. All rights reserved.
Hundreds of
thousands of “trip”
events each day
400+ billion of
viewing-related
events per day
Five billion
training data
points for Price
Tip feature
Disruptors Use a LOT of Data

“We’ve found that when a
host selects a price that’s
within 5% of their tip,
they’re nearly 4 times
more likely to get booked”
“The importance of accuracy and
efficiency […], will continue to
rise as we expand and improve
products like uberPOOL and
beyond.”
“Over 75% of what
people watch come
from our
recommendations”
Data manifests as features in an app

(Data)
Microservices
Loosely coupled
services architecture,
bounded by context
Cloud-Native
Platforms
Enabling continuous
delivery & automated
operations
Open Source
Database
Innovation
Extreme scale &
performance advantages,
built for the cloud
Machine
Learning
Use of predictive
analytics to build
smart apps
How are they accomplishing this?

These companies…
Release new features in minutes, multiple times a day
Support a micro-services architecture
Consume a wide range of data sources and protocols
Store and Analyze all their data
Update algorithms and predictive models daily
Continuously ask lots of questions of their data
Modify data pipelines and add processing steps daily
They are Data Driven, are you?

A lot of reports do
not make you data
driven
A lot of
dashboards do not
make you data
driven A lot of alerts do
not make you data
driven

Converging
Trends
Innovation
New Data New Processes New Insights
The Journey to the Data-Driven Enterprise
Data Science
and Machine
Learning
Big Data
Internet
of Things &
Social Media

HDFS
Data Lake
Ingest Store Analytics
Hard to change
Labor intensive
Inefficient
Coding based
No real-time information
Based on expensive ETL
Migrating from a Reactive, Static and
Constrained Model…

HDFSData Lake
Expert System /
Machine Learning
In-Memory Real-
Time Data
Continuous Learning
Continuous Improvement
Continuous Adapting
Data Stream Pipeline
Multiple Data Sources
Real-Time Processing
Store Everything
To Pro-Active, Self-Improving, Machine
Learning Systems

“Companies need to learn how to
catch
people or things in the act of doing
something and affect the outcome“
PAUL MARITZ
Executive Chairman, Pivotal
Real-time and
Personalised Information
in Context is what Wins!

What does “Customer” mean to you?

The new normal: “an audience of one”
DATA DEVICES
Media
Banks
Delivery
Services
Marketers
Government
Individual
s Employers
Analytic
Services
Advertising
Catalog
Co-ops
List
Brokers
Websites
Information
Brokers
Credit
BureausMedia
Archives
GOVERNMENTGOVERNMENT
PHONE/
TV
PHONE/
TV
INTERNETINTERNET
AD
AGENCY
AD
AGENCY
RETAILRETAIL

Your customers are a new breed

Evolving from digital natives to data
natives

“A data native is someone who expects their world to not just be digital,
but to be smart and to adjust immediately to their taste and habits.”
Beyond tech savvy …..

Digital natives program their
thermostat

Data natives expect the thermostat to
program itself

Do you know the full value of your data
and how to leverage it?

µs ms s hour day month yr+
“Fast Data” “Big Data”
year
Traditional
Systems
Pivotal Data
Science Labs
Value of
Data ($)
Time
Value of data over time…

Analytics - the journey so far (BI)
Complexity
Value of
Analytics ($)
Descriptive
Analytics
Diagnostic
Analytics
What
happened?
Why did it
happen?
Hindsight
Insight

Analytics – where you need to be today
Value of
Analytics ($)
Descriptive
Analytics
Diagnostic
Analytics
Predictive
Analytics
Prescriptive
Analytics
What
happened?
Why did it
happen?
What will
happen?
How can we
make it happen?
Hindsight
Insight
Foresight
Complexity

Interesting questions are found here
Value of
Analytics ($)
Descriptive
Analytics
Diagnostic
Analytics
Predictive
Analytics
Prescriptive
Analytics
What
happened?
Why did it
happen?
What will
happen?
How can we
make it happen?
Hindsight
Insight
Foresight
Complexity
Data science

Data Science at Pivotal
Theory-driven approach
‘start with the problem
and work towards the data’
‘start with the data
and work towards the problem’
Data-driven approach

Data Science at Pivotal
‘start with the data
and work towards the problem’
Data-driven approach

User Feedback
Scoping Data Review
Model Building
Feature Review
Feature Engineering
Model Evaluation
Operationalization
Our tried and tested approach helps extract
value from data

“
THE MAGIC HAPPENS WHEN YOU MARRY THE
TRADITIONAL ENGINEERING APPROACH WITH THE
DATA SCIENCE ENABLED BY THE DATA LAKE. IT
OPENS UP A WHOLE NEW WORLD OF POSSIBLE
‘WHAT IF’ QUESTIONS.
”DAVE BARTLETT, GE AVIATION

IT’S ALL ABOUT
HOW YOU
OPERATIONALISE
YOUR INSIGHTS

Scale-out
analytic
database
Model API Cloud Native
Application
Platform
Data
Sources
0 5
Smart Apps: Models Manifesting as Microservices

OSS
Smart/Data-driven App Technology Needs
Fast Ingest/Pipelining
• Pipelines to consume
streaming and batch data
from various endpoints
Speed/Serving Layer
• In-memory data grids for
real-time and aggregated
data sets
• Low latency
Big Data Analytics Platform
• Scale-out Storage
• Ability to process large structured and unstructured data
• Advanced SQL query capabilities
• Ability to run machine learning at scale
Platform for Scalable
Web/Mobile Apps
• Infrastructure Automation
• Develop, run and manage
Web applications without
the complexity of building
and maintaining the
infrastructure

Fast Ingest/Pipelining Speed/Serving Layer
Big Data Analytics Platform
Platform for Scalable
Web/Mobile Apps
Pivotal’s Answer for Technology Needs
Spring Cloud Data Flow GemFire
Greenplum
Database
Pivotal
HDB
Pivotal
Cloud Foundry
Spring
OSS

● Silo’d and aging database systems
● Spaghetti data pipelines
● Expensive, proprietary data
management systems
● Lack of structured platforms for
continuous software delivery
● Monolithic application architectures
● Batch-oriented data integration
● Limited operationalization of analytics
● Proprietary systems
Today’s Enterprise Challenge

Data Programming Model
Cloud-Native Platform
Microservices FrameworkPlatform Runtime
Stream Data Platform
Hadoop Spark DW
Apps & Microservices
DBMS
IMDG
K/V Store
Relational DB
Big Data & Machine Learning
Modern Cloud-Native Data Architecture

1. Modernize your data platforms:
• Improve performance and scalability for
mission-critical data workloads
• Lower overall TCO by adopting industry-leading open
source database technologies on modern cloud platforms
2. Prove out a specific business use-case:
• Build a smart app end-to-end
• Focus on a specific analytical model / data-microservice
• Automate ingestion and data pipelines
Pivotal can help you to:

Pivotal Offerings
Big Data Suite:
• Best-in-class open source data management software
• GemFire: In-Memory Data Grid
• Greenplum: Data Warehouse
• Pivotal HDB (Apache HAWQ): Hadoop-Native SQL
Pivotal Cloud Foundry
• Industry’s Leading Cloud-Native Platform
Spring Boot, Cloud and Data Flow
• Modern-Java micro-services framework
Pivotal Labs & Data Science
• Agile software development and machine learning

Let’s build something
MEANINGFUL
Let’s build something
MEANINGFUL

APPA Real-World Example

App
Development
Data
analytics
Cloud-native
App platform
Data
Science
& Model
building
Data
Micro-
service
APPA Real-World Example

App
Development
Data
analytics
Cloud-native
App platform
Data
Science
& Model
building
Data
Micro-
service
APP
Must support scale-out
query processing
Must support scale-out
query processing
Must deliver as an APIMust deliver as an API
Must embrace agile development,
focus on outcomes
Must embrace agile development,
focus on outcomes
Must support
microservices, agile dev, and
connect to big data analytics
Must support
microservices, agile dev, and
connect to big data analytics
A Real-World Example

Major Credit Card Issuer Builds Infrastructure Determining
Fraud Markers Across 30 Disparate Sources
Challenge:
• Desire to identify potential fraudulent activity within Fortune
100 credit card issuer with 100K employees
• Identifying markers across 30 internal data sources caused
lags in reporting
• Establish Infrastructure to support Palantir, Platfora and RSA
reporting frameworks
Solution:
• Provided ability to store 3 months of data at 60-70TB, interest
in moving to 3-year retention period
• Build large-scale security analytics platform to scale with data
needs, analytics and reporting requirements
• Support organizational desire to evolve from pure BI to
advanced, predictive analytics
Pivotal Solution includes: GPDB, Hadoop, HAWQ

Leading Swiss Bank Re-shapes Business to Better Monetize and
Regulate Fraudulent Activity with Big Data
Challenge:
• Needed to monetize data around privilege user access and
assure security around fraudulent activity
• Reduce the cost of monitoring privileged user access for
regulatory requirements. (currently taking 45 people to do this)
• Extend across the enterprise to solve business pains around
faster data as opposed to just longer looks in “Hadoop”
Solution:
• Built on top of Big Data Suite to operationalize, automate and
extend across the enterprise
• What took 6 hours should now take less than 1 hour with real
time decision-making
• Capability of monitoring relationship manager’s activity with
analytics to ensure regulatory compliance
Pivotal Solution includes: Big Data Suite and
Data Labs

Pivotal Data Innovation Across Industries
RETAIL / TRANSPORT
MEDIA / SERVICES
TELCO / ISP
FINANCIAL SERVICES
HIGH TECH / ELECTRONICS
EDUCATION
GOVERNMENT
OTHER
INSURANCE
AUTO / INDUSTRIAL
HEALTHCARE
MANUFACTURING

Data and its Role in Your Digital Transformation

More Related Content

What's hot (20)

Viewers also liked (15)

Similar to Data and its Role in Your Digital Transformation (20)

More from VMware Tanzu (20)

Recently uploaded (20)

Data and its Role in Your Digital Transformation