Big data analytics on teradata with revolution r enterprise bill jacobs

1877 Big Data Analytics on
Teradata:
An Introduction to
Revolution R Enterprise
Bill Jacobs
Dir., Product Marketing, Revolution Analytics

Demystifying R
 What is R
 Why is it so

popular?
 Is it only open
source?

Our view: Big Data meets Big Math =
New Business Outcomes

THE
PERFECT
STORM

+ Computing Power
+ Data
+ Pace of Business
+ Customer Expectations

+ Data Science
+ Computer Science
+ Management Science

Confidential to Revolution Analytics and shared
with Siemens under the NDA dated 27/9/2013

Better Business
Decisions

New
Business
Outcomes

4

Big Analytics Delivers Value from Big Data

Volume

Variety

Velocity

The three Vs
of Big Data:

The three V’s of Big Data Big Analytics:
Maximizing Value,
accommodating data Volatility,

while assuring Veracity of insights
5

Confidential to Revolution Analytics

R Open Source
-

Language, Community, Collaboration

-

Robert Gentleman & Ross Ihaka, 1993

-

Version 1.0 released 2000

-

2.5 Million Global Users

-

Over 4,800 add-on ―Packages‖

-

Why R?
R in Universities = New Talent

WELCOME & INTRODUCTIONS
Emerging Modeling/Visualization
Lower Cost Alternative

Open Source = Flexible & Innovative
Access to Free Packages
Confidential to Revolution Analytics and shared with Siemens under the NDA dated 27/9/2013

6

R is Exploding in Popularity & Functionality
Internet Discussion

Package Growth

Mean monthly traffic on email discussion list

Number of R packages listed on CRAN

4,000

2500

R

2000

3,000

1500

2,000

Stata
1000

SAS

1,000

SPSS
S-Plus

0
1995

2000

2005

500

0

2010

Web Site Popularity

Scholarly Activity

Number of links to main web site

Google Scholar hits (’05-’09 CAGR)

R

4,000

SAS

2,000
1,050

SPSS

900

S-Plus
Stata

600

R
SAS

46%
-11%

SPSS -27%
S-Plus
Stata

Source: http://r4stats.com/popularity

0%
10%

7

R is exploding in popularity & functionality
R Usage Growth
Rexer Data Miner Survey, 2007-2013

70% of data miners report using R

“I’ve been astonished by the rate at
which R has been adopted. Four
years ago, everyone in my
economics department [at the
University of Chicago] was using
Stata; now, as far as I can tell, R is
the standard tool, and students learn
it first.”
Deputy Editor for New Products at Forbes

24% use R as primary tool

“A key benefit of R is that it provides
near-instant availability of new and
experimental methods created by its
user base — without waiting for the
development/release cycle of
commercial software. SAS
recognizes the value of R to our
customer base…”

Source: www.rexeranalytics.com
Product Marketing Manager SAS Institute, Inc

R Is The Most Commonly Used Primarly
Analytics Tool

70% of data miners report using R

24% use R as primary tool



Example of advanced visualization with R

Facebook
Network
Graphic

10

R Community, collaboration and breadth:
CRAN task views (sub set of 4800+ packages)

Source: http://www.maths.lancs.ac.uk/~rowlings/R/TaskViews/

Confidential to Revolution Analytics and shared with Siemens under the NDA dated 27/9/2013

11

Key Big Data Challenge: The Analytics Talent Pool

12

The Analytics Talent Pool With R

2 Million R Users

13

R is open source and drives analytic innovation
but….has some limitations for Enterprises

Big Data

In-memory bound

Hybrid memory &
disk scalability

Operates on bigger
volumes & factors

Speed of
Analysis

Single threaded

Parallel threading

Shrinks analysis
time

Enterprise
Readiness

Community
support

Commercial support

Delivers full service
production support

Analytic
Breadth &
Depth

5000+ innovative
analytic packages

Leverage open
source packages
plus Big Data ready
packages

Supercharges R

Commerci
al Viability

Risk of
deployment of
open source

Commercial license

Eliminate risk with
open source

14

Our History & Our Future
V1 through V6.1

V6.2 through V9

V10 through v11

NA Offices
NYC
Dallas

Company
Founding
Relocate HQ
to Palo Alto
250
Customers

2007

500
Customers

2013

Chapter 1
Capture
Mindshare

1000
Customers

2015

Chapter 2
Mobilize with
Market Focus

Company Confidential – Do not distribute

2017

Chapter 3
Scalable
Growth

15

Revolution Confidential

200+ Customer Stories
Finance & Insurance

Academic & Gov’t

Healthcare & Life Sciences

Digital Media & Retail

Manufacturing & High Tech

16

Revolution Analytics - Overview
We are the only provider of a commercial analytics platform based on the open source
R statistical computing language.
Distributed, high performance
analytical algorithms

Power

Easier to build and deploy
analytic applications
Stable, scalable
multi-platform with

Productivity
Enterprise
Readiness

Professional services
enablement

world-wide support
World Wide Support Teams
• Standard and Premium Programs
• Technical Account Managers
• Customer Success Managers

Professional Services
• Architecture planning
• Systems Integration
• Advanced analytic applications
• Full life cycle projects
17

Customers Revolutionize their Business

Power

4X performance
50M records scored daily

“…we saw about a 4x
performance
improvement on 50
million records. It
works brilliantly.”
- CEO, John Wallace,
DataSong

Scalability
TB’s data from 200+ data sources
10’s thousands attributes
100’s millions of scores daily

“We’ve been able to scale
our solution to a problem
that’s so big that most
companies could not
address it…..”
- SVP Analytics, Kevin
Lyons, eXelate

Performance
2X data
2X attributes
no impact on performance

“We need a highperformance analytics
…we can now identify
opportunities for our clients
that would otherwise be
lost.”
- Chief Analytics Officer,
Leon Zemel, [x+1]
19

Revolution R
Enterprise
 What is Revolution R
Enterprise?
 How does Revolution R
Enterprise work with
Teradata Database?

is….
the only big data big analytics platform
based on open source R,
the defacto statistical computing language for
modern analytics

 High Performance, Scalable Analytics
 Portable Across Enterprise Platforms
 Easier to Build & Deploy Analytics

21

How is RRE Used?
Discovering Patterns
with Big Data

Building Models
Efficiently

Flexibly Deploying
Models to Consumers

 Customer
segmentation
 Market basket
analysis
 Social networking
analysis
 Fraud detection
 Marketing attribution
 Sentiment analysis
 …and much more








 Customer lifetime value
 Pricing optimization
 Recommendation
engines
 …and much more

Credit risk
Customer churn
Propensity to buy
Market risk
Operational risk
…and much more

22

Introducing Revolution R Enterprise (RRE)
The Big Data Big Analytics Platform
 Big Data Big Analytics Ready
– Enterprise readiness
DevelopR

ConnectR

DeployR

– High performance analytics
– Multi-platform architecture
– Data source integration
– Development tools

ScaleR

– Deployment tools

DistributedR

23

The Platform Step by Step:
R Capabilities
R+CRAN

RevoR

• Open source R interpreter
• UPDATED R 3.0.2
• Freely-available R algorithms
• Algorithms callable by RevoR
• Embeddable in R scripts
• 100% Compatible with existing
R scripts, functions and
packages

• Performance enhanced R interpreter
• Based on open source R
• Adds high-performance math

Available On:
•
•
•
•
•
•
•
•
•
•
•

PlatformTM LSFTM Linux®
Microsoft® HPC Clusters
Microsoft Azure Burst
Windows® & Linux Servers
Windows & Linux Workstations
Teradata® Database
IBM® Netezza®
IBM BigInsightsTM
Cloudera Hadoop®
Hortonworks Hadoop
Intel® Hadoop

24

Big Data Speed @ Scale with
Revolution R Enterprise (RRE)
In-Hadoop Execution

First, we enhance and
accelerate the Open Source
R interpreter.

In-Database Execution
Parallelized User Code

Parallelized Algorithms
Multi-Core Processing
Multi-Threaded Execution

Memory Management
Fast Math Libraries

25

Open Source R Performance: Multi-threaded
Math
Open
Customers report 5-50x
Source R

Revolution R
Enterprise

performance improvements
compared to Open Source R —
without changing any code

Computation (4-core laptop)

Open Source R

Revolution R

Speedup

Matrix Multiply

176 sec

9.3 sec

18x

Cholesky Factorization

25.5 sec

1.3 sec

19x

Linear Discriminant Analysis

189 sec

74 sec

3x

R Benchmarks (Matrix Functions)

22 sec

3.5 sec

5x

R Benchmarks (Program Control)

5.6 sec

5.4 sec

Not appreciable

Linear Algebra1

General R Benchmarks2

1. http://www.revolutionanalytics.com/why-revolution-r/benchmarks.php
2. http://r.research.att.com/benchmarks/
26

Parallelization & Data Sourcing

ConnectR
• High-speed & direct connectors

Available for:

ScaleR
• Ready-to-Use high-performance
big data big analytics
• Fully-parallelized analytics
• Data prep & data distillation
• Descriptive statistics & statistical
tests
• Correlation & covariance matrices
• Predictive Models – linear, logistic,
GLM
• Machine learning
• Monte Carlo simulation
• NEW Tools for distributing
customized algorithms across nodes

• High-performance XDF
• SAS, SPSS, delimited & fixed format
text data files
• Hadoop HDFS (text & XDF)
• Teradata Database & Aster
• EDWs and ADWs
• ODBC

DistributedR
• Distributed computing framework
• Delivers portability across platforms

Available on:
•
•
•
•
•
•
•
•

Windows Servers
Red Hat and NEW SuSE Linux Servers
IBM Platform LSF Linux
Microsoft HPC Clusters
Microsoft Azure Burst
NEW Teradata Database
NEW Cloudera Hadoop
NEW Hortonworks Hadoop

27

Big Data Speed @ Scale with
Revolution R Enterprise (RRE)
In-Hadoop Execution

Second, we built a platform for hosting
R with Big Data on a variety of
massively parallel platforms.

In-Database Execution
Parallelized User Code

Parallelized Algorithms
Multi-Core Processing
Multi-Threaded Execution

Memory Management
Fast Math Libraries

28

Powering Next Generation Analytics

COMBINE INTERMEDIATE RESULTS

29

SAS HPA Speed comparison* Logistic
Regression

Rows of data

1 billion

Parameters

“just a few”

Time

80 seconds

Data location

In memory

Nodes

32

Cores

384

RAM

1,536 GB

1 billion

Double

7

45%

44 seconds
On disk

1/6th

5

5%

20

5%

80 GB

Revolution R is faster on the same amount of data, despite using approximately a 20th as many cores, a
20th as much RAM, a 6th as many nodes, and not pre-loading data into RAM.

Revolution R Enterprise Delivers Performance at 2%

of the Cost

*As published by SAS in HPC Wire, April 21, 2011
30

Analytics Layer: High Performance Big Data
Analytics with ScaleR

R Data Step

Descriptive
Statistics

Statistical
Tests

Sampling

Predictive
Modeling

Data
Visualization

Machine
Learning

Simulation

31

ScaleR: Fast Parallel External Memory
Algorithms
Data Prep, Distillation & Descriptive Analytics

R Data Step












Data import –
Delimited, Fixed, SAS, SPSS, O
BDC
Variable creation &
transformation
Recode variables
Factor variables
Missing value handling
Sort
Merge
Split
Aggregate by category
(means, sums)
Use any of the functionality of
the R language to transform
and clean data row by row!

Descriptive Statistics















Min / Max
Mean
Median (approx.)
Quantiles (approx.)
Standard Deviation
Variance
Correlation
Covariance
Sum of Squares (cross product
matrix for set variables)
Pairwise Cross tabs
Risk Ratio & Odds Ratio
Cross-Tabulation of Data (standard
tables & long form)
Marginal Summaries of Cross
Tabulations

Company Confidential – Do not distribute

Statistical Tests






Chi Square Test
t-Test
F-Test
Plus 100’s of other tests
available in R!

Sampling




Subsample (observations &
variables)
Random Sampling
High quality, fast, parallel
random number generators

32

ScaleR: Fast Parallel External Memory
Algorithms
Statistical Modeling

Predictive Models










Covariance, Correlation, Sums of
Squares (cross product matrix for set
variables) matrices
Multiple Linear Regression
Generalized Linear Models (GLM) - All
exponential family distributions:
binomial, Gaussian, inverse
Gaussian, Poisson, Tweedie. Standard
link functions including:
cauchit, identity, log, logit, probit. User
defined distributions & link functions.
Logistic Regression
Classification & Regression Trees
Decision Forests
Predictions/scoring for models
Residuals for all models

Machine Learning

Data Visualization





Histogram
Line Plot
Lorenz Curve
ROC Curves (actual data and predicted
values)
Plus numerous tools in R and ScaleR
to generate big data visualizations



Cluster Analysis


K-Means

Classification



Decision Trees
Decision Forests

Simulation




High quality, fast, parallel random
number generators
Use the rich functionality of R
for simulations
33

The Power of Revolution R Enterprise
Performance & Scalability
ScaleR
ScaleR

Moves computation to data

ScaleR

V
a
l
u
e

Moves computation to data

Leverage CRAN

ScaleR

Labor saving power

DistributedR

Maximizes computation

DistributedR

Powerful divide & conquer

DistributedR

Effective memory utilization

RevoR

3-50X faster

Open Source

Leverage latest innovation

34

Why Teradata And Revolution R Enterprise?
 Teradata User Demand
 Data Movement Penalty Growing
 New Analytics Requiring MPP Approach
 R Popularity
 Open Source Limitations
 Arrival of Teradata v14.10

35

+
Revolution Analytics coupled with the Teradata Unified Data Architecture accelerates
big data analytics using the widely-accepted R language.
Available Today:
 Scalable R analytics on servers
connected to Teradata
 High speed, parallel data transfer, 5x
faster than RODBC
 Integrated parallel analytics solution
Teradata Version 14.0

Upcoming Capabilities (4Q13)
 Parallel R in-database for big data
analytics on Teradata
 R programmers can immediately build
parallel R models completely in R
 Revolution parallel in-database
algorithms exclusively available on
Teradata

Revolution R
Enterprise 6.2
High-Speed
TPT Connector

Company Confidential

Teradata
Version 14.10
+
Revolution R
Enterprise V7

36

Introducing Revolution R Enterprise Version 7 on
Teradata Database
 New Teradata Table Operators
 New Parallelized Algorithms
 In-Database Execution of Parallelized Algorithms

 Executes R Scripts From R Workstations or Servers
 Provides Orders of Magnitude Performance Gains
 Supports Multiple Platforms in UDA
 Available Late 2013

37

Revolution Analytics in the UDA
UNIFIED DATA ARCHITECTURE
With Revolution R Enterprise

RODBC

Seamless use of R analytics across the
Teradata UDA
38

Transparent Parallelization of Analytical, Predictive Modeling and
Machine Learning in Teradata

HOW DOES IT WORK?

39

Understanding R’s Compute Workload

R Script
< 1%

Computational
Workload Breakdown

Compute Burden from Script
or Command

Compute Burden from
Algorithmic Computations

Algorithms
99.xxx%

40

ScaleR PEMAs: High Performance Analytical
Algorithms
 Users Script Calls ScaleR PEMA
– No Unique Code or Setup for Parallelism
– ScaleR Algorithms are “just another R package”

– Using PEMAs is Transparent, Automatic, Fast and Scales
Linearly

 PEMAs Transparently Parallelize Algorithm Execution
– Parallelized Versions of Statistics, Predictive Modeling and
Machine Learning Algorithms
– PEMAs Transparently Distribute Computations Across AMPs
– Results are Consolidated Into A Single Result Set
– Provides Write Once Deploy Anywhere (WODA) Portability

41

Transparent Distributed Computing with
RRE ScaleR
Transparent to the Script
 Algorithm Starts A Master Process
 Master Identifies Environment

In Revolution R Enterprise:
 Script Calls ScaleR PEMA


Algorithm Executes

 Algorithm Returns to Script
 Script Continues Execution






Threading?
Cores?
Chips?
Distributed Nodes?

 Master Initializes Algorithm


Prepares Instructions for Nodes

 Master Executes Table Operators In
Each VAMP




VAMPs process each data segment
Table Operator runs in each VAMP
Table Operator returns Intermediate Result
Object (IRO) to master process

 Master Process Combines IROs
 Returns Consolidated Answer to Script

42

ScaleR PEMAs on Teradata:
Transparent Distribution of R Analytics
Desktops &
Servers
Revolution R
Enterprise

 For Each Call to a
ScaleR Algorithm:
– One Request
– Many Subtasks
– One Answer

Corporate
Applications
Revolution R
Enterprise
ODBC

Teradata
Database
+
Revolution
R
Enterprise

Extended Stored
Procedure

Table
Operators

AMPs
43

Revolution R Enterprise Ecosystem
Power of Integration
SI / Service

Deployment / Consumption

MSP / DSP

Advanced Analytics

ETL
Corios

Data / Infrastructure

46

Tools & Deployment
DevelopR

DeployR

• Freely-available R algorithms
• Callable by RevoR
• Embeddable in R scripts

• Web services software
development kit
• Integrates R Into application
infrastructures

Available on:
• Can be called by RevoR
• Can be run singe-node
using RevoR
• Analyze large data using
RDataStep package
• Run on multiple nodes using
rxEXEC package

DevelopR

DeployR

Capabilities:
• Invokes R Scripts from
web services calls
• RESTful interface for
easy integration
• Works with leading desktop
& BI tools

47

DevelopR Integrated Development
Environment
Script with type ahead
and code snippets

Sophisticated debugging
with breakpoints ,
variable values etc.

Solutions window
for organizing code
and data

Objects loaded
in the R
Environment

Packages
installed and
loaded

Object
details

http://www.revolutionanalytics.com/demos/revolution-productivity-environment/demo.htm
48

Data Analysis
DeployR
R / Statistical
Modeling Expert

Deployment
Expert

Business Intelligence

 Seamless

Mobile Web Apps

 Bring the power of R to any web enabled application

 Simple
 Leverage common APIs including JS, Java, .NET

 Scalable
 Robustly scale user and compute workloads

 Secure

Cloud / SaaS

 Manage enterprise security with LDAP & SSO

49

Create Custom, On-Demand Analytical
Apps
Some Examples:
On-demand sales
forecasting

Leveraging the
power of R from
Microsoft tools

Real-time social
media sentiment
analysis

50

Alteryx and Revolution Analytics
Making Predictive Analytics
More Accessible and Scalable
Empowering Analysts with
Easy-to-Use Predictive
Tools combined with the
Leading R Platform

Delivering Enterprise-Scale
Predictive Analytics to Line
of Business Analysts

Enabling a Broader
Audience to Harness the
Universe of R

51

Summary.
 R is Hot.
– Most Broadly Used Analytical Language
– Its Popularity Addresses Critical Talent Gap
– Vast Functionality Via CRAN
– R Needs a Platform For Big Data Big Analytics

 Revolution Provides Enterprise-Capable Platforms for R.
– High Performance.
– Scalable via Transparent Distributed Execution
– Portable – Write Once Deploy Anywhere - WODA
– Commercial Support & Services Cut Project Risks

 Teradata + Revolution Provide a Robust Solution
– Teradata provides stable, high-performane big data environment
– Revolution provides speed, scale, portability and stability for the enterprise

52

Next steps?

The leading commercial provider of software and support for the popular
open source R statistics language.

www.revolutionanalytics.com

650.646.9545

Twitter: @RevolutionR

53

Big data analytics on teradata with revolution r enterprise bill jacobs

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Big data analytics on teradata with revolution r enterprise bill jacobs (20)

Recently uploaded (20)

Big data analytics on teradata with revolution r enterprise bill jacobs

Editor's Notes