IBM-Why Big Data?

Big Data

Paul Zikopoulos, BA, MBA
Director, IBM Information Management Technical Professionals,
WW Competitive Database, and Big Data @BigData_paulz
IBM Certified Advanced Technical Expert (Clusters and DRDA)
IBM Certified Customer Solutions Expert (DBA and BI)
paulz_ibm@msn.com

March 27, 2012 © 2012 IBM Corporation

Paul C. Zikopoulos, B.A., M.B.A., is the
Director of Technical Professionals for IBM
Software Group’s Information Management

division and additionally leads the World
Wide Database Competitive and Big Data
SWAT teams. Paul is an award-winning
writer and speaker with more than 18 years
of experience in Information Management. Paul has written
more than 350 magazine articles and 15 books including
Understanding Big Data: Analytics for Enterprise Class
Hadoop and Streaming Data, DB2 pureScale: Risk Free
Agile Scaling, Break Free with DB2 9.7: A Tour of Cost
Saving Features; Information on Demand: Introduction to
DB2 9.5 New Features; DB2 Fundamentals Certification for
Dummies; DB2 for Dummies; and more. Paul is a DB2
Certified Advanced Technical Expert (DRDA and Clusters)
and a DB2 Certified Solutions Expert (BI and DBA). In his
spare time, he enjoys all sorts of sporting activities,
including running with his dog Chachi, avoiding punches in
his MMA training, and trying to figure out the world
according to Chloë—his daughter. You can reach him at:
paulz_ibm@msn.com.

@BigData_paulz

2 © 2012 IBM Corporation

Why Big Data
How We Got Here


4
In 2005 there were 1.3 billion RFID
tags in circulation…

…by the end of 2011, this was about
30 billion and growing even faster
© 2012 IBM Corporation

An increasingly sensor-enabled and instrumented
business environment generates HUGE volumes of
data with MACHINE SPEED characteristics…

1 BILLION lines of code
5 EACH engine generating 10 TB every 30 minutes!

350B
Transactions/Year

Meter Reads
every 15 min.

120M – meter reads/month 3.65B – meter reads/day

 In August of 2010, Adam
Savage, of “Myth Busters,”
took a photo of his vehicle
using his smartphone. He
then posted the photo to his
Twitter account including the
phrase “Off to work.”

 Since the photo was taken by
his smartphone, the image
contained metadata revealing
the exact geographical
location the photo was taken

 By simply taking and posting
a photo, Savage revealed the
exact location of his home,
the vehicle he drives, and the
time he leaves for work


The Social Layer in an Instrumented Interconnected World
4.6
30 billion RFID billion
tags today
camera
12+ TBs (1.3B in 2005) phones
of tweet data world
every day wide

100s of
millions
of GPS
data every day
? TBs of

enabled
devices
sold
annually

25+ TBs of 2+
log data billion
every day people
on the
76 million smart Web by
meters in 2009… end 2011
200M by 2014

Twitter Tweets per Second Record Breakers of 2011
 Social-media analytics can
be used from healthcare to
predicting votes

 Challenges
– Volume
– Velocity
– Variety
– Language Processing: consider
that Twitter sentences are not
well formed and often use
urban talk


Can a Social Media Persona be Monetized?


Extract Intent, Life Events, Micro Segmentation Attributes

Chloe

Name, Birthday, Family
Tom Sit

Not Relevant - Noise
Tina Mu

Monetizable Intent
Jo Jobs
Not Relevant - Noise

Location Wishful Thinking

11
Relocation
Monetizable Intent
SPAMbots

1.8 ZB

1 ZB
1 ZB=1T GB

4Trillion
8GB
iPods


The Big Data Opportunity
Extracting insight from an immense volume, variety and velocity of data,
in context, beyond what was previously possible.

Variety: Manage the complexity of data
in many different structures,
ranging from relational, to
logs,
to raw text
Velocity:
Streaming data and large
volume data movement
Volume:
Scale from Terabytes to
Petabytes (1K TBs) to
Zettabytes (1B TBs)

Data In Motion
Analyzes and correlates 5M+ 500K/sec, 6B+ IPDRs analyzed
market messages/sec to execute per day on more than 4 PBs/yr.
algorithmic option trades with sustaining 1GBps.
average latency of 30 micro-secs.

height: height: height:
640 1280 640
directory: directory: directory: directory:
width: width: width:
”/img" ”/img" ”/opt" ”/img"
480 1024 480
filename: filename: filename: filename:
data: data: data:
“cow” “bird” “java” “cat”
tuples

The Big Data Conundrum
 The economies of deletion have changed….
– Leading us into new opportunities and challenges

 The percentage of available data an enterprise can analyze is
decreasing proportionately to the available to that enterprise

 Quite simply, this means as enterprises, we are getting
“more naive” about our business over time

Data AVAILABLE to
an organization

Data an organization
can PROCESS

Applications for Big Data Analytics
Smarter Healthcare Multi-channel Finance Log Analysis
sales

Homeland Security Traffic Control Telecom Search Quality

Manufacturing Trading Analytics Fraud and Risk Retail: Churn, NBO


Bigger and Bigger Volumes of Data
 Retailers collect click-stream data from Web site interactions and
loyalty card-drive transaction data
– This traditional POS information is used by retailer for shopping basket analysis,
inventory replenishment, +++
– But data is being provided to suppliers for customer buying analysis

 Healthcare has traditionally been dominated by paper-based systems,
but this information is getting digitized

 Science is increasingly dominated by big science initiatives
– Large-scale experiments generate over 15 PB of data a year and can’t be stored
within the data center; then sent to laboratories

 Financial services are seeing larger volumes through smaller trading
sizes, increased market volatility, and technological improvements in
automated and algorithmic trading

 Improved instrument and sensory technology
– Large Synoptic Survey Telescope’s GPixel camera generates 6PB+ of image
data per year or consider Oil and Gas industry

A Simple Log File
Jun 29 04:03:37 192.168.190.24 nagios: GLOBAL SERVICE EVENT HANDLER: ast-oral-
vip; remedy_oracle_db_latch-contention; (null); (null); (null); handle-all-
critical-event

DATE TIME IP ADDRESS MONITOR
Jun 29 04:03:57 192.168.190.24 nagios: SERVICE ALERT: ast-oral-
vip;remedy_oracle_db_soft-prase-ratio;CRITICAL;HARD;3;CRITICAL – Soft parse
ratio
86.63%

TYPE COMPONENT
Jun 29 07:50:57 192.168.190.24 nagios: SERVICE ALERT: ast-oral
vip;remedy_oracle_db_latch-waiting; OK;SOFT;2;OK – SGA latch xssinfo freelist
(#350)
sleeping 0.000000% of the time

Jun 29 07:50:57 192.168.190.24 nagios: GLOBAL SERVICE EVENT HANDLER: ast-oral-
vip;remedy_oracle_db_latch-waiting; (null); (null); (null); handle-all-
critical-event


Click Stream Analysis Use Case
Goal: Determine how many abandoned shopping carts there are and
where they were abandoned.
Input: Web log data (IP address, timestamp, URL, HTTP return codes).
Output: List of IP addresses and last page visited by user.

Web log input Map Step
K2 V2
201.67.34.67 | 10:24:48 | www.ff.com | 200 Make IP 201.67.34.67 10:24:48,www.ff.com
123.99.54.76 | 11:15:21 | www.ff.com/search? | 200 address the
123.99.54.76 11:15:21,www.ff.com/search?
231.67.66.84 | 02:40:33 | www.ff.com/addItem.jsp | 200
key and
1 line timestamp 231.67.66.84 02:40:33,www.ff.com/addItem.jsp
101.23.78.92 | 07:09:56 | www.ff.com/shipping.jsp | 200
and URL the 101.23.78.92 07:09:56,www.ff.com/shipping.jsp
114.43.89.75 | 01:32:43 | www.ff.com | 200 value. 114.43.89.75 01:32:43,www.ff.com
109.55.55.81 | 09:45:31 | www.ff.com/confirmation.jsp | 200
109.55.55.81 09:45:31,www.ff.com/confirmation.jsp
K1 – Ignored, V1 – 1 line of log

Reduce Step
K3 V3 Ensure K4
Shuffle/Sort/Group
201.67.34.67 10:24:48,www.ff.com something is
Framework 201.67.34.67
in cart and
does this
automatically determine last
10:25:15,www.ff.com/search?
page visited. V4
www.ff.com/shipping
10:26:23,www.ff.com/shipping


Solution Architecture – Social Media Analytics for
Media and Entertainment
Online flow: Data-in-motion analysis

Social Media Stream Computing and Analytics Timely
Decisions

Entity Predictive
Data Ingest Text Analytics: Analytics: Analytics:
and Prep Timely Insights Profile Action
Resolution Determination
Dashboard

Hadoop System and Analytics

Comprehensive
Entity
Social Media Social Media Predictive Customer
Text Analytics Analytics and
Data Customer Analytics Models
Integration
Profiles

Offline flow: Data-at-rest analysis Reports


Most Requested Uses of Big Data
 Log Analytics & Storage

 Smart Grid / Smarter Utilities

 RFID Tracking & Analytics

 Fraud / Risk Management & Modeling

 360° View of the Customer

 Warehouse Extension

 Email / Call Center Transcript Analysis

 Call Detail Record Analysis

 +++
23

Why Didn’t We Use All of the Big Data Before?

“IBM gave us an opportunity to turn our plans into something that was
very tangible right from the beginning. IBM had experts within data mining,
Big Data, and Apache Hadoop and it was clear to use from the beginning we
wanted to improve our business, not only today, but also prepare for the
challenges we will face in three to five years, we had to go with IBM.”
– Lars Christian Christensen VP Plant Siting & Forecasting


Watson’s advanced analytic capabilities can sort through the equivalent of
26 200 MILLION pages of data to uncover an answer in 3 SECONDS.Corporation
© 2012 IBM

IBM Big Data Strategy: Move the Analytics Closer to the Data

 New analytic applications drive Analytic Applications
the requirements for a big data BI / Exploration / Functional Industry Predictive Content
Reporting Visualization App
BI /
App Analytics Analytics
platform Reportin
g
– Integrate and manage the full
IBM Big Data Platform
variety, velocity and volume of data
– Apply advanced analytics to Visualization Application Systems
Development Management
information in its native form & Discovery
– Visualize all available data for ad-
hoc analysis Accelerators
– Development environment for
building new analytic applications Hadoop
Hadoop Stream
Stream Data
– Workload optimization and System
System Computing
Computing Warehouse
scheduling
– Security and Governance

Information Integration & Governance


The IBM Big Data Platform

InfoSphere BigInsights
Hadoop-based low latency
analytics for variety and volume
Data-At-Rest

MPP Hadoop

MPP MPP Stream
Information Computing
InfoSphere Information
Server Integration InfoSphere Streams
Low Latency Analytics for
High volume data streaming data
integration and Velocity, Variety & Volume
transformation
MPP Data Data-In-Motion

Warehouse
Netezza High Netezza 1000
Capacity Appliance BI+Ad Hoc Analytics on
Queryable Archive for Structured Data
Structured Data

InfoSphere Warehouse Smart Analytics System
Large volume structured Operational Analytics on
data analytics Structured Data


Deep Analytics Appliance – Revolutionized Analytics

Purpose-built
analytics appliance
Dedicated High
Performance
Speed: 10-100x faster than
traditional systems Disk Storage

Simplicity: Minimal administration
and tuning

Scalability: Peta-scale user data
capacity Blades With
Custom FPGA
Accelerators
Smart: High-performance
advanced analytics


Complementary Analytics
Traditional Approach New Approach
Structured, analytical, logical Creative, holistic thought, intuition

Data Hadoop
Warehouse Streams
Transaction Data Web Logs

Internal App Data Social Data
Structured
Structured Unstructured
Repeatable Enterprise
Unstructured
Exploratory
Mainframe Data
Repeatable Integration Exploratory Text Data: emails
Linear Iterative
Linear
Monthly sales reports Iterativesentiment
Brand
Profitability analysis Product Sensor data: images
strategy
OLTP System Datasurveys
Customer Maximum asset utilization

ERP data Traditional New RFID
Sources Sources


Most Client Use Cases Combine Multiple Technologies
Pre-processing
Ingest and analyze unstructured data
types and convert to structured data

Combine structured and unstructured analysis
Augment data warehouse with additional external
sources, such as social media

Combine high velocity and historical analysis
Analyze and react to data in motion; adjust
models with deep historical analysis

Reuse structured data for exploratory analysis
Experimentation and ad-hoc analysis with
structured data


Data Governance

ed Some
im es term “Sche
times
t e r me
Somet a Fi rs
t” ma L d
m ater”
“Sche
 Separation of Duties  Separation of Duties
– Users can’t delete their audit logs – No authorized access

 Privilege Users  Privilege Users
– Monitoring authorized and privilege – Monitoring authorized and privilege
user activities user activities

 Sensitive Data (in tables)  Sensitive Data (in the file system)
– Protecting and blocking unauthorized – Protecting and blocking unauthorized
access to the system and the data access to the system and the data

 Workflow for audit and compliance  Workflow for audit and compliance
controls to validate proper security controls to validate proper security
procedures (satisfies regulations!) procedures (satisfies regulations!)
32 Internal Use Only © 2012 IBM Corporation

Data Security Design Goals

ed Some
imes term tim
“Sche es termed
Somet a Fi rs
t” ma L
m ater”
“Sche
 Minimal impact to the Database  Minimal impact to the Big Data
server resources server resources
 No Database configuration changes  No Big Data configuration changes
 Separation of Duties: Preventing  Separation of Duties: Preventing
privilege users from doing malicious privilege users/jobs from doing
activities malicious activities
 Audit trail with very granular details  Audit trail with very granular details
of Database activity of Big Data activity
 Real-time alerting and blocking  Real-time alerting and blocking
 Minimal impact to the network  Minimal impact to the network
 100% Database activity visibility  100% Big Data activity visibility
 Heterogeneous support  Heterogeneous support

33 Internal Use Only © 2012 IBM Corporation

Guardium

Customer Identification Privacy
Master Data Management
Master Data Management Data Privacy
Data Privacy
InfoSphere
InfoSphere Optim for Test Data,
Quality DB2 Redaction, +++
MDM
Stage

Data
Customer Intelligence Appliance
Data
Quality
Quality
Data Models Out-of-the-box analytics
Information Server

Cognos

Pre-built
Customer Integration behavioral IBM Global Business
Appliance attributes Services

IBM Retail Data Model Core Metrics Unica

Enterprise Data Warehouse
Enterprise Data Warehouse Applications and Operational Analytics
Applications and Operational Analytics

Online Archive OLTP and Big Data Integration
Managing Growth
Managing Growth Built-in Integration into Big Data
Built-in Integration into Big Data

Optim Data Archive DB2 SAP - HANA solidDB
Informix
DB2


IBM Flattens the Time to Value Big Data Curve

+ + + + + +
Harden Text/ML
Velocity Hadoop ETL Dev. Tooling Visualization
File System Analytics

Harden
100+Big Data Tooling
Dev. Text/ML
Velocity Hadoop ETL Visualization
File System Analytics
Business Partners
+ + + +
OR

Signed +


First Ever Forrester Wave on Big Data

“IBM has the deepest
Hadoop platform and
application portfolio. IBM,
an established EDW vendor,
has its own Hadoop
distribution; an extensive
professional services force
working on Hadoop projects;
extensive R&D programs
developing Hadoop
technologies; connections to
Hadoop from its EDW.”
–The Forrester Wave™: Enterprise
Hadoop Solutions, 1Q12


Open Source
Technology Behind
Hadoop


A Difference of Processing Models
 SETI@home is a Computational Processing Model
– Service for Extraterrestrial Intelligence (SETI) uses
unused desktop CPU processing power to perform
wide-spread analysis of radio telescope data
– Pushes data to the program for processing
– Data to function
 MapReduce is a Data Processing Model
– Data processing primitives of Mappers and Reducers
– Can be complex to write MapReduce programs, but a
very simple configuration change makes it scale to
1000s of nodes
– Function to data


Building Blocks of Hadoop


Hadoop Framework
 Hadoop Common
– A utility layer that provides access to the HDFS and projects

 HDFS
– Data storage platform for Hadoop framework
– Can scale to massive size when distributed over
multiple computers

 MapReduce
– Framework to process data across a node clusters
– MAP process splits work by first mapping the input
across the control nodes of the cluster, then splitting
the workload into even smaller data sets and distributing
it further throughout the computing cluster
• Allows MPP – think DB2 DPF ‘like’
– REDUCE collects and combines the nodes’ answers to deliver a result


Hadoop Explained
 Hadoop computation model
– Data stored in a distributed file system spanning many inexpensive computers
– Bring function to the data
– Distribute application to the compute resources where the data is stored
 Scalable to thousands of nodes and petabytes of data
public static class TokenizerMapper
public static class TokenizerMapper
Hadoop Data Nodes
extends Mapper<Object,Text,Text,IntWritable> {
extends Mapper<Object,Text,Text,IntWritable> {
private final static IntWritable
private final static IntWritable
one = new IntWritable(1);
one = new IntWritable(1);
private Text word = new Text();
private Text word = new Text();
public void map(Object key, Text val, Context
public void map(Object key, Text val, Context
StringTokenizer itr =
StringTokenizer itr =
new StringTokenizer(val.toString());
new StringTokenizer(val.toString());

1. Map Phase
while (itr.hasMoreTokens()) {
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
word.set(itr.nextToken());
context.write(word, one);
context.write(word, one);
}
}

(break job into small parts)
}
}
}
}
public static class IntSumReducer
public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWrita

2. Shuffle
extends Reducer<Text,IntWritable,Text,IntWrita

Distribute map
private IntWritable result = new IntWritable();
private IntWritable result = new IntWritable();
public void reduce(Text key,
public void reduce(Text key,
Iterable<IntWritable> val, Context context){

tasks to cluster
Iterable<IntWritable> val, Context context){

(transfer interim output
int sum = 0;
int sum = 0;
for (IntWritable v : val) {
for (IntWritable v : val) {
sum += v.get();
sum += v.get();
. . .
. . . for final processing)

MapReduce Application
3. Reduce Phase
(boil all output down to
Shuffle a single result set)

Result Set Return a single result set


Growing the Hadoop Environment
 Avro: Data serialization system that converts
data into a fast, compact binary data format
– Avro data can be stored in a file with versioned schemas

 Chukwa: Large-scale monitoring system that provides
insights into the Hadoop distributed file system
and MapReduce

 HBase is a scalable, column-oriented distributed
database modeled after Google's BigTable distributed
storage system

 Hive is a data warehouse infrastructure that provides
ad hoc query and data summarization for Hadoop
Supported data
– Hive utilizes SQL-like query language call HiveQL
– HiveQL also used by programmers to run custom MapReduce jobs.

Growing the Hadoop Environment
 Mahout is a data mining library designed to work on
the Hadoop framework
– Mahout delivers a core set of algorithms designed for clustering,
classification and batch-based filtering.

 Pig is a high-level programming language and execution
framework for parallel computation
– Pig works within the Hadoop and MapReduce frameworks

 ZooKeeper provides coordination, configuration and
group services for distributed applications working over
the Hadoop stack


Committed to Open Source
 Decade of lineage and contributions
to the open source community
– Apache Hadoop and Jaql, Apache Derby,
Apache Geronimo, Apache Jakarta, +++
– Eclipse: founded by IBM
– Significant Lucene contributions via IBM
Lucene Extension Library (ILEL)
– DRDA, XQuery, SQL, XML4J, XERCES,
HTTP, Java, Linux, +++

 IBM products built on open source
– WebSphere: Apache
– Rational: Eclipse and Apache
– InfoSphere: Eclipse and Apache, +++

 IBM’s BigInsights (Hadoop) is 100%
open source compatible with
no forks


Learn Hadoop: At Your Place, At Your Pace
Making Learning Hadoop Easy and Fun

 Flexible on-line delivery allows
learning @your place and
@your pace

 Free courses, free study materials

 Cloud-based sandbox for
exercises – zero setup

 8500+ registered students

 Hadoop Programming Challenge is
sending 3 students to IOD 2011
Conference, all expenses paid


Why IBM
for Big Data
The Solution Side

Paul Zikopoulos, BA, MBA
Director, IBM Information Management Technical Professionals,
WW Competitive Database, and Big Data
IBM Certified Advanced Technical Expert (Clusters and DRDA)
IBM Certified Customer Solutions Expert (DBA and BI)
paulz@ca.ibm.com


A Big Data Platform
Analytics Excellence In-Motion Operational At-Rest Operational Excellence
Text Analytics Toolkit Excellence Harden Hadoop - GPFS
Machine Learning Toolkit Surface Area Lock Down
Unrivalled….
Industry Accelerators
Development Tooling Embrace and Extend Policy Driven Retention & Immutability
Role-Based Security
Visualization Tooling Adaptive MapReduce
Deployment Tooling (“App Workload Manager
Store”) Fast Splittable CMX Compression
$14B in 5 yrs. on Analytics REST-exposed Administration
+++ Open Source
IBMHadoop
Big Data +++
Platform

In-Motion At-Rest
Analyze extreme amounts of Beyond traditional
data in milliseconds structured data
Uses same analytics as BigInsights uses same analytics as Streams
BigInsights No forked, not ported: Hadoop Extended with
Data can be analyzed on the way operational excellence and security
into the enterprise for earlier Netezza for in-database MapReduce
pattern detection MPP Data Warehouses


Committed to Innovation
 $100M new investment into
analytics announced 2Q11

 IBM spent $14+ billion in 24
analytics acquisitions in 5 years

 IBM has the largest commercial
research organization on Earth
– 200+ mathematicians developing
breakthrough analytics

 Largest patent portfolio in
the industry

For 19 consecutive years IBM inventors have received the most U.S. Patents

6,000
More than
48
patents in a single year © 2012 IBM Corporation

A Big Data Platform for Data In-Motion and At-Rest
Data Ingest

01011001100011101001001001001
110100101010011100101001111001000100100010010001000100101 11000100101001001011001001010
01100100101001001010100010010
Opportunity Cost Starts Here

01100100101001001010100010010
11000100101001001011001001010
01100100101001001010100010010
Bootstrap 01100100101001001010100010010
Enrich 01100100101001001010100010010
01100100101001001010100010010
11000100101001001011001001010
01100100101001001010100010010
01100100101001001010100010010
01100100101001001010100010010
Adaptive 01100100101001001010100010010
01100100101001001010100010010
Analytics 11000100101001001011001001010
01100100101001001010100010010
Model 01100100101001001010100010010
01100100101001001010100010010
11000100101001001011001001010


Data In Motion
 Hear what’s going on miles away to optimize
perimeter displacements

 Perspective: Try to find the word “Zero” in a
1000 MP3 song library in a fraction of a second
– Figure out the difference between the sound of a
human whisper and the wind


Average cluster size is 100 nodes

one
hour
What if of this 100-node
cluster cost

51
$34 © 2012 IBM Corporation

Ease of Use for Developers and Users

End-user Visualization Development Environment
Data exploration, crawling, Familiar coding and tooling
and analytics environment, testing, and optimization


What is BigSheets?
Browser-based Big Data analytics tool for business users

Big Data Challenges… How can BigSheets help?
 Business users need a no  Spreadsheet-like discovery interface lets
programming approach for business users easily analyze Big Data
analyzing Big Data with ZERO PROGRAMMING

 Extremely difficult to find
actionable business insights in  BUILT-IN “readers” can work
data from multiple sources with with data in several common formats
different formats – JSON arrays, CSV, TSV, Web
crawler output, . . .

 Translating untapped data into
actionable business insights is  Users can VISUALLY combine and
a common requirement that
explore various types of data to identify
requires visualization
“hidden” insights


8% spread
change
affecting
brand

Over 1M
tweets
analyzed

2nd most
Tweets/sec
run rate as
of 1Q12


Big Data Made Easy for the Little Guy
 USC’s Film Forecaster correctly predicted a
clamor for "Hangover 2” that resulted in $100
million opening over Memorial Day weekend
– Looked at 250K-500K Tweets and broke down
positive and negative messages using a lexicon
of 1700 words

The Film Forecaster sounds like a
big undertaking for USC, but it really
came down to one communications
masters student who learned Big
Sheets in a day, then pulled in the
tweets and analyzed them - Ryan Kim


Running Applications from the Web Console


A Rich Management Big Data Tool

Quick links to
common functions

Learn more
through external
Where and how to begin
Web resources
performing common administrative
or analytical tasks


A Rich Management Big Data Tool


Rich Big Data Development Environment
 Eclipse base Development tools
– For JQAL, Hive, PIG, Java
MapReduce and Text Analytics


The Path to Efficiency: Declarative Languages
Offline Runtime

Development Environment

Extracted
Declarative Language
Declarative Language Select
Objects
Regex
create view MonetizableIntent as
select P.name as product, Join
I.clue as strength Select
from Intent I, Product P Intent Product
Join
Runtime
Runtime
where
Follows(I.clue, P.name, 0, 20) Regex
Product
and Not(ContainsRegex(/b(not)b/, Cost-based
LeftContext(I.clue, 10))); Intent optimization Input
…

Documents

Join

Select
Product
 Streams, Text Analytics, Machine Regex  High-throughput
Learning, SQL all are declarative,
simple to learn languages Intent  Small memory footprint
 All have strong development
tooling and accelerators  Optimizes for tasks:
 IBM has been optimizing
declarative languages for decades,  Example, Text
IN FACT, IBM INVENTED IT! Analytics needs CPU
optimization


Analytic Accelerators Designed for Variety

Text Simple &
(listen, verb), Acoustic
(radio, noun) Advanced Text

Mining in Advanced
Microseconds Mathematical Models

Predictive ∑ R ( st , at )
population
Statistics

GeoSpatial Image & Video


Accelerators Improve Time to Value
Telecommunications Retail Customer
CDR streaming analytics Intelligence
Deep Network Analytics Customer Behavior and
Lifetime Value Analysis

Finance Social Media
Streaming options trading Analytics
Insurance and banking DW Sentiment Analytics, Intent to
models purchase

Public Data Mining
Transportation Streaming statistical
Real-time monitoring and analysis
routing optimization

Over 100 sample User Defined Standard Industry Data
applications Toolkits Toolkits Models
Banking, Insurance, Telco,
Healthcare, Retail

Telecommunications CDR Analytic Accelerator
Analyze Call Detail
Records in Real Time

Streaming Analytic Accelerators
– CDR dropped call analysis
– Determine VIP customers with service
issues – proactive alerts
– CDR Adapters – ASN.1, Binary, ASCII
– Analytic Operators – CDR de-duplication,
dropped call detection, termination reason,
customer importance
– Visualization – real-time KPI dashboard

Data Warehouse Appliance
– Integrated network, devices, customer, and
services model
– Telecom model, KPIs, and KQIs


Without a Big Data Platform IBM Big Data Platform
You Code…
Over 100 sample applications and
toolkits with industry focused
Event
toolkits with 300+ functions and
Custom SQL
Handling and operators!
Scripts
Multithreading Streams provides development, deployment,
runtime, and infrastructure services
Check Application
Pointing Management
Accelerators
HA and

Toolkits

Performance Debug
Connectors
Optimization

“TerraEchos developers can deliver
Security applications 45% faster due to the agility
of Streams Processing Language…”
– Alex Philip, CEO and President


Streams Runtime Illustrated
Optimizing scheduler assigns PEs Dynamically add
to nodes, and continually manages nodes and jobs
resource allocation

Commodity hardware – laptop, Add in SPSS
blades or high performance clusters jobs in the flow

Meters

Company Usage
Model
Filter Temp
Meters Action

Text Season Daily
Usage Extract Adjust Adjust
Contract

Text
Extract
Degree
History
Compare
History Store
History

X86 Host X86 Host X86 Host X86 Host X86 Host


Streams Runtime Illustrated

PEs on busy nodes, can PEs on failing nodes can be
be moved manually by the moved automatically, with
Streams administrator communications re-routed

Meters

Company Usage
Model
Filter Temp
Meters Action

Text Season Daily
Usage Extract Adjust Adjust
Contract

Text
Extract
Degree
History
Compare
History Store
History

X86 Host X86 Host X86 Host X86 Host X86 Host


How Text Analytics Works
Football World Cup 2010, one team distinguished
themselves well, losing to the eventual champions 1-
0 in the Final. Early in the second half,
Netherlands’ striker, Arjen Robben, had a
breakaway, but the keeper for Spain, Iker Casilas
made the save. Winger Andres Iniesta scored for
Spain for the win.

World Cup 2010 Highlights

Arjen Robben Striker Netherlands
Iker Casilas Keeper Spain
Andres Iniesta Winger Spain


What’s Wrong with Text Analytics Today
 Current alternative approaches and infrastructure for text
analytics present challenges for analysts
– They tend to perform poorly (in terms of accuracy and speed)
– They are difficult to use

 These alternative approaches rely on the raw text flowing only
forward through a system of extractors and filters
– Inflexible and inefficient approach, often resulting in redundant processing

 Existing toolkits are also limited in their expressiveness
– Analysts having to develop custom code
– Programmer  Analyst (think Java Developer  DBA struggles)
– Leads to more delays, complexity, and difficulties getting it right
– Biggest factor hurting analyst productivity is the difficulty in determining
how the system produced a certain result


Extracting Person and Phone Relationships


Text Analytic Toolkit Example
 Semantically search Enron’s emails to support queries such as
“Find Tom’s phone” in order to find his actual phone number
– As opposed to emails containing words Tom and Phone

 Need to be able to accurately extract email entities of type Person
and Phone and the relationship between them


The Enron Email Example
 Start with a naïve set of rules to identify a PersonPhone relationship
built using some sort of editor
– Build out an extractor that know how to find a person’s name: Lorraine Smith
– Build out an extractor that knows how to find a phone number: 607-205-4493

607.205.4493 Euro spacing
xt. ext. … North American spacing
+0119054132112 Globalstar GSP-1600
911, 411 +++


The Tricky Thing About Sentiment…


Text Analytics Toolkit
 System T text analytics engine previously only embedded in IBM products and
hidden from end users
– Found in Lotus Notes, IBM e-discovery Analyzer, CCI, InfoSphere Warehouse,+++
– Almost a decade since initial release

 BigInsights is the first time IBM opens up the Text Analytics Engine
technology for customization and development

 BigInsights Text Analytic Toolkit provides developer tools, an easy to use text
analytics language, and a set of extractors for fast adoption
– Multilingual support, including support for DBCS languages

 BigInsights includes Annotator Query Language (AQL): SQL-like!
– Fully declarative text analytics language
– No “black boxes” or modules that can’t be customized.
– Tooling for easy customization because you are abstracted from the
programmatic details
– Competing solutions make use of locked up black-box modules that cannot be
customized, which restricts flexibility and are difficult to optimize for performance

Accelerating Analytics – Explainability
 Every annotation’s provenance can be visualized

 Provenance of Morgan Stanley Fax: 205-4493 shows the
FullName rule is responsible for generating incorrect annotation
– Solutions?
• Remove Morgan Stanley from FullName dictionary?
• Create new dictionary for CompanyName?



Text Analytics Toolkit Provenance Viewer
 Major challenge for analysts is determining the lineage of changes
that have been applied to text
– REALLY difficult to diskern which extractors need to be adjusted to tweak the
resulting annotations

 Provenance Viewer for interactive visualizations to display output
annotations for regression, debugging, version enhancements, +++
– Reduce development of extractors by days to weeks


Accelerating Analytics – discovery and Pattern Matching
 Contextual Clues discoverer cluster the context surrounding annotation in
order to detect frequently occurring patterns

 Illustrate clustering results between incorrect PersonPhone pairs
– Visualize frequent occurrences (bubble size) of clues: i.e. FAX and TELEFAX
– Developer improves precision of annotator by adding a rule that filters out PersonPhone
pars if the Phone is preceded by a FAX clue
– Developer finds call at text as key clue for rule extraction for PersonPhone



Accelerating Analytics – Assisted Development
 IDE that exposes sophisticated techniques to assist rule developers
throughout all states of the development cycle
– Promotes agile development because it unifies the business analyst and the
developer with a common toolset which fosters understanding
• Business Analyst helps mature and validates extractor results
• Developer understands visually that AQL enhancements needed to refine rules



IBM Text Analytics Toolkit - Development Toolset

Context Sensitive Help
Outline View

Hyperlink Navigation

Procedural
Help


BigInsights Text Analytics Development

Difference
Content Assist Viewer

Regular Expression Builder

Context Sensitive Help
Outline View

Hyperlink Navigation

Procedural
Help Design Time Validation


Text Analytics Toolkit – Development Accelerator
 Eclipse plug-ins enhance analyst productivity
– When writing AQL code, the editor features syntax highlighting, and automatic
detection of syntax error at design time not runtime
– Pre-built extractors and sample test tools, +++

 Reduce coding time and debugging by 30-50%+!


Text Analytics Toolkit – Performance Accelerator
 AQL code is highly optimized for MapReduce because of its
declarative nature

 Unlike other frameworks, AQL optimizer determines order of
execution of the extractor instructions for maximum efficiency

 Deliver analysis up to 10x faster than other leading alternative
frameworks running the same extractors


Business Task: Find the Pictures that Have Cats


BUT! Your Application Returns the Following…
PRECISION
(a measure of exactness)

2/4
RECALL
(a measure of coverage)

2/5

IBM Finds RIGHT Answers Better Than Anyone


IBM’s Hadoop System Provides Unique Business Value

 Optimized beyond open source
Hadoop
– Workload optimization, security, +++

 Integration with enterprise systems
– Connectors for multiple data sources

 Accelerators reduce development
and implementation times
– Industry & application accelerators
– Analytic accelerators

 Visualization tools enable business
users to explore Big Data


University of Southern Cal
Political Debate Monitoring

 Solution to measure public sentiment
during the Republican Primary and
Presidential Debates
 Examines trends, volume, and content
of millions of public Twitter messages
in real-time
 Analytic accelerators to understand
sentiment (positive, negative, neutral)
- Stream Computing and visualization
 Benefits
- Real-time display of public sentiment as
candidates respond to questions
- Debate winner prediction based on
public opinion instead of solely political
analysts

Cisco turns to IBM big
data for intelligent
infrastructure
m anagem ent
 Optimize building energy
consumption with centralized
monitoring and control of
building monitoring system
 Automates preventive and
corrective maintenance of
building corrective systems
 Uses Streams, InfoSphere
BigInsights and Cognos
- Log Analytics
- Energy Bill Forecasting
- Energy consumption optimization
- Detection of anomalous usage
- Presence-aware energy mgt.
90
90 - Policy enforcement 2012 IBM Corporation
©

Stream Computing Provides Unique Business Value

 Real-time answers = low latency
insight
– Better outcomes for time sensitive
applications (e.g. fraud detection, network
management)

 Solution when data is too large or
expensive to store
– Analyze data as it comes to you
– Persist data of interest for deeper analysis

 Insights derived across multiple
streams
– Fuse streams for new insights


Asian telco reduces
billing costs and
improves customer
satisfaction
Capabilities:

Stream Computing
Analytic Accelerators

Real-time mediation and analysis of
6B CDRs per day
Data processing time reduced from
12 hrs to 1 sec

Hardware cost reduced to 1/8th
Proactively address issues
(e.g. dropped calls) impacting
customer satisfaction.

92

Data Warehousing Provides Unique Business Value

 Consolidate, manage and
reconcile data for enterprise
business intelligence

 Establish trust, quality and
governance where necessary
– Financial data
– Credit card data
– Healthcare

 Combine deep and operational
analytics

 Maintain history for trending
and historical reporting


Pacific Northwest Smart
Grid Demonstration Project

Capabilities:

Stream Computing – real-time
control system

Deep Analytics Appliance –
analyze massive data sets

Demonstrates scalability from
100 to 500K homes while
retaining 10 years’ historical data

60k metered customers in 5 states

Accommodates ad hoc analysis of
price fluctuation, energy consumption
profiles, risk, fraud detection, grid
health, etc.


Information Integration Provides Unique Business Value

 Movement of large data sets in
batch and real time
– Parallel processing engine for
efficient data movement

 Governance and trust for
Big Data
– Lineage and meta data of new Big
Data data sources
– Profile sources to determine trust

 Data Quality
– Standardize and transform data


Leaders integrate and govern massive Big Data
Marketing Services
with InfoSphere big
Leader integrates
data for customer
intelligence
Capabilities Utilized:
Information Integration – data
quality, ETL
Deep Analytics Appliance

Complex customer data
integration for
54M records/hour

Processing
5B simultaneous records
96

@BigData_paulz

THINK

PDF http://tinyurl.com/88auawg
Hardcopy http://tinyurl.com/7ef9ubs
97

IBM-Why Big Data?

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to IBM-Why Big Data? (20)

More from Kun Le (14)

Recently uploaded (20)

IBM-Why Big Data?