How to Crunch Petabytes with Hadoop and Big Data using InfoSphere BigInsights and Streams

How to Crunch Petabytes with
Hadoop and Big Data using
InfoSphere BigInsights and
Streams

Tom Deutsch, IBM

Vladimir B
Vl di i Bacvanski, Founder, SciSpike
ki F d S iS ik
vladimir.bacvanski@scispike.com
Stephen Brodsky, Technical Executive and Distinguished Engineer, IBM
sbrodsky@us.ibm.com
b d k @ ib
August 24, 2011 © 2011 IBM Corporation & SciSpike

Who are we?

 Dr. Vladimir Bacvanski
– Consultant, trainer, and mentor focusing on making clients successful in
adopting new data and software approaches
– Over 20 years of experience
y p
– Founder of SciSpike – a training and consulting firm specializing in
advanced software and data technologies

 Stephen Brodsky, Ph.D.
– Di ti
Distinguished E i
i h d Engineer and T h i l E
d Technical Executive f IBM Bi D t
ti for Big Data
initiatives at the IBM Silicon Valley Laboratory
– Previously led the architecture for the Optim Data Studio product line
and pureQuery and was a member of the architecture team for DB2
pureXML, Rational Application Developer (RAD), and WebSphere.

2 © 2011 IBM Corporation & SciSpike

Agenda

 The “Big Data challenge: smarter analytics for a
Big Data”
smarter planet

 How to do it?
– The big data challenge
–FFoundations of Big D
d i f Bi Data approaches
h
– MapReduce and Hadoop
– Real-time data and stream processing
– Integration with existing systems


The “Big Data” Challenge


The World is Changing and Becoming More…
More

INSTRUMENTED

INTERCONNECTED

INTELLIGENT

The
Th resulting explosion of information creates a need for
lti l i fi f ti t df
a new kind of intelligence

…to help build a Smarter Planet

Information is Growing at a Phenomenal Rate . . . .

44x
44 as much data and content
over coming decade 80% Of world’s data
is unstructured

2020
35 zettabytes
(35 billion terabytes)
2009
800,000 petabytes

Source: IDC, The Digital Universe Decade – Are You Ready?, May 2010

The BIG Data Challenge
• Manage and benefit from massive and growing amounts of data
• Handle varied data formats (structured, unstructured, semi-structured) and
increased data velocity
• Exploit BIG Data in a timely and cost effective fashion

COLLECT MANAGE
Collect Manage

Integrate
INTEGRATE Analyze
ANALYZE


What clients are saying . . .

 Lots of potentially valuable data is dormant or discarded
p y
due to size/performance considerations

 Large volume of unstructured or semi-structured data is not worth
semi structured
integrating fully (e.g. Tweets, logs, . . .)

 Not clear what should be analyzed (exploratory iterative)
(exploratory,

 Information distributed across multiple systems and/or Internet

 Some information has a short useful lifespan

 Volumes can be extremely high

 Analysis needed in the context of existing information (not stand
alone)


Big Data Presents Big Opportunities
Extract insight from a high volume, variety and velocity of data
in a timely and cost-effective manner

Variety: Manage and benefit from
diverse data types and data
structures

Velocity: Analyze streaming data and
large volumes of persistent
data

Volume: Scale from terabytes to
zettabytes
ettabytes

9

Streams and Oceans of Information . . . .

Information oceans
Information streams
Information stored outside
High
Hi h speed information flowing in
di f ti fl i i conventional systems. Data may
ti l t D t
real-time, often transient originate from the Web or different
 Information from sensors, instruments, internal different systems
etc.
etc
 Information flowing from real-time logs  Collection of what has streamed
and activity monitors  Information from social media, logs, click
 Streaming content like audio and video streams, emails, etc.
 High speed transactions like tickers,
trades, or traffic systems  Unstructured or mixed schema documents
like claims, forms, desktop applications,
etc.
 Structured data from disparate systems


Applications for Big Data Analytics

Smarter Healthcare Multi-channel sales Finance

Homeland security Traffic Control Telecom

Manufacturing Trading Analytics

Many more!


Use Case Example: Energy Company
Business scenario

 Analyze large volumes of public and
private weather data for alternative
energy business
 E i ti hi h
Existing high-performance computing
f ti
hardware, limited staff

Technical challenges
 High data volume: 2+ PB

 Range of q y types
g query yp
- Avg temp in given location? (Small
result)
- Geo pts where ice may form on wind
turbines? (Large result derived values –
result,
icing determined by humidity + temp.)

 Run on system with non-Hadoop apps


Use Case Example: Global Media Firm
Business scenario
 Identify unauthorized content
streaming in digital media (piracy)
- Quantify annual revenue loss
- Analyze trends
 Monitor social media sites to identify
dissemination of pirated content. Time
sensitive!

Technical challenges

 High variety of unstructured and semi-
structured data.
t t dd t

 Initial focus: text analytics over 1 year’s
worth of social media data. Look for live
streaming URLs, sentiment, event info, etc.

 Complex rules to qualify & classify info

 Future potential for video analysis


IBM Watson

IBM Watson is a breakthrough in analytic innovation, but it is only successful
because of the quality of the information from which it is working.


Big Data and Watson
Big Data technology is used to build Watson technology offers great potential
Watson’s knowledge base for advanced business analytics

Watson uses the Apache Hadoop open
framework to distribute the workload for
loading information into memory. CRM Data
POS Data Social Media

Approx. 200M pages of text
(To compete on Jeopardy!) Distilled Insight
- Spending habits
- Social relationships
- Buying trends
InfoSphere BigInsights
oSp e e g s g ts

Watson’s
Memory Advanced
search and
analysis


Customer Engagements
Use patterns Common requirements
• Customer sentiment analysis (cross-
(cross • Extract business insight from large volumes of
sell, up-sell, campaign management) raw data (often outside operational systems)
• Integrated retail and web customer • Integrate with other existing software
behavior modeling g • Ready for enterprise use
• Predictive modeling (credit card fraud)
• System log analytics (reduce
operational risk)
p )

Consumer
Text, Blog,
Text Blog Weblog
Insight
Click streams
Multi-channel
sales
Log & transactions

Next Gen
Text Analytics
Biological Sequences
Fraud Models

Operational system & streams data sources
p y New Business Stat st ca ode
Statistical Model
Development Building


The approach to
crunching big data


How to approach Big Data analytics?
InfoSphere BigInsights and InfoSphere Streams
• Analytics for data in-motion and at-rest
• Platform for processing large volumes of diverse data
• Complements and integrates with existing software solutions


Addressing the Key Requirements
1. Platform for V3 – Variety, Velocity, Volume
 Variety - manage data & content “As Is”
 Handle any velocity - low-latency streams and large volume batch
 Volume - huge volumes of at-rest or streaming data Big Data Platform
2 Analytics for V3
2.
 Analyze Sources in their native format - text, data, rich content
 Analyze all of the data - not just a subset
 Dynamic analytics - automatic adjustments and actions

3. Ease of Use for Developers and Users
 Developer UIs, common languages & automatic optimization
 End-user UIs & visualization

4. Enterprise Class
 Failure tolerance, Security and Privacy
 Scale Economically

5. Extensive Integration Capabilities
 Integrate wide variety of sources
 Leverage enterprise integration technologies


Big D t I iti ti
Bi Data Initiative

Volumes of diverse persistent data
diverse, Analytic applications for
“Big Data”

InfoSphere
p
BigInsights
Warehouse

Traditional warehouse
applications
IBM Confidential

InfoSphere
Streams

Real-time streaming data

BigInsights Summary

 BigInsights = analytical platform for persistent “Big Data”
– Based on open source & IBM technologies

 Distinguishing characteristics
– Built-in analytics . . . . Enhances business knowledge
– Enterprise soft are integration . . . . Complements and e tends
software extends
existing capabilities
– Production-ready platform . . . . Speeds time-to-value; simplifies
development and maintenance


Big Data Platform Vision
Bringing Big Data to the Enterprise
Data
Big Data Solutions Warehouse

Information
Integration
Big Data User Environments
Developers End Users Administrators Master Data
Mgmt

IN
NTEGRATIO
AGENTS
A

Database

Big Data Enterprise Engines
Content

ON
Analytics

Business
Analytics
Streaming Analytics
g y Internet Scale Analytics
y

Marketing

Open Source Foundational Components
Data Growth
Management


InfoSphere BigInsights v 1.1
Platform for volume,
variety, velocity -- V3
 Hadoop foundation
Analytics for V3
 Text analytics & tooling Enterprise Edition
Licensed
Usability Web admin console, LDAP authentication
 Web administrative

lass
RDBMS, warehouse connectivity

nterprise cl
console Text analytics
Basic Edition
 Integrated install Spreadsheet-style analytic tool
Free download Flexible job scheduler
 Spreadsheet-style
analytic t l
l ti tool Apache 24 x 7 Web
En

Hadoop support
Enterprise Class
 Storage, security,
cluster management
Breadth of capabilities
Integration
 Connectivity to DB2,
Netezza


BigInsights Platform: Key Ideas

 Flexible, enterprise-class support for processing large
volumes of data
– Based on Google’s MapReduce technology
– Inspired by Apache Hadoop; compatible with its ecosystem a d
sp ed pac e adoop; co pat b e t ts ecosyste and
distribution
– Well-suited to batch-oriented, read-intensive applications
– Supports wide variety of data

 Enables applications to work with thousands of nodes and
petabytes of data in a highly parallel, cost effective manner
t b t fd t i hi hl ll l t ff ti
– CPU + disks = “node”
– Nodes can be combined into clusters
– New nodes can be added as needed without changing
• Data formats
• How data is loaded
• How jobs are written


The M R d
Th MapReduce Programming Model
P i M d l
 "Map" step:
Map
– Input split into pieces

– W k nodes process individual pieces i parallel ( d
Worker d i di id l i in ll l (under
global control of the Job Tracker node)

– Each worker node stores its result in its local file system
where a reducer is able to access it

 "Reduce" step:
– Data is aggregated (‘reduced” from the map steps) by
( reduced
worker nodes (under control of the Job Tracker)

– M lti l reduce tasks can parallelize th aggregation
Multiple d t k ll li the ti
25

What is Hadoop?

 Apache Hadoop = free, open source framework for data-
intensive applications
– Inspired by Google technologies (MapReduce, GFS)
– Well-suited to batc o e ted, read-intensive app cat o s
e su ted batch-oriented, ead te s e applications
– Originally built to address scalability problems of Nutch, an open source
Web search technology

 Enables applications to work with thousands of nodes and
petabytes of data in a highly parallel, cost effective manner
– CPU + disks of commodity b = H d
di k f dit box Hadoop “ d ”
“node”
– Boxes can be combined into clusters
– New nodes can be added as needed without changing
• Data formats
• How data is loaded
• How jobs are written


Two Key Aspects of Hadoop

 MapReduce framework
– How Hadoop understands and assigns work to the nodes
(machines)

 Hadoop Distributed File System = HDFS
– Where Hadoop stores data
– A file system that spans all the nodes in a Hadoop cluster
– It links together the file systems on many local nodes to
make them into one big file system


Logical MapReduce Example: Word Count
Content of Input Documents
Hello World Bye World
map(String key, String value):
Hello IBM
// key: document name
// value: document contents Map 1 emits:
< Hello, 1>
for each word w in value: < World, 1>
EmitIntermediate(w, 1 );
EmitIntermediate(w "1"); < Bye, 1>
Bye
< World, 1>

reduce(String key, Iterator values):
( g y, ) Map 2 emits:
< Hello, 1>
// key: a word < IBM, 1>
// values: a list of counts
Reduce (final output):
int result = 0;
< Bye, 1>
for each v in values: < IBM, 1>
result += ParseInt(v); < H ll 2>
Hello, 2
Emit(AsString(result)); < World, 2>

How To Create MapReduce Jobs

 MapReduce development in Java
p p
– Low level, very flexible
– Time consuming development

 Hive
– Open source language / Apache sub-project
sub project
– Provides a SQL-like interface to Hadoop

 Pig
– Data flow language / Apache sub-project

 Jaql
– A query language for JSON
– Useful for loosely structured data

Management Tools: Web Console
 Graphically manage cluster, jobs, HDFS
 Sample administration tasks
– Start/Stop Servers
– Add/Remove Servers
– Server Status Details (Log)


Spreadsheet like
Spreadsheet-like Analysis Tool
 Web-based analysis BigSheets

and visualization tool

 Spreadsheet-like
interface
– Define and manage
long running data
collection j b
ll i jobs

– Analyze content of the
text on the pages that
have been retrieved


Text Analytics
• Distill structured info from unstructured data "Acquisition"
• Sentiment analysis "Address"
Address
"Alliance"
• Consumer behavior "AnalystEarningsEstimate"
• Illegal or suspicious activities "City"
"CompanyEarningsAnnouncement"
CompanyEarningsAnnouncement
• ... "CompanyEarningsGuidance"
"Continent"
"Country"
• Pre-built library of text annotators for common "County"
County
business entities "DateTime"
"EmailAddress"
"JointVenture"
• Rich language and tooling to build custom
g g g "Location"
Location
annotators "Merger"
"NotesEmailAddress"
"Organization"
• Support for Western languages ( g ,
pp g g (English, "Person"
Person
Dutch/Flemish, French, German, Italian, "PhoneNumber"
Portuguese, or Spanish) plus select Asian "StateOrProvince"
languages (Japanese, Chinese) "URL"
"ZipCode"
ZipCode

32

Eclipse based
Eclipse-based Text Analytics Development


So What Does This Result In?

 Easy To Scale

 Fault Tolerant and Self-Healing

 Data Agnostic

 Extremely Flexible


Working with streaming data: a new paradigm

 Conventional processing: static data

Queries Data Results

 Real-time processing: streaming data

Data Queries Results


Real-Time
Real Time Data with InfoSphere Streams
Source Sink
 Streaming analytic applications Adapters Operator Repository Adapters
– M lti l i
Multiple input streams
t t
– Advanced streaming analytics
 Eclipse based IDE
InfoSphere Streams Studio
– Define sources, apply operators, (IDE for Streams Processing Language)
define intermediary and final
output sinks
– User defined operators in Java or
C++
Automated,
Automated Optimized Deploy
O i i i
Optimizing compiler automates
il and Management (Scheduler)
deployment and connections
– Extremely low latency
y y
– Cluster of up to 125 nodes


Scalable stream processing
 InfoSphere Streams provides
– A programming model and IDE f d fi i d t sources and
i d l d for defining data d
software analytic modules called operators that are fused into
process execution units (PEs)
– infrastructure to support the composition of scalable stream
processing applications from these components
– deployment and operation of these applications across distributed
p y p pp
x86 processing nodes, when scaled processing is required
– stream connectivity between data sources and PEs of a stream
processing application


Merging the Traditional and Big Data Approaches
Traditional Approach Big Data Approach
Structured & Repeatable Analysis Iterative & Exploratory Analysis

IT
Business Users
Delivers a platform to
Determine what enable creative
bl ti
question to ask discovery

IT Business
Structures the Explores what
data to answer questions could be
that question
q asked

Monthly sales reports Brand sentiment
Profitability analysis Product strategy
Customer surveys Maximum asset utilization


BigInsights and the data warehouse: filtering and
summarizing “Big Data”

BigInsights

• Broader analytic coverage
• Exploits IT investments while
p Data warehouse
minimizing burden

BigInsights as a “queryable archive for growing
queryable archive”
data warehouses

BigInsights

Data warehouse • Offl d “cold” or dated warehouse info but
Offload “ ld” d t d h i f b t
maintain access for further exploration
• Keep warehouse size manageable and focused
on well-known business analytic needs


Trends and directions
 Enterprise software integration
– Data warehouses, RDBMSs
– ETL platforms
l tf
– Business intelligence tools
– Applications
– ...

 Diverse range of analytics
– Text
– Image / video (e.g., content based user profiling)
(e g content-based
– Predictive modeling (e.g., ranking and classification based on
machine learning)
– ...

 Sophisticated, scalable infrastructure for processing
massive data volumes
– High-performance file system with full POSIX compliance, g
g p y p , granular
security
– Fully recoverable and restartable workflows
– Parallel, distributed indexing for text (“BigIndex”)
– Read-optimized column store
p
– Tooling for administrators, programmers, analysts
– ...

Integrating Relational, Streams, and BigInsights

Traditional /
Traditional Relational
Warehouse
Data Sources
Database & At-rest Results
Warehouse data
analytics

Non-Traditional /
Streams Non-Relational
N R l ti l
Data Sources
In-Motion Ultra Low
Analytics Latencyy
Results
Varied data InfoSphere
Big Insights
formats
Massive Scale
Big Data
Semi-structured, Batch oriented Results
unstructured... data analytics


Typical Strategy for Analytics

ETL SQL Analytics, Mining

Data warehouse / marts

Source
Sources
S
Transform/
Extract Load
subset


Emerging requirements for analytics
SQL Analytics, Mining
ETL, ELT (MR BI, Mining)

Source
Structured Transform,
Analyze Warehouses / marts
Sources
Transform/
Extract subset Load

BigInsights
g g
Source Repository
Other
Sources
Explore large volumes of “raw” or diverse data.
Discover, analyze new insights with BigInsights


Conclusions

– Scale out to crunch petabytes

– We need a mix of technologies
• Data at rest: MapReduce, Hadoop and beyond
• Data in motion: stream processing

– To be successful, integrate with conventional
technologies


Getting in touch

 Stephen Brodsky – IBM
– Email: sbrodsky@us.ibm.com
 InfoSphere BigInsights
– http://www-01.ibm.com/software/data/infosphere/biginsights/
ttp // 0 b co /so t a e/data/ osp e e/b g s g ts/
 InfoSphere Streams
– http://www-01.ibm.com/software/data/infosphere/streams/

 Vladimir Bacvanski - SciSpike
– Email: vladimir.bacvanski@scispike.com
– Blog: http://www.OnBuildingSoftware.com/
– Twitter: http://twitter.com/OnSoftware
– LinkedIn: http://www.linkedin.com/in/VladimirBacvanski
p


How to Crunch Petabytes with Hadoop and Big Data using InfoSphere BigInsights and Streams

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to How to Crunch Petabytes with Hadoop and Big Data using InfoSphere BigInsights and Streams (20)

More from Vladimir Bacvanski, PhD (7)

Recently uploaded (20)

How to Crunch Petabytes with Hadoop and Big Data using InfoSphere BigInsights and Streams