Spark meetup v2.0.5

HUAWEI TECHNOLOGIES CO., LTD.
Spark SQL on HBASE
Spark Meet-up Tech Talks
Yan Zhou/Bing Xiao
March 25, 2015

HUAWEI TECHNOLOGIES Co., Ltd.
 Lead architect of Huawei Big Data Platform
 Apache Pig committer
 ex-Hadooper @ Yahoo!
 15+ years of experience in DB, OLAP, distributed computing fields.
About me
Page ‹2›

 A Fortune Global 500, Private Company
 Annual growth at 15% (revenue up to $46 Billion in 2014)
 More than $6.5 Billion investment on R&D
 Transition from telecom-equipment manufacturer to a leader of ICT
(information and communications technology)
 Big data and open source are part of company-wide strategies
About Huawei
Page ‹3›

 Spark SQL on HBase
o Motivations
o Data Model
o SQL Semantics
o Performance
o Value Proposition
 Demo
 Roadmap
 Q/A
Agenda
Page ‹4›

 Driven by use cases in verticals
including telco
 Telco Data is unique & complex
 Flexible data organization for various
types of queries: range, ad-hoc,
interactive, DW
 Stay in tune: planned sessions in
future events
Customer & Billing Data
• Well Structured
• TB
Session Signaling Data：
• Multi-Device generated
• Hundreds TB-PB
• Real-Time Biz Oriented
MR/CHR Data：
• Semi-structured & Nested
• PB
• Location-centered Data
xDR Data:
• 1~10PB each month
Network Raw Data
• Unstructured
• TB/Sec
• Linear Growth with
Biz
CRM
Billing
Signaling Data
MR/CHR Type Data
xDR Data
Network Raw Data
Note：In a typical network of ~30M subscribers and the data flow around 1TB/Sec.
Page ‹5›
Motivations

 NOSQL Key-Value data store on Hadoop
 Following Google BigTable model
 Emerging platform for scale-out relational data stores on Hadoop:
o Splice Machines
o Trafodion (HP)
o Apache Phoenix (Salesforce)
o Kylin (eBay)
 M/R and API-based data access interfaces only
Page ‹6›
What is HBase?

Spark Core
Spark SQL
Existing HBase Access Path
HadoopRDD
TableInput/OutputFormat
Features: Hadoop M/R plug-in
Inflexible/Hard to use
Limited pushdown capabilities
High Latency
HiveContext
metastore
Page ‹7›
HBase

New Data Access Path
metadata
Featuring:
• Fully distributed processing engine for scalability
and fault tolerance
• Scala/Java/Python APIs
• Pluggable data source to Spark SQL through
Spark SQL API
• Enable systematic and powerful handling of
pushdowns (key range, filters, coprocessor)
• More SQL capabilities made possible (Primary Key,
Update, INSERT INTO … VALUES, 2nd index, Bloom Filter, …)
Page ‹8›
Spark SQL
Spark Core
HBase

 Logical Data Model the same as Spark SQL: relational and type system.
 Physical Data Model:
• Support of composite primary keys
• HBase rowkey of byte representation of composite primary keys
• Logical non-key columns mapped onto <column family, column qualifier>
• Persistent metadata on a special HBase table
• Presplit tables supported
Data Models
Page ‹9›

 Based on Spark SQL syntax, plus …
 DDL:
• CREATE TABLE table_name (col1 TYPE1, col2 TYPE2, …, PRIMARY KEY (col7, col1, col3))
MAPPED BY (hbase_tablename, COLS=[col2=cf1.cq11, col4=cf1.cq12, col5=cf2.cq21,
col6=cf2.cq22])
•ALTER TABLE table_name ADD/DROP column …
 DML:
• INSERT … INTO VALUES …
 Bulk Loading:
• LOAD DATA [PARALLEL] INPATH filePath [OVERWRITE] INTO TABLE tableName [FIELDS
TERMINATED BY char]
SQL Semantics
Page ‹10›

 Precise partition pruning and partition-specific multidimensional predicate
pushdowns based on partial evaluation of filter boolean expressions for queries
Query Optimization Approach
=> Itemid > 300 AND amount < 30
=> customer=‘John’ AND itemid < 100 AND amount > 200
E.g. a sales table with <customer, itemid> as a 2-dimensional primary key
SELECT * from sales WHERE ((customer=‘Joe’ AND itemid > 300 AND amount < 30) OR
(customer=‘John’ AND itemid < 100) AND amount > 200)
The existing partitions/regions are:
1. (, ‘Ashley’)
2. [‘Ashley’, “Iris”)
3. [(‘Joe’, 10), (‘Joe’, 100)),
4. [(‘Joe’, 200), (‘Joe’, 1000))
5. [‘John’, ‘York’)
6. [‘York’, )
 The algorithms are generic and applicable to other organized data sets like
hash-partitioned Hive tables as well.
 Suitable for interactive ad hoc queries
Page ‹11›
for scan range for filtering

 Queries (TPC-DS, 10M records):
Query Performance
Page ‹12›
SQL Query SparkSQL
on HBase
(Seconds)
Phoenix
(Seconds)
1-key-range select count(1) from store_sales where
(ss_item_sk = 99 and ss_ticket_number > 1000);
0.18 0.03
(ss_item_sk = 99 and ss_ticket_number > 1000) or
(ss_item_sk = 5000 and ss_ticket_number < 20000);
0.22 4.29
(ss_item_sk = 99 and ss_ticket_number > 1000) or
(ss_item_sk = 5000 and ss_ticket_number < 20000) or
(ss_item_sk = 28000 and ss_ticket_number <= 10000);
0.27 4.44
Aggregate on
the secondary
key
select count(1) from store_sales group by ss_ticket_number; 37 79
• Cluster:
o 1 master + 6 slaves with 48GB/node
o Xeon 2.4G 16 cores

 Query performance (TPC-DS, 10M records):
Query Performance
0.18 0.22 0.27
37
0.03
4.29 4.44
79
0
10
20
30
40
50
60
70
80
90
1-key-range 2-key-range 3-key-range aggregate on
secondary key
Seconds
Spark SQL on HBase
Phoenix
Page ‹13›

 Performance optimization for tabular data bulk loading
• late materialization of KeyValue cells
 reduction of shuffle data volume
• removal of sorting by reducers
 lightweight reducer
 more scalable
• best effort to colocate reducers with the region servers
 Optional parallel incremental loading after M/R in the bulk loader
Bulk Load Optimization
Page ‹14›

 Loading performance (TPC-DS, 10M records):
Bulk Load Performance
557
185
1093
762
0
200
400
600
800
1000
1200
Load (no presplit) Load (6 presplit regions)
Seconds
Spark SQL on HBase
Phoenix
Page ‹15›
• Cluster:
o 1 master + 6 slaves with 48GB/node
o Xeon 2.4G 16 cores

 Combined capabilities of Spark, SparkSQL and HBase
o Spark Dataframe supported
 More traditional DBMS capabilities made possible on HBase
 Basis to build a highly performing and concurrent distributed big data SQL
system
 Optimized bulk loader for tabular data sets
 Performance excellence
Value Proposition
Page ‹16›

 Source repo: https://github.com/Huawei-Spark/hbase/
 emails: yan.zhou.sc@huawei.com, bing.xiao@huawei.com
 Team members:
Bo Meng, Xinyun Huang, Wang Fei, Stanley Burnitt, Shijun(Ken) Ma, Jacky
Li, Stephen Boesch
Plus our Big Data Teams in India and Hangzhou, China
Project Info
 Join us …
o Comments, tryouts, and contributions are more than welcome
o Open for Joint development in next phase project(s)
o We are hiring: Big Data Engineers/Spark Fans
Page ‹17›

 Environment
 Hardware: 9-node (1 master + 8 slaves) blades
 Software: Linux Enterprise Server 11.1
 Data set: TPC-DS 10M records

 Indexed-range query vs. full scan query through use of the DataFrame
 Join between a Hbase table with an in-memory side table through use of Dataframe
 Partial Evaluation
• Schema = (id : Int, age: Int)
• The row to be partially evaluated on is: ([1,5), null)
• Predicate 1: (id < 1) OR (age > 30) => (age > 30)
• Predicate 2: (id < 6) OR (age > 30) => True
Demo
Page ‹18›

 Targeting Spark 1.4
 Coprocessor/Custom Filter
 Filter/Partitions from CAST values in predicates
 Latency reduction: Spark-3306 for external resource management
 Optimizations of Sorting/Aggregation/Join on primary keys
 Support of salting, timestamp, dynamic columns, nested data type, …
 Views and materialized views
Future Plan
Page ‹19›

 Earliest Corporate sponsor of AMP Lab & its projects including Spark
 One of Leading contributors to Spark: 10 & 11 contributors in Spark 1.2 & 1.3
releases
 Highlighted contributions of New features : Power Iteration Clustering
represents the first use of GraphX routines within MLLIB; ORCFiles Support;
FP-Growth
 Bring Spark & Apps on top of Spark into leading telcos globally as Spark is
cornerstone of Huawei big data vertical solution
Huawei’s Long Term Commitment to Spark & Ecosystem
Page ‹20›

Huawei Planned Spark Roadmap
Spark SQL & Core
1H 2015
 Co-Processor, optimization
 Spark SQL on Hbase
 OrcFiles Support
 Vectorized Processing
ML & Streaming
2H 2015
 Nested Data
Spark on Yarn
 Power iteration
Clustering
 PAM K-Medoids
Streaming Analysis Algorithms
New requirements
from Spark
implementation
 Materialized View
 Spark R
GraphX in Telco Data
Modeling
 SQL99’2003
Compliance features
Page ‹21›

 Reynold Xin, Michael Armbrust at Databricks reviewed Spark SQL
on HBase design document and the code, provided feedback and helped
improve the design
 Xiangrui Meng at Databricks provided the guidance, reviewed and
modified the code MLLib Power Iteration Clustering algorithm (Spark 1.3
release) & FP-Growth algorithm (Spark 1.3 release)
 Huawei Big Data team in India & Hangzhou, China provided the
performance testing/tuning and participated in code development
Acknowledgements
Page ‹22›

Page ‹23›

Phoenix Architecture and Data Access Path
Phoenix Coprocessor Phoenix Coprocessor Phoenix Coprocessor
Phoenix as
HBase Client
Page ‹24›

HBase vs. Cassandra vs. RDBMS
HBase Cassandra RDBMS
Special Nodes Master Seed Coordinator
Synchronization Mechanism Zookeeper Gossip protocol …
CAP Properties CP AP CA
Data Access Shell, REST,
Java/Thrift API
CQL, Shell, Thrift SQL/JDBC/ODBC
Data Size PBs PBs TBs
Coprocessor/In-DB processing Yes No Yes
Origins Google BigTable Amazon Dynamo
+ Google BigTable
IBM System R
Native to Hadoop
(inclusion in Hadoop distributions,
Hadoop/HDFS specific
optimizations, …)
Yes No No
Index-Organization Single row index Single row index On any columns
Dominant Vendor Backing/Lock-in No Datastax Oracle, MS, IBM
Popular use scenarios Range queries; consistency; fast
reads:
Facebook Messenger
Geographical distributed cluster;
large deployments:
Twitter
Transactions, DW/DM

Spark SQL on HBase
Spark SQL
Spark
Master
HBase Master
Spark Slave Spark Slave
HBase Region
Server
HBase Region
Server
Architecture and Data Access Paths
Zookeeper Quorum

Spark meetup v2.0.5

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Spark meetup v2.0.5 (20)

Recently uploaded (20)

Spark meetup v2.0.5

Editor's Notes