SQL in Hadoop To Boldly Go Where no Data Warehouse Has Gone Before
1 like433 views
Actian is a $140M revenue company with 10,000+ customers that provides big data and analytics solutions. One of its key products is Actian Vector in Hadoop, which provides SQL analytics on Hadoop up to 30x faster than Impala. Vector uses single instruction, multiple data processing and storage indexes to minimize I/O and maximize throughput. It also supports trickle updates without impacting read performance through positional delta trees.
Actian Vector in Hadoop Architecture
SQLProcessing
SQL parser
Optimizer
Cross compiler
parsed tree
query plan
Client application
X100 algebra
X100
Distributed rewriter
Builder
Execution engine
annotated query tree
operator tree
Buffer manager
datadata request
HDFS
Masternode
SQL query
I/O
X100
Rewriter
Builder
Execution engine
annotated query tree
partial operator tree
Buffer manager
datadata request
HDFS
I/O
MPI
annotated tree
result
MPI
partial result set
MPI
inter-nodecommunication
HDFS
namenode
HDFS
datanode
X100
X100
X100
X100](https://image.slidesharecdn.com/sqlinhadooptoboldlygowherenodatawarehousehasgonebeforepresentation-150605190110-lva1-app6891/85/SQL-in-Hadoop-To-Boldly-Go-Where-no-Data-Warehouse-Has-Gone-Before-4-320.jpg)
SQL in Hadoop To Boldly Go Where no Data Warehouse Has Gone Before
- 1. SQL in Hadoop: To Boldly Go Where No Data Warehouse has Gone Before Emma McGrattan SVP Engineering, Actian Corp
- 2. $140M Revenues + Profitable 10,000+ Customers Global Presence: 8 world-wide offices, 7x 24 multinational support model 2 “Fast becoming a big data powerhouse to challenge the market.” Forrester “Actian is now very powerfully positioned in the big data and analytics markets.” Bloor Who is Actian?
- 3. Actian Management Console DATAPLATFORM SPARQLverse ANALYTIC APPS Financial Services Health Care Other Verticals SQL Java,C/++, Pythn SOURCE DATA Databases / Marts Warehouses Cloud / SaaS Applications Structured & Unstructured Data Enterprise Applications APPLICATIONDEV Application Development and Tools Deployment Options DataFlow Elastic Data Prep Vector in Hadoop SQL Analytics Predictive Analytics Graph Analytics DataFlow INFRASTRUCTURE Library of Analytic Blueprints Actian Vortex
- 4. X100 X100 X100 X100 HDFS HDFS HDFS HDFS HDFS X100 Workernode[1..n](datanodes) Actian Vector in Hadoop Architecture SQLProcessing SQL parser Optimizer Cross compiler parsed tree query plan Client application X100 algebra X100 Distributed rewriter Builder Execution engine annotated query tree operator tree Buffer manager datadata request HDFS Masternode SQL query I/O X100 Rewriter Builder Execution engine annotated query tree partial operator tree Buffer manager datadata request HDFS I/O MPI annotated tree result MPI partial result set MPI inter-nodecommunication HDFS namenode HDFS datanode X100 X100 X100 X100
- 5. Vector: Built for Warp Speed Time/CyclestoProcess Data Processed DISK RAM CHIP 10GB2-3GB40-400MB 2-20150-250Millions Vectorized End-to-End Single Instruction Multiple Data Update Capability Limit I/O Efficient real time updates Update & Delete individual records Smart Compression Maximize throughput Vectorized decompression Exploiting Chip Cache Process data on chip – not in RAM 1 2 3 4 YARN Integration Intelligent Block Placement Dynamic Resource Management … Storage Indexes Quickly identify candidate data blocks Minimize I/O 5 6 Time/CyclestoProcess
- 6. Vector: To Boldly Go… 0 5 10 15 20 25 30 35 Q3 Q7 Q19 Q27 Q34 Q42 Q43 Q46 Q52 Q53 Q55 Q59 Q63 Q65 Q68 Q73 Q79 Q89 Q98 “Impala Subset” of TPC-DS Queries at Scale Factor 3000 (3TB) Speedup vs Impala Impala Actian 16x faster on average NumberoftimesfasterthanImpala Both Executed on the same hardware and software environment: 5 Node Cluster with 64GB of RAM per node and 24 x 1TB Hard Disks.
- 7. The SQL Behind the Actian Numbers
- 8. The Impala Equivalent Uses “Hints” Note the use of partition keys
- 10. Trickle Update Support The Kobayashi Maru of Hadoop - The design paradigm for HDFS is for data to be written once and read ever after. - Appending updated records to the end of a column/table or rewriting the entire table significantly impacts system performance The Solution – Positional Delta Trees - Enable on-line updates, without impacting read performance - Keep track of the tuple position of Inserts/Modifies/Deletes - Designed to make merging in of these updates fast by providing the tuple positions where differences have to be applied at update time.
- 12. Data Security Access Control Role Separation - System Administrator & Database Administrator should not have access to all data Security Auditing - Ability to audit who accessed, or attempted to access, what and when Encryption - Data at rest – minimize performance impact by enabling at column level - Data in motion - File system
- 13. It’s SQL in Hadoop, Jim, but not as we know it Highest Performing and Fully Industrialized SQL in Hadoop Fully ACID compliant – brings transactional integrity to Hadoop to prevent inaccurate results Full ANSI SQL 92 support – enables use of ALL standard BI tools and apps Native DBMS Security - authentication, user and role-based security, data protection, and encryption Open APIs - allow read access to our block format Highly Performant – up to 30x faster than our closest competitor, Impala Mature, proven planner and fastest optimizer ensures customers can maximize number of nodes, CPU, memory and cacheHadoop distribution agnostic - avoids vendor lock-in and provides customer flexibility Native in-Hadoop YARN – manage Hadoop resources automatically to prevent inefficiencies Collaborative architecture - query native Hadoop file formats (like Parquet) without ingestion Update Capability – provides ability to update without impacting read performance Highest Concurrency – allows your customers to have simultaneous users and tasks run without long wait times
- 14. Beam it Down, Scotty!