Keys to the Kingdom: SQL in Hadoop
6 likes769 views
This document provides an overview of Actian Corporation and their SQL in Hadoop product called Actian Vortex. The summary includes: 1) Actian is a $140M company with over 10,000 customers that provides an end-to-end big data platform and SQL in Hadoop product called Actian Vortex. 2) Actian Vortex is a high performance SQL in Hadoop product that is up to 30x faster than Impala. It provides full ACID compliance, ANSI SQL support, and security features. 3) The document highlights customer case studies and benchmarks showing the high performance of Actian Vortex for analytics use cases in industries like fitness and market risk.
Keys to the Kingdom: SQL in Hadoop
- 1. Confidential © 2015 Actian Corporation1 Keys to the Kingdom: SQL in Hadoop Emma McGrattan, SVP Engineering June 9th, 2015
- 2. Confidential © 2015 Actian Corporation2 I’ve worked with data since before it was “Big” I have no sense of rhythm and was refunded all my money by the Arthur Murray School of Dance because they just couldn’t teach me to dance Upon “retirement” I dream of opening a bar/bistro in Dublin, Amsterdam or Paris I’d like to like olives, they’re so sophisticated, but I just can’t stand the taste I once won an open mic comedy night and would love to do stand-up I was born and raised in Ireland but became an American Citizen in 2011 I would rather walk 150ft to my car to get my phone to control my lights and thermostats and avoid doing it the old fashioned way. I’ve almost drowned four times I joined Mensa at age 12 I once announced on-stage at Comdex that I’d like to be a pornographer – I meant to say “photographer” – honestly I did! Getting to Know Emma McGrattan
- 3. Confidential © 2015 Actian Corporation3 Who is Actian? $140M Revenues + Profitable 10,000+ Customers Global Presence: 8 world-wide offices, 7x 24 multinational support model End to End Big Data Platform. Disruptive Price Performance. 3 “Actian Analytics demonstrates that the company now has an impressive range of offerings that have been rebranded and combined in a pretty comprehensive framework.” 451 Research “Actian is now very powerfully positioned in the big data and data analytics markets.” Bloor Group 3
- 4. Confidential © 2015 Actian Corporation4 Collect the Keys; Win an Apple Watch
- 5. Confidential © 2015 Actian Corporation5 How to Collect the Keys
- 6. Confidential © 2015 Actian Corporation6 “wrapped legacy” “from scratch” Maturity (SQL support, ACID, reliability, security, connectivity, performance) Hadoop IntegrationLow Native High “connections” + + Mature & Integrated + End-to-End SQL In Hadoop Landscape
- 7. Confidential © 2015 Actian Corporation7 Actian Vortex – Architecture
- 8. Confidential © 2015 Actian Corporation8 Actian Vector – Unmatched Innovation Time/CyclestoProcess Data Processed DISK RAM CHIP 10GB2-3GB40-400MB 2-20150-250Millions Vector Processing Single Instruction Multiple Data 2nd Gen Column Store Limit I/O Efficient real time updates Smarter Compression Maximize throughput Vectorized decompression Exploiting Chip Cache Process data on chip – not in RAM 1 2 3 4 YARN Integration Intelligent Block Placement Dynamic Resource Management… Storage Indexes Quickly identify candidate data blocks Minimize IO 5 6
- 9. Confidential © 2015 Actian Corporation9 Up to 30X Faster Than Impala 0 5 10 15 20 25 30 35 Q3 Q7 Q19 Q27 Q34 Q42 Q43 Q46 Q52 Q53 Q55 Q59 Q63 Q65 Q68 Q73 Q79 Q89 Q98 “Impala Subset” of TPC-DS at Scale Factor 3000 (3TB) Actian+HDP2.1 vs Cloudera Impala Impala Actian Background to “Impala Subset “of TPC-DS benchmark can be found here: http://blog.cloudera.com/blog/2015/01/impala-performance-dbms-class-speed/ Average #timesfasterthanImpala Both Executed on the Same Hardware and Software Environment: 5 Node Cluster with 64GB of RAM per node and 24x1TB Hard Disks Highest Performing SQL in Hadoop
- 10. Confidential © 2015 Actian Corporation10 The SQL Behind the Actian Numbers
- 11. Confidential © 2015 Actian Corporation11 The Impala Equivalent Uses “Hints” Note the use of partition keys
- 12. Confidential © 2015 Actian Corporation12 Trickle Update Support The Achilles Heel of Hadoop ■ The design paradigm for HDFS is for data to be written once and read ever after. ■ Appending updated records to the end of a column/table or rewriting the entire table significantly impacts system performance The Solution – Positional Delta Trees ■ Enable on-line updates, without impacting read performance ■ Keep track of the tuple position of Inserts/Modifies/Deletes ■ Designed to make merging in of these updates fast by providing the tuple positions where differences have to be applied at update time.
- 13. Confidential © 2015 Actian Corporation13 Data Security Access Control Role Separation ■ System Administrator & Database Administrator should not have access to all data Security Auditing* ■ Ability to audit who issued what query from where and when Encryption ■ Data at rest – minimal performance impact* ■ Data in motion ■ File system
- 14. Confidential © 2015 Actian Corporation14 Overcoming Challenges with YARN ■ YARN not designed for long running processes ■ Impossible to grow and shrink resources on the fly ■ Requirement to dynamically set priority of the workload ■ Identify data node failure and dynamically reconfigure cluster ■ Intelligent block replica placement And for my next trick….
- 15. Confidential © 2015 Actian Corporation15 Highest Performing and Fully Industrialized SQL in Hadoop Fully ACID compliant – brings transactional integrity to Hadoop to prevent inaccurate results Full ANSI SQL 92 support – enables use of ALL standard BI tools and apps Native DBMS Security - authentication, user and role-based security, data protection, and encryption Open APIs - allow read access to our block format Highly Performant – up to 30x faster than our closest competitor, Impala Mature, proven planner and fastest optimizer ensures customers can maximize number of nodes, CPU, memory and cache Hadoop distribution agnostic - avoids vendor lock-in and provides customer flexibility Native in-Hadoop YARN – manage Hadoop resources automatically to prevent inefficiencies Collaborative architecture - query native Hadoop file formats (like Parquet) without ingestion Update Capability – provides ability to update without impacting read performance Highest Concurrency – allows your customers to have simultaneous users and tasks run without long wait times
- 16. Confidential © 2015 Actian Corporation16 Customer Analytics on Hadoop in the Fitness Industry Actian Vortex Challenge: Leverage wealth of data available from fitness tracking tools and devices to better understand customers, increase wallet share and cross-up sell opportunities. Obstacles: Existing CRM system unable to handle the amount and type of data Wanted to use Hadoop but lacked in- house Hadoop skills. Solution: Actian Vortex Results: • Timely, accurate reports • Ability to leverage existing SQL skills without having to learn new tools • Eliminated need for aggregating or sampling • True ad hoc analysis
- 17. Confidential © 2015 Actian Corporation17 2 billion risk data points in 6 hours (~100k/sec) <30 seconds Hierarchy dimension on 1 million data points in <15 sec Store 80 days (160 billion rows) of data Up to 10 billion rows per day <2 sec Actian Vortex Technical Evaluation 1 hour 40 min (333K per sec) 6 sec on 5 node cluster; 2 sec on 10 node cluster Sub second response times Stored 100 days (200 billion rows) with linear scaling Text book scalability as nodes added to cluster <1 sec Loading Full Day Aggregation Filtered Aggregation Large Data Volumes Horizontal Scalability Drill Up / DrillDown Target Actual Market Risk Analytics on Hadoop
- 18. Confidential © 2015 Actian Corporation18 Download Vortex Today!
Editor's Notes
- #7: internationalization
- #9: 1: We use vectorized processing to exploit modern CPU architecture. We execute one operation at a time on a vector of data, which allows for tight inner code loops without branching. This way, we can use SIMD instructions and, because of the lack of branching, make sure the CPU pipelines are not thrashed. A vector is typically 1024 rows of a single column, so it’s a manageable amount of data while the overhead per row is still negligible. 2: A vector will fit in the CPU cache together with the code for a particular operation, so all execution is in-cache. 3: To feed this engine with enough data, we’re also applying the vectorized paradigm to the storage subsystem. First of all, we’re using a column store, so only relevant columns are read from disk. Data is stored in blocks of typically 512MB and a single block contains only data from a single column (there are exceptions). Blocks of different columns can be interleaved per block, but typically more than one block of the same column is grouped. To keep the stable storage fast and defragmented, we use in-memory overlays to store updates to the data. These overlays are automatically flushed to stable storage when needed. 4: The blocks are stored compressed on-disk. We’ve got a number of lightweight compression algorithms and the most efficient one is chosen per block, depending on the data characteristics. The decompression takes place per vector and can be done in the CPU cache, which neatly ties in with our in-cache execution. We have a buffer manager that predicts what blocks are needed when and makes sure no blocks that will be used in the near future are evicted from the buffer cache. 5: We have min-max indexes on the disk blocks, so when data is not completely random we can narrow down the ranges of blocks we need to read from disk, per column. All in all, the execution engine is able to do about 1.5GB/s per core, and high-end I/O subsystems are able to keep up with this.
- #10: Execution Subset of TPC-DS as chosen by Impala Data size is 3TB (SF3000) Executed on 5-node “rushcluster” in Austin Both Impala and Vector numbers are on the same hardware Comparison with Impala Verified that Impala plans are sensible Currently observed average speedup is 11x Optimal query plans (manually written) gives us 16x speedup These are real numbers! We executed manual plans directly Changes in the cost model would get us to this performance Performance improvements Cost model changes will get us to 16x speedup Pipeline of query execution changes Well into H2 Estimated to get us 2x improvement So, estimated speedup vs Impala would be ~30x (no guarantees) Planning to run TPC-H SF1000 and SF3000 With all planned improvements (end of the year) we should be able to beat the EXASOL cluster numbers.
- #11: Mention that the store_sales table has 8.6 BILLION rows
- #13: Key advantages of the PDT over value-based merging are: Positional merging needs less I/O than value-based merging, because the sort keys do not need to be read. Positional merging is less CPU intensive than value-based merging, especially when the sort-key is a compound key and/or non-numerical attributes are part of the sort-key.
- #17: A leading sports apparel retailer has used analytics, provided by a 3rd party customer relationship marketing service, to promote brand loyalty and identify targeted sales opportunities. Under Armour wanted to turn to analytics to get closer to their customers., increase wallet share and cross & up-sell opportunities. This is a trad’l retailer looking for new ways to leverage wealth of information available to them (especially in fitness tracking tools and devices) They were outsourcing data, which limited flexibility and slowed results.