ApacheKylin_HBaseCon2015
Download as PPTX, PDF1 like438 views
Apache Kylin is an open-source distributed analytics engine that provides a SQL interface for multi-dimensional analysis of large datasets on Hadoop, aimed at achieving sub-second query latency. It features seamless integration with BI tools, high concurrency support, and offers interactive querying capabilities through its MOLAP cubes. The document also outlines Kylin's evolution, performance comparisons with Hadoop Hive, and roadmap for future enhancements.
ApacheKylin_HBaseCon2015
- 1. http://kylin.io Apache Kylin Extreme OLAP Engine Seshu Adunuthula Director, Analytics Platform, eBay | sadunuthula@ebay.com
- 2. http://kylin.io Agenda What’s Apache Kylin? Features Performance Roadmap Q & A
- 3. http://kylin.io Extreme OLAP Engine for Big Data Kylin is an open source Distributed Analytics Engine from eBay that provides SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets What’s Kylin kylin / ˈkiːˈlɪn / 麒麟 --n. (in Chinese art) a mythical animal of composite form • Open Sourced on Oct 1st, 2014 • Be Accepted as Apache Incubator Project on Nov 25th, 2014
- 4. http://kylin.io Business Needs for Big Data Analysis Sub-second query latency on billions of rows ANSI SQL for both analysts and engineers Full OLAP capability to offer advanced functionality Seamless Integration with BI Tools Support of high cardinality and high dimensions High concurrency – thousands of end users Distributed and scale out architecture for large data volume
- 5. http://kylin.io Huge volume data Table scan Big table joins Data shuffling Analysis on different granularity Runtime aggregation expensive Map Reduce job Batch processing Technical Challenges
- 6. http://kylin.io OLAP Cube – Balance between Space and Time time, item time, item, location time, item, location, supplier time item location supplier time, location Time, supplier item, location item, supplier location, supplier time, item, supplier time, location, supplier item, location, supplier 0-D(apex) cuboid 1-D cuboids 2-D cuboids 3-D cuboids 4-D(base) cuboid • Base vs. aggregate cells; ancestor vs. descendant cells; parent vs. child cells 1. (9/15, milk, Urbana, Dairy_land) - <time, item, location, supplier> 2. (9/15, milk, Urbana, *) - <time, item, location> 3. (*, milk, Urbana, *) - <item, location> 4. (*, milk, Chicago, *) - <item, location> 5. (*, milk, *, *) - <item> • Cuboid = one combination of dimensions • Cube = all combination of dimensions (all cuboids)
- 7. http://kylin.io Kylin Architecture Overview 7 Cube Build Engine (MapReduce…) SQL Low Latency - Seconds Mid Latency - Minutes Routing 3rd Party App (Web App, Mobile…) Metadata SQL-Based Tool (BI Tools: Tableau…) Query Engine Hadoop Hive REST API JDBC/ODBC Online Analysis Data Flow Offline Data Flow Clients/Users interactive with Kylin via SQL OLAP Cube is transparent to users Star Schema Data Key Value Data Data Cube OLAP Cube (HBase) SQL REST Server
- 8. http://kylin.io Hive Input source Pre-join star schema during cube building MapReduce Pre-aggregation metrics during cube building HDFS Store intermediated files during cube building. HBase Store data cube. Serve query on data cube. Coprocessor is used for query processing. How Does Kylin Utilize Hadoop Components?
- 9. http://kylin.io Agenda What’s Apache Kylin? Features Performance Roadmap Q & A
- 10. http://kylin.io Extremely Fast OLAP Engine at Scale Kylin is designed to reduce query latency on Hadoop for 10+ billions of rows of data ANSI SQL Interface on Hadoop Kylin offers ANSI SQL on Hadoop and supports most ANSI SQL query functions Seamless Integration with BI Tools Kylin currently offers integration capability with BI Tools like Tableau. Interactive Query Capability Users can interact with Hadoop data via Kylin at sub-second latency, better than Hive queries for the same dataset MOLAP Cube User can define a data model and pre-build in Kylin with more than 10+ billions of raw data records Features Highlights
- 11. http://kylin.io Compression and Encoding Support Incremental Refresh of Cubes Approximate Query Capability for distinct Count (HyperLogLog) Leverage HBase Coprocessor for query latency Job Management and Monitoring Easy Web interface to manage, build, monitor and query cubes Security capability to set ACL at Cube/Project Level Support LDAP Integration Features Highlights…
- 16. http://kylin.io Agenda What’s Apache Kylin? Features Performance Roadmap Q & A
- 17. http://kylin.io Kylin vs. Hive # Query Type Return Dataset Query On Kylin (s) Query On Hive (s) Comments 1 High Level Aggregation 4 0.129 157.437 1,217 times 2 Analysis Query 22,669 1.615 109.206 68 times 3 Drill Down to Detail 325,029 12.058 113.123 9 times 4 Drill Down to Detail 524,780 22.42 6383.21 278 times 5 Data Dump 972,002 49.054 N/A 0 50 100 150 200 SQL #1 SQL #2 SQL #3 Hive Kylin High Level Aggregatio n Analysis Query Drill Down to Detail Low Level Aggregatio n Transactio n Level Based on 12+B records case
- 18. http://kylin.io Performance - Query Latency 90%tile queries <5s Green Line: 90%tile queries Gray Line: 95%tile queries
- 19. http://kylin.io Agenda What’s Apache Kylin? Features Performance Roadmap Q & A
- 20. http://kylin.io Kylin Evolution Roadmap 201520142013 Initial Prototype for MOLAP • Basic end to end POC MOLAP • Incremental Refresh • ANSI SQL • ODBC Driver • Web GUI • ACL • Open Source HOLAP • Streaming OLAP • JDBC Driver • New UI • Excel Support • … more Next Gen • Automation • Capacity Management • In-Memory Analysis (TBD) • Spark (TBD) • … more TBD Future… Sep, 2013 Jan, 2014 Sep, 2014 Q1, 2015
- 21. http://kylin.io Kylin Site: http://kylin.io Twitter: @ApacheKylin Github: apache/incubator-kylin WeChat (微信) ApacheKylin Open Source