Simple Strategies for faster knowledge discovery in big data
0 likes101 views
The document outlines strategies for optimizing data querying and analytics in big data environments, focusing on reducing query latency and resource consumption. Techniques such as partitioning, clustering, and identifying key usage patterns are emphasized to enhance performance. It highlights the importance of ongoing infrastructure optimization in supporting data-driven cultures.
![* 90% of queries using [table A] filter on [column x]
* 70% of queries using [table A] join to [table B] on
column X
* 90% of queries using [table A, Table B] end using
Column X, Y from table A and Column Z from table B
Data Analytics Optimization Forecasting](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-10-320.jpg)
![CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-11-320.jpg)
![CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-12-320.jpg)
![Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—file_00000
|—file_00000
|—date=20170102
|—file_00000
|—file_00000
|—date=20170103
|—file_00000
|—file_00000
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-13-320.jpg)
![Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—file_00000
|—file_00001
|—date=20170102
|—file_00000
|—file_00001
|—date=20170103
|—file_00000
|—file_00001
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-14-320.jpg)
![Optimization > Partitioned By
|—user
|—db
|— table
|—date=20170101
|—event_type=‘click’
|—file_00000
|—file_00001
|—date=20170102
|—event_type=‘click’
|—file_00000
|—file_00001
|—event_type=‘map’
|—file_00000
|—file_00001
|—….
SELECT *
FROM [table]
WHERE
date = 20170102
AND event_type = ‘click’](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-15-320.jpg)
![CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-18-320.jpg)
![CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
[(col_name data_type [COMMENT col_comment], ... [constraint_specification])]
[ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)]
[
CLUSTERED BY (col_name, col_name, ...)
[SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS
]
[
SKEWED BY (col_name, col_name, ...)
ON ((col_value, col_value, ...), (col_value, col_value, ...), ...)
]
[
[ROW FORMAT row_format]
[STORED AS file_format]
]
Data Analytics Optimization Forecasting](https://image.slidesharecdn.com/datasummit2017-170522051249/85/Simple-Strategies-for-faster-knowledge-discovery-in-big-data-22-320.jpg)
Simple Strategies for faster knowledge discovery in big data
- 1. Simple Strategies For Faster Knowledge Discovery In Big Data Dr. Ritesh Agrawal Lead Data Scientist Zheng Shao Staff Software Engineer
- 2. * 6 continents, 70+ countries, 450 cities * 1st billionth trip took 5 years. 2nd billionth trip took 1.5 years * 5+ million trips every day
- 4. Infrastructure Challenge *Help discover *Quickly: having low query latency *Efficiently: minimal cost
- 5. Data eg: log presto queries Analytics eg: 20% compute resources used by timeout queries How to Build An Effective Infrastructure Optimization eg: build query gate Forecasting eg: predict future requirements
- 6. PRESTO HIVE HDFS KAFKA *User *Tables *Join Clause *Filter Clause *Execution Related Metrics Data Analytics Optimization Forecasting
- 7. *Most queried tables *Most expensive queries *Top N users *… Overview Analytics *Filter Clause *Most Joined tables *Hot & Cold Partitions… Detailed Analytics Data Analytics Optimization Forecasting
- 8. Data Analytics Optimization Forecasting others with create select insert PercentQueries 0 20 40 60 80 Number of Joins 0 1 2 3 4 5+ Pct of Total Queries Failure Pct in Bucket
- 9. 90% of compute resources is consumed by 10% most expensive queries
- 10. * 90% of queries using [table A] filter on [column x] * 70% of queries using [table A] join to [table B] on column X * 90% of queries using [table A, Table B] end using Column X, Y from table A and Column Z from table B Data Analytics Optimization Forecasting
- 11. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name [(col_name data_type [COMMENT col_comment], ... [constraint_specification])] [ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [ CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS ] [ SKEWED BY (col_name, col_name, ...) ON ((col_value, col_value, ...), (col_value, col_value, ...), ...) ] [ [ROW FORMAT row_format] [STORED AS file_format] ] Data Analytics Optimization Forecasting
- 12. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name [(col_name data_type [COMMENT col_comment], ... [constraint_specification])] [ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [ CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS ] [ SKEWED BY (col_name, col_name, ...) ON ((col_value, col_value, ...), (col_value, col_value, ...), ...) ] [ [ROW FORMAT row_format] [STORED AS file_format] ] Data Analytics Optimization Forecasting
- 13. Optimization > Partitioned By |—user |—db |— table |—date=20170101 |—file_00000 |—file_00000 |—date=20170102 |—file_00000 |—file_00000 |—date=20170103 |—file_00000 |—file_00000 SELECT * FROM [table] WHERE date = 20170102 AND event_type = ‘click’
- 14. Optimization > Partitioned By |—user |—db |— table |—date=20170101 |—file_00000 |—file_00001 |—date=20170102 |—file_00000 |—file_00001 |—date=20170103 |—file_00000 |—file_00001 SELECT * FROM [table] WHERE date = 20170102 AND event_type = ‘click’
- 15. Optimization > Partitioned By |—user |—db |— table |—date=20170101 |—event_type=‘click’ |—file_00000 |—file_00001 |—date=20170102 |—event_type=‘click’ |—file_00000 |—file_00001 |—event_type=‘map’ |—file_00000 |—file_00001 |—…. SELECT * FROM [table] WHERE date = 20170102 AND event_type = ‘click’
- 16. Optimization > Partitioned By *A query that used to take 10 minutes now completes within 2 minutes. 80% reduction in elapsed time *95% reduction in resource consumption.
- 17. Partitioned By: Key Considerations * Usage: * Identify fields that are often used for filtering (FILTER CLAUSE) * Cardinality: * Low Cardinality — otherwise explodes meta-store * Skewness: * data should evenly distributed if all the keys are popular * skewness is okie if filter value corresponds to smaller dataset
- 18. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name [(col_name data_type [COMMENT col_comment], ... [constraint_specification])] [ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [ CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS ] [ SKEWED BY (col_name, col_name, ...) ON ((col_value, col_value, ...), (col_value, col_value, ...), ...) ] [ [ROW FORMAT row_format] [STORED AS file_format] ] Data Analytics Optimization Forecasting
- 19. Optimization > Clustered By/Bucketing
- 20. Optimization > Clustered By/Bucketing
- 21. * Identify tables that are often joined together. * Cluster tables on join key. Clustered By/Bucketing: Key Considerations
- 22. CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name [(col_name data_type [COMMENT col_comment], ... [constraint_specification])] [ PARTITIONED BY (col_name data_type [COMMENT col_comment], ...)] [ CLUSTERED BY (col_name, col_name, ...) [SORTED BY (col_name [ASC|DESC], ...)] INTO num_buckets BUCKETS ] [ SKEWED BY (col_name, col_name, ...) ON ((col_value, col_value, ...), (col_value, col_value, ...), ...) ] [ [ROW FORMAT row_format] [STORED AS file_format] ] Data Analytics Optimization Forecasting
- 23. StorageSKEWED BY * Row vs Col. * Significant improvement by moving to ORC or parquet format. * splits data so that heavy values are stored in separate files * Helps with query optimization. Data Analytics Optimization Forecasting Other techniques * execution engine: MapReduce Vs Tez * Vectorization * Pre-joined tables
- 24. Key Takeaways ! Fast and efficient infrastructure is key to a business’s success. ! Infrastructure Optimization is a constantly ongoing process. ! Infrastructure Data Science is key to building an efficient infrastructure ! Start simple
- 25. First & Last Name Ritesh Agrawal, Lead Data Scientist, Uber Inc. Thank you