ClickHouse -If Combinators for Fun and Profit-2022-05-04.pdf
0 likes104 views
The document discusses -if combinators in ClickHouse, illustrating their use with aggregation functions like countif, avgif, and sumif to handle data analysis more efficiently, particularly with NYC taxi and GitHub event data. It highlights practical applications for detecting dirty data and performing complex queries faster. Additionally, it notes the limited documentation available and points to resources for further learning about ClickHouse features.
ClickHouse -If Combinators for Fun and Profit-2022-05-04.pdf
- 1. © 2022 Altinity, Inc. ClickHouse -If Combinators for Fun and Profit Robert Hodges, Altinity SF Bay Area ClickHouse Meetup May 4, 2022 1
- 2. © 2022 Altinity, Inc. Let’s make some introductions ClickHouse support and services including Altinity.Cloud Authors of Altinity Kubernetes Operator for ClickHouse and other open source projects Robert Hodges Database geek with 30+ years on DBMS systems. Day job: Altinity CEO Altinity Engineering Database geeks with centuries of experience in DBMS and applications 2
- 3. © 2022 Altinity, Inc. What are -If Combinators? ClickHouse Aggregation Functions ● count() ● min(value) ● max(value) ● sum(value) ● avg(value) ● uniq(value) ● uniqExact(value) ● … 3 -If versions of same functions ● countIf(condition) ● minIf(value, condition) ● maxIf(value, condition) ● sumIf(value, condition) ● avgIf(value, condition) ● uniqIf(value, condition) ● uniqExactIf(value, condition) ● …
- 4. © 2022 Altinity, Inc. Finding dirty data using a histogram – NYC Taxi data: fare amounts WITH histogram(5)(fare_amount) AS hist SELECT arrayJoin(hist) AS hist1, bar(hist1.3, 0, 1000, 20) AS bar FROM tripdata ┌─hist1──────────────────────────────────────────────────┬─bar──────────────────┐ │ (-21474808,-12079576.375,1) │ │ │ (-12079576.375,-1342166.6614073187,163862993.25) │ ████████████████████ │ │ (-1342166.6614073187,177236.82575785994,983177956.125) │ ████████████████████ │ │ (177236.82575785994,502005.5770089285,163863003.875) │ ████████████████████ │ │ (502005.5770089285,825998.625,8.75) │ ▏ │ └────────────────────────────────────────────────────────┴──────────────────────┘ 5 rows in set. Elapsed: 6.853 sec. Processed 1.31 billion rows, 5.24 GB (191.28 million rows/s., 765.11 MB/s.) 4
- 5. © 2022 Altinity, Inc. Finding dirty data with countIf() – NYC Taxi data: fare amounts SELECT countIf(fare_amount < 0) AS fare_less_than_zero, countIf((fare_amount >= 0) AND (fare_amount < 100)) AS fare_0_to_100, countIf((fare_amount >= 100) AND (fare_amount < 1000)) AS fare_100_to_1000, countIf(fare_amount > 1000) AS fare_1000_or_greater FROM tripdata ┌─fare_less_than_zero─┬─fare_0_to_100─┬─fare_100_to_1000─┬─fare_1000_or_greater─┐ │ 128932 │ 1310237048 │ 537609 │ 373 │ └─────────────────────┴───────────────┴──────────────────┴──────────────────────┘ 1 rows in set. Elapsed: 0.898 sec. Processed 1.31 billion rows, 5.24 GB (1.46 billion rows/s., 5.84 GB/s.) 5 Simpler output and faster too!
- 6. © 2022 Altinity, Inc. Searching for GitHub event counts using GROUP BY -- Github event data by year. SELECT toYear(created_at) AS year, event_type, count() FROM github_events WHERE event_type IN ('PullRequestEvent', 'WatchEvent') GROUP BY year, event_type ORDER BY year ASC, event_type ASC ┌─year─┬─event_type───────┬──count()─┐ │ 2011 │ PullRequestEvent │ 476818 │ │ 2011 │ WatchEvent │ 1831742 │ │ 2012 │ PullRequestEvent │ 1807044 │ │ 2012 │ WatchEvent │ 4048676 │ 6
- 7. © 2022 Altinity, Inc. Pivoting event data using countIf() -- Github event data by year. SELECT toYear(created_at) AS year, countIf(event_type = 'PullRequestEvent') AS PRs, countIf(event_type = 'WatchEvent') AS Stars FROM github_events WHERE event_type IN ('PullRequestEvent', 'WatchEvent') GROUP BY year ┌─year─┬──────PRs─┬────Stars─┐ │ 2011 │ 476818 │ 1831742 │ │ 2012 │ 1807044 │ 4048676 │ │ 2013 │ 3103759 │ 7432800 │ │ 2014 │ 5923843 │ 11952935 │ 7 Way slower if you omit the IN clause
- 8. © 2022 Altinity, Inc. Cleanup up multiple inputs on the fly with avgIf() SELECT count() AS rides, avgIf(passenger_count, passenger_count < 10) AS passengers_clean, avgIf(fare_amount, (fare_amount >= 0) AND (fare_amount < 500)) AS fares_clean FROM tripdata ┌──────rides─┬──passengers_clean─┬────────fares_clean─┐ │ 1310903963 │ 1.681498872092554 │ 11.424149328643022 │ └────────────┴───────────────────┴────────────────────┘ 1 rows in set. Elapsed: 1.092 sec. Processed 1.31 billion rows, 6.55 GB (1.20 billion rows/s., 6.00 GB/s.) 8 Multiple conditions in a single scan!
- 9. © 2022 Altinity, Inc. Finding out more tricks for using -If combinators 9 ● ClickHouse docs - Very limited ● Altinity Knowledge Base - Examples using system tables and clever tricks ○ Example: Simple aggregate functions & combinators ● Tests in ClickHouse codebase – Examples of what’s possible but not why cd ~/git/ClickHouse/src/tests grep -r countIf . grep -r avgIf . grep -r weightedSumIf .
- 10. © 2022 Altinity, Inc. © 2022 Altinity, Inc. Thank you! Questions? https://altinity.com rhodges at altinity.com 10 Altinity.Cloud Software and support for ClickHouse We’re hiring!