How to build analytics for 100bn logs a month with ClickHouse. By Vadim Tkachenko.
0 likes791 views
Vectorization improves performance by representing data as arrays that can be processed in tight loops by CPUs. This allows compilers to generate SIMD instructions, enabling modern CPUs to take multiple loop iterations concurrently and improving performance. One study showed vectorized execution improved query performance in a data-intensive workload by a factor of 50.
![* The image taken from [1]](https://image.slidesharecdn.com/sfmeetup-2017-clickhouse-170919183813/85/How-to-build-analytics-for-100bn-logs-a-month-with-ClickHouse-By-Vadim-Tkachenko-12-320.jpg)
How to build analytics for 100bn logs a month with ClickHouse. By Vadim Tkachenko.
- 11. Data is represented as small single-dimensional arrays (vectors), easily accessible for CPUs. The percentage of instructions spent in interpretation logic is reduced by a factor equal to the vector-size The functions that perform work now typically process an array of values in a tight loop Tight loops can be optimized well by compilers, enable compilers to generate SIMD instructions automatically. Modern CPUs also do well on such loops, out-of-order execution in CPUs often takes multiple loop iterations into execution concurrently, exploiting the deeply pipelined resources of modern CPUs. It was shown that vectorized execution can improve data-intensive (OLAP) queries by a factor 50.
- 12. * The image taken from [1]
- 21. SELECT foo FROM distributed_table SELECT foo FROM local_table GROUP BY col1 • Server 1 SELECT foo FROM local_table GROUP BY col1 • Server 2 SELECT foo FROM local_table GROUP BY col1 • Server 3
- 22. N Servers 1 3 140 Time, sec 1.224 0.438 0.043 Speedup x2.8 x28.5
- 28. seconds query
- 37. ClickHouse on MemCloud Kodiak Data and Altinity now Offer a Cloud Version of ClickHouse 38 1. FASTEST MPP Open Source DBMS 2. Cutting Edge Cloud for Big Data Apps and Processing 3. World-class ClickHouse Expertise Try the ClickHouse on MemCloud demo here http://clickhouse-demo.memcloud.works/
Editor's Notes
- #3: Analytic database landscape: Commerical -- fast and expensive: Vertica RedShift Teradata etc. Open Source -- somewhat slow, buggy but free InfiniDB (part of MariaDB now) InfoBright GreenPlum (started as commerical) Hadoop systems ClickHouse: fast as free! ClickHouse story: Yandex -- Russian Google Yandex Metrika -- Russian Google Analytics Interactive Ad Hoc reports at multiple petabytes That's why they developed ClickHouse ClickHouse is extremelly fast and scalable. Why ClickHouse is so fast Popular Yandex answer -- because they had no choice Techical details Vectorised processing (see VectorWise) True MPP True shared nothing True column store with late materialization (like C-Store and Vertica but unlike many others): Data compression Column locality No random reads Some technical details (in Russian): https://clickhouse.yandex/presentations/meetup7/internals.pdf What is column store (I think it is important to explain) Why it is good for quries like range scan + aggregation (look at Yandex presentation above, there are examples) Conclusion -- such an architecture allows very fast queries on a single table with filters and group by's Not enough speed -- let's see how data distribution works Care about reliability -- let's see how replication is set up (again, can use Yandex slides above as the source) Benchmark 1 Benchmark 2 Benchmark 3 Few words on limitations: Custom SQL dialect As a consequence -- limited ecosystem (can not fit to standard one) No deletes/updates: but there are mutable table types (engines) there is a way to connect to external updateble data (dictionaries) Somewhat hard to manage -- no tools Final word: Potential to be MySQL for Analytics Invite to try Need more info -- http://clickhouse.yandex Need consulting/support -- http://www.altinity.com