ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexander Zaitsev
0 likes435 views
The document summarizes a ClickHouse meetup that took place on October 2, 2018 in Paris. It includes an agenda with presentations on ClickHouse and case studies from ContentSquare, Storetail, and Pragma Analytics. The document also provides an introduction to ClickHouse including that it is an open source, real-time, column-oriented database management system developed by Yandex in 2012-2015. It highlights ClickHouse's speed, flexibility, and cost advantages compared to other analytical database systems.
ClickHouse Paris Meetup. ClickHouse Analytical DBMS, Introduction. By Alexander Zaitsev
- 1. ClickHouse Paris Meetup 18:30 ClickHouse DBMS. Introduction and Case Studies. Alexander Zaitsev, Altinity 19:15 ClickHouse at ContentSquare. Christophe Kalenzaga &Vianney Foucault 20:00 Pizza Break 20:20 Storetail Data-warehouse Project. Matthieu Jacquet, Storetail (Criteo) 20:45 Pragma Analytics Software Suite w/ClickHouse. MatthieuTexier, Pragma Innovation 21:15 What's new in ClickHouse. Alexey Milovidov,Yandex Oct 2, 2018
- 2. ClickHouse Analytical DBMS Introduction Alexander Zaitsev, Altinity ContentSquare, Paris, 2 Oct 2018
- 4. ClickHouse DBMS is • Realtime • Column Store • MPP • SQL • Open Source
- 5. ClickHouseTimeline • Developed inYandex in 2012-2015 • Open Sourced June 2016 • First non-Yandex deploymentsQ4 2016 • Hundreds of companies by Q4 2018
- 10. ClickHouse • Fast! • Flexible! • Free!
- 11. How Fast?
- 12. :) select count(*) from T; SELECT count(*) FROM T ┌───────count()─┐ │ 1261705085657 │ └───────────────┘ 1 rows in set. Elapsed: 3.552 sec. Processed 1.26 trillion rows, 1.26 TB (355.22 billion rows/s., 355.22 GB/s.)
- 13. Query 1 Query 2 Query 3 Query 4 Setup 0.034 0.061 0.178 0.498 MapD & 2-node p2.8xlarge cluster 0.051 0.146 0.047 0.794 kdb+/q & 4 Intel Xeon Phi 7210 CPUs 0.762 2.472 4.131 6.041 BrytlytDB 1.0 & 2-node p2.16xlarge cluster 1.034 3.058 5.354 12.748 ClickHouse, Intel Core i5 4670K 1.56 1.25 2.25 2.97 Redshift, 6-node ds2.8xlarge cluster 2 2 1 3 BigQuery 6.41 6.19 6.09 6.63 Amazon Athena 8.1 18.18 n/a n/a Elasticsearch (heavily tuned) 14.389 32.148 33.448 67.312 Vertica, Intel Core i5 4670K 22 25 27 65 Spark 2.3.0 & single i3.8xlarge w/ HDFS 35 39 64 81 Presto, 5-node m3.xlarge cluster w/ HDFS 152 175 235 368 PostgreSQL 9.5 & cstore_fdw “1.1 BillionTaxi Rides Benchmarks” http://tech.marksblogg.com/benchmarks.html
- 14. $ 45.00 $ 104.00 $ 208.00 $ 950.00 0.0 100.0 200.0 300.0 400.0 500.0 600.0 700.0 800.0 900.0 1,000.0 r4.xlarge x1 i3.2xlarge x1 i3.4xlarge x1 RS dc2 x2 Time (s) Price (per week) • 8 queries 1.3B taxi rides dataset
- 15. 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0 ClickHouse Vertica Single Table SELECT GROUP BY JOIN t USING t_key WHERE<t.smth>? JOIN (SELECT * FROM t WHERE <smth>) USING t_key • 19 queries, 1200M rows table, 3-node clusters
- 16. ClickHouse runs at • Bare metal (any Linux) • Virtualization environments: – Kubernets,VMware etc. • Clouds: – Amazon,Azure, Alibaba
- 17. Real companies are using ClickHouse for: • Mobile App and Web analytics • AdTech bidding analytics • Operational Logs analytics • DNS queries analysis • Stock correlation analytics • Telecom • Security audit • Fintech SaaS • Manufactoring process control • BlockChain transactions analysis
- 18. Worldwide * www.altinity.com visits in 2018
- 19. Size does not matter • Yandex: 500+ servers, 25B rec/day • LifeStreet: 60 servers, 75B rec/day • CloudFlare: 36 servers, 200B rec/day • Bloomberg: 102 servers, 1000B rec/day
- 20. Happy Migrations! • From MySQL/InfoBright/Postr eSQL/Spark/Elastic • FromVertica/RedShift SPEED! COST! VENDOR UN-LOCKING!
- 21. Few Case Studies
- 22. • AdTech (ad exchange, ad server, RTB, DMP etc.) • Ad Optimization, programmatic bidding • A lot of data: – 10,000,000,000+ events/day • A lot of queries: users and algorithms
- 23. UsedVertica, but needed to move • Data sizes constantly grow • Estimated PBs • Vertica license would be too expensive
- 24. … migration was not easy * More details at October 2017 Berlin Meetup
- 25. Major Design Decisions • Dictionaries for star-schema design • Extensive use of Arrays • SummingMergeTree for realtime aggregation • Smart query generation • Multiple shards and replicas
- 26. Results • Successful migration, 1y+ in production • Better performance and flexibility – 75B rows/day – 1М rows/sec in peak hours – 1.3M SQL queries /day • 30% hardware cost reduction (less expensive storage) • No license cost and limits: – 3PB of raw data – 6,000 billion rows Powered by:
- 27. Case 2. Fintech Company • Stock Symbols Correlation Analysis • 5000 Symbols • 10 years of data 100B data points
- 29. Challenge • (time, symbol, price) – 100 billion • log_return = runningDifference(log(price)) – 100 billion times • corr(s1,s2) = corr(log_return(s1),log_return(s2)) For every pair (s1,s2) from 5000 s(i), 12.5M pairs overall • Group by hours Calculate 12,500,000 times For every hour!
- 30. Very slow
- 31. Tried… • Hadoop • Spark • Greenplum ClickHouse
- 33. POC Performance Results • 3 servers setup • 2 years, 5000 symbols: – log_return calculations: ~1 h (distributed) – Converting to arrays: ~ 1 h (almost distributed) – Correlations: ~50 hours (also distributed) • 12,5M/50h = 70/sec Distributed => it scales easily!
- 34. Case 3. Ivinco • Mature boardreader application • A lot of data collected from different sources • A lot of operational data (performance monitoring) 200TB in MySQL!
- 35. Operational problems • Hard to scale • Hard to make HA solution • Performance issues: – ‘Manual’ partitioning and sharding – Dozens of indexes per table etc.
- 36. Organizational problems • No development resources to rewrite • Minimal changes to current system are allowed
- 37. Binary log replication from MySQL to ClickHouse MySQL clickhouse-mysql Queries Source Data
- 38. Results • Seamless integration of ClickHouse into the current system • No developers/coding involved, project is done with DevOps • Easy to test performance side by side (ClickHouse is 100 times faster) • Now ready to re-write main system More details at: https://www.altinity.com/blog/2018/6/30/realtime-mysql-clickhouse-replication-in-practice
- 39. ClickHouseToday • Mature Analytic DBMS. Proven by many companies • 2+ years in Open Source • Constantly improves – new cool features were added recently • Many community contributors • Emerging eco-system (tools, drivers, integrations –Tableaux works!) • Support fromAltinity
- 40. Q&A Contact me: alexander.zaitsev@lifestreet.com alz@altinity.com skype: alex.zaitsev telegram: @alexanderzaitsev Altinity