ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
7 likes3,950 views
The document discusses the real-life applications and best practices of ClickHouse, a high-performance analytics database, highlighting its capabilities for fast analytical queries, low-latency data ingestion, and fault-tolerant data warehousing. It presents various case studies from industries such as adtech, fintech, and blockchain, showcasing ClickHouse's successful deployments and migration from other database systems. Key design decisions for using ClickHouse include efficient data ingestion, star-schema designs, and the use of materialized views and dictionaries to enhance performance.
![• Ad Tech (ad exchange, ad server, RTB, DMP etc.)
• Creative optimization, programmatic bidding
• A lot of data:
• 10,000,000,000+ bid requests/day
• 2-3K event record (300+ dimensions)
• 90-120 days of detailed data
10B * 3K * [90-120] = [2.7-3.6]PB
Case 1.](https://image.slidesharecdn.com/clickhouse-180502184605/85/ClickHouse-in-Real-Life-Case-Studies-and-Best-Practices-by-Alexander-Zaitsev-14-320.jpg)
![Distributed Computations
• Distribute data across N servers
• Calculate log_return for every symbol at every server using Arrays:
• (timestamp, Array[String], Array[Float32])
• Distribute correlation computations across all servers
• Batch planning](https://image.slidesharecdn.com/clickhouse-180502184605/85/ClickHouse-in-Real-Life-Case-Studies-and-Best-Practices-by-Alexander-Zaitsev-26-320.jpg)
ClickHouse in Real Life. Case Studies and Best Practices, by Alexander Zaitsev
- 1. ClickHouse In Real Life Case Studies and Best Practices Alexander Zaitsev, LifeStreet/Altinity Percona Live 2018
- 2. Who am I • M.Sc. In mathematics from Moscow State University • Software engineer since 1997 • Developed distributed systems since 2002 • Focused on high performance analytics since 2007 • Director of Engineering in LifeStreet • Co-founder of Altinity – ClickHouse Service Provider
- 4. .. and I am not Peter’s brother
- 5. ClickHouse is •Fast •Flexible •Scalable How does it work in real?
- 6. What Is It For?
- 7. What Is It For? • Fast analytical queries • Low latent data ingestion/aggregation • Distributed computations • Fault-tolerant data warehousing All scaled from 1 to 1000s servers
- 8. Who Is It For?
- 9. Who Is It For? • Analysts/Developers/DevOps • who need analyze huge amounts of data • Startups • build high performance analytics with low investment • Companies • having performance problems with current systems • paying too much for license or infrastructure
- 10. Successful Production Deployments • DNS queries analytics (CloudFlare) • AdTech (multiple companies worldwide) • Operational logs analytics (multiple companies worldwide) • Stock correlation analytics, investor tools (Canadian company) • Hotel booking analytics SaaS (Spanish company) • Security audit (Great Britain, USA) • Fintech SaaS (France) • Mobile App and Web analytics (multiple companies worldwide)
- 11. Evaluating/implementing: • Telecom companies • Satellite data processing • Search engine ranking analytics • Blockchain platform analysis • Manufacturing process control
- 12. Happy Transitions! • From MySQL/InfoBright/PostreSQL/Sp ark to ClickHouse • From Vertica/RedShift to ClickHouse SPEED! COST! VENDOR UN-LOCKING! 24.04 16:50 CLICKHOUSE GATE 2 boarding 24.04 19:30 CLICKHOUSE GATE 3 24.04 20:00 CLICKHOUSE GATE 4
- 13. Case Studies • Migration from Vertica to ClickHouse • Distributed Computations and Analysis of Financial Data • Blockchain Platform Analytics • ClickHouse with MySQL
- 14. • Ad Tech (ad exchange, ad server, RTB, DMP etc.) • Creative optimization, programmatic bidding • A lot of data: • 10,000,000,000+ bid requests/day • 2-3K event record (300+ dimensions) • 90-120 days of detailed data 10B * 3K * [90-120] = [2.7-3.6]PB Case 1.
- 15. Business Requirements • Ad-hoc analytical reports on 3 months of detail data • Low data and query latency • High Availability
- 16. • Tried/used/evaluated: • MySQL (TokuDB, ShardQuery) • InfiniDB • MonetDB • InfoBright EE • Paraccel (now RedShift) • Oracle • Greenplum • Snowflake DB • Vertica ClickHouse
- 17. Main Migration Challenges • Efficient star-schema for OLAP • Reliable data ingestion • Sharding and replication • Client interfaces
- 18. Data Load Diagram Temp tables (local) Fact tables (shard) SummingMergeTree (shard) SummingMergeTree (shard) Log Files INSERT MV MV INSERT Buffer tables (local) Realtime producers INSERT Buffer flush MySQL Dictionaries CLICKHOUSE NODE
- 19. Sharding and Replication S1 S2 S3 S4 SnTable1 S1 S2 S3 S4 SnTable1 Replica1 Replica2 Altinity Ltd. S1 S2 S3 S4 SnTable1 Replica3
- 20. Major Design Decisions • Dictionaries for star-schema design • Extensive use of Arrays • SummingMergeTree for realtime aggregation • Smart query generation • Multiple shards and replicas
- 21. Project Results • Successful migration and cost reduction • Increased performance and flexibility • 60 servers in 3 replicas • 2-3PB of data • 6,000B+ rows in fact and aggregate tables (50B+ daily load) • 1M+ SQL-queries/day Powered by:
- 22. Case 2. Fintech Company • Stock Symbols Correlation Analysis • 5000 Symbols • 100ms granularity • 10 years of data 100B data points
- 24. Main Challenge • Symbols S(1)..S(5000) • Time points Т(1)…T(300M) • log_return(n)(m) = runningDifference(log(price(n))) • corr(n1,n2) = corr(log_return(n1),log_return(n2)) • For every tuple (n1,n2), 12.5M tuples altogether calculate 12,500,000 times!
- 25. Tried… • Hadoop • Spark • Greenplum ClickHouse
- 26. Distributed Computations • Distribute data across N servers • Calculate log_return for every symbol at every server using Arrays: • (timestamp, Array[String], Array[Float32]) • Distribute correlation computations across all servers • Batch planning
- 27. POC Performance Results • 3 servers setup • 2 years, 5000 symbols: • log_return calculations: ~1 h • Converting to arrays: ~ 1 h • Correlations: ~50 hours • 12,5M/50h = 70/sec And is scales easily!
- 28. Case 3. Bloxy.info - Etherium network analysis • 450M transactions • Transaction level interactive reports • Transaction graph navigation • Aggregate reports • Rich visualization
- 30. Main Challenge: ClickHouse is bad for point queries!
- 31. Main Design Decisions • Encode transaction IDs to binary • ClickHouse MergeTree with low index_granularity • Materialized Views for different sort orders • Apache SuperSet for visualization
- 35. And more: http://bloxy.info • Etherium Mixer Analysis • Token Dynamics • Token Distribution • ERC721 Token and Collectibles • ICO Analysis and Trends • Smart Contract Events and Methods • Etherium Mining • DAO Efficiency Analytics Powered by:
- 36. Case 4. ClickHouse with MySQL • Accessing MySQL from ClickHouse • Accessing ClickHouse from MySQL • Streaming data from MySQL to ClickHouse • Analyzing MySQL logs with ClickHouse
- 37. Accessing MySQL from ClickHouse • External dictionaries from MySQL table • Map mysql table to in-memory structure • Mysql() function select * from MySQL('host:port', 'database', 'table', 'user', 'password'); https://www.altinity.com/blog/2018/2/12/aggregate-mysql-data-at-high-speed-with-clickhouse
- 38. Accessing ClickHouse from MySQL
- 39. Streaming Data from MySQL to ClickHouse https://github.com/Altinity/clickhouse-mysql-data-reader
- 41. Analyzing MySQL logs with ClickHouse • MySQL Logs may grow large • https://www.percona.com/blog/2018/02/28/analyze-raw-mysql- query-logs-clickhouse/ • https://www.percona.com/blog/2018/03/29/analyze-mysql-audit- logs-clickhouse-clicktail/
- 42. Main Lessons • Schema is the most important • Proper data types • Arrays • Dictionaries • Summing/Aggregating MergeTree for realtime aggregation • Materialized Views if one key is not enough • Reduce Index granularity for point queries • Distribute data and load as uniform as possible • Integrate smartly
- 43. ClickHouse is •Fast •Flexible •Scalable And it really works!