OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Engagement Platform - Prafulla Gupta.pdf
0 likes109 views
GrowthRx is an analytics and customer engagement platform developed by Times Internet, leveraging ClickHouse for high-volume data processing and user engagement strategies. It supports real-time user segmentation, detailed trend analysis, and efficient data handling with features like materialized views and parametric aggregate functions. The architecture enables the ingestion of vast amounts of data, targeting millions of users through effective analytics tools.
OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Engagement Platform - Prafulla Gupta.pdf
- 1. Using Clickhouse Database for GrowthRx, an analytics and customer engagement platform Prafulla Gupta Principal Architect Times Internet
- 2. //01 // 2 News Live Events Brand Capital Magazines Outdoor Radio Print Music TV Film Education Internet Established in 1838, The Times Group is India’s largest media conglomerate. Its flagship newspaper, The Times of India, is the most read English newspaper in the world. 609 MN Monthly Active Users Entertainment Marketplace Fintech 133 MN Daily Active Users 54 BN Monthly Page Views India’s leading digital consumer company
- 3. //01 // 3 What’s GrowthRx and its features? C u s t o m e r A n a l y t i c s a n d E n g a g e m e n t T o o l U s e d a c r o s s m u l t i p l e p r o d u c t s a t s c a l e Helps convert, retain and engage more users through self serve analytics GrowthRx Project Features Can ingest 40-50k records per sec Low cash- burn on Infra Analyze user trends & generate queries on > 5 billion data Push notifications powered by a segmentation engine that can target 10-15M users/segment View & Analyze multiple charts in Dashboard
- 4. //01 // 4 Overall architecture ● Ingests events data from multiple websites and apps. ● Create real time user segments and target them via multiple mediums ● View and analyse campaign reports. ● Create funnel, cohorts, pivots etc ● Create dashboards for easy trend analysis.
- 5. //01 // 5 Overall architecture - Data ingest ● Receives data at the rate of more than 30-40K Req/Sec. ● More than 1 Trillion data points are collected per month. ● Achieved this on just 3 VM Clickhouse replica setup with 10TB SSD on each machine. ● Clickhouse efficient compression results in very good reduction in size of data at disk.
- 6. //01 // 6 Overall architecture - User segmentation For user segmentation we have queries with multiple AND/OR condition on Events table along with Join on User Profile Table ● Eg: Subscribed Users read Tech News >5 Times in 2 days and not visited Sports section Used Bitmap functions provided by Clickhouse Query processing speed - 200 million rows/sec
- 7. //01 // 7 Overall architecture- Dashboard and Trend Analysis ● Multiple charts on a single screen ● Async execution to throttle number of queries executing parallely. ● Sampling feature to make queries run faster on larger dataset
- 8. //01 // 8 01 02 03 ClickHouse Key Features MATERIALIZED VIEWS SAMPLE CLAUSE PARAMETRIC AGGREGATE FUNCTION
- 9. //01 // 9 ClickHouse Key Features - MATERIALISED VIEWS ● Generate summarised view of raw data ● Multiple Mat Views ● Minimal Space & Max duration storage of data ● Easy to generate campaign reports & power user curve CREATE MATERIALIZED VIEW campaign_user_mv ( `tenantId` LowCardinality(String), `campaignId` UUID, `date` Date, `count` UInt64 ) ENGINE=ReplicatedSummingMergeTree('/{cluster} /{shard}/campaign_user_mv','{replica}') ORDER BY (tenantId, campaignId, date) AS SELECT tenantId, campaignId, toDate(timestamp) AS date, count(*) as count FROM growthrx.campaign GROUP BY tenantId, campaignId, date
- 10. //01 // 10 ClickHouse Key Features - SAMPLE CLAUSE ● SAMPLE clause in Clickhouse query proves very efficient for approximate SELECT query processing ● For large dataset, results using SAMPLE clause are near to accurate but query processing time improves drastically. ● Business trends where approximate results are good enough is an ideal use case. ● Select Sampling Factor based on the expected dataset size. SELECT eventname, round(count(*) * 10) AS count FROM <table> SAMPLE 0.1 WHERE tenantid = ‘x’ and …. GROUP BY eventname
- 11. //01 // 11 ClickHouse Key Features - PARAMETRIC AGGREGATE FUNCTION ● Fast and easy to use. ● Window Funnel function provided by Clickhouse 1. Funnel analysis 2. Creating segments for targeting users who are part of a funnel section e.g. send email to users who are part of Responded section. SELECT funnel_level,count() AS c FROM (SELECT uidint, windowFunnel(2678400)(grx_ timestamp, eventname = ‘lead’, eventname = 'prospect',....) AS funnel_level FROM <table> WHERE (....)
- 12. Thank You!