Abstract image of a woman sitting on books and studying on a laptop

Building interactive audience analytics with Spark

E
Eugene Zhulenev

This document discusses using Spark to build interactive audience analytics by summarizing log data. It proposes using HyperLogLog to estimate cardinalities and handle distinct counting. The document shows how to model impression and segment data as Spark DataFrames, extend the DataFrame DSL to support HyperLogLog aggregations, and use Spark as an in-memory SQL database to serve analytics queries in 1-2 seconds. Spark allows building a complicated analytics structure, caching data in memory, and handling queries faster than other options like Hive, Impala, or Druid.

1 of 22
Download to read offline
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Building interactive audience analytics with Spark Eugene Zhulenev @ezhulenev
Problem definition 1 / 22
2 / 22 Impression log Ad Site Cookie Impressions Clicks Segments bmw_X5 forbes.com 13e835610ff0d95 10 1 [a.m, b.rk, c.rh, d.sn, ...] mercedes_2015 forbes.com 13e8360c8e1233d 5 0 [a.f, b.rk, c.hs, d.mr, ...] nokia gizmodo.com 13e3c97d526839c 8 0 [a.m, b.tk, c.hs, d.sn, ...] apple_music reddit.com 1357a253f00c0ac 3 1 [a.m, b.rk, d.sn, e.gh, ...] nokia cnn.com 13b23555294aced 2 1 [a.f, b.tk, c.rh, d.sn, ...] apple_music facebook.com 13e8333d16d723d 9 1 [a.m, d.sn, g.gh, s.hr, ...] * - a.m : Male - a.f : Female - b.tk : $75k-$100k annual income - b.rk : $100k-$150k annual income - c.hs : High School - c.rh : College - d.sn : Single - d.mr : Married
3 / 22 What we want to know What is Male/Female ratio for people who have seen 'bmw_X5' ad on forbes.com Income distribution for people who have seen Apple Music ad Nokia click distribution across different education levels
4 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m'
5 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple
6 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple Unfortunately it doesn't work
7 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple Unfortunately it doesn't work It takes minutes to run this type of queries with Impala or Hive It's impossible to pre-generate all reports, number of filter combinations is huge We need instant response for reporting UI
8 / 22 HyperLogLog for cardinality estimation
9 / 22 HyperLogLog trait HyperLogLog { def add(cookieId: String): Unit // |A| def cardinality(): Long // |A ∪ B| def merge(other: HyperLogLog): HyperLogLog // |A ∩ B| = |A| + |B| - |A ∪ B|, def intersect(other: HyperLogLog): Long } Algorithm for the count-distinct problem, approximating the number of distinct elements (cardinality) Uses finite space (configurable precision) Able to estimate cardinalities of >10^9 with a typical accuracy of 2%, using 1.5kB of memory
10 / 22 From cookies to HyperLogLog Ad Site Cookies HLL Impressions Clicks bmw_X5 forbes.com HyperLogLog@23sdg4 5468 35 bmw_X5 cnn.com HyperLogLog@84jdg4 8943 29 Segment Cookies HLL Impressions Clicks Male HyperLogLog@85sdg4 235468 335 $100k-$150k HyperLogLog@35jdg4 569473 194 Percent of college and high school education in BMW campaignPercent of college and high school education in BMW campaign val adImpressions: Seq[Audience(ad, site, hll, imp, clk)] = ... val segmentImpressions: Seq[Segment(name, hll, imp, clk)] = ... val bmwCookies: HyperLogLog = adImpressions .filter(_.ad = "bmw_X5") .map(_.hll).reduce(_ merge _) val educatedCookies: HyperLogLog = segmentImpressions .filter(_.segment in Seq("College", "High School")) .map(_.hll).reduce( _ merge _)
val p = (bmwCookies intersect educatedCookies) / bmwCookies.count() 11 / 22 Spark DataFrames with HyperLogLog
12 / 22 Spark DataFrames Inspired by R data.frame and Python/Pandas DataFrame Distributed collection of rows organized into named columns SchemaRDD in Spark < 1.3.0 High-Level Operations Selecting required columns Filtering Joining different data sets Aggregation (count, sum, average, etc)
13 / 22 Spark DataFrames val adImpressions: DataFrame = sqlContext.parquetFile("/aa/audience") adImpressions.printSchema() // root // | -- ad: string (nullable = true) // | -- site: string (nullable = true) // | -- hll: binary (nullable = true) // | -- impressions: long (nullable = true) // | -- clicks: long (nullable = true) val segmentImpressions: DataFrame = sqlContext.parquetFile("/aa/segments") segmentImpressions.printSchema() // root // | -- segment: string (nullable = true) // | -- hll: binary (nullable = true) // | -- impressions: long (nullable = true) // | -- clicks: long (nullable = true)
14 / 22 Spark DataFrames Percent of college and high school education in BMW campaignPercent of college and high school education in BMW campaign import org.apache.spark.sql.functions._ import org.apache.spark.sql.HLLFunctions._ val bmwCookies: HyperLogLog = adImpressions .filter(col("ad") === "bmw_X5") .select(mergeHll(col("hll")).first() // -- sum(clicks) val educatedCookies: HyperLogLog = hllSegments .filter(col("segment") in Seq("College", "High School")) .select(mergeHll(col("hll")).first() val p = (bmwCookies intersect educatedCookies) / bmwCookies.count()
15 / 22 Extending DataFrames DSL looks like 'native' DataFrame code works faster than RDD[Row] transformations easy to manage mutable state inside partition/merge function case class MergeHLLPartition(child: Expression) extends AggregateExpression with trees.UnaryNode[Expression] { ... } case class MergeHLLMerge(child: Expression) extends AggregateExpression with trees.UnaryNode[Expression] { ... } case class MergeHLL(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression] { override def asPartial: SplitEvaluation = { val partial = Alias(MergeHLLPartition(child), "PartialMergeHLL")() SplitEvaluation( MergeHLLMerge(partial.toAttribute), partial :: Nil ) } }
def mergeHLL(e: Column): Column = MergeHLL(e.expr) 16 / 22 Extending DataFrames DSL Building complicated structure with DSL type SegmentName = String val dailyEstimates: RDD[(SegmentName, Map[LocalDate, SegmentEstimate])] = segments.groupBy(segment_name).agg( segment_name, mergeDailySegmentEstimates( mkDailySegmentEstimate( // -- Map[LocalDate, SegmentEstimate] dt, mkSegmentEstimate( // -- SegmentEstimate(cookieHLL, clickHLL) cookie_hll, click_hll) ) ) ) HyperLogLog is a Monoid SegmentEstimate is a Monoid
Map[K, SegmentEstimate] is a Monoid 17 / 22 Extending DataFrames DSL custom aggregation functions needs to be in org.apache.spark.sql package no guarantee that it will work in next Spark release org.apache.spark.sql.catalyst.expressions.Sum as example
18 / 22 Spark as in-memory SQL database restart overnight to load new data (data preprocessed with Hive) cache all the data in memory serve client requests during business day simple Spray Http/Json Api 40 Spark worker nodes in Yarn cluster 100+ gigabytes cached in memory average response time ~2 seconds
19 / 22 Spark configuration spark.scheduler.mode=FAIR spark.yarn.executor.memoryOverhead=4000 spark.sql.autoBroadcastJoinThreshold=300000000 // ~300mb spark.serializer=org.apache.spark.serializer.KryoSerializer spark.speculation=true repartition datasets (4-6 partitions for core)
20 / 22 Other options HiveHive Too slow for interactive API backend ImpalaImpala Requires writing aggregation functions in C++ Not so sure that 1-2 seconds latency is easy to achieve DruidDruid Managing separate Druid cluster We have batch oriented process Bad support for some of type of queries that we need Not clear how get data back from Druid?
21 / 22 Thank you http://collective.com

More Related Content

PDF
Audience counting at Scale
b0ris_1
 
PDF
Scalding big ADta
b0ris_1
 
PDF
HyperLogLog in Hive - How to count sheep efficiently?
bzamecnik
 
PDF
Big Data Day LA 2015 - Large Scale Distinct Count -- The HyperLogLog algorith...
Data Con LA
 
PDF
Spark-driven audience counting by Boris Trofimov
JavaDayUA
 
PDF
Big Audience at Scale — Spark and Big Data
GlobalLogic Ukraine
 
PDF
Predicting model for prices of used cars
HARPREETSINGH1862
 
PDF
3D for the modern web: declarative3d and gltf
Brian Coughlin
 
Audience counting at Scale
b0ris_1
 
Scalding big ADta
b0ris_1
 
HyperLogLog in Hive - How to count sheep efficiently?
bzamecnik
 
Big Data Day LA 2015 - Large Scale Distinct Count -- The HyperLogLog algorith...
Data Con LA
 
Spark-driven audience counting by Boris Trofimov
JavaDayUA
 
Big Audience at Scale — Spark and Big Data
GlobalLogic Ukraine
 
Predicting model for prices of used cars
HARPREETSINGH1862
 
3D for the modern web: declarative3d and gltf
Brian Coughlin
 

Similar to Building interactive audience analytics with Spark (20)

PDF
Best Practices & Lessons Learned from the field on EMC Documentum xCP 2.0
Haytham Ghandour
 
PDF
Leverage the power of machine learning on windows
José António Silva
 
PDF
Easy path to machine learning
wesley chun
 
ODP
Desenvolvimento Mobile Híbrido
Juliano Martins
 
PDF
[1D6]RE-view of Android L developer PRE-view
NAVER D2
 
PDF
How to Pass the Google Analytics Individual Qualification Test by Slingshot SEO
Aarif Nazir
 
PDF
WebGL demos showcase
Yukio Andoh
 
DOCX
cbse 12 computer science investigatory project
D. j Vicky
 
DOCX
cbse 12 computer science investigatory project
D. j Vicky
 
PPTX
Visualizing large data sets with wijmo enterprise webinar
Ross Dederer
 
PPTX
Deepak-Computational Advertising-The LinkedIn Way
yingfeng
 
PPT
Introduction to 2D/3D Graphics
Prabindh Sundareson
 
PDF
How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow
Daniel Zivkovic
 
PDF
Universal Declarative Services - Simon Chemouil
mfrancis
 
PDF
Google App Engine in 40 minutes (the absolute essentials)
Python Ireland
 
PPTX
The Sizmek_Tech solutions
Karunakar Ravirala
 
PDF
Machine learning on streams of data
Tomasz Sosiński
 
PDF
Chrysler Smart Screen
Mostafa Sameh
 
PPTX
2019 12 14 Global AI Bootcamp - Auto ML with Machine Learning.Net
Bruno Capuano
 
PDF
Conf42 - AI Augmented Platform Engineering.pdf
cfaa0001
 
Best Practices & Lessons Learned from the field on EMC Documentum xCP 2.0
Haytham Ghandour
 
Leverage the power of machine learning on windows
José António Silva
 
Easy path to machine learning
wesley chun
 
Desenvolvimento Mobile Híbrido
Juliano Martins
 
[1D6]RE-view of Android L developer PRE-view
NAVER D2
 
How to Pass the Google Analytics Individual Qualification Test by Slingshot SEO
Aarif Nazir
 
WebGL demos showcase
Yukio Andoh
 
cbse 12 computer science investigatory project
D. j Vicky
 
cbse 12 computer science investigatory project
D. j Vicky
 
Visualizing large data sets with wijmo enterprise webinar
Ross Dederer
 
Deepak-Computational Advertising-The LinkedIn Way
yingfeng
 
Introduction to 2D/3D Graphics
Prabindh Sundareson
 
How to build unified Batch & Streaming Pipelines with Apache Beam and Dataflow
Daniel Zivkovic
 
Universal Declarative Services - Simon Chemouil
mfrancis
 
Google App Engine in 40 minutes (the absolute essentials)
Python Ireland
 
The Sizmek_Tech solutions
Karunakar Ravirala
 
Machine learning on streams of data
Tomasz Sosiński
 
Chrysler Smart Screen
Mostafa Sameh
 
2019 12 14 Global AI Bootcamp - Auto ML with Machine Learning.Net
Bruno Capuano
 
Conf42 - AI Augmented Platform Engineering.pdf
cfaa0001
 
Ad

Building interactive audience analytics with Spark

  • 1. Building interactive audience analytics with Spark Eugene Zhulenev @ezhulenev
  • 3. 2 / 22 Impression log Ad Site Cookie Impressions Clicks Segments bmw_X5 forbes.com 13e835610ff0d95 10 1 [a.m, b.rk, c.rh, d.sn, ...] mercedes_2015 forbes.com 13e8360c8e1233d 5 0 [a.f, b.rk, c.hs, d.mr, ...] nokia gizmodo.com 13e3c97d526839c 8 0 [a.m, b.tk, c.hs, d.sn, ...] apple_music reddit.com 1357a253f00c0ac 3 1 [a.m, b.rk, d.sn, e.gh, ...] nokia cnn.com 13b23555294aced 2 1 [a.f, b.tk, c.rh, d.sn, ...] apple_music facebook.com 13e8333d16d723d 9 1 [a.m, d.sn, g.gh, s.hr, ...] * - a.m : Male - a.f : Female - b.tk : $75k-$100k annual income - b.rk : $100k-$150k annual income - c.hs : High School - c.rh : College - d.sn : Single - d.mr : Married
  • 4. 3 / 22 What we want to know What is Male/Female ratio for people who have seen 'bmw_X5' ad on forbes.com Income distribution for people who have seen Apple Music ad Nokia click distribution across different education levels
  • 5. 4 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m'
  • 6. 5 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple
  • 7. 6 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple Unfortunately it doesn't work
  • 8. 7 / 22 SQL solution select count(distinct cookie_id) from impressions where site = 'forbes.com' and ad = 'bmw_X5' and segment contains 'a.m' Looks pretty simple Unfortunately it doesn't work It takes minutes to run this type of queries with Impala or Hive It's impossible to pre-generate all reports, number of filter combinations is huge We need instant response for reporting UI
  • 9. 8 / 22 HyperLogLog for cardinality estimation
  • 10. 9 / 22 HyperLogLog trait HyperLogLog { def add(cookieId: String): Unit // |A| def cardinality(): Long // |A ∪ B| def merge(other: HyperLogLog): HyperLogLog // |A ∩ B| = |A| + |B| - |A ∪ B|, def intersect(other: HyperLogLog): Long } Algorithm for the count-distinct problem, approximating the number of distinct elements (cardinality) Uses finite space (configurable precision) Able to estimate cardinalities of >10^9 with a typical accuracy of 2%, using 1.5kB of memory
  • 11. 10 / 22 From cookies to HyperLogLog Ad Site Cookies HLL Impressions Clicks bmw_X5 forbes.com HyperLogLog@23sdg4 5468 35 bmw_X5 cnn.com HyperLogLog@84jdg4 8943 29 Segment Cookies HLL Impressions Clicks Male HyperLogLog@85sdg4 235468 335 $100k-$150k HyperLogLog@35jdg4 569473 194 Percent of college and high school education in BMW campaignPercent of college and high school education in BMW campaign val adImpressions: Seq[Audience(ad, site, hll, imp, clk)] = ... val segmentImpressions: Seq[Segment(name, hll, imp, clk)] = ... val bmwCookies: HyperLogLog = adImpressions .filter(_.ad = "bmw_X5") .map(_.hll).reduce(_ merge _) val educatedCookies: HyperLogLog = segmentImpressions .filter(_.segment in Seq("College", "High School")) .map(_.hll).reduce( _ merge _)
  • 12. val p = (bmwCookies intersect educatedCookies) / bmwCookies.count() 11 / 22 Spark DataFrames with HyperLogLog
  • 13. 12 / 22 Spark DataFrames Inspired by R data.frame and Python/Pandas DataFrame Distributed collection of rows organized into named columns SchemaRDD in Spark < 1.3.0 High-Level Operations Selecting required columns Filtering Joining different data sets Aggregation (count, sum, average, etc)
  • 14. 13 / 22 Spark DataFrames val adImpressions: DataFrame = sqlContext.parquetFile("/aa/audience") adImpressions.printSchema() // root // | -- ad: string (nullable = true) // | -- site: string (nullable = true) // | -- hll: binary (nullable = true) // | -- impressions: long (nullable = true) // | -- clicks: long (nullable = true) val segmentImpressions: DataFrame = sqlContext.parquetFile("/aa/segments") segmentImpressions.printSchema() // root // | -- segment: string (nullable = true) // | -- hll: binary (nullable = true) // | -- impressions: long (nullable = true) // | -- clicks: long (nullable = true)
  • 15. 14 / 22 Spark DataFrames Percent of college and high school education in BMW campaignPercent of college and high school education in BMW campaign import org.apache.spark.sql.functions._ import org.apache.spark.sql.HLLFunctions._ val bmwCookies: HyperLogLog = adImpressions .filter(col("ad") === "bmw_X5") .select(mergeHll(col("hll")).first() // -- sum(clicks) val educatedCookies: HyperLogLog = hllSegments .filter(col("segment") in Seq("College", "High School")) .select(mergeHll(col("hll")).first() val p = (bmwCookies intersect educatedCookies) / bmwCookies.count()
  • 16. 15 / 22 Extending DataFrames DSL looks like 'native' DataFrame code works faster than RDD[Row] transformations easy to manage mutable state inside partition/merge function case class MergeHLLPartition(child: Expression) extends AggregateExpression with trees.UnaryNode[Expression] { ... } case class MergeHLLMerge(child: Expression) extends AggregateExpression with trees.UnaryNode[Expression] { ... } case class MergeHLL(child: Expression) extends PartialAggregate with trees.UnaryNode[Expression] { override def asPartial: SplitEvaluation = { val partial = Alias(MergeHLLPartition(child), "PartialMergeHLL")() SplitEvaluation( MergeHLLMerge(partial.toAttribute), partial :: Nil ) } }
  • 17. def mergeHLL(e: Column): Column = MergeHLL(e.expr) 16 / 22 Extending DataFrames DSL Building complicated structure with DSL type SegmentName = String val dailyEstimates: RDD[(SegmentName, Map[LocalDate, SegmentEstimate])] = segments.groupBy(segment_name).agg( segment_name, mergeDailySegmentEstimates( mkDailySegmentEstimate( // -- Map[LocalDate, SegmentEstimate] dt, mkSegmentEstimate( // -- SegmentEstimate(cookieHLL, clickHLL) cookie_hll, click_hll) ) ) ) HyperLogLog is a Monoid SegmentEstimate is a Monoid
  • 18. Map[K, SegmentEstimate] is a Monoid 17 / 22 Extending DataFrames DSL custom aggregation functions needs to be in org.apache.spark.sql package no guarantee that it will work in next Spark release org.apache.spark.sql.catalyst.expressions.Sum as example
  • 19. 18 / 22 Spark as in-memory SQL database restart overnight to load new data (data preprocessed with Hive) cache all the data in memory serve client requests during business day simple Spray Http/Json Api 40 Spark worker nodes in Yarn cluster 100+ gigabytes cached in memory average response time ~2 seconds
  • 20. 19 / 22 Spark configuration spark.scheduler.mode=FAIR spark.yarn.executor.memoryOverhead=4000 spark.sql.autoBroadcastJoinThreshold=300000000 // ~300mb spark.serializer=org.apache.spark.serializer.KryoSerializer spark.speculation=true repartition datasets (4-6 partitions for core)
  • 21. 20 / 22 Other options HiveHive Too slow for interactive API backend ImpalaImpala Requires writing aggregation functions in C++ Not so sure that 1-2 seconds latency is easy to achieve DruidDruid Managing separate Druid cluster We have batch oriented process Bad support for some of type of queries that we need Not clear how get data back from Druid?
  • 22. 21 / 22 Thank you http://collective.com