OSA Con 2022 - Building Event Collection SDKs and Data Models - Paul Boocock - Snowplow.pdf

Building Event Collection
SDKs and Data Models
OSA Con ‘22
Paul Boocock
@paul_boocock
2022 www.snowplow.io

Table of
contents
What is Snowplow?
A Quick Introduction
Tracking SDKs
How we build them and our decisions
Data Models
Considerations for modeling the raw data
_01
_02
_03

What is Snowplow?
A quick intro
_01
www.snowplow.io

Snowplow:
we build
tech to
enable
companies
to Create
Data
www.snowplow.io

But first, a
small detour…
www.snowplow.io
{
"$schema":
"http://iglucentral.com/schemas/com.snowplowana
lytics.self-desc/schema/jsonschema/1-0-0#",
"description": "JSON schema for
a button click event",
"self": {
"vendor":
"com.acme",
"name": "click",
"format":
"jsonschema",
"version": "1-0-
0"
},
"type": "object",
"properties": {
"button": {
"type":
["string", "null"],
"maxLength":
255
}
},
"additionalProperties": false
}

www.snowplow.io
The data created looks a bit like this
user_id
page_url
timestamp
device
city
mkt_campaign
Default
content_id
content_title
author
date_created
button_click
Custom Custom
Entity
Event

But can crucially be evolved over time
www.snowplow.io
{
"$schema":
"self": {
"vendor":
"com.acme",
"name": "click",
"format":
"jsonschema",
"version": "1-0-
0"
},
"type": "object",
"properties": {
"button": {
"type":
["string", "null"],
"maxLength":
255
}
},
}
{
"$schema":
"self": {
"vendor":
"com.acme",
"name": "click",
"format":
"jsonschema",
"version": "1-0-
1"
},
"type": "object",
"properties": {
"button": {
"type":
["string", "null"],
"maxLength":
255
},
"index": {
"type":
["integer", "null"]
}
},
}

www.snowplow.io
The real benefit of that approach is how the
data then looks
Default
fields (1/130)
Custom Event data
event_name unstruct_event_com_acme_click_1
click {
"button": "open_article"
}
Default fields (1/130) Custom Event data
event_name unstruct_event_com_acme_click_1
click {
"button": "open_article"
}
click {
"button": "open_article",
"index": "2"
}

Tracking SDKs
Lorem ipsum
_02.1
www.snowplow.io

Snowplow Tracking SDKs
An Introduction

Multi-language Support
1.
Quickly developed beyond a
web only platform
Web Tracker is unique as
different challenges on the
web vs other platforms
So, what do we do?
2.
Tempting to auto
generate
- Works well for
some trackers
(server)
- Less well for
others (client)
https://quicktype.io/?
www.snowplow.io

Building SDKs
Hand craft SDKs Autogenerate SDKs Can we do both?
www.snowplow.io

Server vs Client SDKs
How do they differ?

Configuration
Challenges

Sensible defaults
Whilst the SDKs are incredibly
configurable, opting for sensible
defaults keeps users happy
Where possible we now try to be more
opinionated and don’t offer
configuration
Works well until someone asks us if
they can configure it on Github!
www.snowplow.io

Data Models
_03
www.snowplow.io

Data modeling is the
process of using
business logic to
aggregate or
otherwise transform
raw data.
www.snowplow.io

1
Process clickstream data from raw events to
create aggregate tables of views, sessions and
users, reducing the volume of data massively
while adding quality
2
Deal with user stitching across sessions and
devices
3
Compatible with schema evolution, so
when you add new fields to your
contexts they immediately show up in
your data models
users
sessions
views
raw events
3%
6%
20%
Rows relative to raw
events
www.snowplow.io
Modeling is not an afterthought

This end-to-end approach results in
Clean, structured
data always available
in your warehouse
Consistent business logic
defined once means no
qualms about what metrics
mean
More time spent building
out ML/AI models instead of
toiling with data problems
www.snowplow.io

Structure data to match your product
video_play
view
video_pause
heartbeat
click
content
id: ‘abc123’
type: ‘video’
date updated: 2019-06...
title: ‘Birthday…’
creator: ‘Nicholas Witchell’
theme: ‘royals’
native: FALSE
personalities: [‘Kate’,
‘Wills’,
‘George’]
Events Entity
search content content content

Going Incremental
_03.2
www.snowplow.io

Consolidation
Similarities between queries:
● Common levels of aggregation
● Repeated logic, like joins
Consolidate ad-hoc queries into a set of
derived tables.
These generalised tables can be used to
solve a variety of use cases.
raw events
page views
sessions
users
20%
6%
3%
Rows relative to raw
events

Value of a page or screen view
PAGE VIEW
HEARTBEAT
ENGAGEMENT
time_engaged: 25s
scroll_depth(x): 100%
scroll_depth(y): 37%
clicks:
1
shares:
0
time_engaged: 15s
clicks:
2
shares:
0
time_engaged: 35s
clicks:
0
shares:
1

One Step Further - Incremental Models
As the business continues to grow:
● The size of the events table grows.
● Behavioural data is more business critical
and needs to be processed more
regularly.
Meaning:
● Running the derived tables in a drop and
compute manner is becoming
increasingly costly and taking
considerable time.
● Processing events in an incremental
manner becomes a necessity.

#page_views.sql
{{ config(
materialized='table'
)
}}
select
page_view_id,
...
row_number() over (
partition by session_id
order by derived_tstamp
) AS page_view_in_session_index
from {{ ref('events') }}
where event_name = 'page_view'
#page_views.sql
{{ config(
materialized='incremental',
unique_key='page_view_id'
)
}}
with sessions_with_new_events as (
select distinct
session_id
{% if is_incremental() %}
and derived_tstamp > (
select max(derived_tstamp) from
{{this}}
)
{% endif %}
)
select
e.page_view_id,
...
row_number() over (
partition by e.session_id
order by e.derived_tstamp
from {{ ref('events') }} e
inner join sessions_with_new_events s
on e.session_id = s.session_id
where e.event_name = 'page_view'
Drop & Recompute
Incremental
Page views - Incremental

www.snowplow.io
Process the least data possible
#page_views.sql
{{ config(
)
}}
select distinct
session_id
{{this}}
)
{% endif %}
)
Reduce the amount of data by:
● Ensure you filter on the
partition/sort keys of the source

www.snowplow.io
#page_views.sql
{{ config(
)
}}
select distinct
session_id
and collector_tstamp > (
select max(collector_tstamp)
from {{this}}
)
{% endif %}
)

www.snowplow.io
#page_views.sql
{{ config(
)
}}
select ...
)
select
e.page_view_id,
...
row_number() over (
partition by e.session_id
order by e.derived_tstamp
● Restrict table scans on all source
tables if possible

www.snowplow.io
#page_views.sql
{{ config(
)
}}
select ...
)
select
e.page_view_id,
...
-- limit table scans
and collector_tstamp > (
select
dateadd(
day,
-3,
max(collector_tstamp))
from {{this}}
)
{% endif %}
tables if possible

www.snowplow.io
#page_views.sql
{{ config(
)
}}
select distinct
session_id
{{this}}
)
{% endif %}
)
Incremental
tables if possible
● Understanding your warehouse

Wrap up Considered the Snowplow Tracking SDKs
- Why we hand craft them
- Differences in Server vs Client SDKs
How do we model billions of rows of atomic data?
- Incrementally!
- Aggregation brings many benefits for analysis
Configuration is painful for everyone
- Easy to get carried away with configurable trackers
but then Data Models need to support it too
Schema’d events make it easy to make type safe objects
for us with the Tracking SDKs
- Great for tracking engineers
Understand your full pipeline to extract the most
- How the data is tracked and processed impacts
how you can model the data

OSA Con 2022 - Building Event Collection SDKs and Data Models - Paul Boocock - Snowplow.pdf

More Related Content

Similar to OSA Con 2022 - Building Event Collection SDKs and Data Models - Paul Boocock - Snowplow.pdf (20)

More from Altinity Ltd (20)

Recently uploaded (20)

OSA Con 2022 - Building Event Collection SDKs and Data Models - Paul Boocock - Snowplow.pdf