How to build and run a big data platform in the 21st century

How to build and run a big
data platform in the 21st
century
Ali Dasdan - Atlassian - adasdan@atlassian.com
Dhruba Borthakur - Rockset - dhruba@rockset.com
This tutorial was presented at the IEEE BigData Conference in Los Angeles,
CA, USA on Dec 9th, 2019.
1

Disclaimers
● This tutorial presents the opinions of the authors. It does
not necessarily reﬂect the views of our employers.
● This tutorial presents content that is already available in
the public domain. It does not contain any conﬁdential
information.
2

Speaker: Ali Dasdan
3
Ali Dasdan is the head of engineering for Conﬂuence Cloud at Atlassian. Prior to Atlassian, he worked as the head of engineering and CTO in three
startups (Turn in real-time online advertising, Vida Health in chronic disease management, Poynt in payments platform) and as an engineering leader
in four leading tech companies (Synopsys for electronic design automation; Yahoo! for web search and big data; Ebay for e-commerce search,
discovery, and recommendation; and Tesco for big data for retail). He has been working in big data and large distributed systems since 2006
(releasing the ﬁrst production application on Hadoop in 2006). He led the building of three big data platforms on-prem from scratch (100PB+ capacity
at Turn, multi-PB capacity for a leading telco, and 30PB+ capacity at Tesco). He is active in multiple areas of research related to big data, namely, big
data platforms, distributed systems, machine learning. He received his PhD in Computer Science from University of Illinois at Urbana-Champaign in
1999. During part of his PhD, he worked as a visiting scholar at the University of California, Irvine on embedded real-time systems.
Linkedin: https://www.linkedin.com/in/dasdan/
Scholarly publications: https://dblp.org/pers/hd/d/Dasdan:Ali
Citations: https://scholar.google.com/citations?user=Bol1-tMAAAAJ&hl=en
Courses taught: Multiple courses at the graduate school, many paper presentations at conferences, 3-day training on embedded real-time systems at
a SoCal company, two tutorials in the 18th and 19th International World Wide Web Conferences.

Speaker: Dhruba Borthakur
4
Dhruba Borthakur is cofounder and CTO at Rockset (https://rockset.com/, a company building software to enable data-powered applications.
Dhruba was the founding engineer of the open source RocksDB [1] database when we was an engineer at Facebook. Prior to that, he is one of the
founding engineers of the Hadoop File System [2] at Yahoo!. Dhruba was also an early contributor to the open source Apache HBase project.
Previously, he was a senior engineer at Veritas Software, where he was responsible for the development of VxFS and Veritas SanPointDirect storage
system; was the cofounder of Oreceipt.com, an ecommerce startup based in Sunnyvale; and was a senior engineer at IBM-Transarc Labs, where he
contributed to the development of Andrew File System (AFS). Dhruba holds an MS in computer science from the University of Wisconsin-Madison.
Linkedin: https://www.linkedin.com/in/dhruba/
Scholarly publications: https://dblp.uni-trier.de/pers/hd/b/Borthakur:Dhruba
Citations: https://scholar.google.com/citations?user=NYgWyv0AAAAJ&hl=en.
Rockset: : https://rockset.com/
RocksDB: https://en.wikipedia.org/wiki/RocksDB
Apache HDFS: https://svn.apache.org/repos/asf/hadoop/common/tags/release-0.10.0/docs/hdfs_design.pdf

Abstract
We want to show that building and running a big data platform for both streaming
and bulk data processing for all kinds of applications involving analytics, data
science, reporting, and the like in today’s world can be as easy as following a
checklist. We live in a fortunate time that many of the components needed are
already available in the open source or as a service from commercial vendors. We
show how to put these components together in multiple sophistication levels to
cover the spectrum from a basic reporting need to a full ﬂedged operation across
geographically distributed regions with business continuity measures in place. We
plan to provide enough information to the audience that this tutorial can also serve
as a high-level goto reference in the actual process of building and running.
5

Motivation
We have been working on big data platforms for more than a decade now. Building
and running big data platforms have become relatively more manageable now. Yet
our interactions at multiple forums have shown us that a comprehensive and
checklist-style guideline for building and running big data platforms does not exist
in a practical form. With this tutorial we are hoping to close this gap, within the
limitations of the tutorial length.
6

Targeted audience
● Targets
○ Our first target is the developers and practitioners of software or analytics or data science.
However, we do provide multiple research issues and provide a review of the research literature
so that researchers and students can significantly benefit from this tutorial too.
● Content level
○ 25% beginner, 50% intermediate, 25% advanced
● Audience prerequisites
○ Introductory software engineering or data science knowledge or experience as an individual
contributor or as a manager of such individuals. Interest in building and running big data
platforms.
7

Terminology and notation
● API: Application Programming Interface
● BCP: Business Continuity Planning
● DAG: Directed Acyclic Graph
● DB: Database
● DR: Disaster Recovery
● GDPR: General Data Protection Regulation
● ML: Machine Learning
● On premise (on-prem): Computing services offered by the 1st party
● Public cloud: Computing services offered by a 3rd party over the public internet
○ Possibly from Alibaba, Amazon, Google, IBM, Microsoft, Oracle, etc.
● RPO: Recovery Point Objective; RTO: Recovery Time Objective
● SLA: Service Level Agreement; SLI: Service Level Indicator; SLO: Service Level
Objective
● TB (terabyte); PB (petabyte) = 1000 TB; EB (exabyte) = 1000 PB
8

Tutorial outline
Section Time allocated Presenter
1 Introduction 15 Ali
2 Architecture 30 Ali
3 Input layer 10 Dhruba
4 Storage layer 10 Dhruba
5 Data transformation 10 Ali
6 Compute layer 10 Ali
7 Access layer 10 Dhruba
8 Output layer 10 Dhruba
9 Management and operations 10 Ali
10 Examples, lessons learned 5 Ali 9

Introduction
Section 1 of 10
10

● Clive Humby, UK mathematician,
founder of Dunnhumby, and an
architect of Tesco’s Clubcard, 2006
○ “Data is the new oil. It’s valuable, but
if unreﬁned it cannot really be used. It
has to be changed into gas, plastic,
chemicals, etc to create a valuable
entity that drives proﬁtable activity;
so must data be broken down,
analyzed for it to have value.”
● The Economist (2019): “Data is the
new oil.”
● Forbes (2018), Wired (2019): “No,
data is not the new oil.”
● Ad Exchanger (2019): “Consent, not
data, is the new oil.”
11
What is big data?
https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data

What is big data?
12
● No standard definition
● According to my then 6-yr old daughter
○ “You get data; then you get more data; and
you get more and more data; finally you get
big data.”
● Another definition
○ If the size of your data is part of the
problem, you may have big data.
● “Big” along one or more of 3 V’s
Data volumes
● 10+ of terabytes
○ A high end server
● 10+ of petabytes
○ US Library of Congress (10PB?)
● 100+ of petabytes
○ Turn (now Amobee) (2016)
○ Dropbox (2016)
○ Netflix (2017)
○ Twitter (2017)
● 1+ of exabytes
○ Amazon, Google, Facebook
○ 1 exabyte = 1000 petabytes
● 1+ of zettabytes
○ (per Cisco) global internet traffic

13
How big is a petabyte of data?

14
How big is a petabyte of data?
● Length of US land and water boundary:
19,937mi = 24,483km
○ Length of US-Canada land and water
boundary: 5,525mi = 8,892km
○ Length of US-Mexico land and water
boundary: 1,933mi = 3,112km
○ Length of US coastline: 12,479mi =
20,083km
● Recall 1 PB of text = 50M ﬁle cabinets or
23,000km of ﬁle cabinets
● So 1PB of text almost covers all around US
land and water boundary
● https://fas.org/sgp/crs/misc/RS21729.pdf

● A high-end server in 2017
○ Main memory size: 0.5TB
○ Hard disk size: 45TB
○ Number of cores: 40
● 2 racks with 15 servers each
○ Total main memory size: 15TB
○ Total hard disk size: 1.4PB
○ Total number of cores: 1,200
● Now consider the total number of racks in
the row in this picture
● Then consider the total number of rows in
a typical datacenter
● Finally consider the total number of data
centers owned by one or more companies
● That is a lot of data capacity
15
Where to ﬁt a petabyte of data in a datacenter?

16
Generic big data platform architecture (1 of 3)

17

18

19
Generic big data platform
● To discuss
○ how this generic architecture is realized in
the real world, or
○ how the real world architectures ﬁt into
this generic architecture
● To discuss the six key components of a
big data platform
Six key components to discuss
1. Input
2. Compute (or process)
3. Data transformations
4. Storage
5. Access
6. Output
7. Management and operations

Gartner: Analytics maturity (2017)
20

Gartner: Types of analytics (2017)
21

22
Gartner: Analytics maturity (2017)
https://www.gartner.com/en/newsroom/press-releases/2018-02-05-gartner-survey-shows-organizations-are-slow-to-advance-in-data-and-analytics

23
Cons of a big data platform
● Hope that pros are obvious already
● Cons and how to counter them
○ Costly
■ But the beneﬁts are immense, just a little more patience
○ Not many users
■ But they will come, guaranteed
○ Centralized
■ But this opens up more innovation opportunities through sometimes surprising
connections between data sources and applications
○ Single point of failure
■ But the platform can be made more reliable using disaster recovery and other availability
and reliability enhancement techniques
○ Big security vulnerability
■ But the platform can be made more secure using many security enhancement techniques

Public cloud vs on premise?
24
Public cloud
● Can start now
● Elastic with resources and cost
● Inexpensive for small scale
● Expensive for huge scale
● Ease of operations
On premise
● Wait until it is built (a few months)
● Inelastic with resources, ﬁxed cost
● Expensive for small scale
● Relatively inexpensive for huge scale
● Cost of operations

25
Apache Hadoop ecosystem to build and run a big data platform
https://www.apache.or
g/

Big data platform
architecture: Self built
Section 2 of 10
26

27
Airbnb: Big data platform (2016)
https://medium.com/airbnb-engineering/data-infrastructure-at-airbnb-8adfb34f169c

28
Airbnb: Big data platform (2018)
https://medium.com/airbnb-engineering/druid-airbnb-data-platform-601c312f2a4c

29
Netﬂix: Big data platform (2013)
https://medium.com/netﬂix-techblog/hadoop-platform-as-a-service-in-the-cloud-c23f35f965e7

30
Netﬂix: Big data platform (2017)
https://www.youtube.com/watch?v=CSDIThSwA7s

31
Facebook: Big data platform (2008)
https://research.fb.com/wp-content/uploads/2010/06/data-warehousing-and-analytics-infrastructure-at-facebook.pdf

32
Facebook: Real-time processing (2016)
https://research.fb.com/wp-content/uploads/2016/11/realtime_data_processing_at_facebook.pdf
● Scribe: Message bus,
transport mechanism for
sending data to batch
and real-time systems
● Puma: Stream processing
system with apps written
in a SQL-like language
● Swift: Stream processing
engine that provides
checkpointing for Scribe
● Stylus: Low-level stream
processing framework
● Laser: K/V store built on
RocksDB
● Scuba: Slice-and-dice
analysis data store
● Hive: Exabyte-scale data
warehouse

33
Slack: Big data platform (2016)
https://slack.engineering/data-wrangling-at-slack-f2e0ff633b69

34
Uber: Big data platform (2016)
https://eng.uber.com/tech-stack-part-two/

35
Uber: Real-time processing (2016)
https://eng.uber.com/ureplicator/

36
Uber: Big data platform (2019)
“The third generation of our Big Data platform incorporates faster, incremental data ingestion (using our open source Marmaray framework), as
well as more eﬃcient storage and serving of data via our open source Hudi library.”
https://eng.uber.com/uber-big-data-platform/

Big data platform
architecture: Cloud built
Section 2 of 10
37

IBM Cloud: Big data platform
38
https://www.ibm.com/cloud/garage/architectures/dataAnalyticsArchitecture/reference-architecture

39
Microsoft Cloud: Big data platform
https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/

40
Microsoft Cloud: Batch processing

41
Microsoft Cloud: Real-time streaming

42
Google Cloud: Big data platform
https://cloud.google.com/solutions/build-a-data-lake-on-gcp

43
Google Cloud: Batch processing

44
Google Cloud: Real-time processing

45
Amazon AWS: Big data platform
https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/svoc_data_platform_mdm_ra.pdf?did=wp_card&trk=wp_card

Big data platform
architecture: Consulting built
Section 2 of 10
46

47
Cloudera: Big data using Hadoop ecosystem
https://www.cloudera.com/products/open-source/apache-hadoop.html

48
Cloudera: Real-time processing
https://blog.cloudera.com/introducing-cloudera-dataﬂow-cdf/

49
ThoughtWorks: Big data platform

50
NIST Big data reference architecture (NBDRA)
https://bigdatawg.nist.gov/_uploadﬁles/NIST.SP.1500-6r1.pdf

51
NIST NBDRA: Top level roles and fabrics

52
NIST NBDRA: Activities view

53
NIST NBDRA: Functional view

Big data platform
architecture: Batch vs
Streaming
Section 2 of 10
54

Generic: Lambda architecture
55https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/
Nathan Marz, creator of Storm

Generic: Kappa architecture
56
Jay Kreps, creator of Kafka

57
Facebook: Aggregator Leaf Tailer Architecture
https://engineering.fb.com/production-engineering/serving-facebook-multifeed-eﬃciency-performance-gains-through-redesign/

58
Streaming: Design decisions and their impact
● Language paradigms
○ Declarative, functional,
procedural
● Data transfer between nodes
of the DAG
○ Direct
○ Broker based
○ Persistent storage based
● Processing (state vs output)
semantics
○ At least once, at most once,
exactly once
● State-saving mechanism
○ Replication, persistence via
local DB or remote DB,
upstream backup, global
consistent snapshot
● Reprocessing
○ Stream only, separate
stream and batch, stream
also as batch

59
Uber: Generic big data architecture for future
“Building a more extensible data transfer platform allowed us to easily aggregate all data pipelines in a standard way under one service as well as
support any-to-any connectivity between any data source and data sink.”

60
Conclusion on batch vs streaming
In an established company interfacing with the systems of many partners, it is
important to support both batch and streaming for data transfer and operations.
The main reason is that this company needs to adapt to the available external
interfaces and get the job done, as it will be almost impossible to force every
external interface to adapt to what this company wants to support.
One way to minimize effort is to ensure an application can run both in batch and
streaming mode to a large extent.
Having said that, the future is far more real-time, i.e., far more streaming but it may
take years for some companies.

Input layer
Section 3 of 10
61

Top 5 sources of input
63https://www.allerin.com/blog/top-5-sources-of-big-data

64
Getting data from a database
● Open source tools
○ Apache Sqoop
○ Debezium
● Commercial tools
○ Elastic Beats
○ ??
● Public cloud tools
○ Internal

65
Getting files / logs
● Stream the file into a streaming system
○ Apache Kafka
● Get files / logs directly
○ Apache Flume
○ Internal

● Open source tools
○ Apache Flume
○ Kafka connector for HDFS
○ Kafka connector to <output>
○ Internal
66
Getting from streaming

Batch
● Volume: Huge
● Velocity: Non-real time
● Variety: N/A
● Cost: Lower
67
Getting data in batch vs streaming
Streaming
● Volume: Medium to large
● Velocity: Real time or near real time
● Variety: N/A
● Cost: Higher

Storage layer
Section 4 of 10
68

Choices for public cloud vs on-prem
69
Public cloud
● AWS
○ S3
● Google Public Cloud
○ Google Cloud Storage
● Azure
○ Azure Storage
● Also see other vendors
On premise
● Apache Hadoop Distributed File System
(HDFS)

Public cloud vs on-prem storage only comparison (2017)
https://databricks.com/blog/2017/05/31/top-5-reasons-for-choosing-s3-over-hdfs.html
70

A comparison of the complete package
71
● At a minimum, the complete package includes
○ Storage cost
○ Compute cost
○ Network cost
○ Operations cost (cost to operate daily and time to ﬁrst query)
○ People cost
● For simplicity, let us consider only the compute and storage costs of building a
big data cluster on a public cloud and on prem
○ Number of servers: 800, equally divided into two data centers
○ Server details: 512GB memory, 45TB disk, 2 CPUs with 20 cores each
○ Cluster details: 400TB memory, 36PB disk, 32,000 CPU cores
○ On prem: 800 * $12,500 = $10M with 3-year amortization
○ Public cloud: depends but $1M or more per month
● For large big data clusters, public cloud seems more expensive but it has its
own advantages, as covered before

Refresher: Row vs column storage format
72
https://bigdatawg.nist.gov/_uploadfiles/NIST.SP.1500-6r1.pdf
● Row format
○ Good for reading all
ﬁelds in a record
○ Good for updates
○ Good for more
transactional needs
● Columnar format
○ Good for reading few
ﬁelds in a scan
○ Good for compression
○ Not ideal for updates
○ Good for more analytics
needs

Data organization choices
73https://bigdatawg.nist.gov/_uploadfiles/NIST.SP.1500-6r1.pdf

74
Data organization comparison

Data formats
75
● Row format
○ Apache Avro
● Columnar
○ Apache Arrow
○ Apache Parquet
○ Apache ORC
● Columnar and in-memory optimization
○ Apache Arrow

● Data encryption possibilities
○ Data at rest: Inactive data stored physically in persistent storage
○ Data in process: Active data in memory manipulated by an application
○ Data in transit: Active data traveling between servers or systems
● Key management options
○ Customers manage their own keys
○ You manage all keys
● Key management tools
○ Use the relevant public cloud solutions
○ Use the solution provided by the open source distribution partner
● Processing over encrypted data
○ Decrypt using keys, managed by the application
○ Run processing over encrypted data without decrypting
■ Limited scope
■ Research solutions
76
Encryption

Data transformation
Section 5 of 10
77

Four kinds of data
78
Raw data
● Data or logs ‘as is’
● “Source of truth”
● Data source for tables, profiles, and
graphs
● Archival
● Transformed for unified formats
● Ex: All event logs
Tables
● For focused, fast access
● Ex (advertising): Clicks, actions
● Ex (retail): Sales
(Knowledge) Graphs
● Modeling relationships between profiles
● Having given and derived fields for both
nodes and edges
● Reveals new observations
● Ex (retail): Customers to products to
suppliers to ...
Profiles
● For each key object in the space
● 360-degree view of each object
● For fast access to relevant fields
● Having given and derived fields
● Ex (advertising): Anonymized users
● Ex (retail): Customer, product, supplier

● Pull from source systems of raw data into
the platform ‘as is’
● Transform raw data into a unified format
for unified and easy access
● Stream new data into existing tables, or
create new tables if needed
● Stream new data into existing profiles, or
create new profiles if needed
● Drive new fields or enhance field values for
profiles
● Stream changes into graphs, or create new
graphs if needed
● Enhance properties of graph nodes and
edges
● Push data and insights into destination
systems
79
Pipelines to create four kinds of data
Graphs
Profiles
Raw data in
“Parquet”
Raw data
Tables

● ETL
● Reporting
● Analytics
● Aggregates for serving DB or tools
● ML feature engineering
● ML model building
● … and other application speciﬁc pipelines
80
Other types of data pipelines

Compute & Access layers
Section 6 and 7 of 10
81

Custom applications
82
● Multiple languages and tools
available for creating custom
applications
● Java and later Pig were more
common in the past
○ There is probably a large number of
applications in these languages
now that work just fine.
○ They were built on top of Apache
Hadoop’s MapReduce functionality.
○ If large, the cost of migrating to a
more popular language or tool can
be costly.
● Python is more common now
● Prototyping can also be done in SQL
or Python.
● For input data flows, Apache Nifi can
be used.

83
Applications via systems
● Apache Spark is the preferred choice right
now.
○ Has support for Graphs, ML, SQL, and
Streaming
○ Can use Python or Scala or SQL on Spark.
● Apache Hive is another choice for SQL
interface.
● Prototyping can be done in SQL on Spark
and can later move to Python or Scala for
speed and ease of expressiveness.
● Notebooks via Python or R are also very
popular for their ease of use and
integrated approach in data science
applications.
● Established tools like Matlab and SPSS
also support Hadoop and Spark now.
● Lots of commercial tools with SQL-based
access

Apache Spark
84https://spark.apache.org/

85
Workflow management
● Many tools are available
○ Apache Airflow (from Airbnb)
○ Apache Oozie
○ Netflix Genie
○ Spotify Luigi
○ Linkedin’s Azkaban
● Public clouds have their own tools.
● Each of these tools uses a DAG, with each
node representing a job and each directed
edge representing precedence or
dependence between jobs.
● These tools are compared based on their
expressiveness, input language, the ease
of use with or without a GUI, reliability, etc.
● Apache Airflow seems to be the preferred
choice now.

86
A comparison of workflow management tools
https://medium.com/@xunnan.xu/workflow-processing-engine-overview-2018-airflow-vs-azkaban-vs-conductor-vs-oozie-vs-amazon-step-90affc54d53b

87
Metadata and access management
● Many tools
○ Apache Atlas
○ Apache Ranger
○ Apache Knox
○ Netﬂix Metacat
○ Ab Initio (commercial)
○ Alation (commercial)
○ Collobra (commercial)
● Many other tools have their internal
management functionality
○ E.g., Rockset

Output layer
Section 8 of 10
88

Refresher: Types of databases
89
Operational applicationsAnalytics applicationsTransactional applications
https://rockset.com/blog/operational-analytics-what-every-software-engineer-should-know/

Refresher: Types of databases
90

Output choices
91
● Keep the output in the platform
● Push the output to a serving db
○ Reporting
○ Timeseries
○ Key/value
○ Search and/or document
○ Graph
● Push the output to another
system
○ Another data platform
○ Streaming system
○ Another destination,
possibly external

● Two key use cases
○ Reporting
○ Operational
● Reporting use case
○ Usually for analytics
○ More periodic, slower pace
○ Connected to more static dashboards
● Operational use case
○ Usually for operations
○ More real-time
○ Connected to more dynamic / live
dashboards
● Data stores for reporting
○ Postgres, MariaDB, Druid
○ Commercial solutions: Oracle, Teradata.
○ Public cloud solutions: Internal
● Data stores for operational
○ Tsdb
○ Commercial solutions: Rockset, Elastic
Search, Redis, InﬂuxDB
○ Public cloud solutions: Internal 92
Serving DB choices

● Reporting
○ Looker
○ Microstrategy
○ Tableau
93
Business intelligence interfaces
● Operational
○ Grafana
○ Apache superset

94
Serving DB interface example: Overview

95
Serving DB interface example: Metadata

96
Serving DB interface example: Query

97
Serving DB interface example: Management

Management and operations
Section 9 of 10
98

Service Level Indicators, Objectives, and Agreements
99
● Data platforms usually start service without any guarantees.
● Users also do not expect any guarantees as they usually experiment or
prototype.
● However, within six or so months, as more production jobs are moved to the
data platform, service level guarantees will be demanded.
● In our experience, a sign of success with data platformization is that more and
more users start complaining about such service level experiences as the
speed of job execution, the number of users, reliability, and availability of data
sources.
● Service level indicators are common for all users.
○ Every user expects reliable operation.
● Service level objectives may be user or task dependent.
○ One user may require her job to execute once a day, another once every hour.

Service Level Indicators (SLIs) to guarantee
100
● Latency
● Frequency
● Data correctness, exactly-once semantics
globally
● Availability
● Durability
● Reliability
● Mean time to resolve (MTTR) incidents
● Mean time to detect (MTTD) incidents
● Mean time to communicate incidents and
outages
● Change management
● Resource usage, cost
● Ease of development
● Ease of deployment
● Ease of experimentation
● Ease of operation
● Ease of metadata management
○ for data, jobs, models, and features
● Ease of data ingress, data egress
● Availability of tools for access and insights
● Number of users supported
Usually the durability objective for data is far more stringent than the availability of the platform.
In other words, the platform may be down but should not lose data.

● Need for multiple clusters
○ Different sets of users or applications, e.g., ﬁnancial data with insiders vs not
○ Disaster recovery
○ Load reduction or balancing
○ Cluster size limitations due to, e.g., Hadoop software scaling limitations
● Explanation using the disaster recovery example: Given two clusters A and B
○ The same data on both at any given time
■ How? Constantly replicate data from A to B and vice versa
■ For improved data consistency, ingest data to A and then replicate to B or vice versa
○ The same applications ready to go on both at any given time
■ How? Deploy to both continuously and test at regular intervals
○ Each is sized to support the desired traﬃc load if the other goes down
○ Due to the cost of these clusters, it is not recommended to run them as active-passive. Both
can be active for different sets of applications and users while each is ready to take the load
from the other in case of a disaster
101
Multi-cluster management

Disaster recovery
102
● RPO: Recovery Point Objective
○ Time during which data got lost due to
an incident
● RTO: Recovery Time Objective
○ Time to recovery from an incident
● RPO and RTO may depend on the
cluster, the data, the applications, etc.
● One way to minimize RPO
○ If raw data comes from sources
outside the data platform, have these
sources to keep a history longer than
RPO
● RTO will include multiple activities
○ Ingesting the “lost” data from the
sources
○ Starting the applications
○ Running the applications to produce
the “lost” output
○ Running the applications to process
the current incoming data
https://en.wikipedia.org/wiki/Disaster_recovery

● Users just want to focus on getting their reports and insights
● Need full operations support to ensure users are happy
● As a data platform gets more production usage and dependence, the
expectations from and pressures on operations team will increase
signiﬁcantly so start early to build the necessary operations muscles
● Need an operations team just for the data platform
○ If the team is small and/or inexperienced, also get maintenance and service deals from
vendors for hardware and software support
● Make sure the data, the pipelines, the applications are owned by
non-operations team; the operations team needs to focus on ensuring that the
platform runs well as an enabler for its users
● One sign of success of a data platform: Users complain about limited data
platform resources, how long their applications wait in queues, how long their
applications take time to run, etc.
103
Operations

Other concerns
● Data governance
● Security
● Data locality
● Privacy
● GDPR
We will not cover these in this tutorial due to time and focus limitations.
104
The rest of management concerns

Example: Machine Learning
Section 10 of 10
105

Microsoft Cloud: Big data architecture for IoT
106

107
Google Cloud: Machine learning processing

108
Apple: Machine learning processing
https://dl.acm.org/citation.cfm?id=3314050

109
Apple: ML from conventional to integrated

110
Apple: ML data platform architecture

111
Example:
Web Search Log Analysis
The 1st production application on Hadoop,
internally called “key site monitoring”,
deployed in the summer of 2006 at Yahoo!
Section 10 of 10

Refresher: Web Search Engine Architecture
112
https://www.slideshare.net/adas0693

Crawler Log Analysis: Key Site Monitoring
● A web search engine gets content by crawling the web sites on the WWW.
● Key sites were the most important sites that our crawlers were configured to
always crawl for these sites’ quality content.
○ Key sites were derived using a PageRank-like algorithm on the web graph.
● Key site monitoring (KSM) was to ensure that these sites were always crawled
satisfactorily every day.
● The (simplified) KSM process worked as follows:
○ Pull in a day’s log files from a sample of crawler nodes into HDFS
○ Scan these log files for daily statistics for key sites in parallel using MapReduce on Hadoop
○ Add daily statistics to the existing time series for key sites on a DB outside Hadoop
○ Provide graphical visualization and alerting over these time series
● v1.0 vs v2.0
○ v1.0: Move large log files from a sample of crawler nodes into HDFS; scan them in parallel on
Hadoop; send tiny stats data into monitoring DB
○ v2.0: Scan log files in each crawler node in parallel for daily stats; send tiny stats data into
monitoring DB 113

Key Site Monitoring Architecture
114
CrawlerLogs
https://www.linkedin.com/pulse/ﬁrst-production-application-hadoop-ali-dasdan/

115
Example:
Online Real-Time Advertising
Section 10 of 10

116
Resources to learn the details: Systems
● How does online advertising work?
○ https://www.linkedin.com/pulse/how-does-online-real-time-advertising-work-ali-dasdan/
● Architecture of a generic online advertising platform
○ https://www.linkedin.com/pulse/architecture-generic-online-real-time-advertising-platform-ali-dasdan/
● From 0.5 Million to 2.5 Million: Eﬃciently Scaling up Real-Time Bidding
○ https://ieeexplore.ieee.org/document/7373421
○ http://www0.cs.ucl.ac.uk/staff/w.zhang/rtb-papers/turn-throatling.pdf
● Overview of Turn Data Management Platform for Digital Advertising
○ http://www.vldb.org/pvldb/vol6/p1138-elmeleegy.pdf
● A High Performance, Custom Data Warehouse on Top of MapReduce (similar
to Hive)
○ https://www.vldb.org/pvldb/vldb2010/pvldb_vol3/I08.pdf
● Leveraging Materialized Views For Very Low-Latency Analytics Over
High-Dimensional Web-Scale Data
○ http://www.vldb.org/pvldb/vol9/p1269-liu.pdf

117
Resources to learn the details: Data Science
● Estimating Conversion Rate in Display Advertising from Past Performance
Data
○ https://dl.acm.org/citation.cfm?id=2339651
○ https://s.yimg.com/ge/labs/v2/uploads/kdd20122.pdf
● Online Model Evaluation in a Large-Scale Computational Advertising Platform
○ https://arxiv.org/abs/1508.07678
● Multi-Touch Attribution Based Budget Allocation in Online Advertising
● User Clustering in Online Advertising via Topic Models
● Scalable Audience Reach Estimation in Real-time Online Advertising
● Real Time Bid Optimization with Smooth Budget Delivery in Online Advertising

Lessons learned
1. Start small, build it, and they will come
2. Put all data in one place
3. Build a general platform
4. Do not overbuild as you will rebuild
5. Use your own infrastructure
6. Ship early and often
7. A/B test relentlessly
8. Be patient: Innovation takes time to get accepted
119

References (1 of 12)
https://www.ibm.com/cloud/garage/architectures/dataAnalyticsArchitecture/reference-architecture
https://www.oracle.com/assets/oracle-wp-big-data-refarch-2019930.pdf#targetText=The%20reference%20architecture%20is%20deﬁned,best%20practices%20in%20the%20industry.
http://eprints.bournemouth.ac.uk/30020/1/SKIMA2016_paper_76.pdf
https://core.ac.uk/download/pdf/83943361.pdf
https://www.emc.com/collateral/white-paper/h12878-rsa-pivotal-security-big-data-reference-architecture-wp.pdf
https://docs.cloudera.com/documentation/other/reference-architecture.html
https://www.dellemc.com/resources/en-us/asset/white-papers/solutions/cdh-architecture-guide.pdf
https://h20195.www2.hpe.com/v2/GetPDF.aspx/4AA5-9086ENW.pdf
https://www.cisco.com/c/dam/m/en_sg/dc-innovation/assets/pdfs/cloudera-enterprise-data-lake-presentation.pdf
https://lenovopress.com/tips1329.pdf
121

https://eng.uber.com/managing-data-workflows-at-scale/
https://medium.com/netflix-techblog/hadoop-platform-as-a-service-in-the-cloud-c23f35f965e7
https://www.youtube.com/watch?v=CSDIThSwA7s
https://medium.com/netflix-techblog/genie-2-0-second-wish-granted-d888d79455c6
https://medium.com/netflix-techblog/evolving-the-netflix-data-platform-with-genie-3-598021604dda
https://medium.com/netflix-techblog/announcing-suro-backbone-of-netflixs-data-pipeline-5c660ca917b6
https://medium.com/netflix-techblog/how-data-inspires-building-a-scalable-resilient-and-secure-cloud-infrastructure-at-netflix-c14ea9f2d00c
https://slack.engineering/data-wrangling-at-slack-f2e0ff633b69
https://medium.com/airbnb-engineering/data/home
https://medium.com/airbnb-engineering/data-infrastructure-at-airbnb-8adfb34f169c
https://cloud.google.com/data-catalog/
https://landing.google.com/sre/workbook/chapters/data-processing/
https://ai.google/research/pubs/pub37795 122

123
https://ai.google/research/pubs/pub48449

124
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/42851.pdf
https://arxiv.org/pdf/1905.12133.pdf
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2017/the-infrastructure-behind-twitter-scale.html
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/partly-cloudy-architecture.html

125
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/democratizing-data-analysis-with-google-bigquery.html
https://blog.twitter.com/engineering/en_us/topics/insights/2016/discovery-and-consumption-of-analytics-data-at-twitter.html
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2019/interactive-analytics-at-mopub.html
http://www.vldb.org/pvldb/vol6/p1138-elmeleegy.pdf
http://www.vldb.org/pvldb/vol9/p1269-liu.pdf
https://www.vldb.org/pvldb/vldb2010/pvldb_vol3/I08.pdf
https://research.fb.com/publications/realtime-data-processing-at-facebook/
https://research.fb.com/publications/presto-sql-on-everything/
https://research.fb.com/publications/a-large-scale-study-of-data-center-network-reliability/
https://research.fb.com/publications/providing-streaming-joins-as-a-service-at-facebook/
https://research.fb.com/publications/sve-distributed-video-processing-at-facebook-scale/
https://research.fb.com/publications/canopy-end-to-end-performance-tracing-at-scale/

126
https://research.fb.com/publications/dqbarge-improving-data-quality-tradeoffs-in-large-scale-internet-services-2/
https://research.fb.com/publications/cubrick-indexing-millions-of-records-per-second-for-interactive-analytics/
https://research.fb.com/publications/one-trillion-edges-graph-processing-at-facebook-scale/
https://research.fb.com/publications/cubrick-a-scalable-distributed-molap-database-for-fast-analytics/
https://research.fb.com/publications/wormhole-reliable-pub-sub-to-support-geo-replicated-internet-services/
https://research.fb.com/publications/an-analysis-of-facebook-photo-caching/
https://research.fb.com/publications/unicorn-a-system-for-searching-the-social-graph/
https://research.fb.com/publications/tao-facebooks-distributed-data-store-for-the-social-graph-2/
https://research.fb.com/publications/tao-facebooks-distributed-data-store-for-the-social-graph/
https://research.fb.com/publications/apache-hadoop-goes-realtime-at-facebook/
https://research.fb.com/publications/ﬁnding-a-needle-in-haystack-facebooks-photo-storage/
https://research.fb.com/publications/data-warehousing-and-analytics-infrastructure-at-facebook/
https://research.fb.com/publications/hive-a-warehousing-solution-over-a-map-reduce-framework/
https://research.fb.com/publications/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/

127
https://research.fb.com/publications/facebooks-data-center-network-architecture/
https://research.fb.com/publications/presto-sql-on-everything/
https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
https://engineering.linkedin.com/blog/topic/stream-processing
https://engineering.linkedin.com/architecture/brief-history-scaling-linkedin
https://engineering.linkedin.com/blog/2016/06/voices--a-text-analytics-platform-for-understanding-member-feedb
https://engineering.linkedin.com/blog/topic/espresso
https://engineering.linkedin.com/teams/data/projects/data-management
https://engineering.linkedin.com/teams/data/projects/wherehows
https://engineering.linkedin.com/blog/2016/03/open-sourcing-wherehows--a-data-discovery-and-lineage-portal
https://engineering.linkedin.com/teams/data/projects/pinot
https://engineering.linkedin.com/teams/data/projects/xlnt
https://engineering.linkedin.com/ab-testing/xlnt-platform-driving-ab-testing-linkedin

128
https://engineering.linkedin.com/teams/data/projects/dr-elephant
https://engineering.linkedin.com/teams/data/projects/gobblin
https://engineering.linkedin.com/teams/data/projects/dali
https://engineering.linkedin.com/blog/topic/Spark
https://engineering.linkedin.com/blog/topic/hadoop
https://docs.microsoft.com/en-us/azure/architecture/data-guide/
https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/batch-processing
https://docs.microsoft.com/en-us/azure/architecture/data-guide/big-data/real-time-processing
https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/analytical-data-stores
https://azure.microsoft.com/en-us/solutions/architecture/advanced-analytics-on-big-data/
https://azure.microsoft.com/en-us/solutions/architecture/modern-data-warehouse/
https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/
https://blogs.msdn.microsoft.com/azurecat/2018/12/24/azure-data-architecture-guide-blog-8-data-warehousing/

129
https://docs.microsoft.com/en-us/azure/architecture/guide/architecture-styles/big-data
http://www.vldb.org/pvldb/vol12/p2170-tan.pdf
https://www.vldb.org/pvldb/vldb2010/pvldb_vol3/I07.pdf
http://www.vldb.org/pvldb/vol12/p2143-antonopoulos.pdf
https://azure.microsoft.com/en-us/blog/microsoft-azure-data-welcomes-attendees-to-vldb-2018/
http://www.vldb.org/pvldb/vol9/p1245-lang.pdf
http://www.vldb.org/pvldb/vol8/p401-chandramouli.pdf
http://www.vldb.org/pvldb/vol6/p1178-lomet.pdf
http://www.cs.ucf.edu/~kienhua/classes/COP5711/Papers/MSazure2017.pdf
https://www.microsoft.com/en-us/research/uploads/prod/2019/05/socrates.pdf
https://www.microsoft.com/en-us/research/uploads/prod/2019/05/QuickInsights-camera-ready-ﬁnal.pdf
http://pages.cs.wisc.edu/~remzi/Classes/739/Spring2004/Papers/p215-dageville-snowﬂake.pdf
http://www.vldb.org/pvldb/vol11/p1317-mahajan.pdf
http://www.vldb.org/pvldb/vol12/p681-zou.pdf

130
http://www.benstopford.com/2012/10/28/the-best-of-vldb-2012/
http://pages.cs.wisc.edu/~jignesh/publ/Quickstep.pdf
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.640.7744&rep=rep1&type=pdf
https://www.betterevaluation.org/sites/default/ﬁles/data_cleaning.pdf
http://www.vldb.org/pvldb/vol6/p1009-aji.pdf
https://www.scitepress.org/papers/2017/65871/65871.pdf
https://db.in.tum.de/~kipf/papers/fastdata.pdf
https://people.csail.mit.edu/matei/papers/2015/sigmod_spark_sql.pdf
https://cs.stanford.edu/~matei/papers/2013/sigmod_shark.pdf
https://hal.archives-ouvertes.fr/tel-02059437v2/document
http://www.cs.cornell.edu/~ragarwal/pubs/zipg.pdf
http://dbgroup.cs.tsinghua.edu.cn/ligl/papers/sigmod19-tutorial-paper.pdf
https://homes.cs.washington.edu/~magda/papers/wang-cidr17.pdf

131
https://www.comp.nus.edu.sg/~memepic/pdfs/mem-survey.pdf
http://www.vldb.org/pvldb/vol11/p1414-agrawal.pdf
http://da.qcri.org/rheem/pdf/paper/rheemtutorial.pdf
http://www.ijsrp.org/research-paper-0613/ijsrp-p18143.pdf
https://www.slideshare.net/YunyaoLi/enterprise-search-in-the-big-data-era-recent-developments-and-open-challenges-38801004
http://cidrdb.org/cidr2017/papers/p123-barber-cidr17.pdf
https://adalabucsd.github.io/papers/Slides_2017_Tutorial_SIGMOD.pdf
http://static.druid.io/docs/druid.pdf
http://www.vldb.org/pvldb/vol11/p1822-cai.pdf
http://cs.brown.edu/~xhu3/paper/diffprivacy.pdf
https://e.huawei.com/us/solutions/cloud-computing/big-data
http://www.vldb.org/pvldb/vol7/p1754-huawei.pdf

132
http://conferences.cis.umac.mo/icde2019/wp-content/uploads/2019/06/icde-2019-keynote-jianjun-chen.pdf
https://astro.temple.edu/~tua86772/zhang14SIGMOD_DEMO.pdf
https://users.wpi.edu/~yli15/Includes/SIGMOD15Telco.pdf
https://15721.courses.cs.cmu.edu/spring2016/papers/p337-soliman.pdf
https://aws.amazon.com/big-data/datalakes-and-analytics/
http://www.vldb.org/pvldb/vol11/p1781-schelter.pdf
https://www.allthingsdistributed.com/ﬁles/p1041-verbitski.pdf
https://lovvge.github.io/Forecasting-Tutorial-VLDB-2018/
http://lunadong.com/
https://databricks.com/wp-content/uploads/2018/12/adam-sigmod-2015.pdf

End of tutorial -- Thank you
133

How to build and run a big data platform in the 21st century

More Related Content

What's hot (20)

Similar to How to build and run a big data platform in the 21st century (20)

More from Ali Dasdan (8)

Recently uploaded (20)

How to build and run a big data platform in the 21st century