Real-time Distributed Stream Processing@ Scale

Take Control of Your Data: CaliStream.com © CaliStream.com 2016
Meetup
Real-time
Distributed Stream Processing
@ Scale
Mountain-View Meetup (March 2016)
Jerome Boulon

Meetup
CaliStream.com
Take Control of Your Data
Jerome Boulon
CEO/Founder
jboulon@caliStream.com https://www.linkedin.com/in/jboulon

Meetup
Quick History
1999 2008 2009 2010 2012
Yahoo!: Chukwa
Hadoop Monitoring Solution
Netflix: Honu
Data Collection Pipeline
CaliStream: Founder
Honu: Data As a Service
Monitoring Solution
for cable modems/TV
network
Ontology/Semantic Search
Acquired by Microsoft

Meetup
CaliStream.com
TO ENABLE ANY COMPANY
TO QUICKLY LEVERAGE BIG DATA
AS A STRATEGIC ADVANTAGE

Meetup
Big Data
• LIMITED TALENT WITH HADOOP KNOWLEDGE
• LIMITED TALENT TO BUILD SCALABLE SYSTEMS
• A TORRENT OF DATA
… BUT A STRATEGIC ADVANTAGE
THAT YOU CANNOT IGNORE

Meetup
CONSULTING CALISTREAM SAAS
• Architecture Design & Review
• Big Data Projects
• Distributed & Large Scale
Projects
• Research Projects
Schema-less data processing pipeline
to easily stream large volume of events
from your applications directly
to Hive/Hadoop

Meetup
Agenda
• CaliStream Data Pipeline
• Stream Analysis:
– Samza
– Live Monitoring
– Network Analysis
– Samza New Features
• Q & A

Meetup
Data Pipeline,
Some Challenges …

Meetup
Challenges …
• Lack of talents
• Event format
• Schema evolution
• Time to market
• Rapidly changing infrastructure
• Collecting massive amount of data live
• Batch & Real-time Analysis
• Load balancing, Auto-scaling, Discovery, etc
• Being up 24/7 (no downtime!)
• Cost
• …

Meetup
Traditional Pipeline Vs. Ease of Use
• Schema Based
• Uncompressed
• Dedicated team
• Global schema
• Synchronization
• Upgrade/Downtime
TRADITIONAL PIPELINE EASE OF USE
• Schema-Less
• Hadoop Bin. Compressed
• Self-Service
• No Synchronization
• No Upgrade/Downtime
• Continuous integration
• Continuous delivery

Meetup
Our Solution: CaliStream
CaliStream provides
a SaaS data processing pipeline
to easily stream large volume of events
from your applications directly to Hive/Hadoop
in a robust, scalable and cost effective way
without any prior Hadoop Knowledge

Meetup
CaliStream: Native integration with Hive
Big
Data
Sensor
Data
Social
Click
Stream
Location
logs
Sensor
Data
Click
Stream
Location logsSocial
… …
CaliStream

Meetup
Real-time Analysis
- Log Search
- User Activity
- Click Stream
- Customer Support
- …
Stream Processing
Events Routing
Data augmentation
Data Security
…
BI Analytics
Data Science
- Sensors Data
- Telemetry Data
- Users behavior
- Location
- Social
- …

Meetup
CaliStream
Everything in CaliStream
is represented as a Hive table
so you can easily analyze your data
Select […]
from […]
where […]
group by […] ;
• Managed Service
• Self-Service SDK
• Schema-Less API
• SQL Compliant
• Scalable

Meetup
CaliStream: Data Lake
CaliStream
Rest API
JS API
Java API
AppsClusters
CaliStream
RT
…
DataCollectionAnalysis

Meetup
Ease of Use
1. Start
+ Online Signup
+ Download from CaliStream.com
2. Prototype
Implement few lines of code
3. Deploy and collect data
Less than a week deployment cycle
1. Start
Bring/train talent in-house
and hire consultants
2. Prototype/Iterate
Understand Hadoop
Ecosystem
Deploy/Fix cycles
3. Deploy and collect data
14-24 months deployment cycle
VS
CaliStreamOthers

Meetup
Ease of Use (Java, REST, MQTT, …)
CaliStream
Hive Table

Meetup
Streams Analysis

Meetup
Stream Analysis
• Gaming:
– Real-time Deployment Monitoring & Alerting
– Billion of events to analyze
Number of Downloads/Installs/Errors
GroupBy Version/Region/Country
Over 1, 5, 10 Minutes window

Meetup
Apache Samza
• App logic decoupled from Msg
Transport
• Native Kafka Integration
• Scalable & Fault Tolerant
• Automatic State Management
• Etc.
• Top Level Apache Project
http://samza.apache.org/

Meetup
Architecture
Kibana
• Data augmentation
(IP -> Region, Country, ISP, etc.)
• Native Operators
• GroupBy
• State Management
• Windowed aggregation

Meetup
Stream Analysis
• Network Security:
– Real-time DDOS and Breaches Analysis
• Monitor, Detect, Mitigate
– Billion of events to analyze
• Players behaviors
• Network events
• Audit Trail
• Etc.

Meetup
Samza
• Native Stream-stream join: Join Players,
Network and Audit trail activity Streams
• Native RocksDB Integration
(Key/Value local Storage)
• Native Windowed aggregation :
Aggregate over 1, 5, 10 minutes window
• Native Log Compaction & Broadcast
Stream

Meetup
Apache Samza (New Features)
• Coordinator Stream (Dyn. config, etc.)
• Broadcast Stream (Update/Change Behavior)
• Host affinity & State reuse (Faster re-start)
• ElasticSearch Producer (Write output to ES)
• Samza @ scale on EC2 (Netflix)

Meetup
CaliStream.com
Jerome Boulon
CEO/Founder
jboulon@caliStream.com

Real-time Distributed Stream Processing@ Scale

More Related Content

What's hot (20)

Similar to Real-time Distributed Stream Processing@ Scale (20)

Recently uploaded (20)