Check Point Big Data Forum m3

1©2017 Check Point Software Technologies Ltd.
BIG DATA FORUM
ALEXANDER FOK, BIG DATA
ARCHITECT
MARCH 2017 MEETING

• Big Data Forum Scope
• Big Data Projects in CP
• No SQL Overview
Agenda

Big Data Forum Scope
• Engineers talk about Engineering
• Bring ideas, dilemmas, problems, technologies, reference architectures
• Ask QUESTIONS
• What is Big Data Problem?

Simple Exercise
• What is more expensive
• Network IO
• Disk IO
• RAM Memory IO
• CPU Cache IO
• CPU Context switch
• Google interview question

How long would it take to copy 1TB file?
• USB 2 ~480 Mbps
• Network 1 Gbps
• SATA drive 6 Gbps
• SAS 12 Gbps
• SSD PCIe 10 Gbps
• Disk to disk
1TB/500Mbps = ~5h
• Over LAN 1TB/250Mbps
= ~10h
• Over WAN
1TB/100Mbps = ~25h

Some World Numbers
• NYSE – 1TB/day
• Airbus A380 640TB/flight (10GB/s)
• Verizon – 1M Events/s, 20 B/day

Big Data Forum for You
• What is Big Data Forum for you?
• What is Big Data for you?
• What do you expect from this forum?
• What are you willing to contribute to this forum?

Some CP Projects
• Threat Cloud infrastructure – distributed threats DBs handles hundreds
of millions of threat indicators (IPs, files, domains, etc)
• Threat Cloud DataLake
• MTP infrastructure – AWS based applications store
• GWs TI Access Logs (AV, URL filtering, AB, etc) – 300 GB/day, 20K
events/s
• CPDiag 300GB/day diagnostic data from CPDiag enabled GWs

Rule of Thumb
• If you can solve the problem with more RAM – do it
• If you can solve the problem with more CPUs (threads) – do it
• Else – you have to distribute the solution

Machine Learning vs Big Data Problems
Big Data
Machine
Learning

• Volume – a lot of data to collect and make accessible
• Velocity – processed quickly at a rapid pace (correlations, enrichments,
etc)
• Variety – no predefined data schema (veracity)
Big Data Three V’s Challenges

• Sizing
• Duplication prevention
• Correlation
• Data Integrity, Consistency
• Visualization
• Integration with other systems
• Retention Policy, Distribution, Regulations, Security
• Storage and Backup
Big Data Three V’s Challenges in Real Life
Main Challenge - How can we
handle x10, 100, 1000 times more
load?

• Scale out architecture
• Cheap storage
• Reasonable computation times for various data exploration scenarios:
̶ Key-value lookup
̶ Documents search
̶ Generic filtering and aggregation analysis - batch processing
̶ Interactive queries
• Stream Processing
̶ (Near) Real Time Complex Events Processing
NoSQL Selected Problems

NoSQL Trend Overview
• Database providing mechanism for storage and retrieval of data other than
tabular relations used by RDBMS
• Data collection, visualization, access management – not always are part of
the solution ecosystem
• What now – everybody is looking for SQL over noSQL, or NewSQL
• Compromised Consistency (CAP Theorem)
• How many NoSQL DBs are there?

CAP (Brewer's)
Theorem

NoSQL – Where To?
• We have a lot of data
• What next?
• Emerging Computational Models

NoSQL Types
• Key-Value stores
• Document stores
• Search Engines
• Column stores
• Graph DBs
• Time Series
• RDF, Object Oriented, Multivalue
• Cloud Provided Solutions

Key-Value Stores
• map concept – key->value
̶ Value can be any object (blob)
• Cache – in memory
• Store – have solid persistency model
• Additional query mechanisms (secondary indexes, time series)
• Abstract Data Types (data structure and algorithms on them) – sets,
hashes, lists, queues, etc
• Distribution
• Aerospike, Couchbase, Dynamo, Redis, Riak, Hbase, Cassandra

Document stores
• Semi structured data store
• key-value retrieval and APIs based on document properties
• Collections, tags, metadata, etc
• Search engines
• Couchbase (CouchDB + Memcached), MongoDB, , Riak, ElasticSearch

Search Engines
• Limited Document Stores with poor raw data storage capabilities
• Strong indexing mechanisms at ingestion time
• Complex query capabilities
• ElasticSearch, SolR, Splunk

Column stores
• Good for append only scenarios
• Good for batch data insertion
• Good for analytics queries requiring massive data partial reads
• Bad for updates
• Bad for specific objects searches
• Bad for real time analytics
• Vertica, GreenPlum, Cassandra, File systems based solutions
• OLAP usage, except Cassandra

Column stores

Graph DBs
• Graph traversal queries
̶ Social Networks Recommendation engines
• Data Modeling - not about implementation mechanism
• Scale problems
• Query Standardization issues – Cypher, SPARQL, XQuery, Gremlin
• Neo4j, TITAN
• Modern approach – computational engine oriented solutions – Spark
Graph, ELK Graph

Time Series
• Used for meters performance counters analysis – e.g. monitoring systems
(CPU, RAM, etc)
• Riak TS, TSDB,
• https://prometheus.io/
• http://influxdb.com/
• http://opentsdb.net/
• https://github.com/kairosdb/kairosdb

Cloud Provided Solutions
• Azure Tables
• Amazon DynamoDB
• Google BigTable, Spanner

Tip of The Day
• Apache is cemetery full of dead bodies
̶ No one size fits all solutiontechnologyarchitecture
̶ COOL is not always RIGHT
• http://nosql-database.org/
• http://db-engines.com

• Use of low language query languages (C++, Java, Python) and tools
(Kibana, Jupyter Notebooks)
• Lack of standardization
• Integrative challenge
• No all size fits all solutions – REAL NEED to KNOW and UNDERSTAND the
underlying technology
NoSQL Adaptation Barriers

Yuval Noah Harari on Big Data, Google and the end of
free will
http://www.ft.com/cms/s/2/50bb4830-6a4c-11e6-ae5b-a7cc5dd5a28c.html

“Listen, Google,”
• “Well, I know you from the day you were born. I have read all your emails, recorded all your phone calls, and
know your favourite films, your DNA and the entire biometric history of your heart. I have exact data about each
date you went on, and I can show you second-by-second graphs of your heart rate, blood pressure and sugar
levels whenever you went on a date with John or Paul. And, naturally enough, I know them as well as I know you.
Based on all this information, on my superb algorithms and on decades’ worth of statistics about millions of
relationships — I advise you to go with John, with an 87 per cent probability of being more satisfied with him in
the long run.
• “Indeed, I know you so well that I even know you don’t like this answer. Paul is much more handsome than John
and, because you give external appearances too much weight, you secretly wanted me to say ‘Paul’. Looks
matter, of course, but not as much as you think. Your biochemical algorithms — which evolved tens of thousands
of years ago in the African savannah — give external beauty a weight of 35 per cent in their overall rating of
potential mates. My algorithms — which are based on the most up-to-date studies and statistics — say that looks
have only a 14 per cent impact on the long-term success of romantic relationships. So, even though I took Paul’s
beauty into account, I still tell you that you would be better off with John.”

Alexander Fok, Big Data Architect
THANK YOU

Check Point Big Data Forum m3

More Related Content

What's hot (20)

Similar to Check Point Big Data Forum m3 (20)

Recently uploaded (20)

Check Point Big Data Forum m3

Editor's Notes