The Hive Think Tank: "Stream Processing Systems" by M.C. Srivas of MapR
Download as PPTX, PDF0 likes765 views
The document discusses the MapR converged data platform, highlighting its components like Apache Hadoop, Spark, and NoSQL technologies for big data processing. It emphasizes features such as unlimited scale, global namespace, and unified security, catering to real-time data analytics and micro-batching. Additionally, it details MapR Streams for reliable message delivery across geo-dispersed clusters, enabling advanced data integration and operational intelligence.
![© 2014 MapR Technologies
Records
JSON messages allow SQL queries by Apache Drill
Pub1
{
“event” : “CLICK”,
“url” : “/about/”,
“user” : “Will”,
“ip” : “192.168.1.1”,
“loggedin” : true
}
/stream/web:events
[
{“seq001”: { … } },
{“seq002”: { … } },
…
]
SELECT user, ip, url FROM marlin.`/stream/web:events` WHERE event=CLICK
user ip url
--------------------------------------------------------------------------------------------
Will 192.168.1.1 /about/
Neeraja 192.168.1.2 /logout/
Mitesh 192.168.1.3 /prod01/buy](https://image.slidesharecdn.com/mapr-thehive-2016-1-20-v2-2-160121204826/85/The-Hive-Think-Tank-Stream-Processing-Systems-by-M-C-Srivas-of-MapR-11-320.jpg)
The Hive Think Tank: "Stream Processing Systems" by M.C. Srivas of MapR
- 1. © 2014 MapR Technologies 1© 2014 MapR Technologies
- 2. © 2014 MapR Technologies 2 American express American express MapR at the Core of Big Data 700+ CustomersCloud Leadership American Express Wells Fargo UnitedHealth Group
- 3. © 2015 MapR Technologies 3 The Growing Need For Platform Thinking As it happens Near real time Micro batch Batch Developer Production Aspirational Goal Batch Micro batch New System of Record AND Operational Intelligence Operational analytics Scale Reliability Trust Latency Multi-tenancy Streams
- 4. © 2015 MapR Technologies 4 Their Strategy: Customer Do-It-Yourself APACHE HADOOP Security YARN Spark Streaming Storm StreamingNoSQL & Search Juju Provisioning & coordination Savanna h ML, Graph Mahout MLLib GraphX EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS Workflow & Data Governanc e Pig Cascadin g Spark Batch MapRedu ce v1 & v2 Tez HBase Solr Hive Impala Spark SQL Drill SQL Sentry Oozie ZooKeepe r Sqoop Flume Data Integration & Access HttpFS Hue Search cluster HA duplicate Kafka cluster NAS storage NoSQL cluster Not pre-integrated More expensive Latencies everywhere Unreliability grows
- 5. © 2015 MapR Technologies 5 BIG STORAGE What Makes A Big Data Platform? Hadoop SQL Search Messaging BIG PROCESSING NoSQL
- 6. © 2015 MapR Technologies 6 MapR CONVERGED DATA PLATFORM APACHE HADOOP Security YARN Spark Streaming Storm StreamingNoSQL & Search Provisioning & coordination Sahara ML, Graph Mahout MLLib GraphX EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS Workflow & Data Governance Pig Spark Batch MapReduce v1 & v2 HBase Solr Hive Impala Spark SQL Drill SQL Sentry Oozie ZooKeeperSqoop Flume Data Integration & Access HttpFS Hue HDFS APIs MapR STORAGE PLATFORM Unlimited Scale Multi Data-center Global Namespace Data Protection Unified Security
- 7. © 2015 MapR Technologies 7 MapR CONVERGED DATA PLATFORM MapR- DB JSON Search RealTime Streams Apache Drill : ANSI SQL as the Unified Access Layer File System POSIX APACHE HADOOP Security YARN Spark Streaming Storm StreamingNoSQL & Search Provisioning & coordination Sahara ML, Graph Mahout MLLib GraphX EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS Workflow & Data Governance Pig Spark Batch MapReduce v1 & v2 HBase Solr Hive Impala Spark SQL Drill SQL Sentry Oozie ZooKeeperSqoop Flume Data Integration & Access HttpFS Hue HDFS APIs MapR STORAGE PLATFORM Unlimited Scale Multi Data-center Global Namespace Data Protection Unified Security
- 8. © 2015 MapR Technologies 8 MapRCONVERGEDDATAPLATFORM MapR DB Search MapR Streams APACHE HADOOP AND OSS ECOSYSTEM Security YARN Spark Streaming Storm StreamingNoSQL & Search Provisioning & coordination Sahara ML, Graph Mahout MLLib GraphX EXECUTION ENGINES DATA GOVERNANCE AND OPERATIONS Workflow & Data Governance Pig Spark Batch MapReduce v1 & v2 HBase Solr Hive Impala Spark SQL Drill SQL Sentry Oozie ZooKeeperSqoop Flume Data Integration & Access HttpFS Hue File System POSIX Apache Drill: Unified Access Layer MapR STORAGE PLATFORM Unlimited Scale Multi Data-center Global Namespace Data Protection Unified Security DATABASES&APPLICATIONS EMail HPC Document Mgmt Custom Apps Custom Apps
- 9. © 2015 MapR Technologies 9 MapR Converged Data Platform Open Source Engines & Tools Commercial Engines & Applications Utility-Grade Platform Services DataProcessing Enterprise Storage MapR-FS MapR-DB MapR Streams Database Event Streaming Global Namespace High Availability Data Protection Self-healing Unified Security Real-time Multi-tenancy Search & Others Cloud & Managed Services Custom Apps UnifiedManagementandMonitoring
- 10. © 2015 MapR Technologies 10 MapR Streams – Publish/Subscribe Listeners Publish billions of messages per sec to a topic in a stream. Reliable delivery to all consumers. Immediately. JSON data format, direct data access from analytics frameworks L Simple, standard API (Kafka). Topi c Stream Tie together geo-dispersed clusters. Worldwide. Topic Producers Consumers
- 11. © 2014 MapR Technologies Records JSON messages allow SQL queries by Apache Drill Pub1 { “event” : “CLICK”, “url” : “/about/”, “user” : “Will”, “ip” : “192.168.1.1”, “loggedin” : true } /stream/web:events [ {“seq001”: { … } }, {“seq002”: { … } }, … ] SELECT user, ip, url FROM marlin.`/stream/web:events` WHERE event=CLICK user ip url -------------------------------------------------------------------------------------------- Will 192.168.1.1 /about/ Neeraja 192.168.1.2 /logout/ Mitesh 192.168.1.3 /prod01/buy
- 12. © 2015 MapR Technologies MapR Streams: Global Messaging Fabric ...Edge Aggregation ... Sensors Apps Analytics ● Arbitrary interconnection of thousands of clusters ● Seamless disconnection/reconnection ● Globally synchronized sequence numbers & cursors: Listener & Producer failover ● Global applications ● Data filtering/aggregation at the edge ● Distributed analytics NYC LONDON CHI TOK