Spark Summit EU talk by Michael Nitschinger
0 likes594 views
This document discusses using Apache Spark and Couchbase together. It provides an overview of use cases for combining the two technologies, such as operationalizing analytics and machine learning models, data integration, and recommendations. It then covers various access patterns for moving data between Spark and Couchbase, including key-value access, queries, views, streaming, and full text search. Finally, it discusses the Couchbase Spark connector and resources for using it.
Spark Summit EU talk by Michael Nitschinger
- 1. SPARK SUMMIT EUROPE2016 SPARK AND COUCHBASE AUGMENTINGTHE OPERATIONAL DATABASE WITH SPARK Michael Nitschinger Couchbase
- 3. WHY SPARK AND COUCHBASE Overview & Use-Cases
- 4. Use Cases Operations Analytics CB § Recommendations § Predictive analytics § Fraud detection § Catalog § Personalization § Mobile applications
- 5. Use Case: Operationalize Analytics / ML Hadoop ML Model Data Warehouse Training Data CB Model Online Data Serving Predictions
- 6. Adapted from: Databricks – Not Your Father’s Database https://www.brighttalk.com/webcast/12891/196891
- 7. Use Case: Data Integration RDBMSS3HDFS ES NoSQL
- 10. ACCESS PATTERNS From Spark to Couchbase and Back Again
- 12. Key-Value Fetch/Store by Document ID N1QL Query Fetch by Criteria “SQL”
- 13. Key-Value Fetch/Store by Document ID N1QL Query Fetch by Criteria “SQL” Map-Reduce Views Materialized Indexes (Aggregation)
- 14. Key-Value Fetch/Store by Document ID N1QL Query Fetch by Criteria “SQL” Map-Reduce Views Materialized Indexes (Aggregation) Streaming Mutation Streams For Processing
- 15. Key-Value Fetch/Store by Document ID N1QL Query Fetch by Criteria “SQL” Map-Reduce Views Materialized Indexes (Aggregation) Streaming Mutation Streams For Processing Full Text Search on Freeform Text
- 16. Key-Value Fetch/Store by Document ID N1QL Query Fetch by Criteria “SQL” Map-Reduce Views Materialized Indexes (Aggregation) Streaming Mutation Streams For Processing
- 18. Data Locality • RDD Location Hints based on the Cluster Map • Not available for N1QL or Views – Round robin - can’t give location hints – Back end is scatter gather with 1 node responding
- 19. N1QL Query • N1QL is a SQL service with JSON extensions • Uses Couchbase’s Global Secondary Indexes • Can run on any nodes within the cluster • Nodes with differing services can be added and removed as needed on the fly
- 20. Data Service Projector & Router Couchbase Query Architecture Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 Query Processor cbq-engine Bucket#1 Bucket#2 DCP Stream Index#4Index#3 ... B u c k e t # 2 B u c k e t # 1 ForestDB Storage Engine
- 21. Spark SQL Sources TableScan Scan all of the data and return it PrunedScan Scan an index that matches only relevant data to the query at hand. PrunedFilteredScan Scan an index that matches only relevant data to the query at hand.
- 23. Schema Inference
- 24. Schema Inference N1QLRelation:28 - Inferring schema from bucket travel- sample with query 'SELECT META(`travel-sample`).id as `META_ID`, `travel-sample`.* FROM `travel-sample` WHERE `type` = 'airline' LIMIT 1000' N1QLRelation:28 - Executing generated query: 'SELECT `name`,`callsign` FROM `travel-sample` WHERE `type` = 'airline''
- 25. Schema Inference
- 26. DCP and Spark Streaming Replica Indexing …
- 27. Structured Streaming Source 27Adapted from https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html DCP Stream Unbounded Table
- 31. Couchbase Spark Connector 1.2.1 • Spark 1.6.x support, including Datasets • DCP Flow Control • Enhanced Java APIs 31
- 32. Couchbase Spark Connector 2.0.0 • Spark 2.0.x Support • Enhanced DCP Client • Experimental Structured Streaming 32
- 33. Resources • Spark Packages https://spark-packages.org/package/couchbase/couchbase-spark- connector • Docs http://docs.couchbase.com • Source https://github.com/couchbase/couchbase-spark-connector • Bugs https://issues.couchbase.com/browse/SPARKC 33
- 34. SPARK SUMMIT EUROPE2016 THANK YOU. Michael Nitschinger @daschl michael.nitschinger@couchbase.com