Couchbase Data Pipeline

Editor's Notes

• #5: Couchbase is an open source NoSQL Data Management platform. It specializes in operational data management to run the full range of online scenarios – web, mobile, and Internet of Things It is open source, and available in community and enterprise editions. It is built upon an integrated object-managed cache, which is both memory centric and distributed. That’s the core. All three of these together pretty well cover the web, mobile, and IOT use cases Couchbase was designed for. KEY POINT: COUCHBASE PROVIDES A SET OF MULTI-PURPOSE, CORE CAPABILITIES THAT SUPPORT A BROAD RANGE OF APPLICATIONS AND USE CASES, ALL IN A SINGLE DATA MANAGEMENT PLATFORM. Couchbase provides a set of technology capabilities to support a broad range of applications and use cases: High Availability Cache: Couchbase provides an integrated managed object cache, so you can start out using Couchbase as a high availability cache on top of your existing relational database. For example, you can use Couchbase as a session store in front of your relational database, if your relational DB is struggling to keep up with the load required for online interactive applications. Key-Value Store: Many customers start with Couchbase as a cache and then broaden their usage to other capabilities, like using Couchbase as a Key-Value Store for things like Profile Management. Document Database: From there, you can grow into using Couchbase as a Document Database, where you can do more with capabilities like indexing and Cross Data Center Replication. Embedded Database: Couchbase also provides an embedded database called Couchbase Lite. It’s a purpose-built database for the device, so you can build applications that are always available and always work, whether offline or online. Sync Management: Finally, as part of our solution for mobile applications, we provide Couchbase Sync Gateway, which automatically synchronizes data on the device with Couchbase Server in the cloud so your developer doesn’t have to write code to manage the complex sync process. Starting with cache and then expanding to other capabilities is often a good way to learn the technology and get comfortable with Couchbase for a wider set of use cases. Couchbase Open Source Projects (Apache 2.0 Public License) Community Editions Enterprise Editions NoSQL Database (Couchbase Server) Document Database + Key / Value Store + Distributed Cache Cross Data Center Replication Mobile Database (Couchbase Lite) iOS, Android, Java, and .NET Mobile Synchronization (Couchbase Sync Gateway)
• #6: Couchbase Server is a NoSQL document database for interactive web applications. It has a flexible data model, is easily scalable, provides consistent high performance and is “always-on,” meaning it can serve application data 24 hours, 7 days a week. KEY POINTS: COUCHBASE DELIVERS ALL THE CAPABILITIES NEEDED TO MEET TODAY’S REQUIREMENTS FOR PERFORMANCE, SCALABILITY, AVAILABILITY, AND DATA MODEL FLEXIBILITY. THESE TRANSLATE INTO MAJOR BENEFITS FOR YOUR BUSINESS. Couchbase was purpose-built to solve today’s requirements for enterprise-class, mission-critical, web and mobile applications. Specifically, Couchbase delivers the following capabilities: Fast performance at scale -- submillisecond latency to enable highly responsive applications, for millions or even hundreds of millions of users. Easy, affordable scalability – Couchbase is a distributed database that scales out on commodity hardware with push button simplicity. We make it very easy to add or remove capacity on demand with no system downtime. On premises, in the cloud, wherever you want. High availability – Couchbase automatically replicates your data across your servers, clusters, and data centers, so it’s always available, 24x7. And Couchbase doesn’t require any downtime to maintain. Flexible data model – Couchbase gives you complete flexibility to handle any kind of data, and to change your data model on the fly to accommodate new data attributes or new data types. It’s the kind of flexibility that developers love, because it gets rid of the rigid schemas that slow them down. So developers can build applications faster and easier. All this adds up to powerful benefits for your enterprise: Faster development & time to market Better business agility Improved customer experience Increased loyalty and revenue Lower IT costs and increased efficiency
• #8: Not an either or decision… I’m going to argue why
• #9: Latency - Everyone says real time, but what do mean? For an operational system, this means: Extremely fast (in-memory) reads Extremely fast (log append) writes For Couchbase, complete millions of ops / second (these are gets / sets) at latencies of under 1ms, compare LinkedIn figures from Jerry Franz’s session Tuned to LinkedIn’s specific workload: 75% writes (sets + incr) / 25% reads – 13 byte values, 25 byte keys on average 2.5 billion items (+ 1 replica) 600 Gbytes of RAM /  3 Tbytes of disk in use on average Average store latency ~ 0.4 milliseconds 99th percentile store latency ~ 2.5 milliseconds Average get latency ~ 0.8 milliseconds 99th percentile get latency ~ 8 milliseconds
• #10: Users and consumers of information increasingly demand an always on, low latency access to their data. As well as providing a framework for businesses to understand what’s happening in real time while addressing Polyglot Persistence in managing data. The conceptual framework Lambda Architecture evolved out of Twitter and coined by Nathan Marz for a generic data processing architecture. In a way the architecture is an extended event sourced system but aims to accommodate streaming data at large scale. 1. All data entering the system is dispatched to both the batch layer and the speed layer for processing. 2. The batch layer has two functions: (i) managing the master dataset (an immutable, append-only set of raw data), and (ii) to pre-compute the batch views. 3. The serving layer indexes the batch views so that they can be queried in low-latency, ad-hoc way. 4. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only. 5. Any incoming query can be answered by merging results from batch views and real-time views.
• #11: - Improve the animation for data handling – refresh existing sales deck and deep dives.
• #12: Using Couchbase as the high performance, low latency, scalable data store to support personalized interactions Couchbase, as real-time operational database may be generating real-time data to feed into both the batch and real-time layers In some use cases, Couchbase is used to perform real-time processing and analytics – M/R views Some customers are using Couchbase as the data store to for stream processing Email from Michael on streaming data from CB to Spark via Kafka: well, Kafka is one way but it also uses DCP under the covers. I actually need to make some changes to the DCP implementation in the java client and my plan is to have DCP support in dp2 (a month later or so). So once we are GA, there will be a 100% way to stream data directly into a DStream (spark streaming). And of course you can easily implement simple polling of let’s say a view, and grab the full docs that match for example a time interval. 
• #13: Couchbase Server is engineered with a unique, memory-centric architecture for increased scalability and performance. As data is written to an in-memory cache for low latency read/write access, it’s persisted and replicated for durability and availability. Memory-centric architecture removes unnecessary disk IO from read and write paths. For example, updating the index in-memory based on writes to the cache when it’s queried. Database Change Protocol (DCP) streams writes from the in-memory cache to in-memory queues within the indexing and replication components to increase performance. The same protocol we use to replicate changes between remote clusters is now being used for replicated mutations outside of Couchbase Database Change Protocol Since Couchbase Server 3.x internal de-facto standard to handle changes within a Bucket Clients: Intra-Cluster Replication, Indexing, XDCR Mutation Event that’s raised in case of a creation, update or delete Each mutation that occurs in a vBucket has a sequence number Important: Not yet officially exposed, but used to implement Connectors those are provided by Couchbase
• #15: The data generated by users is published to Apache Kafka. Next, it’s pulled into Apache Storm for real time analysis and processing as well as into Hadoop. Finally, Storm writes the data to Couchbase Server for real-time access by LivePerson agents while the data in Hadoop is eventually accessed via HP Vertica and MicroStrategy for offline business intelligence and analysis.
• #17: Each Couchbase node is exactly the same. All nodes are broken down into two components: A data manager (on the left) and a cluster manager (on the right). It’s important to realize that these are separate processes within the system specifically designed so that a node can continue serving its data even in the face of cluster problems like network disruption. The data manager is written in C and C++ and is responsible both for the object caching layer, persistence layer and querying engine. It is based off of memcached and so provides a number of benefits; -The very low lock contention of memcached allows for extremely high throughput and low latencies both to a small set of documents (or just one) as well as across millions of documents -Being compatible with the memcached protocol means we are not only a drop-in replacement, but inherit support for automatic item expiration (TTL), atomic incrementer. -We’ve increased the maximum object size to 20mb, but still recommend keeping them much smaller -Support for both binary objects as well as natively supporting JSON documents -All of the metadata for the documents and their keys is kept in RAM at all times. While this does add a bit of overhead per item, it also allows for extremely fast “miss” speeds which are critical to the operation of some applications….we don’t have to scan a disk to know when we don’t have some data. The cluster manager is based on Erlang/OTP which was developed by Ericsson to deal with managing hundreds or even thousands of distributed telco switches. This component is responsible for configuration, administration, process monitoring, statistics gathering and the UI and REST interface. Note that there is no data manipulation done through this interface.
• #20: Now, as you fill up memory (click), some data that has already been written to disk will be ejected from RAM to make room for new data. (click) Couchbase supports holding much more data than you have RAM available. It’s important to size the RAM capacity appropriately for your working set: the portion of data your application is working with at any given point in time and needs very low latency, high throughput access to. In some applications this is the entire data set, in others it is much smaller. As RAM fills up, we use a “not recently used” algorithm to determine the best data to be ejected from cache.
• #21: Should a read now come in for one of those documents that has been ejected (click), it is copied back from disk into RAM and sent back to the application. The document then remains in RAM as long as there is space and it is being accessed.
• #29: The application makes a call for a key called NYC MQ1 We run the key through the crc 32 function and the result of that hash function is that it points to vbucket3 Which in turn points to couchbase server number 1
• #30: We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
• #31: We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
• #37: Build-in cache with persistence. Performance & Scalability Always on – constant uptime even during maintenance cycles Solution for Cookie had to be one that is transparent to existing Paypal applications. A solution that is highly available , low latency service. Looking at the Data Volume -> more than a billion documents, each document approx. 10K size, 10TB of data Always available with multi data center replication. Availability requirements of 4-9’s.
• #40: Cookieservice connects to local Couchbase cluster. Upon write failure it writes to seconday cluster. We read our own writes for immediate consistency. Uni-directional XDCR to backup cluster for our DR needs.