Under the Hood - Couchbase Server Architecture - June 2015
Download as PPTX, PDF1 like518 views
The document provides a comprehensive overview of Couchbase Server 4.0 architecture, including its major components such as connectivity, replication, cluster management, and various services like data, index, and query services. It highlights the functionality and structure of the cluster, detailing the cluster manager's role in coordinating operations and managing node interactions. Additionally, it discusses data service operations, indexing capabilities, and the N1QL query execution process, emphasizing the system's performance and scalability.
Under the Hood - Couchbase Server Architecture - June 2015
- 1. UNDER THE HOOD COUCHBASE SERVER 4.0 ARCHITECTURE Cihan Biyikoglu Director of Product Management
- 2. ©2015 Couchbase Inc. 2 Agenda Goals Give you a full tour of the mansion! Zoom into major components and services in Couchbase Server 4.0 Impress your date…
- 3. ©2015 Couchbase Inc. 3 Agenda Overview Cluster Architecture – 10k ft view Deep Dive Connectivity Architecture with Client SDKs ReplicationArchitecture with DCP Cluster Management Services Data Service, Index Service, Query Service Recap Q&A
- 4. ©2015 Couchbase Inc. 4 Disclaimer Couchbase Server 4.0 and ForestDB are still in development and the final version of the products may not be identical in details discussed on this session.
- 5. Overview
- 6. ©2015 Couchbase Inc. 6 Couchbase Server 4.0 - Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage
- 7. ©2014 Couchbase Inc. Couchbase Server 4.0 - Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage
- 8. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 9. ©2015 Couchbase Inc. 9 Connectivity and Client Libraries Type Port Endpoint REST 8091, 18091 Admin Connections Pointed at any node in the cluster REST 8091, 18092 Query withView Load balanced across node of the cluster that runs data service REST 8093, 18093 Query with N1QL Load balanced across node of the cluster that runs query service ONLINE 11210, 11207 Core Data Operations State-full connections from client app to nodes of the cluster that runs data service
- 10. ©2015 Couchbase Inc. 10 Connectivity and Client Libraries Connectivity Phases 1. Auth 2. Discovery Get cluster map 3. Service Connection Auth to Service Run operation If (topology_change) Rerun #2 10 1 2 …
- 11. ©2015 Couchbase Inc. 11 Discovery and Cluster Map
- 12. ©2015 Couchbase Inc. 12 Discovery and Cluster Map
- 13. ©2015 Couchbase Inc. 13 Discovery and Cluster Map – 2 New Nodes
- 14. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 15. ©2015 Couchbase Inc. 15 Database Change Protocol (DCP) Fast Streaming Replication DCP - An open streaming protocol that conveys the consistent database state to all consumers Ordering (vbucket based seq.number) Re-startable, Resumable (version histories and rollbacks) Consistent (snapshots) High Performance (memory based with dedup) Master Local Replica Index Map/R educe Remot e Replica IndexMap/Re duce Source Cluster Cross Data Center Cluster Hadoop Client/Appl ication Notification In future Integration Backup/Ex port Tooling
- 16. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 17. ©2015 Couchbase Inc. 17 Cluster Manager Cluster Manager = Governor of the Cluster Manages cluster level operations and coordination among nodes Cluster Membership & Service Layout Node Status & Failover Data Placement & Rebalance Auth 17
- 18. ©2015 Couchbase Inc. 18 Cluster Manager Inside Cluster Manager per-node-&-bucket services generic distributed facilities generic local facilities Logging and Other Services distributed node discovery Master Services - cluster level operations - data placement - rebalancer - auto-failover Admin Portal – RESTAPI Global Config (gossip replication) Local Config Store Per-node Services - Heartbeats, - Babysitter Bucket services - dcp init and teardown - stats collectors, Auth
- 19. ©2014 Couchbase Inc. Adding Nodes to Cluster Online 19 ACTIVE ACTIVE ACTIVE REPLICA REPLICA REPLICA Couchbase Server 1 Couchbase Server 2 Couchbase Server 3 ACTIVE ACTIVE REPLICA REPLICA Couchbase Server 4 Couchbase Server 5 SHARD 5 SHARD 2 SHARD SHARD SHARD 4 SHARD SHARD SHARD 1 SHARD 3 SHARD SHARD SHARD 4 SHARD 1 SHARD 8 SHARD SHARD SHARD SHARD 6 SHARD 3 SHARD 2 SHARD SHARD SHARD SHARD 7 SHARD 9 SHARD 5 SHARD SHARD SHARD SHARD 7 SHARD SHARD 6 SHARD SHARD 8 SHARD 9 SHARD READ/WRITE/UPDATE Cluster Manager receives the new nodes - Node inherit cluster settings - Move active and replica vbuckets using DCP - As vbuckets catch up, Initiate online handoff from “existing node” to “new node” Clients Receive Topology Change Notification - Trap not_my_vbucket errors - Refresh cluster map and retry operation
- 20. ©2015 Couchbase Inc. 20 Deeper Dive into Architecture THUR @1.00 - ArchitectureTrack Deep Dive into Cluster Manager in Couchbase Server 4.0 Dave Finlay, Senior Director of Development, Couchbase
- 21. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 22. ©2015 Couchbase Inc. 22 Data Service Data Service = GET/SET + Map-ReduceViews* Tackles fast core data operations with efficient caching and disk persistence Core Database Operations Core GET/SET operations Couchstore Based Storage Terms: Bucket = database reside within a cluster vBucket = hash partition of the database that reside within a node 22
- 23. ©2015 Couchbase Inc. 23 Data Manager Architecture … Database Engine (ep-engine) Listener vBucket Manager Item Pager Expiry PagerCheckpoint Manager Cache Partition Hash Tables (Active and Replica) Partition Hash Tables (Active and Replica) Partition Hash Tables (Active and Replica) AuthNetwork IO Flusher Scheduler Reader IO Writer IO Non IO Batch Reader
- 24. ©2014 Couchbase Inc. APPLICATION SERVER MANAGED CACHE DISK DISK Data Manager - Update Operation 24 DOC 1 DOC 1 DOC 1 DOC 1DOC 1 Caching based on Memcached: App gets an ACK when update is successfully in RAM Or RAM+Replicated Or RAM+Persisted Or RAM+Replicated+Persisted Or … DCP base Replication: updates queued to other nodes Couchstore based Storage: updates queued for storage DCP
- 25. ©2014 Couchbase Inc. Data Manager - Cache Ejection 25 APPLICATION SERVER MANAGED CACHE DISK DISK DOC 1 DOC 2DOC 3DOC 4DOC 5 DOC 1 DOC 2 DOC 3 DOC 4 DOC 5 DCP
- 26. ©2014 Couchbase Inc. APPLICATION SERVER MANAGED CACHE DISK DISK DCP DOC 1 Data Manager - Cache Miss 26 DOC 2 DOC 3 DOC 4 DOC 5 DOC 2 DOC 3 DOC 4 DOC 5 GET DOC 1 DOC 1 DOC 1
- 27. ©2015 Couchbase Inc. 27 Deeper Dive into Architecture WED @4.30 - ArchitectureTrack Next Generation Storage Engine: ForestDB Chiyoung Seo, Software Engineer, Couchbase
- 28. ©2015 Couchbase Inc. 28 Deeper Dive into Architecture THUR @2.30 - ArchitectureTrack Under the Hood ForestDB: Performance on SSDs and File Systems Sundar Sridharan, Senior Developer, Couchbase Prof. Sang-Won Lee, Sungkyunkwan University, Suwon, Korea
- 29. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 30. ©2015 Couchbase Inc. 30 Indexing Indexers Views: Incremental Map/Reduce with customer JavaScript for complex indexing logic for online reporting and analytics SpatialViews: Incremental/Reduce with R-tree Indexes for Bounding- Box queries GSI (Global Secondary Indexes): Efficient indexes for secondary lookups and ad-hoc query processing
- 31. ©2015 Couchbase Inc. 31 Indexing withViews Incremental Map/ReduceViews Queries with incremental map/reduce processing that can execute custom JS Flexible Indexer Indexer & Query Processor Couchstore Based Storage 31
- 32. ©2015 Couchbase Inc. 32 Views 32 APPLICATION SERVER VIEW INDEXER Query Set 1. SET operation • In Memory SET: queued for DCP Replication • View Indexer: Executes incremental map/reduce on a batch of updates • Couchstore based Storage: updates queued for storage 2. ViewQuery Engine: REST Based queries with filters, limit and more executed with scatter-gather
- 33. ©2015 Couchbase Inc. 33 Index Service Global Secondary Indexes (NEW in 4.0) Tackles indexer for fast query execution with efficient index maintenance for N1QLQueries High Performance Indexing Projector and Router : Coordinate and communicate efficient index change notifications between data service and index service. Supervisor – Indexer and scanner Indexer : Maintain large number of indexes as change notifications arrive Scanner: Respond to Query Service index-scan requests with rich set of consistency dials Index Storage &Caching ForestDB: Brand new storage engine for high performance index caching and storage 33
- 34. ©2015 Couchbase Inc. 34 Data Service Projector & Router Indexing Service Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 Query Processor cbq-engine Bucket#1 Bucket#2 DCP Stream Index#4Index#3 ... B u c k e t # 2 B u c k e t # 1 Projector and Router: 1 Projector and Router per node 1 stream of changes per buckets per supervisor ForestDB Storage Engine Supervisor 1 Supervisor per node Many indexes per Supervisor
- 35. ©2015 Couchbase Inc. 35 Deeper Dive into Architecture WED @1.45 - ArchitectureTrack Global Secondary Indexing in Couchbase Server 4.0 Cihan Biyikoglu, Director of Product Management, Couchbase
- 36. ©2015 Couchbase Inc. 36 Deeper Dive into Architecture WED @3.45 - ArchitectureTrack Deep Dive into Global Secondary Indexing Architecture in Couchbase Server 4.0 John Liang, Architect, Couchbase
- 37. Deep Dive Connectivity Replication Cluster Manager Data Service Indexing & Index Service Query Service
- 38. ©2015 Couchbase Inc. 38 Query Service Query Service = N1QL Tackles N1QLQuery execution Query Execution N1QL Parser & Optimizer: tokenize N1QL statement, and generate an execution plan based utilizing indexes Query Execution Engine: Assigns resources to query and coordinates query execution. Data Sources: Pluggable “data source driver” layer for accessing data sources in Couchbase Server (data and index service) and other external data provides 38
- 39. ©2015 Couchbase Inc. 39 Query Service N1QL Query Processing Query Engine Query Processor Listeners Parser Optimizer Data Stores Execution Engine Couchbase Server Auth Data Indexers GSI Views Others… 8093/18903 File system Data Service Index Service ...... Cluster Manager Bucket#2 Bucket#2 Index#2Index#1
- 40. ©2015 Couchbase Inc. 40 Deeper Dive into Architecture WED @2.30 - ArchitectureTrack Deep Dive into N1QL Internals in Couchbase Server 4.0 Keshav Murthy, Director of Development, Couchbase
- 41. ©2015 Couchbase Inc. 41 Recap Couchbase Server decouples competing workloads into Services Each Service tunes itself with caching and storage services Each Service can be independently scale
- 44. Thank you.
Editor's Notes
- #12: The application makes a call for a key called NYC MQ1 We run the key through the crc 32 function and the result of that hash function is that it points to vbucket3 Which in turn points to couchbase server number 1
- #13: We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
- #14: We now run a different key through through the has and we now come up with differnet vbucket, vbucket 4 and that points to server 3
- #16: Protocol seemlessly consolidate/encapsulate working set management and storage cache Restartable – enable client to stop and resume work Universal – stream “Database change record”, not just data – enable disparate clients to have common agreement on a consistent state Effiicient – Multi-plex connection, de-duplication, memory-to-memory, flow control
- #19: Large scale structure of ns_server. Similar / identical to Steve’s slide. Composed of different subsystems: libraries master services per node services per node – per bucket services generic services events To zoom in further, we need to talk a little about the language. Erlang.
- #26: Now, as you fill up memory (click), some data that has already been written to disk will be ejected from RAM to make room for new data. (click) Couchbase supports holding much more data than you have RAM available. It’s important to size the RAM capacity appropriately for your working set: the portion of data your application is working with at any given point in time and needs very low latency, high throughput access to. In some applications this is the entire data set, in others it is much smaller. As RAM fills up, we use a “not recently used” algorithm to determine the best data to be ejected from cache.
- #27: Should a read now come in for one of those documents that has been ejected (click), it is copied back from disk into RAM and sent back to the application. The document then remains in RAM as long as there is space and it is being accessed.