Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!

Why Developing with Couchbase is
Easy
Karthik Babu Sekar
Solutions Architect, World Wide Technical Field Operations @ Couchbase
- LinkedIn - https://www.linkedin.com/in/karthikbabusekar
- Twitter- https://twitter.com/SKBhere
- E-mail – karthik@couchbase.com

Change is
happening:
the way we
engage has
evolved

Technology is Further Driving Digital Innovation
Internet
of
Things
Big
Data
Mobile
Cloud

Personalized Feature-rich Quick Anywhere Anytime
Customers now expect exceptional experiences

Agility to adapt to changes in demand
Responsive, context-aware customer experiences
Multi-channel, multi-device engagement
Ability to support a large and growing number of
users
What businesses need to succeed in this new world

But these requirements cannot be met just at the application tier:
they need to be applied to the database

©2017 Couchbase Inc. 7
History of Couchbase

The Rise of the System of Engagement Database
8©2017 Couchbase Inc.
System of
Record
1980’
s
CRM ERP ECM
System of
Record
1995
CACHE
CRM ERP ECM
NoSQLSystem of
Record
2005
CACHE
CRM ERP ECM
TODAY
NoSQL
System of
Engagement
MOBILEQUERY
INDEX
SEARCH REPLICATION
ANALYTICS
DATA CACHE

Database requirements to power engaging experiences
9
Adaptable.
Flexible schema
Responsive.
Consistent
High-Performance
Scale as needed.
Able to handle any demand
(operations, queries, data)
Secure.
Device to cloud,
auditable, pluggable
Global.
Data locality,
Geo-distributed,
Multi-Data Center
Always on.
FaultTolerant. Able to
continue operations
under failure conditions
Mobile experience.
Offline, sync’d,
secure, low latency,
personalized
Intelligent.
FullText Search, Real
Time Analytics, Big
Data enabled
©2017 Couchbase Inc.

Couchbase Data Platform
Unified Administration
Unified Programming Interface
Data Query Index SearchMobileReplication Analytics
{N1QL}
The System of Engagement Database

Developing with Couchbase
Couchbase supports a wide range of frameworks, languages,
platforms and infrastructure choices.
Frameworks Languages Platforms Infrastructure
MOBILE MOBILE

Languages and Interfaces for Couchbase
§ Official SDKs
– Java
– .NET
– Node.js
– Python
§ For each of these we have
– Full Document support
– Interoperability
– Common Programming Model
Others: Erlang, Perl, TCL, Clojure, Scala
- PHP
- C
- Go
- Ruby
JDBC and ODBC

Couchbase Server Architecture
Overview

Service-Based Architecture
§ Multi-Dimensional Scalability (MDS) – Optionally Scale each service
independently:
– Data
– Index
– Query
MDS is the architecture
that enables independent
scaling of data, query, and
indexing workloads while
being managed as one
cluster.

Full Cluster Architecture
15
STORAGE
Couchbase Server 1
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Query Service STORAGE
Couchbase Server 2
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Couchbase Server 3
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Couchbase Server 4
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Couchbase Server 5
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Couchbase Server 6
SHARD
7
SHARD
9
SHARD
5
SHARDSHARDSHARD
Managed
Cache
Cluster
Manager
Cluster
Manager
Managed
Cache
Storage
Data Service
Index Service
Query Service

Couchbase Data Access
• Everything is built on top of Key Value
• A Document store is a special case of Key-Value
• Views provide aggregation and real-time analytics
through incremental map-reduce
• Global Secondary Indexes provide low latency/high
throughput indexes
• N1QL is a language that provides a powerful and
expressive way of accessing documents

Cluster
Bucket
CRUD
View
Query
N1QL
Query
Interacting with Data

Couchbase SDKs
§ What does it mean to be a Couchbase SDK?
Cluster
Bucket
CRUD
Functional
Give the application developer a concurrent
API for basic (k-v) or document management
API
get()
insert()
replace()
upsert()
remove()

The Document
§ Documents are integral to the SDKs.
§ All SDKs support JSON format
§ In addition: Serialized objects, Unquoted Strings, Binary pass-through
§ A Document contains:
19
Property Description
ID The bucket-unique identifier
Content The value that is stored
Expiry An expiration time
CAS Check-and-Set identifier

Bucket API – Modifying Documents
§ insert() the document if it does not exist
§ replace() the document if it does exist
§ upsert() the document (insert or replace)
§ remove() the document
§ append() data to the document
§ prepend() data to the document
§ counter() for increment/decrement type operations

Bucket API – Retrieving Documents
§ get() the document
§ getFromReplica() if the active is not available
§ getAndTouch() to load the document and reset the expiry
§ getAndLock() to load the document and write-lock it

Unified API – DML, CAS Example
1. Two Clients retrieve the same document "XYZ"
2. Client A retrieves it first.
3. Client B then retrieves XYZ. Both clients will have the same CAS value for document XYZ
4. Client B tries to perform an update to document XYZ. The update succeeds as the CAS value
was unchanged from when Client B initially retrieved the document. Once the update
succeeds, the CAS value for XYZ changes.
5. Client A then tries to perform an update on XYZ immediately after Client B. The update will
fail as Client A's CAS value is out of date. When Client B updated XYZ, the CAS value
changed.
©2014 Couchbase, Inc.

Durability Requirements – Mutations
§ By default, a write operation is successful when the data has been written to the
memory of the node with the active vBucket.
– Eventually (but pretty quickly), the data will also be written to persistent storage and to replica
vBuckets
§ persistTo – write operation is successful when data has been written to persistent
storage
– Can be set to require writes to persistent storage for replicas, too
– Using persistTo will slow performance
§ replicateTo – write operation is successful when data has been written to both the
active vBucket and replica vBuckets
– If more than one replica, can set the number of replicas that must be written for success

Bucket API – Querying
§ query() is possible for
§ Views
§ N1QL
§ Streams N response rows as they arrive from the server
§ Loads more than one Document based on Criteria
§ Typically used to satisfy secondary and advanced querying use cases

©2017 Couchbase Inc. 25©2016 Couchbase Inc. 25
Bootstrapping your application - Service List
25
• Dynamic Distributed Services
• Dynamic Configuration Updates – No additional
work from the developer
• FaultTolerant/Durable Connectivity (we are paranoid)
Client Connectivity Characteristics:

Key/Value
§ Fastest way to access & mutate documents
§ Access by Document ID only
§ Includes subdocument manipulation
§ Efficient binary protocol
Latency Flexibility

Key/Value – API DML Methods (Create/Delete/Update)
• insert-Insert a document or binary key/value. Fails if the item exists.
• upsert-Stores a document or binary key/value to the bucket, or updates if a
document exists.
• replace-Replaces a document or binary key/value in a bucket. Fails if the
item doesn’t exist.
• remove-Deletes an item from the bucket. Fails if the item doesn’t exist
• append/prepend-Appends or prepends in place the value of a binary k/v
item. Does NOT work with documents
• touch-Updates the ttl of a documet.
• getAndTouch-Retrieves a document or binary key/value and updates the
expiry of the item at the same time.
• counter-Increments or decrements a key's numeric value.

Key/Value – API DML Methods (Retrieval)
• get-Retrieves a document or binary key/value.
• getAndLock-Lock the document or binary key/value on the server and
retrieve it. When an document is locked, its CAS changes and
subsequent operations on the document (without providing the current
CAS) will fail until the lock is no longer held.
• getReplica-Get a document binary key/value from a replica server in
your cluster.
• unlock-Unlock a previously locked document or binary key/value on
within a bucket.

Key/Value – Durability Methods
§ By default, a write operation is successful when the data has been written to the
memory of the node with the active vBucket.
– Eventually (but pretty quickly), the data will also be written to persistent storage and to replica vBuckets
persistTo – A write operation is
successful when data has been
written to persistent storage.
• Can be set to require writes to persistent
storage for replicas, too
• Using persistTo will slow performance
replicateTo – A write operation is
successful when data has been written
to both the active vBucket and replica
vBuckets.
• If more than one replica, can set the number of
replicas that must be written to for success

Key/Value - Sub-Document Operations
§ Document Mutations:
• Atomic Operate on individual
fields
• Identical syntax behavior to
regular bucket methods
(upsert, insert, get, replace)
• Support for JSON fragments.
• Support for Arrays with
uniqueness guarantees and
ordinal placement
(front/back)

Views
§ Distributed Map-Reduce
§ Great for Aggregations (fast)
§ Spatial Support
Latency Flexibility

N1QL
§ Very flexible, but get your index right
§ Global Index instead of Map/Reduce
§ Similar to SQL
Latency Flexibility

Full Text Search
§ Very Flexible, but more concepts to learn initially
§ Inverted Index
§ Still Developer Preview
§ Similar to Elasticsearch
Latency Flexibility

Show me some code ...
ü Create connection
ü Open bucket
ü Create JSON object
ü Create doc from object, assign
key, and "upsert"
• if exists, update
• if not, insert
ü Get the document by key,
and display
// Connect to localhost
Cluster cluster = CouchbaseCluster.create();
// Open a bucket connection
Bucket bucket = cluster.openBucket("customers");
// Create a document
JsonObject user = JsonObject.empty()
.put("first", "Walter")
.put("last", "White")
.put("job", "chemistry teacher")
.put("age", 50);
// Store a document
JsonDocument stored =
bucket.upsert(JsonDocument.create("walter", user));
// Get the document
JsonDocument walter = bucket.get("walter");
System.out.println("Found: " + walter.getString(“first”));("first"));
§ Connect, create, and retrieve
a document (Java)

What’s New and Improved in 4.6.0

Agenda
Cross Datacenter Replication (XDCR)
– Timestamp-based Conflict Resolution
Security
– Secret Management and support for Pluggable Authentication Modules (PAM)
Tools [Developer Preview]
– New CBImport and CBExport utilities
Search [Developer Preview 2] (FTS)
– New Sort and Document “type” features
Query (N1QL)
– New String functions, Date functions, Array functions and Object functions
– Performance Improvements

Cross Datacenter Replication (XDCR)
Timestamp-based Conflict Resolution

XDCR Conflict Resolution Modes
● Revision-based Conflict Resolution [Default]
Current XDCR conflict resolution uses the revision ID (part of document metadata) as the
first field to resolve conflicts between two writes across clusters. Revision IDs keep track
of the number of mutations to a key, thus the current XDCR conflict resolution can be
best characterized as “the most updates wins”.
● Timestamp-based Conflict Resolution [New]
Timestamp-based conflict resolution uses the hybrid logical clock (part of
document metadata) as the first field to resolve conflicts between two writes across
clusters. Timestamp has both physical time (NTP) and a logical counter, thus the new
XDCR conflict resolution is also known as Last Write Wins (LWW) and is best
characterized as “the most recent update wins”.

What is Hybrid Logical Clock?
§ Hybrid Logical Clock is combination of physical time and logical counter.
§ Hybrid Logical Clock is represented as 64 bit integer
– First 48 bit – physical time
– Last 16 bit - logical counter
§ Hybrid Logical Clock is stored in CAS

XDCR Use Cases Supported for Conflict Resolution
● Unidirectional Replication
○ Hot spare/ Disaster Recovery
● Bidirectional Replication
○ Datacenter Locality
● High Availability
○ Cross-Cluster FailOver and FailBack

Security
Secret Management

Secret Management in 4.6
Secret management provides encryption of system secrets
• Zero-knowledge secret management system driven by user specified
master password
• Allows online password rotation without application downtime
PBKDF2 Encrypt/Decrypt
Data Key
Administrator password
Bucket password
Erlang cookie
E-mail password (alerts)
Master password
Non-persisted key

Setting Master Password & Rotating Data Key
Setting Master Password
-Via REST API
POST - /node/contorller/changeMasterPassword
- Via CLI
couchbase-cli master-password -c 192.168.0.1:8091 -u Administrator -p password --new-password
Rotating Data Key
-Via REST API
POST - /node/controller/rotateDataKey
- Via CLI
couchbase-cli master-password -c 192.168.0.1:8091 -u Administrator -p password --rotate-data-key

Pluggable Authentication Modules (PAM)
▪ Allows UNIX local accounts to authenticate as Couchbase administrators
▪ Pluggable authentication architecture that is policy driven
Centralized
Management
Centralized and synchronize
administrator account
management using UNIX user
management services
Security Policy
Enforcement
Allows configuration of strong
security policies such as strong
password requirements

Tools
CBImport & CBExport

CBImport
- ‘cbimport' is a new tool to import data into a Couchbase cluster from different data sources. In Spock,
CBImports supports importing data from CSV file or JSON documents.
- Each supported format is a sub-command of the cbimport tool
- ‘cbimport csv’: Imports data into Couchbase from a CSV file
- ‘cbimport json’: Imports data into Couchbase from a JSON file
- ‘cbimport' tool also supports custom key-generation for each document in the imported file. Key
generation is done with a combination of pre-existing fields in a document and custom generator
functions supplied by cbimport.
- ‘cbimport' tool supports importing JSON document of different structure
- JSON object on each line,
- files that contain a JSON list (that is array) where each element is a document,
- and the Couchbase Samples files

CBExport
- ‘cbexport' is a new tool to export data from a Couchbase cluster. In Spock, CBExports supports exporting
JSON data to a file.
- Each supported format is a sub-command of the cbexport tool
- ‘cbimport json’: Exports JSON data from Couchbase to a file
- ‘cbexport' tool supports exporting JSON document of different structure
- JSON object on each line,
- files that contain a JSON list (that is array) where each element is a document,
- and the Couchbase Samples files

Search
Full Text Search (FTS)

Custom Sort Ordering
Sort search results your way
§ Before 4.6, all results sorted by score
§ Now, sort on any indexed field, id, or score in ascending / descending
order
§ Control how to handle fields with multiple or missing values
{
...
"sort": [
"country",
{ "by" : "field", "field" : "reviews.ratings.Overall", "mode" : "max", "missing" : "last" },
{ "by" : "field", "field" : "reviews.ratings.Location", "mode" : "max", "missing" : "last" },
"-_score”
]
}

Type Mapping By ID
Type mapping = specifying different indexing rules for different doc types
Pre-4.6, type defined by a type attribute in the JSON doc body
Now, doc types can also be determined by parsing doc key
– prefix or regular expression

Query

New Date Functions
▪ CLOCK_LOCAL()
– Returns the local time at the server.
▪ CLOCK_UTC()
– Returns the Coordinated UniversalTime
▪ CLOCK_TZ()
– Returns the time in the specified timezone
▪ DATE_FORMAT_STR(expr, fmt)
– This function converts a given date string parameter to the specified format

Couchbase Mobile 1.4

Couchbase Mobile: The Complete Mobile Database Solution
Lightweight
embedded NoSQL
database with full
CRUD and query
functionality.
Secure web
gateway with sync,
REST, stream, batch
and event APIs for
syncing data over
the web.
Highly scalable,
Highly available,
High performance
NoSQL database
servers.
Couchbase Lite Sync Gateway Couchbase Server
EMBEDDED DATABASE SYNCHRONIZATION DATABASE SERVER

Couchbase Lite Overview
§Cross-platform support for all major operating
systems and platforms
§Built native from the ground up
§500kb for most platforms
§256-bit AES full database encryption
55
Couchbase Lite
Embedded Database

Sync Gateway Overview
§Synchronize data between Couchbase Lite &
Couchbase Server
§Pluggable Authentication
§Fine grained user and role based access
control
§Elastically scalable in real-time
56
Sync Gateway
Synchronization

Security
§Encryption of Local Database
–AES-256 / PBKDF2
• HTTPS / TLS
• Separate Read & Write Permission
with Fine Grain Document Control
• Pluggable Authentication
Couchbase Lite
Embedded Database
Sync Gateway
Synchronization

Load
Balancer
CRUD ,
Security &
Seq. Number
HA Data
Storage &
Channel Index
Couchbase Mobile - Before
20,000 Max
Concurrent

Problems – Single Global Sequence Number
Updating a single key at scale
causes too many problems

Bucket = Database
vB
Data buckets (default)
vB
1 ….. 1024
Virtual buckets
or
(shards / files)

Solution - DCP (Database Change Protocol)
Cluster data replication via sequential streaming
feed from each vBucket (shard/file) on each node
called “DCP”

Sync Gateway Accelerator (Enterprise Only)
Couchbase Mobile v1.2+
Accelerator listens to DCP to create k/v
channel indexes & uses DCP sequence numbers
Same vLAN

Horizontally Scale All Three Parts in v1.2+
LB
x,000,000+
Concurrent
CRUD &
Security
HA Data
Storage
Channel Index &
Seq. Number

Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !! (20)

Recently uploaded (20)

Couchbase Singapore Meetup #2: Why Developing with Couchbase is easy !!