Northeastern DB Class Introduction to Marklogic NoSQL april 2016

© COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Matt Turner, CTO Media & Entertainment
Introduction to MarkLogic NoSQL

SLIDE: 2 © COPYRIGHT 2016 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.
Outline
• Something’s Happening Here
• The Old and the New
 Data models
 Data access
• Discussion

Analysis Operations Access
DATA MAKES AN IMPACT

Stress on Traditional Data Approaches
Complexity
 Structured
 Unstructured
 Semi-structured
 Raw
 Streams of data
 Constant change
 Agile analytics
 Fail-fast
Volume
Velocity Variety
Volume • Many months of system log files
• Every tweet
• Years of articles
• Relative to current size of
operation
Velocity • Streams of customer feedback
to determine sentiment
• Real-time risk analysis
• Real-time Business Intelligence
Variety • Database feeds
• Raw logs
• Web crawl data
• Articles
• Multi-media
• ALSO: questions!
Examples
Big Data: Gartner coined the “three V’s” description
 Data: Petabyte
scale
 Nodes:
Thousands

Leader Quadrant
Online Transaction
Processing RDBS
(May 2002)

Leader Quadrant
Operational DBMS
(Oct 2014)
Traditional
Mainstays
Upstarts Storm
the Field

MarkLogic:
Best Operational
Data Warehouse
(Aug 2014)

A Unified, Actionable
360 View of Data
WHAT BUSINESSES WANT

Northeastern DB Class Introduction to Marklogic NoSQL april 2016

Data Is In Silos
 Data is spread across disconnected databases
 M&A outpaces the speed of data integration
 Data needs to be delivered in real time
THE REALITY

80% OF TIME
By data scientists just
wrangling data
WASTED
In 2015 on creating relational
data silos
Of data warehouse projects
is on ETL
The Massive Cost of Integrating Data From Silos
36BILLION IN
SPENDING
$% OF THE
COST60

Relational Databases with
ETL Sacrifice Agility,
Timeliness, and Cost
 All future data needs must be predictable
 New SQL queries require database re-indexing
 Siloed database changes require ETL re-writes
THE IT CHALLENGE
ETL
OLTP
ARCHIVES
ETL
ETL
ETL
DATA MARTS
ETL
WAREHOUSE
REFERENCE DATA

OLTP
Warehouse
Data MartsArchives
“Unstructured”
“ ”
Video
Audio
Signals,
Logs,
Streams
Social
Documents,
Messages
{ }
Metadata
Search🔍
Reference
Data
It’s Complicated

The OLD:
Let’s Design the Application
(And pretend it’s the 80s)

Name Hair Colour Fulltime Employee? Car type
Paul Blond Y
Alex Auburn Y Porsche
Dom Black Y Hummer
name hr_colr flltme_empl car_tp
Let’s Begin… Cast Members
{
How many characters wide should this be? 8? 16? 32?
{
{
{

New Schema – Extend Ours!
name hr_colr flltme_empl car_tp
Paul Blond Y
Alex Auburn Y porsche
Dom Black Y Hummer
house_road town city postcode
11d Yonge Pk Finsbury London N4 3NU
Reading
London N43
• Hang on
• If this table had 10k rows, issues?
• First create new big schema
• Then import rows across
• Delete old table?
• Maybe not, legacy programs might use it!
• What if we want to select “Road” only?
• Split out again
• More extensions?
• House name and number?

There is another way!
Create a new table and point to it from the old one!
name hr_colr flltme_empl car_tp Address
Paul Blond Y
Alex Auburn Y porsche
Dom Black Y Hummer
house_road town city postcode
11d Yonge Pk Finsbury London N4 3NU
Reading
London N43

…

Now Let’s Store Something More . . . Complicated
Transcript / Book
Info
Title = “NL April 14”
Author = “SNL Cast”
Section
• Chapter
Page
Paragraph =
“I love penguins because…”
Page
Paragraph =
“On the subject of food…”
• Chapter
Page
Section
• Chapter
• Chapter
• Chapter
• Paragraph
• Paragraph
title author Section
I love
Penguins
S. Lion
Issues with Sections? How many columns?

Don’t Forget Taxonomies
Hierarchical levels of metadata
Fixed to a specific business purpose
 Can’t be re-used in new contexts
Each record can only be associated with
one level
 How many category fields?
Category
Feature
Series
Action
Drama
Comedy
Documentary
…
Cable
Broadcast
Drama
Comedy
…
Action
Drama
Family
Documentary
…

Result
Requires everything to be defined up front
Data to be transformed and processed to
fit the system
Needs to be redone as information
changes
Costly to create, maintain and only
captures part of the data!
Title ProductionDate Category AssetType Length
Film1 3/1/14 Feature HD Master 2:40
Show1 6/4/13 Series HD720 0:40
Film2 6/4/05 Feature Archive 1:55
Category
Feature
Series
Action
Drama
Comedy
Documentary
…
Cable
Broadcast
Drama
Comedy
…
Action
Drama
Family
Documentary
…
?

OLTP
Warehouse
Data MartsArchives
“Unstructured”
“ ”
Video
Audio
Signals,
Logs,
Streams
Social
Documents,
Messages
{ }
Metadata
Search🔍
Reference
Data
*NOTE: We only did this
little bit!
Remember?

The NEW! Enter NoSQL
Category Description Examples
Key-value • Persistent hash-table “on steroids”
• Typically no single modeling paradigm (e.g. columns
can be primitives, data structures, binaries, etc.)
• Amazon
DynamoDB
• Redis
• Riak
Columnar • Similar to K-V in some ways
• Column may be arranged in groups (families)
• Data types are usually the expected “primitives”
• Works well with “value crunching” (e.g. time series)
• HBase
• Cassandra
Document • URI-mapped (i.e. keyed) documents in lieu of rows
• Supports structured and unstructured content
• Nested context
• MarkLogic
• MongoDB
• Couchbase
Graph • Deals with inter-object graphs
• Relationship oriented
• Think object cache (with pointers) “on steroids”
• Neo4J
• AllegroGraph
• InfoGrid

A Database That
Integrates Data Better,
Faster, with Less Cost
THE DESIRED SOLUTION

The MarkLogic Alternative
An Operational and Transactional Enterprise NoSQL Database
 Data ingested as is (no ETL)
 Structured and unstructured data
 Data and metadata together
 Adapts to changing data
and changing data structures
EASY TO
GET DATA IN
Flexible Data Model
 Index once and query endlessly
 Real-time and lightning fast
 Query across JSON, XML, text,
geospatial, and semantic triples
in one database
EASY TO
GET DATA OUT
Ask Anything Universal Index
 Reliable data and transactions
(100% ACID compliant)
 Out-of-the-box automatic
failover, replication, and
backup/recovery
 Enterprise-grade security and
Common Criteria certified
100%
TRUSTED
Enterprise Ready

The SNL App

No need to define up front
Matched to complex content and
metadata data modeling
Data is managed in its most
accessible, natural form
XML, JSON, RDF, geospatial
Flexible Data Model
Schema-agnostic, structure-aware

Instead of THIS

Do it like THIS!

Search and Query
Search to find answers in documents, relationships, and metadata
 Automatic indexing of every data value, text and data
structure
 Specialized indexes for data values (analytics, facets,
sorting), geospatial and triples
 All updated in the context of ACID transactions to
ensure data integrity and real-time access
 Accessible via fully programmable search API with full-
text search, type-ahead suggestions, facets, snippeting,
highlighted search terms, proximity boosting, relevance
ranking, and language support
JavaScript XQuery SPARQL
Rich Query
Capability
In-database
MapReduce
Full-text
Search
Semantic
Search
Geospatial
Search

Do domestic dogs interpret pointing as a command?
Animal Cognition (2012): 1-12 , November 09, 2012
By Scheider, Linda; Kaminski, Juliane; Call, Josep; Tomasello, Michael
Context!

© COPYRIGHT 2015 MARKLOGIC CORPORATION. ALL RIGHTS RESERVED.SLIDE: 39
Machines Don’t Get Context . . .
Manu Sporny Founder/CEO - Digital Bazaar, Inc.
http://www.cambridgesemantics.com/semantic-university/what-is-linked-data

Enter Semantics!
Manu Sporny Founder/CEO - Digital Bazaar, Inc.
http://www.cambridgesemantics.com/semantic-university/what-is-linked-data

Semantics
Enterprise triple store, document store, and database combined
 Store and query billions of facts and relationships
 Leverage ontologies for domain and role specific
context access to data and documents
 Efficient metadata management with relationships
to ontologies
 Standards-based for ease of use and integration
– RDF, SPARQL, and standard REST
interfaces

Semantics to Model Relationships
Data model to manage relationships and link together data
‘triples’ describe single facts
Collections of facts describe complex real-world scenarios
“Chevy” ”NBC"
isOn
”SNL"
isOn
isOn
!

Ontologies Instead of Categories
Actually model information as it is in
the real world
Not limited to a single purpose
 Ontologies for all categories of
metadata
 Even ‘impossible’ categories
like fictional worlds

NoSQL and Semantics!

Real-Time Analytics
Range indexes can be used for
 Faceted search
 Aggregation and visualization
 Analytics…
…including custom user-defined functions
 Co-occurrence
 SQL, ODBC, and BI integration

Scalability, Elasticity and Cloud
Massive enterprise scalability and elasticity
 Scale horizontally in clusters on commodity
hardware to hundreds of nodes, petabytes of
data, and billions of documents
 Process thousands of multi-document multi-
statement transactions per second
 Start small and scale up or down to meet capacity
and performance demands without over-
provisioning or over-spending
 Fully cloud enabled for automated deployment
and management on EC2
 Leverage dynamic configurations with Tiered
Storage
D-NODE D-NODE
E-NODE E-NODE
D-NODE
Result: Enterprise-ready to power mission critical products

Use Case: Deliver Better Information
Present information based on
relationships
Go beyond traditional technology with
depth of content
Drive efficiency using semantic approach
to tagging

Use Case: Go Beyond Search
• Concept instead of keyword search
• Related content and information
drive the content discovery and new
interactions
 SNL40 continuous viewing
• Dynamically tailored to the users
specific attributes or activity

Use Case: Integrate Data
• Integrate data across the automoti

Bob Pilz
Taxonomy Manager
Mitchell1

Semantics-driven search
Talent
Kristen Wiig
Acted in
Episode 4
Anne Hathaway and Killers
Part of
Played
Character
Maharelle Sister
Season 34
Segment
The Lawrence Welk Show
Aired on
Date
10/4/08
Era
Acted in
Includes
Part of

Intelligent recommendation engine

Northeastern DB Class Introduction to Marklogic NoSQL april 2016

More Related Content

What's hot (20)

Viewers also liked (19)

Similar to Northeastern DB Class Introduction to Marklogic NoSQL april 2016 (20)

More from Matt Turner (20)

Recently uploaded (20)

Northeastern DB Class Introduction to Marklogic NoSQL april 2016

Editor's Notes