Hadoop World 2011: Lily: Smart Data at Scale, Made Easy

Lily
A SMART DATA PLATFORM
MAKING BIG DATA APPS EASY

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org

volume
data

need for
distributed
processing
moore

time

IIC » TECHNOLOGIEPARK 3 » B-9052 ZWIJNAARDE (GENT) » www.outerthought.org 2

need for
distributed
processing


distributed
systems are
hard.

data shuffling, data duplication

database data warehouse analytics


“Top-performing
organizations are twice
as likely to apply
analytics to activities.”

(MIT Sloan Management
Review, Winter 2011)


Heavy Math?
8

lots of
data?
9

what drives insights?


your audience does.


data

audience data
recommendations

insights

LILY combines
scalable storage,
indexing and search
with real-time usage
metrics, insights and
recommendations.

...on top of CDH

Cloudera’s Distribution including Apache Hadoop


lily features


schema
16

lily data model
» adds a high-level data model on top of HBase’s byte[ ]’s


value types
» basic value types » datetime

» string » blob

» integer » uri

» long » parametrized value types
» double » list
» decimal » path
» boolean » link
» date » record


why a schema?

» contract for client apps
» validation
» application lifecycle mgmt
» schema versioning
» cfr. Avro
» content-based indexing


versioning
20

record version schema version


Versioning scopes

data gets overwritten versioned data version status data


API

builder API

other apis

Java Builder
REST API
» REST (HTTP + json) API

» Java (Avro) Java API


indexing
19

features

» indexing conﬁguration
» denormalization + link dereferencing
» indexing of multiple versions
» incremental and batch (MR) index updates
» blob content extraction (Apache Tika)
» Solr index sharding (!)


indexer configuration


search

30

simple:

consistency

32

consistencY

» many things can go wrong
» service failure: HBase, HDFS, Solr, Lily
» network failure, time skew
» node failure
» no Lily master node
» services can pick up where died ones left
» Zookeeper as service coordinator + lookup


architecture


High-level data model / easy API indexes

UI Framework SDK
(HUE) (HUE SDK)

Search
Dev2Dev
Workflow Scheduling Metadata tutoring,
(OOZIE) (oozie) (HIVE) integrated
deployment
and
Languages / enterprise
Data Compilers Fast usage metrics, support
Integration (PIG, HIVE) Read/Write analytics &
(FLUME, Access recommen-
SQOOP) (HBASE) dations
(PIG, HIVE)

Coordination
(ZOOKEEPER)

CDH

falling in love with Hbase
HBase = Google BigTable, open source
» datamodel with column families and cell versioning: ﬂexible, for sparse data
» ordered tables with range scans
» HDFS for blob storage
» Apache
» consistent!
» atomic single-row updates
» automatic scaling to large data sets
» fault-tolerance
» commodity hardware
» efﬁcient random access M/R for index regeneration


Lily Architecture
(deployment)


Lily Architecture
(components)


HBase indexing & RowLog Library
» building and querying » need for sync/async
indexes, GAE-style operations
» updating of secondary indexes
rowkey col col
content
A val3 foo6 (e.g. link tables)
table
B val2 foo7
» feeding of Indexer
(= indexes Lily-content into Solr)
rowkey col » not: transactions
order

index
table A val2-B
val3-A » need for distribution and
durability


WHERE?

www.lilyproject.org


lily enterprise

» adds tools:
» Redhat/Debian package repo
» cluster deploy tools
(based on Whirr)
» Administration UI
» + enterprise support


food for
(outer)thought


from analytics to recommendations
» shorten distance between transactional and analytical aspects
» (near) real-time analytics > Insights
» "people are buying this now"
» algorithmical feedback > data-backed feedback
» based on Insights > recommendations
» shorten feedback cycle
» store data + metadata + attention data in one data system
» basis for analytical queries
» single point of growth
» incremental insights


challenges ahead
» solving real-time aspects
» in a distributed environment
» terra-byte-sized snapshots & backups?
» build resilience into data architecture
(i.e. against operator malfunction)
» presenting bigdata insights: new UIs
» less static / report-driven
» more 'Minority Report'
» from near-real-time data to real-time decision systems


conclusion: 4 trends

» real-time is rapidly becoming really important
» agility with complex/dynamic information
» predictive analysis
» one store for operational data + analytics


Thank you !
for your attention
for your questions

» stevenn@outerthought.org

» @stevenn


Hadoop World 2011: Lily: Smart Data at Scale, Made Easy

More Related Content

Viewers also liked (18)

Similar to Hadoop World 2011: Lily: Smart Data at Scale, Made Easy (20)

More from Cloudera, Inc. (20)

Recently uploaded (20)

Hadoop World 2011: Lily: Smart Data at Scale, Made Easy