Chen li asterix db： 大数据处理开源平台
Download as PPTX, PDF1 like386 views
AsterixDB is an open source "Big Data Management System" (BDMS) that provides flexible data modeling, efficient query processing, and scalable analytics on large datasets. It uses a native storage layer built on LSM trees with indexing and transaction support. The Asterix Data Model (ADM) supports semistructured data types like records, lists, and bags. Queries are written in the Asterix Query Language (AQL) which supports features like spatial and temporal predicates. AsterixDB is being used for applications like social network analysis, education analytics, and more.
![create dataverse TinySocial;
use dataverse TinySocial;
create type MugshotUserType as {
id: int32,
alias: string,
name: string,
user-since: datetime,
address: {
street: string,
city: string,
state: string,
zip: string,
country: string
},
friend-ids: {{ int32 }},
employment: [EmploymentType]
}
ASTERIX Data Model (ADM)
13
create dataset MugshotUsers(MugshotUserType)
primary key id;
Highlights include:
• JSON++ based data model
• Rich type support (spatial, temporal, …)
• Records, lists, bags
• Open vs. closed types
create type EmploymentType as open {
organization-name: string,
start-date: date,
end-date: date?
}](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-14-320.jpg)
![create dataverse TinySocial;
use dataverse TinySocial;
create type MugshotUserType as {
id: int32,
alias: string,
name: string,
user-since: datetime,
address: {
street: string,
city: string,
state: string,
zip: string,
country: string
},
friend-ids: {{ int32 }},
employment: [EmploymentType]
}
create dataverse TinySocial;
use dataverse TinySocial;
create type MugshotUserType as {
id: int32
}
ASTERIX Data Model (ADM)
14
create dataset MugshotUsers(MugshotUserType)
primary key id;
Highlights include:
• JSON++ based data model
• Rich type support (spatial, temporal, …)
• Records, lists, bags
• Open vs. closed types
create type EmploymentType as open {
organization-name: string,
start-date: date,
end-date: date?
}](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-15-320.jpg)
![create dataverse TinySocial;
use dataverse TinySocial;
create type MugshotUserType as {
id: int32,
alias: string,
name: string,
user-since: datetime,
address: {
street: string,
city: string,
state: string,
zip: string,
country: string
},
friend-ids: {{ int32 }},
employment: [EmploymentType]
}
create dataverse TinySocial;
use dataverse TinySocial;
create type MugshotUserType as {
id: int32
}
create type MugshotMessageType
as closed {
message-id: int32,
author-id: int32,
timestamp: datetime,
in-response-to: int32?,
sender-location: point?,
tags: {{ string }},
message: string
}
ASTERIX Data Model (ADM)
15
create dataset MugshotUsers(MugshotUserType)
primary key id;
create dataset
MugshotMessages(MugshotMessageType)
primary key message-id;
Highlights include:
• JSON++ based data model
• Rich type support (spatial, temporal, …)
• Records, lists, bags
• Open vs. closed types
create type EmploymentType as open {
organization-name: string,
start-date: date,
end-date: date?
}](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-16-320.jpg)
![16
{ "id":1, "alias":"Margarita", "name":"MargaritaStoddard", "address”:{
"street":"234 Thomas Ave", "city":"San Hugo", "zip":"98765",
"state":"CA", "country":"USA" }
"user-since":datetime("2012-08-20T10:10:00"),
"friend-ids":{{ 2, 3, 6, 10 }}, "employment":[{
"organization-name":"Codetechno”, "start-date":date("2006-08-06") }] }
{ "id":2, "alias":"Isbel", "name":"IsbelDull", "address":{
"street":"345 James Ave", "city":"San Hugo", "zip":"98765”,
"state":"CA", "country":"USA" },
"user-since":datetime("2011-01-22T10:10:00"),
"friend-ids":{{ 1, 4 }}, "employment":[{
"organization-name":"Hexviafind”, "start-date":date("2010-04-27") }] }
{ "id":3, "alias":"Emory", "name":"EmoryUnk", "address":{
"street":"456 Jose Ave", "city":"San Hugo", "zip":"98765",
"state":"CA", "country":"USA" },
"user-since”: datetime("2012-07-10T10:10:00"),
"friend-ids":{{ 1, 5, 8, 9 }}, "employment”:[{
"organization-name":"geomedia”,
"start-date":date("2010-06-17"), "end-date":date("2010-01-26") }] }
...
Ex: MugshotUsers Data](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-17-320.jpg)
![Updates (and Transactions)
21
• Key-value store-
like transaction
semantics
• Insert/delete ops
with indexing
• Concurrency
control (locking)
• Crash recovery
• Backup/restore
• Ex: Add a new user to Mugshot.com:
insert into dataset MugshotUsers
( {
"id":11, "alias":"John", "name":"JohnDoe",
"address":{
"street":"789 Jane St", "city":"San Harry",
"zip":"98767", "state":"CA", "country":"USA"
},
"user-since":datetime("2010-08-15T08:10:00"),
"friend-ids":{ { 5, 9, 11 } },
"employment":[{
"organization-name":"Kongreen",
"start-date":date("20012-06-05")
}] } );](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-22-320.jpg)
![LSM-Based Filters
Memory
Disk
T1, T2, T3,
T4, T5, T6
T7, T8, T9,
T10, T11
T12, T13,
T14, T15
T16, T17
Oldest Component
[ T12, T15 ] [ T7, T11 ] [ T1, T6 ]
Intuition: Do NOT touch unneeded records
Idea: Utilize LSM partitioning to prune disk components
Q: Get all tweets > T14
26](https://image.slidesharecdn.com/chenli-asterixdb-160616080628/85/Chen-li-asterix-db-27-320.jpg)
Chen li asterix db： 大数据处理开源平台
- 1. and the Big Data Landscape Chen Li Information Systems Group CS Department UC Irvine 0
- 2. Big Data / Web Warehousing 1#AsterixDB So what went on – and why? What’s going on right now? What’s going on
- 3. Notes: • Storage manager per node • Upper layers orchestrate them • One way in/out: via the SQL door Big Data in the Database World • Enterprises needed to store and query historical business data (data warehouses) – 1980’s: Parallel database systems based on “shared- nothing” architectures (Gamma/GRACE, Teradata) – 2000’s: Netezza, Aster Data, DATAllegro, Greenplum, Vertica, ParAccel (“Big $”acquisitions!) • OLTP is another category (a source of Big Data) – 1980’s: Tandem’s NonStop SQL system 2
- 4. Big Data in the Systems World • Late 1990’s brought a need to index and query the rapidly exploding content of the Web – DB technology tried but failed (e.g., Inktomi) – Google, Yahoo! et al needed to do something • Google responded by laying a new foundation – Google File System (GFS) • OS-level byte stream files spanning 1000’s of machines • Three-way replication for fault-tolerance (availability) – MapReduce (MR) programming model • User functions: Map and Reduce (and optionally Combine) • “Parallel programming for dummies” – MR runtime does the heavy lifting via partitioned parallelism 3
- 5. Input Splits (distributed) Mapper Outputs Reducer Inputs Reducer Outputs (distributed) SHUFFLE PHASE (based on keys) (MapReduce: Word Count Example) 4 Partitioned Parallelism! . . . . . .
- 6. Soon a Star Was Born… • Yahoo!, Facebook, and friends read the papers – HDFS and Hadoop MapReduce now in wide use for indexing, clickstream analysis, log analysis, … • Higher-level languages subsequently developed – Pig (Yahoo!), Hive (Facebook), Jaql (IBM) • Key-value (“NoSQL”) stores are another category – Used to power scalable social sites, online games, … – BigTableHBase, DynamoCassandra, MongoDB, … 5 Notes: •Giant byte sequence files at the bottom •Map, sort, shuffle, reduce layer in middle •Possible storage layer in middle as well •Now at the top: HLL’s
- 7. Apache Pig (PigLatin) • Scripting language inspired by the relational algebra – Compiles down to a series of Hadoop MR jobs – Relational operators include LOAD, FOREACH, FILTER, GROUP, COGROUP, JOIN, ORDER BY, LIMIT, ... 6
- 8. Apache Hive (HiveQL) 7 • Query language inspired by an old favorite: SQL – Compiles down to a series of Hadoop MR jobs – Supports various HDFS file formats (text, columnar, ...) – Numerous contenders appearing that take a non-MR- based runtime approach (duh!) – these include Impala, Stinger, Spark SQL, ...
- 9. Other Up-and-Coming Platforms (I) 8 Distributed memory Input query 1 query 2 query 3 . . . one-time processing • Spark for in-memory cluster computing – for doing repetitive data analyses, iterative machine learning tasks, ... iter. 1 iter. 2 . . . Input iterative processing (Especially gaining traction for scaling Machine Learning)
- 10. Other Up-and-Coming Platforms (II) • Bulk Synchronous Programming (BSP) platforms, e.g., Pregel, Giraph, GraphLab, ..., for Big Graph analytics 9 (“Big” is the platform’s concern) “Think Like a Vertex” – Receive messages – Update state – Send messages • Quite a few BSP-based platforms available – Pregel (Google) – Giraph (Facebook, LinkedIn, Twitter, Yahoo!, ...) – Hama (Sogou, Korea Telecomm, ...) – Distributed GraphLab (CMU, Washington) – GraphX (Berkeley) – Pregelix (UCI) – ...
- 11. No Shortage of “NoSQL” Big Data Analysis Platforms... 10 Query/Scripting Language High-Level API Compiler/Optimizer Low-Level API Execution Engine Resource Management Data Store SQL SQL Dataflow Processor Relational Row/ Column Storage SCOPE SCOPE Dryad DryadLINQ TidyFS Quincy AQL Algebricks Hyracks Hyracks LSM Storage PigLatin Jaql Cascading Pig Jaql Tez MapReduce HBase HDFS Hadoop MapReduce Google MapReduce Spark RDDs Spark Mesos Meteor Sopremo Java/Scala Nephele YARN Sawzall Dremel FlumeJava Dremel Dremel Bigtable GFS Omega Cosmos PACT Cascading FlumeJava
- 12. 11 (Pig) Also: Today’s Big Data Tangle
- 13. AsterixDB: “One Size Fits a Bunch” 12 Semistructured Data Management Parallel Database Systems World of Hadoop & Friends BDMS Desiderata: • Flexible data model • Efficient runtime • Full query capability • Cost proportional to task at hand (!) • Designed for continuous data ingestion • Support today’s “Big Data data types” • • •
- 14. create dataverse TinySocial; use dataverse TinySocial; create type MugshotUserType as { id: int32, alias: string, name: string, user-since: datetime, address: { street: string, city: string, state: string, zip: string, country: string }, friend-ids: {{ int32 }}, employment: [EmploymentType] } ASTERIX Data Model (ADM) 13 create dataset MugshotUsers(MugshotUserType) primary key id; Highlights include: • JSON++ based data model • Rich type support (spatial, temporal, …) • Records, lists, bags • Open vs. closed types create type EmploymentType as open { organization-name: string, start-date: date, end-date: date? }
- 15. create dataverse TinySocial; use dataverse TinySocial; create type MugshotUserType as { id: int32, alias: string, name: string, user-since: datetime, address: { street: string, city: string, state: string, zip: string, country: string }, friend-ids: {{ int32 }}, employment: [EmploymentType] } create dataverse TinySocial; use dataverse TinySocial; create type MugshotUserType as { id: int32 } ASTERIX Data Model (ADM) 14 create dataset MugshotUsers(MugshotUserType) primary key id; Highlights include: • JSON++ based data model • Rich type support (spatial, temporal, …) • Records, lists, bags • Open vs. closed types create type EmploymentType as open { organization-name: string, start-date: date, end-date: date? }
- 16. create dataverse TinySocial; use dataverse TinySocial; create type MugshotUserType as { id: int32, alias: string, name: string, user-since: datetime, address: { street: string, city: string, state: string, zip: string, country: string }, friend-ids: {{ int32 }}, employment: [EmploymentType] } create dataverse TinySocial; use dataverse TinySocial; create type MugshotUserType as { id: int32 } create type MugshotMessageType as closed { message-id: int32, author-id: int32, timestamp: datetime, in-response-to: int32?, sender-location: point?, tags: {{ string }}, message: string } ASTERIX Data Model (ADM) 15 create dataset MugshotUsers(MugshotUserType) primary key id; create dataset MugshotMessages(MugshotMessageType) primary key message-id; Highlights include: • JSON++ based data model • Rich type support (spatial, temporal, …) • Records, lists, bags • Open vs. closed types create type EmploymentType as open { organization-name: string, start-date: date, end-date: date? }
- 17. 16 { "id":1, "alias":"Margarita", "name":"MargaritaStoddard", "address”:{ "street":"234 Thomas Ave", "city":"San Hugo", "zip":"98765", "state":"CA", "country":"USA" } "user-since":datetime("2012-08-20T10:10:00"), "friend-ids":{{ 2, 3, 6, 10 }}, "employment":[{ "organization-name":"Codetechno”, "start-date":date("2006-08-06") }] } { "id":2, "alias":"Isbel", "name":"IsbelDull", "address":{ "street":"345 James Ave", "city":"San Hugo", "zip":"98765”, "state":"CA", "country":"USA" }, "user-since":datetime("2011-01-22T10:10:00"), "friend-ids":{{ 1, 4 }}, "employment":[{ "organization-name":"Hexviafind”, "start-date":date("2010-04-27") }] } { "id":3, "alias":"Emory", "name":"EmoryUnk", "address":{ "street":"456 Jose Ave", "city":"San Hugo", "zip":"98765", "state":"CA", "country":"USA" }, "user-since”: datetime("2012-07-10T10:10:00"), "friend-ids":{{ 1, 5, 8, 9 }}, "employment”:[{ "organization-name":"geomedia”, "start-date":date("2010-06-17"), "end-date":date("2010-01-26") }] } ... Ex: MugshotUsers Data
- 18. create index msUserSinceIdx on MugshotUsers(user-since); create index msTimestampIdx on MugshotMessages(timestamp); create index msAuthorIdx on MugshotMessages(author-id) type btree; create index msSenderLocIndex on MugshotMessages(sender-location) type rtree; create index msMessageIdx on MugshotMessages(message) type keyword; create type AccessLogType as closed { ip: string, time: string, user: string, verb: string, path: string, stat: int32, size: int32 }; create external dataset AccessLog(AccessLogType) using localfs (("path"="{hostname}://{path}"), ("format"="delimited-text"), ("delimiter"="|")); create feed socket_feed using socket_adaptor (("sockets"="{address}:{port}"), ("addressType"="IP"), ("type-name"="MugshotMessageType"), ("format"="adm")); connect feed socket_feed to dataset MugshotMessages; Other DDL Features 17 External data highlights: • Common HDFS file formats + indexing • Feed adaptors for sources like Twitter
- 19. 18 • Ex: List the user name and messages sent by those users who joined the Mugshot social network in a certain time window: from $user in dataset MugshotUsers where $user.user-since >= datetime('2010-07-22T00:00:00') and $user.user-since <= datetime('2012-07-29T23:59:59') select { "uname" : $user.name, "messages" : from $message in dataset MugshotMessages where $message.author-id = $user.id select $message.message }; 18 ASTERIX Query Language (AQL)
- 20. AQL (cont.) 19 • Ex: Identify active users and group/count them by country: with $end := current-datetime( ) with $start := $end - duration("P30D") from $user in dataset MugshotUsers where some $logrecord in dataset AccessLog satisfies $user.alias = $logrecord.user and datetime($logrecord.time) >= $start and datetime($logrecord.time) <= $end group by $country := $user.address.country with $user select { "country" : $country, "active users" : count($user) } AQL highlights: • Lots of other features (see website!) • Spatial predicates and aggregation • Set-similarity (fuzzy) matching • And plans for more…
- 21. Fuzzy Queries in AQL 20 • Ex: Find Tweets with similar content: for $tweet1 in dataset('TweetMessages') for $tweet2 in dataset('TweetMessages') where $tweet1.tweetid != $tweet2.tweetid and $tweet1.message-text ~= $tweet2.message-text return { "tweet1-text": $tweet1.message-text, "tweet2-text": $tweet2.message-text } • Or: Find Tweets about similar topics: for $tweet1 in dataset('TweetMessages') for $tweet2 in dataset('TweetMessages') where $tweet1.tweetid != $tweet2.tweetid and $tweet1.referred-topics ~= $tweet2.referred-topics return { "tweet1-text": $tweet1.message-text, "tweet2-text": $tweet2.message-text }
- 22. Updates (and Transactions) 21 • Key-value store- like transaction semantics • Insert/delete ops with indexing • Concurrency control (locking) • Crash recovery • Backup/restore • Ex: Add a new user to Mugshot.com: insert into dataset MugshotUsers ( { "id":11, "alias":"John", "name":"JohnDoe", "address":{ "street":"789 Jane St", "city":"San Harry", "zip":"98767", "state":"CA", "country":"USA" }, "user-since":datetime("2010-08-15T08:10:00"), "friend-ids":{ { 5, 9, 11 } }, "employment":[{ "organization-name":"Kongreen", "start-date":date("20012-06-05") }] } );
- 24. ASTERIX Software Stack 23 Hivesterix Apache VXQuery Algebricks Algebra Layer M/R Layer Pregelix Hyracks Data-Parallel Platform Hyracks Job Hadoop M/R JobPregel Job AQL HiveQL XQuery AsterixDB
- 25. Native Storage Management Transaction Manager Transaction Sub-System Recovery Manager Lock Manager Log Manager IO Scheduler Disk 1 Disk n Memory Buffer Cache In-Memory Components Working Memory Datasets Manager ( ) + 24
- 26. LSM-Based Storage + Indexing Memory Disk Sequential writes to disk Periodically merge disk trees 25
- 27. LSM-Based Filters Memory Disk T1, T2, T3, T4, T5, T6 T7, T8, T9, T10, T11 T12, T13, T14, T15 T16, T17 Oldest Component [ T12, T15 ] [ T7, T11 ] [ T1, T6 ] Intuition: Do NOT touch unneeded records Idea: Utilize LSM partitioning to prune disk components Q: Get all tweets > T14 26
- 28. • Recent/projected use case areas include – Behavioral science (at UCI) – Social data analytics – Cell phone event analytics – Education (MOOC analytics) – Power usage monitoring – Public health (joint effort with UCLA) – Cluster management log analytics 27 Some Example Use Cases
- 29. Behavioral Science (HCI) • First study to use logging and biosensors to measure stress and ICT use of college students in their real world environment (Gloria Mark, UCI Informatics) – Focus: Multitasking and stress among “Millennials” • Multiple data channels – Computer logging – Heart rate monitors – Daily surveys – General survey – Exit interview 28 Learnings for AsterixDB: • Nature of their analyses • Extended binning support • Data format(s) in and out • Bugs and pain points
- 30. Social Data Analysis (Based on 2 pilots) #AsterixDB 29 Learnings for AsterixDB: • Nature of their analyses • Real vs. synthetic data • Parallelism (grouping) • Avoiding materialization • Bugs and pain points The underlying AQL query is: use dataverse twitter; for $t in dataset TweetMessagesShifted let $region := create-rectangle(create-point(…, …), create-point(…, …)) let $keyword := "mind-blowing" where spatial-intersect($t.sender-location, $region) and $t.send-time > datetime("2012-01-02T00:00:00Z”) and $t.send-time < datetime("2012-12-31T23:59:59Z”) and contains($t.message-text, $keyword) group by $c := spatial-cell($t.sender-location, create-point(…), 3.0, 3.0) with $t return { "cell” : $c, "count”: count($t) }
- 31. Current Status • 4 year initial NSF project (250+ KLOC @ UCI/UCR) • AsterixDB BDMS is here! (Shared on June 6th, 2013) – Semistructured “NoSQL” style data model – Declarative (parallel) queries, inserts, deletes, … – LSM-based storage/indexes (primary & secondary) – Internal and external datasets both supported – Rich set of data types (including text, time, location) – Fuzzy and spatial query processing – NoSQL-like transactions (for inserts/deletes) – Data feeds and external indexes in next release • Performance competitive (at least!) with a popular parallel RDBMS, MongoDB, and Hive (see papers) • Now in Apache incubation mode! 30
- 32. For More Info AsterixDB project page: http://asterixdb.ics.uci.edu Open source code base: • ASTERIX: http://code.google.com/p/asterixdb/ • Hyracks: http://code.google.com/p/hyracks • (Pregelix: http://hyracks.org/projects/pregelix/) 31