SlideShare a Scribd company logo

Survey of Accumulo Techniques for Indexing Data

Download as PPTX, PDF
Donald Miner, profile picture
Donald Miner

The document provides an overview of Apache Accumulo, a high-performance, distributed key/value store designed for scalable data storage and retrieval. It describes various indexing techniques, performance benchmarks, and compares its capabilities with HBase, highlighting features like faster write throughput and customizable server-side processing via its iterator framework. Additional functionalities such as visibility controls, locality groups, and encryption options are also discussed.

Technology
1 of 49
Downloaded 60 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
SURVEY OF ACCUMULO TECHNIQUES FOR INDEXING DATA Donald Miner @donaldpminer January 18th, 2015
INTRODUCTION TO ACCUMULO
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Adelaide Bartkowski Alyssa Files Beatriz Palmore Cecilia Ours Craig Avalos Dianna Lapointe Erma Davis Fermina Smead Garrett Harsh Gaylene Sherry Gilberto Pardue Hui Nodal Janell Tomita Jannette Betters Jeana Delk Madlyn Radke Peggie Allis Rhona Zygmont Tran Degarmo Wilhelmina Papp
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Janell Tomita Jannette Betters Jeana Delk Madlyn Radke Peggie Allis Rhona Zygmont Tran Degarmo Wilhelmina Papp Adelaide Bartkowski Alyssa Files Beatriz Palmore Cecilia Ours Craig Avalos Dianna Lapointe Erma Davis Fermina Smead Garrett Harsh Gaylene Sherry Gilberto Pardue Hui Nodal -inf to D E to H J to +inf
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Accumulo Master TabletServer TabletServer TabletServer ZooKeeper
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. KEY VALUE Adelaide Bartkowski 91294124 Alyssa Files 491294 Beatriz Palmore 4124124124 Cecilia Ours 419120 Craig Avalos 940124 Dianna Lapointe 4921 Erma Davis 050194 Fermina Smead 10024599949 Garrett Harsh 140095931 Gaylene Sherry 914815 Gilberto Pardue 412414124124 Hui Nodal 962195192 Janell Tomita 12121 Jannette Betters 9192012 Jeana Delk 9120150 Madlyn Radke 4921 Peggie Allis 944944 Rhona Zygmont 123103 Tran Degarmo 9499494 Wilhelmina Papp 11221 Lookup “Garret Harsh” FAST Lookup “4921” SLOW
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. MIT Lincoln Lab study: 100 Million inserts per second using Accumulo http://arxiv.org/ftp/arxiv/papers/1406/1406.4923.pdf http://sqrrl.com/media/Accumulo-Benchmark-10312013-1.pdf Booz Allen Hamilton study: 942 tablet servers, 7.56 trillion entries, 408TB, 26 hours 94MB/Sec, 15TB/hr, 80million inserts per second 11 tablet servers went down with no interruption Showed linear scalability for write throughput 22,000 queries per second
HBase vs. Accumulo • Subtle yet important differences in visibility implementation • Coprocessors vs. Iterators • Accumulo has faster write throughput* • HBase’s reads are faster* • HBase has more ecosystem integration • Accumulo can shift around column families and locality groups after the fact • Accumulo has shown to work with no problems at 1,000 nodes (BAH paper). Facebook and others run a “cell” design for HBase. Largest clusters in the hundreds*. * We believeDisclaimer: I am biased
Column Visibility Syntax Label Description A & B Both ‘A’ and ‘B’ are required A | B Either ‘A’ or ‘B’ is required A & (C | B) ‘A’ and ‘C’ or ‘A’ and ‘B’ is required A | (B & C) ‘A’ or ‘B’ and ‘C’ is required (A | B) & (C & D) ? A & (B & (C | D)) ? Patient has schizophrenia: insurer | MD & psych Patient has stomach ulcers: insurer | doctor Patient has cavity: insurer | dentist Patient has consent for general anesthesia: surgeon
More cool features • Iterator framework: customizable server-side processing • Constraints: user-defined Java functions that allow or prevent new writes based on a condition • Large rows: no limit on data stored in a row • MapReduce InputFormats • Thrift proxy: access Accumulo through Ruby, Python, … • Monitor page: shows performance, status, errors, more • Locality groups: group column families together on disk for performance tuning (changeable later) • On-HDFS at rest encryption (work in progress) • Table import and export
Scalability & Performance • Multiple HDFS volumes: Accumulo can use multiple NameNodes to store its data • Master stores metadata in an Accumulo table • Native in-memory map: data is first written into a buffer written in C++, outside of Java • Relative encoding: consecutive keys with the same values are flagged instead of rewritten • Scan pipelines: stages of the read path are parallelized into separate threads • Caching: data recently scanned is cached
HOW IT WORKS
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Lookup key
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Collection of data that is kept together
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P What the data is
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Who can see the data
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P When the data was created
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P UNIQUENESS
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P SORTED
Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Some piece of information
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo
Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo Text rowID = new Text(”don"); Text colFam = new Text(”info"); Text colQual = new Text(”picture"); ColumnVisibility colVis = new ColumnVisibility("public"); long timestamp = System.currentTimeMillis(); Value value = new Value(MyPictureObj.getBytes()); Mutation mutation = new Mutation(rowID); mutation.put(colFam, colQual, colVis, timestamp, value); BatchWriterConfig config = new BatchWriterConfig(); BatchWriter writer = conn.createBatchWriter(”usertable", config) writer.add(mutation); writer.close();
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Family Visibilities don-don info public Reading data
Range Family Visibilities don-don info public Reading data Authorizations auths = new Authorizations("public”); Scanner scan = conn.createScanner(”usertable", auths); scan.setRange(new Range(”don",”don")); scan.fetchFamily(”info"); for(Entry<Key,Value> entry : scan) { String row = entry.getKey().getRow(); Value value = entry.getValue(); }
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Family Visibilities don-don info public, user, tech Reading data
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Visibilities don-don public, user, tech Reading data
Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Visibilities d-e public, user, tech Reading data Scan
INDEXING & TABLE DESIGN
Basic Structured Data Row ID Column Family Column Qualifier Column Visibility Timestam p Value bob attribute height public Jun 2012 5’11” bob attribute surname public Jul 2013 doe bob insurance dental private Sep 2009 MetLife jane attribute bloodType public Jul 2011 ab- jane attribute surname public Aug 2013 doe jane contact cellPhone public Dec 2010 (808) 345- 9876 jane insurance vision private Jan 2008 VSP john allergy major private Feb 1988 amoxicillin john attribute weight public Sep 2013 180 john contact homeAddr public Mar 2003 34 Baker LN
Basic Structured Data Row ID Column Family Column Qualifier Column Visibility Timestam p Value bob attribute height public Jun 2012 5’11” bob attribute surname public Jul 2013 doe bob insurance dental private Sep 2009 MetLife jane attribute bloodType public Jul 2011 ab- jane attribute surname public Aug 2013 doe jane contact cellPhone public Dec 2010 (808) 345- 9876 jane insurance vision private Jan 2008 VSP john allergy major private Feb 1988 amoxicillin john attribute weight public Sep 2013 180 john contact homeAddr public Mar 2003 34 Baker LN
Indexing Everything Row ID Column Fam Column Qual Visibility Time value index Column Fam Column Qual:Row ID Visibility Time - to Column Fam Column Qual:Row ID Visibility Time - values Column Fam Column Qual:Row ID Visibility Time - Event Table Index Table
Index Table Row ID Column Family Column Qualifier Column Visibility Timestam p Value (808) 345- 9876 contact cellPhone:jane public Dec 2010 - 180 attribute weight:john public Sep 2013 - 34 Baker LN contact homeAddr:john public Mar 2003 - 5’11” attribute height:bob public Jun 2012 - MetLife insuranc e dental:bob private Sep 2009 - VSP insuranc e vision:jane private Jan 2008 - ab- attribute bloodType:jane public Jul 2011 - amoxicillin allergy major:john private Feb 1988 - doe attribute surname:bob public Jul 2013 - doe attribute surname:jane public Aug 2013 -
Data Lake PATIENTS MEDICINES DOCTORS INDEX
Data Lake PATIENTS MEDICINES DOCTORS INDEX Tell me everything you know of amoxicillin amoxicillin
Data Lake PATIENTS DISEASES DOCTORS INDEX amoxicillin bob:allergy:amoxicillin larry:takes:amoxicillin Stomach ulcer: treatment:amoxicillin smith: prescribed:amoxicillinInfection: treatment:amoxicillin Diarrhea: side effect:amoxicillin Visibility labels help converge data sources but still protect who can see them.
Graphs a bc d e a b c d e a - 1 b 1 - c - 1 d 1 1 - 1 e - Start Nodes EndNodes Row ID Column Family Column Qualifier Value a edge b 1 a edge d 1 c edge a 1 c edge d 1 d edge c 1 e edge d 1 • Random walk • Neighborhoods • Traversals Each edge can have a visibility label!
Term-Partitioned Index Tablet Server 1 Row ID Column Family Value baseball document docid_3 baseball document docid_2 bat document docid_2 Tablet Server 2 Row ID Column Family Value football document docid_1 football document docid_3 glove document docid_1 Tablet Server 3 Row ID Column Family Value nba document docid_1 shoes document docid_1 soccer document docid_3 RESULTS: [docid_2, docid_3] RESULTS: [docid_1, docid_3] RESULTS: [docid_3] Tablet Server knows about the terms “baseball” Tablet Server knows about the terms “football” Tablet Server knows about the terms “soccer” Query: “baseball” AND “football” AND “soccer” Client Client-side Set Intersection [docid_2, docid_3] [docid_1, docid_3] [docid_3] Visibility labels allow protected search Iterators can maintain stats about docs
Geospacial Indexing: Grid Squares
Geospacial Indexing: Z-Order Curve 33.333W, 55.555N = 3535.353535 3535.353535 is the rowkey
Temporal Indexing Row ID Column Family Column Qualifier Value Router37 2014-12 1418624102 cold Router37 2015-01 1421633979 cold Router37 2015-01 1421634319 hot Router37 2015-01 1421635001 cold Server92 2014-12 1418555102 cold Server92 2014-12 1418556999 hot Server92 2014-12 1418651002 cold Server92 2014-12 1418756987 hot Server92 2014-12 1418853304 cold Server98 2014-12 1418555104 cold Server98 2015-01 1421633319 cold Note: Dynamically adding column families
Resources Apache Accumulo website accumulo.apache.org Accumulo Summit 2014 accumulosummit.com slideshare.net/AccumuloSummit Accumulo Summit 2015 End of April! accumulosummit.com

More Related Content

KEY
Introduction to Riak - Red Dirt Ruby Conf Training
Sean Cribbs
 
KEY
Introducing Riak
Kevin Smith
 
KEY
Riak Training Session — Surge 2011
DstroyAllModels
 
PPTX
Coding with Riak (from Velocity 2015)
Basho Technologies
 
PDF
Relational Databases to Riak
Basho Technologies
 
PDF
Spark Cassandra Connector Dataframes
Russell Spitzer
 
PDF
Apache Spark and DataStax Enablement
Vincent Poncet
 
DOCX
Apache Drill with Oracle, Hive and HBase
Nag Arvind Gudiseva
 
Introduction to Riak - Red Dirt Ruby Conf Training
Sean Cribbs
 
Introducing Riak
Kevin Smith
 
Riak Training Session — Surge 2011
DstroyAllModels
 
Coding with Riak (from Velocity 2015)
Basho Technologies
 
Relational Databases to Riak
Basho Technologies
 
Spark Cassandra Connector Dataframes
Russell Spitzer
 
Apache Spark and DataStax Enablement
Vincent Poncet
 
Apache Drill with Oracle, Hive and HBase
Nag Arvind Gudiseva
 

What's hot (14)

KEY
Embrace NoSQL and Eventual Consistency with Ripple
Sean Cribbs
 
PPTX
Maximum Overdrive: Tuning the Spark Cassandra Connector
Russell Spitzer
 
PDF
Cassandra Day Atlanta 2015: Data Modeling In-Depth: A Time Series Example
DataStax Academy
 
PDF
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
PDF
Cassandra and Spark: Optimizing for Data Locality-(Russell Spitzer, DataStax)
Spark Summit
 
PDF
Spark and Cassandra 2 Fast 2 Furious
Russell Spitzer
 
PPTX
SQL-on-Hadoop with Apache Drill
MapR Technologies
 
PPTX
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
randyguck
 
PDF
Time series with Apache Cassandra - Long version
Patrick McFadin
 
KEY
Introduction to Riak and Ripple (KC.rb)
Sean Cribbs
 
PDF
Analytics with Cassandra & Spark
Matthias Niehoff
 
PDF
Escape From Hadoop: Spark One Liners for C* Ops
Russell Spitzer
 
PDF
Data Exploration with Apache Drill: Day 1
Charles Givre
 
KEY
Riak with node.js
Sean Cribbs
 
Embrace NoSQL and Eventual Consistency with Ripple
Sean Cribbs
 
Maximum Overdrive: Tuning the Spark Cassandra Connector
Russell Spitzer
 
Cassandra Day Atlanta 2015: Data Modeling In-Depth: A Time Series Example
DataStax Academy
 
Big data analytics with Spark & Cassandra
Matthias Niehoff
 
Cassandra and Spark: Optimizing for Data Locality-(Russell Spitzer, DataStax)
Spark Summit
 
Spark and Cassandra 2 Fast 2 Furious
Russell Spitzer
 
SQL-on-Hadoop with Apache Drill
MapR Technologies
 
Strata Presentation: One Billion Objects in 2GB: Big Data Analytics on Small ...
randyguck
 
Time series with Apache Cassandra - Long version
Patrick McFadin
 
Introduction to Riak and Ripple (KC.rb)
Sean Cribbs
 
Analytics with Cassandra & Spark
Matthias Niehoff
 
Escape From Hadoop: Spark One Liners for C* Ops
Russell Spitzer
 
Data Exploration with Apache Drill: Day 1
Charles Givre
 
Riak with node.js
Sean Cribbs
 
Ad

Similar to Survey of Accumulo Techniques for Indexing Data (20)

PDF
Accumulo design
Koverse, Inc.
 
PDF
Accumulo design
scsorensen
 
PPTX
Adam Fuchs' Accumulo Talk at NoSQL Now! 2013
Sqrrl
 
PDF
Introduction to Accumulo
Mario Pastorelli
 
PDF
Accumulo Tutorial — Up and Running (or at Least Walking) in 90 Minutes
Accumulo Summit
 
PPTX
Accumulo meetup 20130109
Sqrrl
 
PPTX
An Introduction to Accumulo
Donald Miner
 
PPTX
Introduction to Apache Accumulo
busbey
 
PDF
Sqrrl October Webinar: Data Modeling and Indexing
Sqrrl
 
PDF
Apache Accumulo and the Data Lake
Aaron Cordova
 
PPTX
HBase and Accumulo | Washington DC Hadoop User Group
Cloudera, Inc.
 
PDF
Sqrrl real time_big_data_20130411
Sqrrl
 
PDF
What's Next for Google's BigTable
Sqrrl
 
PPTX
An Exploration of 3 Very Different ML Solutions Running on Accumulo
Accumulo Summit
 
PPTX
01 hbase
Subhas Kumar Ghosh
 
PDF
Accumulo Summit 2015: Event-Driven Big Data with Accumulo - Leveraging Big Da...
Accumulo Summit
 
PPTX
Sqrrl and Accumulo
John Dougherty
 
ODP
Introduciton to Apache Cassandra for Java Developers (JavaOne)
zznate
 
PPTX
Accumulo: A Quick Introduction
James Salter
 
PDF
Hugaccumulo 121018192044-phpapp02
Sqrrl
 
Accumulo design
Koverse, Inc.
 
Accumulo design
scsorensen
 
Adam Fuchs' Accumulo Talk at NoSQL Now! 2013
Sqrrl
 
Introduction to Accumulo
Mario Pastorelli
 
Accumulo Tutorial — Up and Running (or at Least Walking) in 90 Minutes
Accumulo Summit
 
Accumulo meetup 20130109
Sqrrl
 
An Introduction to Accumulo
Donald Miner
 
Introduction to Apache Accumulo
busbey
 
Sqrrl October Webinar: Data Modeling and Indexing
Sqrrl
 
Apache Accumulo and the Data Lake
Aaron Cordova
 
HBase and Accumulo | Washington DC Hadoop User Group
Cloudera, Inc.
 
Sqrrl real time_big_data_20130411
Sqrrl
 
What's Next for Google's BigTable
Sqrrl
 
An Exploration of 3 Very Different ML Solutions Running on Accumulo
Accumulo Summit
 
01 hbase
Subhas Kumar Ghosh
 
Accumulo Summit 2015: Event-Driven Big Data with Accumulo - Leveraging Big Da...
Accumulo Summit
 
Sqrrl and Accumulo
John Dougherty
 
Introduciton to Apache Cassandra for Java Developers (JavaOne)
zznate
 
Accumulo: A Quick Introduction
James Salter
 
Hugaccumulo 121018192044-phpapp02
Sqrrl
 
Ad

More from Donald Miner (10)

PPTX
Machine Learning Vital Signs
Donald Miner
 
PPTX
10 concepts the enterprise decision maker needs to understand about Hadoop
Donald Miner
 
PPTX
EDHREC @ Data Science MD
Donald Miner
 
PPTX
Hadoop with Python
Donald Miner
 
PPTX
SQL on Accumulo
Donald Miner
 
PPTX
Data, The New Currency
Donald Miner
 
PPTX
The Amino Analytical Framework - Leveraging Accumulo to the Fullest
Donald Miner
 
PPTX
Hadoop for Data Science
Donald Miner
 
PPTX
MapReduce Design Patterns
Donald Miner
 
PPTX
Data science and Hadoop
Donald Miner
 
Machine Learning Vital Signs
Donald Miner
 
10 concepts the enterprise decision maker needs to understand about Hadoop
Donald Miner
 
EDHREC @ Data Science MD
Donald Miner
 
Hadoop with Python
Donald Miner
 
SQL on Accumulo
Donald Miner
 
Data, The New Currency
Donald Miner
 
The Amino Analytical Framework - Leveraging Accumulo to the Fullest
Donald Miner
 
Hadoop for Data Science
Donald Miner
 
MapReduce Design Patterns
Donald Miner
 
Data science and Hadoop
Donald Miner
 

Recently uploaded (20)

PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Approach and Philosophy of On baking technology
30anthuong17
 
PPTX
Spectroscopy.pptx food analysis technology
nawahassan45
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PDF
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
PDF
KodekX | Application Modernization Development
KodekX
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
PPTX
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
PPTX
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Approach and Philosophy of On baking technology
30anthuong17
 
Spectroscopy.pptx food analysis technology
nawahassan45
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Chapter 3 Spatial Domain Image Processing.pdf
Getnet Tigabie Askale -(GM)
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Review of recent advances in non-invasive hemoglobin estimation
IAESIJAI
 
KodekX | Application Modernization Development
KodekX
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
SynapseIndia
 
Detection-First SIEM: Rule Types, Dashboards, and Threat-Informed Strategy
ReZa AdineH
 
Effective Security Operations Center (SOC) A Modern, Strategic, and Threat-In...
ReZa AdineH
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Diabetes mellitus diagnosis method based random forest with bat algorithm
IAESIJAI
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 

Survey of Accumulo Techniques for Indexing Data

  • 1. SURVEY OF ACCUMULO TECHNIQUES FOR INDEXING DATA Donald Miner @donaldpminer January 18th, 2015
  • 3. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 4. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Adelaide Bartkowski Alyssa Files Beatriz Palmore Cecilia Ours Craig Avalos Dianna Lapointe Erma Davis Fermina Smead Garrett Harsh Gaylene Sherry Gilberto Pardue Hui Nodal Janell Tomita Jannette Betters Jeana Delk Madlyn Radke Peggie Allis Rhona Zygmont Tran Degarmo Wilhelmina Papp
  • 5. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Janell Tomita Jannette Betters Jeana Delk Madlyn Radke Peggie Allis Rhona Zygmont Tran Degarmo Wilhelmina Papp Adelaide Bartkowski Alyssa Files Beatriz Palmore Cecilia Ours Craig Avalos Dianna Lapointe Erma Davis Fermina Smead Garrett Harsh Gaylene Sherry Gilberto Pardue Hui Nodal -inf to D E to H J to +inf
  • 6. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. Accumulo Master TabletServer TabletServer TabletServer ZooKeeper
  • 7. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. KEY VALUE Adelaide Bartkowski 91294124 Alyssa Files 491294 Beatriz Palmore 4124124124 Cecilia Ours 419120 Craig Avalos 940124 Dianna Lapointe 4921 Erma Davis 050194 Fermina Smead 10024599949 Garrett Harsh 140095931 Gaylene Sherry 914815 Gilberto Pardue 412414124124 Hui Nodal 962195192 Janell Tomita 12121 Jannette Betters 9192012 Jeana Delk 9120150 Madlyn Radke 4921 Peggie Allis 944944 Rhona Zygmont 123103 Tran Degarmo 9499494 Wilhelmina Papp 11221 Lookup “Garret Harsh” FAST Lookup “4921” SLOW
  • 8. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 9. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 10. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 11. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 12. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system.
  • 13. The Apache Accumulo sorted, distributed key/value store is a robust, scalable, high performance data storage and retrieval system. MIT Lincoln Lab study: 100 Million inserts per second using Accumulo http://arxiv.org/ftp/arxiv/papers/1406/1406.4923.pdf http://sqrrl.com/media/Accumulo-Benchmark-10312013-1.pdf Booz Allen Hamilton study: 942 tablet servers, 7.56 trillion entries, 408TB, 26 hours 94MB/Sec, 15TB/hr, 80million inserts per second 11 tablet servers went down with no interruption Showed linear scalability for write throughput 22,000 queries per second
  • 14. HBase vs. Accumulo • Subtle yet important differences in visibility implementation • Coprocessors vs. Iterators • Accumulo has faster write throughput* • HBase’s reads are faster* • HBase has more ecosystem integration • Accumulo can shift around column families and locality groups after the fact • Accumulo has shown to work with no problems at 1,000 nodes (BAH paper). Facebook and others run a “cell” design for HBase. Largest clusters in the hundreds*. * We believeDisclaimer: I am biased
  • 15. Column Visibility Syntax Label Description A & B Both ‘A’ and ‘B’ are required A | B Either ‘A’ or ‘B’ is required A & (C | B) ‘A’ and ‘C’ or ‘A’ and ‘B’ is required A | (B & C) ‘A’ or ‘B’ and ‘C’ is required (A | B) & (C & D) ? A & (B & (C | D)) ? Patient has schizophrenia: insurer | MD & psych Patient has stomach ulcers: insurer | doctor Patient has cavity: insurer | dentist Patient has consent for general anesthesia: surgeon
  • 16. More cool features • Iterator framework: customizable server-side processing • Constraints: user-defined Java functions that allow or prevent new writes based on a condition • Large rows: no limit on data stored in a row • MapReduce InputFormats • Thrift proxy: access Accumulo through Ruby, Python, … • Monitor page: shows performance, status, errors, more • Locality groups: group column families together on disk for performance tuning (changeable later) • On-HDFS at rest encryption (work in progress) • Table import and export
  • 17. Scalability & Performance • Multiple HDFS volumes: Accumulo can use multiple NameNodes to store its data • Master stores metadata in an Accumulo table • Native in-memory map: data is first written into a buffer written in C++, outside of Java • Relative encoding: consecutive keys with the same values are flagged instead of rewritten • Scan pipelines: stages of the read path are parallelized into separate threads • Caching: data recently scanned is cached
  • 19. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P
  • 20. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Lookup key
  • 21. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Collection of data that is kept together
  • 22. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P What the data is
  • 23. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Who can see the data
  • 24. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P When the data was created
  • 25. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P UNIQUENESS
  • 26. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P SORTED
  • 27. Data Model KEY ROW ID COLUMN FAMILY QUALIFIER VISIBILITY VALUE Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … TIMESTAM P Some piece of information
  • 28. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info SSN private 12314514 123-45-6789 erica … … … … … Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo
  • 29. Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo Text rowID = new Text(”don"); Text colFam = new Text(”info"); Text colQual = new Text(”picture"); ColumnVisibility colVis = new ColumnVisibility("public"); long timestamp = System.currentTimeMillis(); Value value = new Value(MyPictureObj.getBytes()); Mutation mutation = new Mutation(rowID); mutation.put(colFam, colQual, colVis, timestamp, value); BatchWriterConfig config = new BatchWriterConfig(); BatchWriter writer = conn.createBatchWriter(”usertable", config) writer.add(mutation); writer.close();
  • 30. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Row ID Family Qualifier Visibility Timestamp Value don info picture public 13119103 dd3ae1d3b951a33f… Writing data into Accumulo
  • 31. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Family Visibilities don-don info public Reading data
  • 32. Range Family Visibilities don-don info public Reading data Authorizations auths = new Authorizations("public”); Scanner scan = conn.createScanner(”usertable", auths); scan.setRange(new Range(”don",”don")); scan.fetchFamily(”info"); for(Entry<Key,Value> entry : scan) { String row = entry.getKey().getRow(); Value value = entry.getValue(); }
  • 33. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Family Visibilities don-don info public, user, tech Reading data
  • 34. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Visibilities don-don public, user, tech Reading data
  • 35. Row ID Family Qualifier Visibility Timestamp Value derek … … … … … don contact email admin | private 11905014 dminer@gopivotal.com don contact email admin | private 12412412 dminer@clearedgeit.com don contact email public 12412412 dm…@cl....com don contact twitter public 12423523 @donaldpminer don info height public 12314514 5’ 9” don info picture public 13119103 dd3ae1d3b951a33f… don info SSN private 12314514 123-45-6789 erica … … … … … Range Visibilities d-e public, user, tech Reading data Scan
  • 37. Basic Structured Data Row ID Column Family Column Qualifier Column Visibility Timestam p Value bob attribute height public Jun 2012 5’11” bob attribute surname public Jul 2013 doe bob insurance dental private Sep 2009 MetLife jane attribute bloodType public Jul 2011 ab- jane attribute surname public Aug 2013 doe jane contact cellPhone public Dec 2010 (808) 345- 9876 jane insurance vision private Jan 2008 VSP john allergy major private Feb 1988 amoxicillin john attribute weight public Sep 2013 180 john contact homeAddr public Mar 2003 34 Baker LN
  • 38. Basic Structured Data Row ID Column Family Column Qualifier Column Visibility Timestam p Value bob attribute height public Jun 2012 5’11” bob attribute surname public Jul 2013 doe bob insurance dental private Sep 2009 MetLife jane attribute bloodType public Jul 2011 ab- jane attribute surname public Aug 2013 doe jane contact cellPhone public Dec 2010 (808) 345- 9876 jane insurance vision private Jan 2008 VSP john allergy major private Feb 1988 amoxicillin john attribute weight public Sep 2013 180 john contact homeAddr public Mar 2003 34 Baker LN
  • 39. Indexing Everything Row ID Column Fam Column Qual Visibility Time value index Column Fam Column Qual:Row ID Visibility Time - to Column Fam Column Qual:Row ID Visibility Time - values Column Fam Column Qual:Row ID Visibility Time - Event Table Index Table
  • 40. Index Table Row ID Column Family Column Qualifier Column Visibility Timestam p Value (808) 345- 9876 contact cellPhone:jane public Dec 2010 - 180 attribute weight:john public Sep 2013 - 34 Baker LN contact homeAddr:john public Mar 2003 - 5’11” attribute height:bob public Jun 2012 - MetLife insuranc e dental:bob private Sep 2009 - VSP insuranc e vision:jane private Jan 2008 - ab- attribute bloodType:jane public Jul 2011 - amoxicillin allergy major:john private Feb 1988 - doe attribute surname:bob public Jul 2013 - doe attribute surname:jane public Aug 2013 -
  • 42. Data Lake PATIENTS MEDICINES DOCTORS INDEX Tell me everything you know of amoxicillin amoxicillin
  • 43. Data Lake PATIENTS DISEASES DOCTORS INDEX amoxicillin bob:allergy:amoxicillin larry:takes:amoxicillin Stomach ulcer: treatment:amoxicillin smith: prescribed:amoxicillinInfection: treatment:amoxicillin Diarrhea: side effect:amoxicillin Visibility labels help converge data sources but still protect who can see them.
  • 44. Graphs a bc d e a b c d e a - 1 b 1 - c - 1 d 1 1 - 1 e - Start Nodes EndNodes Row ID Column Family Column Qualifier Value a edge b 1 a edge d 1 c edge a 1 c edge d 1 d edge c 1 e edge d 1 • Random walk • Neighborhoods • Traversals Each edge can have a visibility label!
  • 45. Term-Partitioned Index Tablet Server 1 Row ID Column Family Value baseball document docid_3 baseball document docid_2 bat document docid_2 Tablet Server 2 Row ID Column Family Value football document docid_1 football document docid_3 glove document docid_1 Tablet Server 3 Row ID Column Family Value nba document docid_1 shoes document docid_1 soccer document docid_3 RESULTS: [docid_2, docid_3] RESULTS: [docid_1, docid_3] RESULTS: [docid_3] Tablet Server knows about the terms “baseball” Tablet Server knows about the terms “football” Tablet Server knows about the terms “soccer” Query: “baseball” AND “football” AND “soccer” Client Client-side Set Intersection [docid_2, docid_3] [docid_1, docid_3] [docid_3] Visibility labels allow protected search Iterators can maintain stats about docs
  • 47. Geospacial Indexing: Z-Order Curve 33.333W, 55.555N = 3535.353535 3535.353535 is the rowkey
  • 48. Temporal Indexing Row ID Column Family Column Qualifier Value Router37 2014-12 1418624102 cold Router37 2015-01 1421633979 cold Router37 2015-01 1421634319 hot Router37 2015-01 1421635001 cold Server92 2014-12 1418555102 cold Server92 2014-12 1418556999 hot Server92 2014-12 1418651002 cold Server92 2014-12 1418756987 hot Server92 2014-12 1418853304 cold Server98 2014-12 1418555104 cold Server98 2015-01 1421633319 cold Note: Dynamically adding column families
  • 49. Resources Apache Accumulo website accumulo.apache.org Accumulo Summit 2014 accumulosummit.com slideshare.net/AccumuloSummit Accumulo Summit 2015 End of April! accumulosummit.com

Editor's Notes

  • #2: This talk will go over table design and row key design approaches for indexing large amounts of data in Apache Accumulo. We'll do an overview of how to store geographical data, entity relationship graphs, natural language text, numbers, and more in Accumulo. This will serve as a starting point to learning how to effectively store different types of data in Accumulo as well as showcase the capabilities of Accumulo for handling varying situations.
  • #16: Two basic operators AND operator represented by & OR operator represented by | In the examples A,B, C, and D are security tokens Security Tokens are strings of alphanumeric characters Tokens are user defined Parenthesis are required to use nested logic
  • #38: This data is our original health care data Rows in red are rows that are only viewable to users with the ”private” authorization
  • #39: This data is our original health care data Rows in red are rows that are only viewable to users with the ”private” authorization
  • #40: It is easy to create a text index by splitting large values into constituent words This is similar to the previous example except we are indexing unstructured data (text) instead of structured data (single value)
  • #41: After indexing our data Row IDs are storing the data of all of our different types It may be necessary to transform these values to get the desired sort order It may also be handy to prepend type information to the Row ID like INT, or CHAR
  • #45: Matrix representation is efficient for densely connected graphs Matrix allows weights of nodes to be stored Not great for sparse graphs since most cells will be empty or null
  • #46: Process Tablets are partitioned on row boundaries, in this example the row boundaries are the terms Tablets are assigned to one Tablet Server each, distributing document information across many servers To perform a query that searches for multiple terms, all of the Tablet Servers need to be searched for each term Each tablet server will return a list of document IDs that contain the terms The client needs to perform a set intersection to determine which document was returned from all of the tablet servers Problems This method requires a lot of network traffic to return all the documents found, from each Tablet Server to the client The client will filter out many documents, probably a majority of what is returned If a lot of documents are contained in the table, the client could run out of memory before completing the intersection
  • #49: After indexing our data Row IDs are storing the data of all of our different types It may be necessary to transform these values to get the desired sort order It may also be handy to prepend type information to the Row ID like INT, or CHAR