NIST Big Data Working Group.pdf

Brainstorming (2006): Combining Subgroup Deliverables
by Bob Marcus
Table of Contents
1. Introduction and Definition - Page 1
2. Reference Architecture - Page 2
3. Requirements, Gap Analysis, and Suggested Best Practices - Page 10
4. Future Directions and Roadmap - Page 21
5. Security and Privacy (10 Top Challenges) - Page 22
6. Conclusions and General Advice - Page 23
7. References - Page 24
Appendix A. Terminology Glossary - Page 24
Appendix B. Use Case Examples - Page 28
Appendix C. Actors and Roles - Page 39
Appendix D. Questions and Answers from a Customer Perspective - Page 40
Appendix E. Solutions Glossary - Page 44
1. Introduction and Definition
The purpose of this outline is to illustrate how some initial brainstorming documents
might be pulled together into an integrated deliverable. The outline will follow the
diagram below.
Section 1 introduces a definition of Big Data.
Section 2 describes two Reference Architecture diagrams (high level and detailed)
Section 3 maps Requirements from Use Case building blocks to the detailed Reference
Architecture. A description of the Requirement, a gap analysis, and suggested best
practice is included with each mapping.
Section 4 lists future improvements in Big Data technology mapped to the high level
Reference Architecture.
Section 5 is a placeholder for an extended discussion of Security and Privacy.
Section 6 gives an example of some general advice.
The Appendices provide Big Data Terminology and Solutions Glossaries, Use Case
examples, some possible Actors and Roles, and Big Data questions and answers from
a Customer Perspective.
1

Big Data Definition - “Big Data refers to the new technologies and applications
introduced to handle increasing Volume, Velocity and Variety of data while enhancing
data utilization capabilities such as Variability, Veracity, and Value.”
The large Volume of data available forces horizontal scalability of storage and
processing and has implications for all the other V-attributes. The increasing velocity of
data ingestion and change implies the need for stream processing, filtering, and
processing optimizations. The variety of data types (e.g. multimedia) being generated a
requires the use of non-traditional data stores and processing.
Some changes driven the V-attributes are given below:
Volume - Driving requirement for robust horizontal scalability of storage/processing
Velocity - Driving optimization such as parallel stream processing and performance optimized databases
Variety - Driving move to non-relational data models (e.g. key-value)
Variability - Driving need for adaptive infrastructure
Value - Driving need for new querying and analytics tools
Veracity - Driving need for ensuring accuracy, relevance, and security of Big Data stores and processing
2. Reference Architecture
The High Level Reference Architecture below is the most abstract that can capture the
essential feature of Big Data architectures. See References for highest level consensus
Reference Architecture
2

Descriptions of the Components of the High Level Reference Architecture. Points for
future discussion are in bold. Examples are from the Apache Big Data Ecosystem.
A. External Data Sources and Sinks - Provides external data inputs and output to the
internal Big Data components. Note that Big Data stores can output data flows to
external systems e.g. feeding external databases.
B. Stream and ETL Processing - Filters and transforms data flows between external
data resources and internal Big Data systems. This processing should also be
scalable by adding additional processors
C. Highly Scalable Foundation - Horizontally scalable data stores and processing that
form the foundation of Big Data Architectures. It is essential to represent this layer
explicitly in the Reference Architecture.
D. Operational and Analytics Databases - Databases integrated into the Big Data
architecture. These can be horizontally scalable databases or single platform
databases with data extracted from the foundational data store. Big Data databases
(e.g. NoSQL, NewSQL) must be explicitly represented in the Big Data
architecture.
E. Analytics and Database Interfaces - These are the interfaces to the data stores for
queries, updates, and analytics. They are included in the Reference Architecture
because data stores may have multiple interfaces (e.g. Pig and Hive to HDFS)
and this is an area of possible standardization.
F. Applications and User Interfaces - These are the applications (e.g. machine
learning) and user interfaces (e.g. visualization) that are built on Big Data components.
Applications must have an underlying horizontally scalable data storage and
processing foundation to qualify as Big Data applications.
G. Supporting Services - These services include the components needed for the
implementation and management of robust Big Data systems. The key difference is
that services must be enhanced to handle scalable horizontally distributed data
sources and processing deployed on relatively low reliability platforms.
The Lower Level Reference Architecture below expands on some of the layers in the
High Level Reference Architecture and shows some of the data flows. It can be drilled
down if necessary.
3

A. External Data Sources and Sinks - Data sources might be separated from
sinks in a more detailed model
4. Data Sources and Sinks - These are components in a complete data architecture
that clearly defined interfaces to Big Data horizontally scalable internal data stores and
applications.
B. Stream and ETL Processing - Stream Processing and Extract, Transform,
Load (ETL) processing might be split in a more detailed architecture
5. Scalable Stream Processing - This is processing of “data in movement” between
data stores. It can be used for filtering, transforming, or routing data. For Big Data
streams, the stream processing must be scalable to support distributed and/or
pipelined processing.
C. Highly Scalable Foundation - This is the core of Big Data Technology. There
are several aspects including data stores (e.g. Hadoop File System), e.g. data
processing (e.g. MapReduce) and infrastructure (e.g. Clouds)
4

1. Scalable Infrastructure - To support scalable Big Data stores and processing, it is
necessary to have an infrastructure that can support the easy addition of new
resources. Possible platforms include public and/or private Clouds.
2. Scalable Data Stores - This is the essence of Big Data architecture. Horizontal
scalability using less expensive components can support the unlimited growth of
data storage. However there must be fault tolerance capabilities available to handle
component failures.
3 . Scalable Processing - To take advantage of scalable distributed data stores, it is
necessary to have scalable distributed parallel processing with similar fault tolerance.
In general, processing should be configuring to minimize unnecessary data movement.
D. Operational and Analytics Databases - The databases are split because
this is currently a differentiator that effects interfaces and applications
6. Analytics Databases - Analytics databases are generally highly optimized for read-
only interactions (e.g. columnar storage, extensive indexing, and denormalization). It is
often acceptable for database responses to have high latency (e.g. invoke scalable
batch processing over large data sets).
7. Operational Databases - Operation databases generally support efficient write and
read operations. NoSQL databases are often used in Big Data architectures in this
capacity. Data can be later transformed and loaded into analytic databases to support
analytic applications.
8. In Memory Data Grids - These are very high performance data caches and stores
that minimize writing to disk. They can be used for large scale real time applications
requiring transparent access to data.
The diagram below from http://blogs.the451group.com/information_management/
2011/04/15/nosql-newsql-and-beyond/ provides a multiple attribute classification of
databases including analytic databases, operational databases, and in memory data
grids. Big Data databases are in the shaded boxes.
SPRAIN in the diagram (451group terminology) stands for some of the drivers for
using new Big Data databases.
• Scalability – hardware economics
• Performance – MySQL limitations
• Relaxed consistency – CAP theorem
• Agility – polyglot persistence
• Intricacy – big data, total data
• Necessity – open source
5

E. Analytics and Database Interfaces - Interfaces and simple analytics
are bundled together because there are overlaps e.g. SQL interfaces and SQL-based
analytics, The interfaces and analytics are split into subgroups by latency. There is a
definite distinction between interfaces requiring batch processing (e.g. current Hive,
Pig), end-user interactive responses (e.g. HBase), and ultrafast real-time responses
(e.g machine-based Complex Event Processing).
9. Batch Analytics and Interfaces - These are interfaces that use batch scalable
processing (e.g. Map-Reduce) to access data in scalable data stores (e.g Hadoop File
System). These interfaces can be SQL-like (e.g. Hive) or programmatic (e.g. Pig).
10. Interactive Analytics and Interfaces - These interfaces avoid directly access data
stores to provide interactive responses to end-users. The data stores can be
horizontally scalable databases tuned for interactive responses (e.g. HBase) or query
languages tuned to data models (e.g. Drill for nested data).
11. Real-Time Analytics and Interfaces - There are applications that require real time
responses to events occurring within large data streams (e.g. algorithmic trading). This
complex event processing is machine-based analytics requiring very high performance
data access to streams and data stores.
6

F. Applications and User Interfaces - Visualization might be split from
applications in a more detailed model.
12. Applications and Visualization - The key new capability available to Big Data
analytic applications is the ability to avoid developing complex algorithms by utilizing
vast amounts of distributed data (e.g. Google statistical language translation). However
taking advantage of the data available requires new distributed and parallel processing
algorithms.
G. Supporting Services - These services are available in all robust enterprise
architectures. The key extension for Big Data is the ability to handle horizontal
components. They could all be expanded in a more detailed Reference Architecture.
13. Design, Develop, and Deploy Tools - High level tools are limited for the
implementation of Big Data applications (e.g. Cascading). This will have to change
to lower the skill levels needed by enterprise and government developers.
14. Security - Current Big Data security and privacy controls are limited (e.g. only
Kerberos authentication for Hadoop, Knox). They must be expanded in the future
by commercial vendors (e.g. Cloudera Sentry) for enterprise and government
applications.
15. Process Management - Commercial vendors are supplying process management
tools to augment the initial open source implementations (e.g. Oozie)
16. Data Resource Management - Open Source data governance tools are still
immature (e.g. Apache Falcon). These will be augmented in the future by
commercial vendors.
17. System Management - Open source systems management tools are also
immature (e.g. Ambari). Fortunately robust system management tools are
commercially available for scalable infrastructure (e.g. Cloud-based).
7

Apache’s Big Data Offerings are mapped to the Reference Architecture in the diagram
below for reference. An overview of the Apache ecosystem is at
http://www.revelytix.com/?q=content/hadoop-ecosystem
Two Other Reference Architectures are shown below for Comparison
8

From http://www.slideshare.net/Hadoop_Summit/dont-be-hadooped-when-looking-for-big-data-roi
From “ Big Data Governance” Book by Sunil Soares
9

3. Requirements, Gap Analysis, and Suggested Best Practices
In the Requirements discussion, building block components for use cases will be
mapped to elements of the Reference. These components will occur in many use
cases across multiple application domains. A short description, possible requirements,
gap analysis, and suggested best practices is provided for each building block.
1. Data input and output to Big Data File System (ETL, ELT)
Example Diagram:
Description: The Foundation Data Store can be used as a repository for very large
amounts of data (structured, unstructured, semi-structured). This data can be imported
and exported to external data sources using data integration middleware.
Possible Requirements: The data integration middleware should be able to do high
performance extraction, transformation and load operations for diverse data models and
formats.
Gap Analysis: The technology for fast ETL to external data sources (e.g Apache Flume,
Apache Sqoop) is available for most current data flows. There could be problems in the
future as the size of data flows increases (e.g. LHC). This may require some filtering or
summation to avoid overloading storage and processing capabilities
Suggested Best Practices: Use packages that support data integration. Be aware of the
possibilities for Extract-Load-Transform (ELT) where transformations can be done using
data processing software after the raw data has been loaded into the data store e.g,
Map-Reduce processing on top of HDFS.
10

2. Data exported to Databases from Big Data File System
Example Diagram:
Description: A data processing system can extract, transform, and transmit data to
operational and analytic databases.
Possible Requirements: For good through-put performance on very large data sets, the
data processing system will require multi-stage parallel processing
Gap Analysis: Technology for ETL is available (e.g. Apache Sqoop for relational
databases, MapReduce processing of files). However if high performance multiple
passes through the data are necessary, it will be necessary to avoid rewriting
intermediate results to files as is done by the original implementations of MapReduce.
Suggested Best Practices: Consider using data processing that does not need to write
intermediate results to files e.g. Spark.
11

3 Big Data File Systems as a data resource for batch and interactive queries
Example Diagram:
Description: The foundation data store can be queried through interfaces using batch
data processing or direct foundation store access.
Possible Requirements: The interfaces should provide good throughput performance for
batch queries and low latency performance for direct interactive queries.
Gap Analysis: Optimizations will be necessary in the internal format for file storage to
provide high performance (e.g. Hortonworks ORC files, Cloudera Parquet)
Suggested Best Practices: If performance is required, use optimizations for file formats
within the foundation data store. If multiple processing steps are required, data
processing packages that retain intermediate values in memory.
12

4. Batch Analytics on Big Data File System using Big Data Parallel Processing
Example Diagram:
Description: A data processing system augmented by user defined functions can
perform batch analytics on data sets stored in the foundation data store.
Possible Requirements: High performance data processing is needed for efficient
analytics.
Gap Analysis: Analytics will often use multiple passes through the data. High
performance will require the processing engine to avoid writing intermediate results to
files as is done in the original version of MapReduce
Suggested Best Practices: If possible, intermediate results of iterations should be kept
in memory. Consider moving data to be analyzed into memory or an analytics optimized
database.
13

5. Stream Processing and ETL
Example Diagram:
Description: Stream processing software can transform, process, and route data to
databases and real time analytics
Possible Requirements: The stream processing software should be capable of high
performance processing of large high velocity data streams.
Gap Analysis: Many stream processing solutions are available. In the future, complex
analytics will be necessary to enable stream process to perform accurate filtering and
summation of very large data streams.
Suggested Best Practices: Parallel processing is necessary for good performance on
large data streams.
14

6. Real Time Analytics (e.g. Complex Event Processing)
Description: Large high velocity data streams and notifications from in memory
operational databases can be analyzed to detect pre-determined patterns, discover new
relationships, and provide predictive analytics.
Possible Requirements: Efficient algorithms for pattern matching and/or machine
learning are necessary.
Gap Analysis: There are many solutions available for complex event processing. It
would be useful to have standards for describing event patterns to enable portability.
Suggested Best Practices: Evaluate commercial packages to determine the best fit for
your application.
15

7. NoSQL (and NewSQL) DBs as operational databases for large-scale updates
and queries
Example Diagram:
Description: Non-relational databases can be used for high performance for large data
volumes (e.g. horizontally scaled). New SQL databases support horizontal scalability
within the relational model.
Possible Requirements: It is necessary to decide on the level of consistency vs.
availability is needed since the CAP theorem demonstrates that both can not be
achieved in horizontally scaled systems.
Gap Analysis: The first generation of horizontal scaled databases emphasized
availability over consistency. The current trend seems to be toward increasing the role
of consistency. In some cases (e.g. Apache Cassandra), it is possible to adjust the
balance between consistency and availability.
Suggested Best Practices: Horizontally scalable databases are experiencing rapid
advances in performance and functionality. Choices should be based on application
requirements and evaluation testing. Be very careful about choosing a cutting edge
solution that has not been used in applications similar to your use case. SQL (or SQL-
like) interfaces will better enable future portability until there are standards for NoSQL
interfaces.
16

8. NoSQL DBs for storing diverse data types
Example Diagram:
Description: Non-relational databases can store diverse data types (e.g. documents,
graphs, heterogeneous rows) that can be retrieved by key or queries.
Possible Requirements: The data types to be stored depend on application data usage
requirements and query patterns.
Gap Analysis: In general, the NoSQL databases are not tuned for analytic applications.
Suggested Best Practices: There is a trade off when using non-relational databases.
Usually some functionality is given up (e.g. joins, referential integrity) in exchange for
some advantages (e.g. higher availability, better performance). Be sure that the trade-off
meets application requirements.
17

9. Databases optimized for complex ad hoc queries
Example Diagram:
Description: Interactive ad hoc queries and analytics to specialized databases are key
Big Data capabilities
Possible Requirements: Analytic databases need high performance on complex queries
which require optimizations such as columnar storage, in memory caches, and star
schema data models.
Gap Analysis: There is a need for embedded analytics and/or specialized databases for
complex analytics applications.
Suggested Best Practices: Use databases that have been optimized for analytics and/or
support embedded analytics. It will often be necessary to move data from operational
databases and/or foundation data stores using ETL tools.
18

10. Databases optimized for rapid updates and retrieval (e.g. in memory or SSD)
Example Diagram:
Description: Very high performance operational databases are necessary for some
large-scale applications.
Possible Requirements: Very high performance will often require in memory databases
and/or solid state drive (SSD) storage.
Gap Analysis: Data retrieval from disk files is extremely slow compared in memory,
cache, or SSD access. There will be increased need for these faster options as
performance requirements increase.
Suggested Best Practices: In the future, disk drives will be used for archiving or for non-
performance oriented applications. Evaluate the use of data stores that can reside in
memory, caches, or on SSDs.
19

11 Visualization Tools for End-Users
Description: Visualization of data and relationships is a very important capability for
end-users trying to understand and use the data
Possible Requirements: Ability to extract data from multiple databases and present in a
meaningful user-customizable fashion.
Gap Analysis: There are good first generation visualization tools available for analytic
and applications. The increasing volume, velocity, and variety of data sources will
provide a challenge for the future
Suggested Best Practices: Choose a visualization tools that satisfies current
requirements and has extension and customization capabilities to meet future needs.
20

4. Future Directions and Roadmap
In the Big Data Technology Roadmap, the results of the gap analysis should be
augmented with a list of future developments that will help close the gaps. Ideally some
timelines should be included to aid in project planning. This sections lists ongoing
improvements mapped to elements of Reference Architecture with links for more detail
1. Processing Performance Improvements
(Reference Architecture: Data Processing)
Data in memory or stored on Solid State Drive (SSD)
http://www3.weforum.org/docs/GITR/2012/GITR_Chapter1.7_2012.pdf
http://www.datanami.com/datanami/2012-02-13/big_data_and_the_ssd_mystique.htm’
Enhancements to first generation Map-Reduce
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
http://incubator.apache.org/mesos/
Use of GPUs
http://www.networkworld.com/news/tech/2013/062413-hadoop-gpu-271194.html
2. Application Development Improvements
(Reference Architecture: Development Tools)
Big Data PaaS and data grids
http://searchsoa.techtarget.com/feature/Look-out-Big-Data-In-memory-data-grids-start-to-go-mainstream
http://aws.amazon.com/elasticmapreduce/
Visual design, development, and deploy tools
http://www.pentahobigdata.com/overview
Unified interfaces using data virtualization
http://www.compositesw.com/company/pages/composite-software-next-generation-data-virtualization-platform-composite-6/
http://www.marklogic.com/solutions/data-virtualization/
3. Complex Analytics Improvements
(Reference Architecture: Analytics)
Embedded analytics
http://www.slideshare.net/InsideAnalysis/embedded-analytics-the-next-megawave-of-innovation
Stream analytics, filtering, and complex event processing
http://www.sqlstream.com/
Integrated data ingestion, processing, storage, and analytics
www.teradata.com/products-and-services/unified-data-architecture/
4. Interoperability Improvements
(Reference Architecture: integration across components)
21

Data sharing among multiple Hadoop tools and external tools (e.g. using HCatalog)
http://hortonworks.com/hdp/hdp-hcatalog-metadata-services/
Queries across Hadoop and legacy databases (e.g. EDW)
http://hadapt.com/product/
Data exchanges and ETL among diverse data stores
http://sqoop.apache.org/
http://www.talend.com/products/data-integration
5. Possible Alternative Deployment Improvements
(Reference Architecture: Infrastructure)
Cloud
http://www.cloudstandardscustomercouncil.org/031813/agenda.htm
HPC clusters
http://insidehpc.com/2013/06/19/cray-launches-complete-lustre-storage-solution-across-hpc-and-big-data-computing-markets/
Appliances
http://nosql.mypopescu.com/post/15729871938/comparing-hadoop-appliances-oracles-big-data
6. Applications
(Reference Architecture: Applications)
Internet of Things
http://en.wikipedia.org/wiki/Internet_of_Things
Big Data for Vertical Applications (e.g. science, healthcare)
http://jameskaskade.com/?p=2708
Big Data Society Applications and Issues
www.parl.gc.ca/HousePublications/Publication.aspx?DocId=6094136&Language=E&Mode=1&Parl=41&Ses=1
7. Interface Improvements
(Reference Architecture: Interfaces)
SQL interfaces to NoSQL databases
http://qconsf.com/sf2012/dl/qcon-sanfran-2012/slides/MaryHolstege_and_StephenBuxton_TheDesignOfASQLInterfaceForANoSQLDatabase.pdf
Performance optimizations for querying (e.g. columnar storage)
http://searchdatamanagement.techtarget.com/definition/columnar-database
Querying and analytics interfaces for end-user
http://www.tableausoftware.com/
5. Security and Privacy
Top 10 Challenges from CSA at https://cloudsecurityalliance.org/download/expanded-
top-ten-big-data-security-and-privacy-challenges/
1. Secure computations in distributed programming frameworks
2. Security best practices for non-relational data stores
22

3. Secure data storage and transactions logs
4. End-point input validation/filtering
5. Real-time security monitoring
6. Scalable and composable privacy-preserving data mining and analytics
7. Cryptographically enforced data centric security
8. Granular access control
9. Granular audits
10. Data provenance
6. Conclusions and General Advice
From Demystifying Big Data by TechAmerica
http://www.techamerica.org/Docs/fileManager.cfm?f=techamerica-bigdatareport-final.pdf
23

7. References
Demystifying Big Data by TechAmerica
http://www.techamerica.org/Docs/fileManager.cfm?f=techamerica-bigdatareport-final.pdf
Consumer Guide to Big Data from ODCA
http://www.opendatacenteralliance.org/docs/Big_Data_Consumer_Guide_Rev1.0.pdf
Another Word for it - Indexed collection of articles on Big Data solutions + Much more
http://tm.durusau.net/?m=201309
Highest Level Consensus Infrastructure Architecture from NIST Big Data Working Group
Appendix A. Terminology Glossary
The description and links for terms are listed to help in understanding other sections.
ACiD - Atomicity, Consistency, Isolation, Durability are a group of properties that
together guarantee that database transactions are processed reliably
http://en.wikipedia.org/wiki/ACID
Analytics - “The discovery and communication of meaningful patterns in data”
24

http://en.wikipedia.org/wiki/Analytics
Avatarnode - Fault-tolerant extension to Namenode
http://gigaom.com/2012/06/13/how-facebook-keeps-100-petabytes-of-hadoop-data-online/
BASE - Basically Available, Soft state, Eventual consistency semantics
http://en.wikipedia.org/wiki/Eventual_consistency
Big Data - “A collection of data set so large and complex that it is difficult to process
using on-hand database management tools or traditional data processing applications.”
http://en.wikipedia.org/wiki/Big_data
BSON - Binary coding of JSON
http://bsonspec.org/
BSP (Bulk Synchronous Parallel) - A programming model for distributed computation
that avoid writing intermediate results to files
http://en.wikipedia.org/wiki/Bulk_synchronous_parallel
B-Tree Indexing - Commonly used Indexing algorithm and data structure
http://en.wikipedia.org/wiki/B-tree
Business Analytics - “Refers to the skills, technologies, applications and practices for
continuous iterative exploration and investigation of past business performance to gain
insight and drive business planning”
http://en.wikipedia.org/wiki/Business_analytics
Cache - Intermediate storage between files and memory used to improve performance
http://en.wikipedia.org/wiki/Database_caching
(CEP) Complex Event Processing - “Event processing that combines data from
multiple sources[2] to infer events or patterns that suggest more complicated
circumstances”.
http://en.wikipedia.org/wiki/Complex_event_processing
Consistent Hashing - A hashing algorithm that is resilient to dynamic changes
http://en.wikipedia.org/wiki/Consistent_hashing
Descriptive Analytics -”The discipline of quantitatively describing the main features of
a collection of data.”
https://en.wikipedia.org/wiki/Descriptive_statistics
Discovery Analytics - Data mining and related processes
http://en.wikipedia.org/wiki/Data_mining
25

ELT (Extract, Load, Transform) - “A process architecture where a bulk of the
transformation work occurs after the data has been loaded into the target database in its
raw format”
http://it.toolbox.com/wiki/index.php/ELT
ETL (Extract, Transform Load) - Extracting data from source databases, transforming
it, and then loading it into target databases.
http://en.wikipedia.org/wiki/Extract,_transform,_load
Erasure Codes - (Alternate to file replication for availability) Replicates fragments.
http://oceanstore.cs.berkeley.edu/publications/papers/pdf/erasure_iptps.pdf
Fractal Tree Indexing - New indexing with better peformance than B-tree
http://en.wikipedia.org/wiki/TokuMX#Fractal_tree_indexes
In Memory Database - A database that primarily resides in computer main memory.
http://en.wikipedia.org/wiki/In-memory_database
JSON (Javascript Object Notation) - Hierarchical serialization format derived from
Javascript.
http://www.json.org/
MapReduce - A programming model for processing large data sets. It consists of a
mapping processing to distributed resources, followed by a sorting phase of
intermediate results, and parallel reduction to a final result.
http://en.wikipedia.org/wiki/MapReduce
MPP (Massively Parallel Processing) - “Refers to the use of a large number of
processors to perform a set of coordinated computations in parallel”
http://en.wikipedia.org/wiki/Massive_parallel_processing
NewSQL - Big Data databases supporting relational model and SQL
http://en.wikipedia.org/wiki/NewSQL
NoSQL - Big Data databases not supporting relational model
https://en.wikipedia.org/wiki/NoSQL
OLAP (Online Analytic Processing) - “OLAP tools enable users to analyze
multidimensional data interactively from multiple perspective”
http://en.wikipedia.org/wiki/Online_analytical_processing
OLTP (Online Transactional Processing) - “A class of information systems that
facilitate and manage transaction-oriented applications”
http://en.wikipedia.org/wiki/Online_transaction_processing
Paxos - A distributed coordination protocol
26

http://en.wikipedia.org/wiki/Paxos_%28computer_science%29
PMML (Predictive Markup Model Language) - Model exchange using XML
http://en.wikipedia.org/wiki/Predictive_Model_Markup_Language
Predictive Analytics - “Encompasses a variety of techniques fthat analyze facts to
make predictions about future, or otherwise unknown, events”
http://en.wikipedia.org/wiki/Predictive_analytics
Prescriptive Analytics - “Automatically synthesizes big data, multiple disciplines of
mathematical sciences and computational sciences, and business rules, to make
predictions and then suggests decision options to take advantage of the predictions”
http://en.wikipedia.org/wiki/Prescriptive_Analytics
Semi-Structured Data - Unstructured data combine with structured data (e.g.
metadata)
http://en.wikipedia.org/wiki/Semi-structured_data
SSD (Solid State Drive) - “ A data storage device using integrated circuit assemblies
as memory to store data persistently”
http://en.wikipedia.org/wiki/Solid-state_drive
Stream Processing - “Given a set of data (a stream), a series of operations (kernel
functions) is applied to each element in the stream”
http://en.wikipedia.org/wiki/Stream_processing
Structured Data - Schema can be in part of data store or within application. Some
examples are data described by a formal data model or formal grammar.
http://www.webopedia.com/TERM/S/structured_data.html
Unstructured Data - Data stored with no schema and at most Implicit structure. The
data can be in a standard format (e.g. ASCII, JPEG) or binary.
http://en.wikipedia.org/wiki/Unstructured_data
Vector Clocks - An algorithm that generates partial ordering of events in distributed
systems
http://en.wikipedia.org/wiki/Vector_clock
Web Analytics - “The measurement, collection, analysis and reporting of Internet data
for purposes of understanding and optimizing web usage.:
http://en.wikipedia.org/wiki/Web_analytics
27

Appendix B. Application Use Case Examples
From http://thebigdatainstitute.wordpress.com/2013/05/23/our-favorite-40-big-data-use-
cases-whats-your/ “While there are extensive industry-specific use cases, here are
some for handy reference:
EDW Use Cases
• Augment EDW by offloading processing and storage
• Support as preprocessing hub before getting to EDW
Retail/Consumer Use Cases
• Merchandizing and market basket analysis
• Campaign management and customer loyalty programs
• Supply-chain management and analytics
• Event- and behavior-based targeting
• Market and consumer segmentations
Financial Services Use Cases
• Compliance and regulatory reporting
• Risk analysis and management
• Fraud detection and security analytics
• CRM and customer loyalty programs
• Credit risk, scoring and analysis
• High speed arbitrage trading
• Trade surveillance
• Abnormal trading pattern analysis
Web & Digital Media Services Use Cases
• Large-scale clickstream analytics
28

• Ad targeting, analysis, forecasting and optimization
• Abuse and click-fraud prevention
• Social graph analysis and profile segmentation
• Campaign management and loyalty programs
Health & Life Sciences Use Cases
• Clinical trials data analysis
• Disease pattern analysis
• Campaign and sales program optimization
• Patient care quality and program analysis
• Medical device and pharma supply-chain management
• Drug discovery and development analysis
Telecommunications Use Cases
• Revenue assurance and price optimization
• Customer churn prevention
• Campaign management and customer loyalty
• Call detail record (CDR) analysis
• Network performance and optimization
• Mobile user location analysis
Government Use Cases
• Fraud detection
• Threat detection
• Cybersecurity
• Compliance and regulatory analysis
• http://www.whitehouse.gov/sites/default/files/microsites/ostp/
big_data_press_release_final_2.pdf
New Application Use Cases
• Online dating
• Social gaming
Fraud Use-Cases
• Credit and debit payment card fraud
• Deposit account fraud
• Technical fraud and bad debt
• Healthcare fraud
• Medicaid and Medicare fraud
• Property and casualty (P&C) insurance fraud
• Workers’ compensation fraud
29

E-Commerce and Customer Service Use-Cases
• Cross-channel analytics
• Event analytics
• Recommendation engines using predictive analytics
• Right offer at the right time
• Next best offer or next best action”
•
http://www.theequitykicker.com/2012/03/12/looking-to-the-use-cases-of-big-data/
discusses some Big Data Use Case examples.
Case Studies from TechAmerica
From http://www.techamerica.org/Docs/fileManager.cfm?f=techamerica-bigdatareport-final.pdf
30

Requirements Subgroup Use Cases Summary
Government Operation
1. Census 2010 and 2000 – Title 13 Big Data; Vivek Navale & Quyen
Nguyen, NARA
Application: Preserve Census 2010 and 2000 – Title 13 data for a long term in order to
provide access and perform analytics after 75 years. One must maintain data “as-is”
with no access and no data analytics for 75 years; one must preserve the data at the
bit-level; one must perform curation, which includes format transformation if
necessary; one must provide access and analytics after nearly 75 years. Title 13 of U.S.
code authorizes the Census Bureau and guarantees that individual and industry
specific data is protected.
Current Approach: 380 terabytes of scanned documents
2. National Archives and Records Administration Accession NARA,
Search, Retrieve, Preservation; Vivek Navale & Quyen Nguyen, NARA
Application: Accession, Search, Retrieval, and Long term Preservation of Government
Data.
Current Approach: 1) Get physical and legal custody of the data; 2) Pre-process data
for virus scan, identifying file format identification, removing empty files; 3) Index; 4)
Categorize records (sensitive, unsensitive, privacy data, etc.); 5) Transform old file
formats to modern formats (e.g. WordPerfect to PDF); 6) E-discovery; 7) Search and
retrieve to respond to special request; 8) Search and retrieve of public records by
public users. Currently 100’s of terabytes stored centrally in commercial databases
supported by custom software and commercial search products.
Futures: There are distributed data sources from federal agencies where current
solution requires transfer of those data to a centralized storage. In the future, those
data sources may reside in multiple Cloud environments. In this case, physical custody
should avoid transferring big data from Cloud to Cloud or from Cloud to Data Center.
Commercial
3. Cloud Eco-System, for Financial Industries (Banking, Securities &
Investments, Insurance) transacting business within the United
States; Pw Carey, Compliance Partners, LLC
Application: Use of Cloud (Bigdata) technologies needs to be extended in Financial
Industries (Banking, Securities & Investments, Insurance).
31

Current Approach: Currently within Financial Industry, Bigdata and Hadoop are used
for fraud detection, risk analysis and assessments as well as improving the
organizations knowledge and understanding of the customers. At the same time, the
traditional client/server/data warehouse/RDBM (Relational Database Management)
systems are used for the handling, processing, storage and archival of the entities
financial data. Real time data and analysis important in these applications.
Futures: One must address Security and privacy and regulation such as SEC mandated
use of XBRL (extensible Business Related Markup Language) and examine other cloud
functions in the Financial industry.
4. Mendeley – An International Network of Research; William Gunn ,
Mendeley
Application: Mendeley has built a database of research documents and facilitates the
creation of shared bibliographies. Mendeley uses the information collected about
research reading patterns and other activities conducted via the software to build more
efficient literature discovery and analysis tools. Text mining and classification systems
enables automatic recommendation of relevant research, improving the cost and
performance of research teams, particularly those engaged in curation of literature on
a particular subject
Current Approach: Data size is 15TB presently, growing about 1 TB/month.
Processing on Amazon Web Services with Hadoop, Scribe, Hive, Mahout, Python.
Standard libraries for machine learning and analytics, Latent Dirichlet Allocation,
custom built reporting tools for aggregating readership and social activities per
document.
Futures: Currently Hadoop batch jobs are scheduled daily, but work has begun on
real-time recommendation. The database contains ~400M documents, roughly 80M
unique documents, and receives 5-700k new uploads on a weekday. Thus a major
challenge is clustering matching documents together in a computationally efficient way
(scalable and parallelized) when they’re uploaded from different sources and have been
slightly modified via third-part annotation tools or publisher watermarks and cover
pages.
5. Netflix Movie Service; Geoffrey Fox, Indiana University
Application: Allow streaming of user selected movies to satisfy multiple objectives (for
different stakeholders) -- especially retaining subscribers. Find best possible ordering
of a set of videos for a user (household) within a given context in real-time; maximize
movie consumption. Digital movies stored in cloud with metadata; user profiles and
rankings for small fraction of movies for each user. Use multiple criteria – content
based recommender system; user-based recommender system; diversity. Refine
algorithms continuously with A/B testing.
32

Current Approach: Recommender systems and streaming video delivery are core
Netflix technologies. Recommender systems are always personalized and use logistic/
linear regression, elastic nets, matrix factorization, clustering, latent Dirichlet
allocation, association rules, gradient boosted decision trees and others. Winner of
Netflix competition (to improve ratings by 10%) combined over 100 different
algorithms. Uses SQL, NoSQL, MapReduce on Amazon Web Services. Netflix
recommender systems have features in common to e-commerce like Amazon.
Streaming video has features in common with other content providing services like
iTunes, Google Play, Pandora and Last.fm.
Futures: Very competitive business. Need to aware of other companies and trends in
both content (which Movies are hot) and technology. Need to investigate new business
initiatives such as Netflix sponsored content
6. Web Search; Geoffrey Fox, Indiana University
Application: Return in ~0.1 seconds, the results of a search based on average of 3
words; important to maximize quantities like “precision@10” or number of great
responses in top 10 ranked results.
Current Approach: Steps include 1) Crawl the web; 2) Pre-process data to get
searchable things (words, positions); 3) Form Inverted Index mapping words to
documents; 4) Rank relevance of documents: PageRank; 5) Lots of technology for
advertising, “reverse engineering ranking” “preventing reverse engineering”; 6)
Clustering of documents into topics (as in Google News) 7) Update results efficiently.
Modern clouds and technologies like MapReduce have been heavily influenced by this
application. ~45B web pages total.
Futures: A very competitive field where continuous innovation needed. Two important
areas are addressing mobile clients which are a growing fraction of users and
increasing sophistication of responses and layout to maximize total benefit of clients,
advertisers and Search Company. The “deep web” (that behind user interfaces to
databases etc.) and multimedia search of increasing importance. 500M photos
uploaded each day and 100 hours of video uploaded to YouTube each minute
7. IaaS (Infrastructure as a Service) Big Data Business Continuity &
Disaster Recovery (BC/DR) Within a Cloud Eco-System; Pw Carey,
Compliance Partners, LLC
Application: BC/DR (Business Continuity/Disaster Recovery) needs to consider the role
that the following four overlaying and inter-dependent forces will play in ensuring a
workable solution to an entity's business continuity plan and requisite disaster
recovery strategy. The four areas are; people (resources), processes (time/cost/ROI),
technology (various operating systems, platforms and footprints) and governance
(subject to various and multiple regulatory agencies).
33

Current Approach: Cloud Eco-systems, incorporating IaaS (Infrastructure as a Service),
supported by Tier 3 Data Centers provide data replication services. Replication is
different from Backup and only moves the changes since the last time a replication
occurs, including block level changes. The replication can be done quickly, with a five
second window, while the data is replicated every four hours. This data snap shot is
retained for seven business days, or longer if necessary. Replicated data can be moved
to a Fail-over Center to satisfy an organizations RPO (Recovery Point Objectives) and
RTO (Recovery Time Objectives). Technologies from VMware, NetApps, Oracle, IBM,
Brocade are some of those relevant. Data sizes are terabytes up to petabytes
Futures: The complexities associated with migrating from a Primary Site to either a
Replication Site or a Backup Site is not fully automated at this point in time. The goal is
to enable the user to automatically initiate the Fail Over sequence. Both organizations
must know which servers have to be restored and what are the dependencies and
inter-dependencies between the Primary Site servers and Replication and/or Backup
Site servers. This requires a continuous monitoring of both.
8. Cargo Shipping; William Miller, MaCT USA
Application: Monitoring and tracking of cargo as in Fedex, UPS and DHL.
Current Approach: Today the information is updated only when the items that were
checked with a bar code scanner are sent to the central server. The location is not
currently displayed in real-time.
Futures: This Internet of Things application needs to track items in real time. A new
aspect will be status condition of the items which will include sensor information, GPS
coordinates, and a unique identification schema based upon a new ISO 29161
standards under development within ISO JTC1 SC31 WG2.
9. Materials Data for Manufacturing; John Rumble, R&R Data Services
Application: Every physical product is made from a material that has been selected for
its properties, cost, and availability. This translates into hundreds of billion dollars of
material decisions made every year. However the adoption of new materials normally
takes decades (two to three) rather than a small number of years, in part because data
on new materials is not easily available. One needs to broaden accessibility, quality,
and usability and overcome proprietary barriers to sharing materials data. One must
create sufficiently large repositories of materials data to support discovery.
Current Approach: Currently decisions about materials usage are unnecessarily
conservative, often based on older rather than newer materials R&D data, and not
taking advantage of advances in modeling and simulations.
Futures: Materials informatics is an area in which the new tools of data science can
have major impact by predicting the performance of real materials (gram to ton
quantities) starting at the atomistic, nanometer, and/or micrometer level of
34

description. One must establish materials data repositories beyond the existing ones
that focus on fundamental data; one must develop internationally-accepted data
recording standards that can be used by a very diverse materials community, including
developers materials test standards (such as ASTM and ISO), testing companies,
materials producers, and R&D labs; one needs tools and procedures to help
organizations wishing to deposit proprietary materials in data repositories to mask
proprietary information, yet to maintain the usability of data; one needs multi-variable
materials data visualization tools, in which the number of variables can be quite high
10. Simulation driven Materials Genomics; David Skinner, LBNL
Application: Innovation of battery technologies through massive simulations spanning
wide spaces of possible design. Systematic computational studies of innovation
possibilities in photovoltaics. Rational design of materials based on search and
simulation. These require management of simulation results contributing to the
materials genome.
Current Approach: PyMatGen, FireWorks, VASP, ABINIT, NWChem, BerkeleyGW, and
varied materials community codes running on large supercomputers such as 150K core
Hopper machine at NERSC produce results that are not synthesized.
Futures: Need large scale computing at scale for simulation science. Flexible data
methods at scale for messy data. Machine learning and knowledge systems that
integrate data from publications, experiments, and simulations to advance goal-driven
thinking in materials design. The current 100TB of data will become 500TB in 5 years.
Healthcare and Life Sciences
11. Electronic Medical Record (EMR) Data; Shaun Grannis, Indiana
University
Application: Large national initiatives around health data are emerging, and include
developing a digital learning health care system to support increasingly evidence-
based clinical decisions with timely accurate and up-to-date patient-centered clinical
information; using electronic observational clinical data to efficiently and rapidly
translate scientific discoveries into effective clinical treatments; and electronically
sharing integrated health data to improve healthcare process efficiency and outcomes.
These key initiatives all rely on high-quality, large-scale, standardized and aggregate
health data. One needs advanced methods for normalizing patient, provider, facility
and clinical concept identification within and among separate health care organizations
to enhance models for defining and extracting clinical phenotypes from non-standard
discrete and free-text clinical data using feature selection, information retrieval and
machine learning decision-models. One must leverage clinical phenotype data to
support cohort selection, clinical outcomes research, and clinical decision support.
35

Current Approach: Clinical data from more than 1,100 discrete logical, operational
healthcare sources in the Indiana Network for Patient Care (INPC) the nation's largest
and longest-running health information exchange. This describes more than 12 million
patients, more than 4 billion discrete clinical observations. > 20 TB raw data. Between
500,000 and 1.5 million new real-time clinical transactions added per day.
Futures: Teradata, PostgreSQL, MongoDB running on Indiana University supercomputer
supporting information retrieval methods to identify relevant clinical features (tf-idf,
latent semantic analysis, mutual information). Natural Language Processing techniques
to extract relevant clinical features. Validated features will be used to parameterize
clinical phenotype decision models based on maximum likelihood estimators and
Bayesian networks. Decision models will be used to identify a variety of clinical
phenotypes such as diabetes, congestive heart failure, and pancreatic cancer
12. Pathology Imaging/digital pathology; Fusheng Wang, Emory
University
Application: Digital pathology imaging is an emerging field where examination of high
resolution images of tissue specimens enables novel and more effective ways for
disease diagnosis. Pathology image analysis segments massive (millions per image)
spatial objects such as nuclei and blood vessels, represented with their boundaries,
along with many extracted image features from these objects. The derived information
is used for many complex queries and analytics to support biomedical research and
clinical diagnosis.
Current Approach: 1GB raw image data + 1.5GB analytical results per 2D image. MPI
for image analysis; MapReduce + Hive with spatial extension on supercomputers and clouds. GPU’s used
effectively.
Futures: Recently, 3D pathology imaging is made possible through 3D laser
technologies or serially sectioning hundreds of tissue sections onto slides and
scanning them into digital images. Segmenting 3D microanatomic objects from
registered serial images could produce tens of millions of 3D objects from a single
image. This provides a deep “map” of human tissues for next generation diagnosis.
1TB raw image data + 1TB analytical results per 3D image and 1PB data per moderated
hospital per year.
13. Computational Bioimaging; David Skinner, Joaquin Correa, Daniela
Ushizima, Joerg Meyer, LBNL
Application: Data delivered from bioimaging is increasingly automated, higher
resolution, and multi-modal. This has created a data analysis bottleneck that, if
resolved, can advance the biosciences discovery through Big Data techniques.
Current Approach: The current piecemeal analysis approach does not scale to
situation where a single scan on emerging machines is 32TB and medical diagnostic
36

imaging is annually around 70 PB excluding cardiology. One needs a web-based one-
stop-shop for high performance, high throughput image processing for producers and
consumers of models built on bio-imaging data.
Futures: Our goal is to solve that bottleneck with extreme scale computing with
community-focused science gateways to support the application of massive data
analysis toward massive imaging data sets. Workflow components include data
acquisition, storage, enhancement, minimizing noise, segmentation of regions of
interest, crowd-based selection and extraction of features, and object classification,
and organization, and search. Use ImageJ, OMERO, VolRover, advanced segmentation
and feature detection software.
14. Genomic Measurements; Justin Zook, NIST
Application: NIST/Genome in a Bottle Consortium integrates data from multiple
sequencing technologies and methods to develop highly confident characterization of
whole human genomes as reference materials, and develop methods to use these
Reference Materials to assess performance of any genome sequencing run.
Current Approach: The storage of ~40TB NFS at NIST is full; there are also PBs of
genomics data at NIH/NCBI. Use Open-source sequencing bioinformatics software
from academic groups (UNIX-based) on a 72 core cluster at NIST supplemented by
larger systems at collaborators.
Futures: DNA sequencers can generate ~300GB compressed data/day which volume
has increased much faster than Moore’s Law. Future data could include other ‘omics’
measurements, which will be even larger than DNA sequencing. Clouds have been
explored
15. Comparative analysis for metagenomes and genomes; Ernest
Szeto, LBNL (Joint Genome Institute)
Application: Given a metagenomic sample, (1) determine the community composition
in terms of other reference isolate genomes, (2) characterize the function of its genes,
(3) begin to infer possible functional pathways, (4) characterize similarity or
dissimilarity with other metagenomic samples, (5) begin to characterize changes in
community composition and function due to changes in environmental pressures, (6)
isolate sub-sections of data based on quality measures and community composition.
Current Approach: integrated comparative analysis system for metagenomes and
genomes, front ended by an interactive Web UI with core data, backend
precomputations, batch job computation submission from the UI. Provide interface to
standard bioinformatics tools (BLAST, HMMER, multiple alignment and phylogenetic
tools, gene callers, sequence feature predictors…).
Futures: Management of heterogeneity of biological data is currently performed by
relational database management system (Oracle). Unfortunately, it does not scale for
37

even the current volume 50TB of data. NoSQL solutions aim at providing an
alternative but unfortunately they do not always lend themselves to real time
interactive use, rapid and parallel bulk loading, and sometimes have issues regarding
robustness.
16. Individualized Diabetes Management; Ying Ding , Indiana
University
Application: Diabetes is a growing illness in world population, affecting both
developing and developed countries. Current management strategies do not
adequately take into account of individual patient profiles, such as co-morbidities and
medications, which are common in patients with chronic illnesses. Need to use
advanced graph-based data mining techniques applied to EHR converted into a RDF
graph, to search for Diabetes patients and extract their EHR data for outcome
evaluation.
Current Approach: Typical patient data records composed of 100 controlled
vocabulary values and 1000 continuous values. Most values have a timestamp. Need to
change traditional paradigm of relational row-column lookup to semantic graph
traversal.
Futures: Identify similar patients from a large Electronic Health Record (EHR) database,
i.e. an individualized cohort, and evaluate their respective management outcomes to
formulate most appropriate solution suited for a given patient with diabetes. Use
efficient parallel retrieval algorithms, suitable for cloud or HPC, using open source
Hbase with both indexed and custom search to identify patients of possible interest.
Use Semantic Linking for Property Values method to convert an existing data
warehouse at Mayo Clinic, called the Enterprise Data Trust (EDT), into RDF triples that
enables one to find similar patients through linking of both vocabulary-based and
continuous values. The time dependent properties need to be processed before query
to allow matching based on derivatives and other derived properties.
17. Statistical Relational Artificial Intelligence for Health Care; Sriraam
Natarajan, Indiana University
Application: The goal of the project is to analyze large, multi-modal medical data
including different data types such as imaging, EHR, genetic and natural language. This
approach employs the relational probabilistic models that have the capability of
handling rich relational data and modeling uncertainty using probability theory. The
software learns models from multiple data types and can possibly integrate the
information and reason about complex queries. Users can provide a set of descriptions
– say for instance, MRI images and demographic data about a particular subject. They
can then query for the onset of a particular disease (say Alzheimer’s) and the system
will then provide a probability distribution over the possible occurrence of this disease.
Current Approach: A single server can handle a test cohort of a few hundred patients
with associated data of 100’s of GB.
38

Futures: A cohort of millions of patient can involve petabyte datasets. Issues include
availability of too much data (as images, genetic sequences etc) that can make the
analysis complicated. A major challenge lies in aligning the data and merging from
multiple sources in a form that can be made useful for a combined analysis. Another
issue is that sometimes, large amount of data is available about a single subject but
the number of subjects themselves is not very high (i.e., data imbalance). This can
result in learning algorithms picking up random correlations between the multiple data
types as important features in analysis.
18. World Population Scale Epidemiological Study; Madhav Marathe,
Stephen Eubank or Chris Barrett, Virginia Tech
Application: One needs reliable real-time prediction and control of pandemic similar
to the 2009 H1N1 influenza. In general one is addressing contagion diffusion of
various kinds: information, diseases, social unrest can be modeled and computed. All
of them can be addressed by agent-based models that utilize the underlying
interaction network to study the evolution of the desired phenomena.
Current Approach: (a) Build a synthetic global population. (b) Run simulations over the
global population to reason about outbreaks and various intervention strategies.
Current 100TB dataset generated centrally with MPI based simulation system written in
Charm++. Parallelism is achieved by exploiting the disease residence time period.
Futures: Use large social contagion models to study complex global scale issues
19. Social Contagion Modeling for Planning, Public Health and
Disaster Management; Madhav Marathe or Chris Kuhlman, Virginia
Tech
Application: Model Social behavior including national security, public health, viral
marketing, city planning, disaster preparedness. In a social unrest application, people
take to the streets to voice unhappiness with government leadership. There are
citizens that both support and oppose government. Quantify the degrees to which
normal business and activities are disrupted owing to fear and anger. Quantify the
possibility of peaceful demonstrations, violent protests. Quantify the potential for
government responses ranging from appeasement, to allowing protests, to issuing
threats against protestors, to actions to thwart protests. To address these issues,
must have fine-resolution models (at level of individual people, vehicles, and
buildings) and datasets.
Current Approach: The social contagion model infrastructure includes different types
of human-to-human interactions (e.g., face-to-face versus online media) to be
simulated. It takes not only human-to-human interactions into account, but also
interactions among people, services (e.g., transportation), and infrastructure (e.g.,
39

internet, electric power). These activity models are generated from averages like
census data.
Futures: Data fusion a big issue; how should one combine data from different sources
and how to deal with missing or incomplete data? take into account heterogeneous
features of 100s of millions or billions of individuals, models of cultural variations
across countries that are assigned to individual agents? How to validate these large
models?
20.Biodiversity and LifeWatch; Wouter Los, Yuri Demchenko,
University of Amsterdam
Application: Research and monitor different ecosystems, biological species, their
dynamics and migration with a mix of custom sensors and data access/processing and
a federation with relevant projects in area. Particular case studies: Monitoring alien
species, monitoring migrating birds, wetlands. See ENVRI for integration of LifeWatch
with other environmental e-infrastructures.
Futures: LifeWatch initiative will provide integrated access to a variety of data,
analytical and modeling tools as served by a variety of collaborating initiatives. Another
service is offered with data and tools in selected workflows for specific scientific
communities. In addition, LifeWatch will provide opportunities to construct
personalized ‘virtual labs', also allowing one to enter new data and analytical tools.
New data will be shared with the data facilities cooperating with LifeWatch. LifeWatch
operates the Global Biodiversity Information facility and Biodiversity Catalogue that is
Biodiversity Science Web Services Catalogue. Data includes ‘omics, species information,
ecological information (such as biomass, population density etc.), ecosystem data
(such as CO2 fluxes. Algal blooming, water and soil characteristics)
Deep Learning and Social Media
21. Large-scale Deep Learning; Adam Coates , Stanford University
Application: Increase the size of datasets and models that can be tackled with deep
learning algorithms. Large models (e.g., neural networks with more neurons and
connections) combined with large datasets are increasingly the top performers in
benchmark tasks for vision, speech, and Natural Language Processing. One needs to
train a deep neural network from a large (>>1TB) corpus of data (typically imagery,
video, audio, or text). Such training procedures often require customization of the
neural network architecture, learning criteria, and dataset pre-processing. In addition
to the computational expense demanded by the learning algorithms, the need for rapid
prototyping and ease of development is extremely high.
Current Approach: Largest applications so far are to face recognition with 10 million
images and up to 11 billion parameters on a 64 GPU HPC Infiniband cluster. Both
40

supervised (using existing classified images) and unsupervised applications
investigated.
Futures: Large datasets of 100TB or more may be necessary in order to exploit the
representational power of the larger models. Training a self-driving car could take 100
million megapixel images. Deep Learning shares many characteristics with the broader
field of machine learning. The paramount requirements are high computational
throughput for mostly dense linear algebra operations, and extremely high
productivity for researcher exploration. One needs integration of high performance
libraries with high level (python) prototyping environments.
22. Organizing large-scale, unstructured collections of consumer
photos; David Crandall, Indiana University
Application: Produce 3d reconstructions of scenes using collections of millions to
billions of consumer images, where neither the scene structure nor the camera
positions are known a priori. Use resulting 3d models to allow efficient and effective
browsing of large-scale photo collections by geographic position. Geolocate new
images by matching to 3d models. Perform object recognition on each image. 3d
reconstruction can be posed as a robust non-linear least squares optimization problem
in which observed (noisy) correspondences between images are constraints and
unknowns are 6-d camera pose of each image and 3-d position of each point in the
scene.
Current Approach: Hadoop cluster with 480 cores processing data of initial
applications. Note over 500 billion images on Facebook and over 5 billion on Flickr
with over 500 million images added to social media sites each day.
Futures: Need many analytics including feature extraction, feature matching, and
large-scale probabilistic inference, which appear in many or most computer vision and
image processing problems, including recognition, stereo resolution, and image
denoising. Need to visualize large-scale 3-d reconstructions, and navigate large-scale
collections of images that have been aligned to maps.
23. Truthy: Information diffusion research from Twitter Data; Filippo
Menczer, Alessandro Flammini, Emilio Ferrara, Indiana University
Application: Understanding how communication spreads on socio-technical networks.
Detecting potentially harmful information spread at the early stage (e.g., deceiving
messages, orchestrated campaigns, untrustworthy information, etc.)
Current Approach: 1) Acquisition and storage of a large volume (30 TB a year
compressed) of continuous streaming data from Twitter (~100 million messages per
day, ~500GB data/day increasing over time); (2) near real-time analysis of such data,
for anomaly detection, stream clustering, signal classification and online-learning; (3)
data retrieval, big data visualization, data-interactive Web interfaces, public API for
41

data querying. Use Python/SciPy/NumPy/MPI for data analysis. Information diffusion,
clustering, and dynamic network visualization capabilities already exist.
Futures: Truthy plans to expand incorporating Google+ and Facebook. Need to move
towards Hadoop/IndexedHBase & HDFS distributed storage. Use Redis as a in-memory
database as a buffer for real-time analysis. Need streaming clustering, anomaly
detection and online learning.
24. CINET: Cyberinfrastructure for Network (Graph) Science and
Analytics; Madhav Marathe or Keith Bisset, Virginia Tech
Application: CINET provides a common web-based platform for accessing various (i)
network and graph analysis tools such as SNAP, NetworkX, Galib, etc. (ii) real-world
and synthetic networks, (iii) computing resources and (iv) data management systems to
the end-user in a seamless manner.
Current Approach: CINET uses an Infiniband connected high performance computing
cluster with 720 cores to provide HPC as a service. It is being used for research and education.
Futures: As the repository grows, we expect a rapid growth to lead to over 1000-5000
networks and methods in about a year. As more fields use graphs of increasing size,
parallel algorithms will be important. Data manipulation and bookkeeping of the
derived data for users is a challenge there are no well-defined and effective models
and tools for management of various graph data in a unified fashion
25. NIST Information Access Division analytic technology performance
measurement, evaluations, and standards; John Garofolo, NIST
Application: Develop performance metrics, measurement methods, and community
evaluations to ground and accelerate the development of advanced analytic
technologies in the areas of speech and language processing, video and multimedia
processing, biometric image processing, and heterogeneous data processing as well as
the interaction of analytics with users. Typically employ one of two processing models:
1) Push test data out to test participants and analyze the output of participant systems,
2) Push algorithm test harness interfaces out to participants and bring in their
algorithms and test them on internal computing clusters.
Current Approach: Large annotated corpora of unstructured/semi-structured text,
audio, video, images, multimedia, and heterogeneous collections of the above
including ground truth annotations for training, developmental testing, and summative
evaluations. The test corpora exceed 900M Web pages occupying 30 TB of storage,
100M tweets, 100M ground-truthed biometric images, several hundred thousand
partially ground-truthed video clips, and terabytes of smaller fully ground-truthed test
collections.
42

Futures: Even larger data collections are being planned for future evaluations of
analytics involving multiple data streams and very heterogeneous data. As well as
larger datasets, future includes testing of streaming algorithms with multiple
heterogeneous data. Use of clouds being explored.
The Ecosystem for Research
26. DataNet Federation Consortium DFC; Reagan Moore, University of
North Carolina at Chapel Hill
Application: Promote collaborative and interdisciplinary research through federation of
data management systems across federal repositories, national academic research
initiatives, institutional repositories, and international collaborations. The
collaboration environment runs at scale: petabytes of data, hundreds of millions of
files, hundreds of millions of metadata attributes, tens of thousands of users, and a
thousand storage resources.
Current Approach: Currently 25 science and engineering domains have projects that
rely on the iRODS (Integrated Rule Oriented Data System) policy-based data
management system including major NSF projects such as Ocean Observatories
Initiative (sensor archiving); Temporal Dynamics of Learning Center (Cognitive science
data grid); the iPlant Collaborative (plant genomics); Drexel engineering digital library;
Odum Institute for social science research (data grid federation with Dataverse). iRODS
currently manages petabytes of data, hundreds of millions of files, hundreds of
millions of metadata attributes, tens of thousands of users, and a thousand storage
resources. It interoperates with workflow systems (NCSA Cyberintegrator, Kepler,
Taverna), cloud and more traditional storage models and different transport protocols.
27. The ‘Discinnet process’, metadata <-> big data global experiment;
P. Journeau, Discinnet Labs
Application: Discinnet has developed a web 2.0 collaborative platform and research
prototype as a pilot installation now becoming deployed to be appropriated and tested
by researchers from a growing number and diversity of research fields through
communities belonging a diversity of domains.
Its goal is to reach a wide enough sample of active research fields represented as
clusters–researchers projected and aggregating within a manifold of mostly shared
experimental dimensions–to test general, hence potentially interdisciplinary,
epistemological models throughout the present decade.
Current Approach: Currently 35 clusters started with close to 100 awaiting more
resources and potentially much more open for creation, administration and animation
by research communities. Examples range from optics, cosmology, materials,
microalgae, health to applied math, computation, rubber and other chemical products/
issues.
43

Futures: Discinnet itself would not be Bigdata but rather will generate metadata when
applied to a cluster that involves Bigdata. In interdisciplinary integration of several
fields, the process would reconcile metadata from many complexity levels.
28. Semantic Graph-search on Scientific Chemical and Text-based
Data; Talapady Bhat, NIST
Application: Establish social media-based infrastructure, terminology and semantic
data-graphs to annotate and present technology information using ‘root’ and rule-
based methods used primarily by some Indo-European languages like Sanskrit and
Latin.
Futures: Create a cloud infrastructure for social media of scientific information where
many scientists from various parts of the world can participate and deposit results of
their experiment. Some of the issues that one has to resolve prior to establishing a
scientific social media are: a) How to minimize challenges related to local language and
its grammar? b) How to determine the ‘data-graph’ to place an information in an
intuitive way without knowing too much about the data management? c) How to find
relevant scientific data without spending too much time on the internet? Start with
resources like the Open Government movement and protein databank. This effort
includes many local and networked resources. Developing an infrastructure to
automatically integrate information from all these resources using data-graphs is a
challenge that we are trying to solve. Good database tools and servers for data-graph
manipulation are needed.
29. Light source beamlines; Eli Dart, LBNL
Application: Samples are exposed to X-rays from light sources in a variety of
configurations depending on the experiment. Detectors (essentially high-speed digital
cameras) collect the data. The data are then analyzed to reconstruct a view of the
sample or process being studied.
Current Approach: A variety of commercial and open source software is used for data
analysis – examples including Octopus for Tomographic Reconstruction, Avizo (http://
vsg3d.com) and FIJI (a distribution of ImageJ) for Visualization and Analysis. Data
transfer is accomplished using physical transport of portable media (severely limits
performance) or using high-performance GridFTP, managed by Globus Online or
workflow systems such as SPADE.
Futures: Camera resolution is continually increasing. Data transfer to large-scale
computing facilities is becoming necessary because of the computational power
required to conduct the analysis on time scales useful to the experiment. Large
number of beamlines (e.g. 39 at LBNL ALS) means that aggregate data load is likely to
increase significantly over the coming years and need for a generalized infrastructure
for analyzing gigabytes per second of data from many beamline detectors at multiple
facilities.
44

Astronomy and Physics
30. Catalina Real-Time Transient Survey (CRTS): a digital, panoramic,
synoptic sky survey; S. G. Djorgovski, Caltech
Application: The survey explores the variable universe in the visible light regime, on
time scales ranging from minutes to years, by searching for variable and transient
sources. It discovers a broad variety of astrophysical objects and phenomena,
including various types of cosmic explosions (e.g., Supernovae), variable stars,
phenomena associated with accretion to massive black holes (active galactic nuclei)
and their relativistic jets, high proper motion stars, etc. The data are collected from 3
telescopes (2 in Arizona and 1 in Australia), with additional ones expected in the near
future (in Chile).
Current Approach: The survey generates up to ~ 0.1 TB on a clear night with a total of
~100 TB in current data holdings. The data are preprocessed at the telescope, and
transferred to Univ. of Arizona and Caltech, for further analysis, distribution, and
archiving. The data are processed in real time, and detected transient events are
published electronically through a variety of dissemination mechanisms, with no
proprietary withholding period (CRTS has a completely open data policy). Further data
analysis includes classification of the detected transient events, additional observations
using other telescopes, scientific interpretation, and publishing. In this process, it
makes a heavy use of the archival data (several PB’s) from a wide variety of
geographically distributed resources connected through the Virtual Observatory (VO)
framework.
Futures: CRTS is a scientific and methodological testbed and precursor of larger
surveys to come, notably the Large Synoptic Survey Telescope (LSST), expected to
operate in 2020’s and selected as the highest-priority ground-based instrument in the
2010 Astronomy and Astrophysics Decadal Survey. LSST will gather about 30 TB per
night.
31. DOE Extreme Data from Cosmological Sky Survey and Simulations;
Salman Habib, Argonne National Laboratory; Andrew Connolly,
University of Washington
Application: A cosmology discovery tool that integrate simulations and observation to
clarify the nature of dark matter, dark energy, and inflation, some of the most exciting,
perplexing, and challenging questions facing modern physics including the properties
of fundamental particles affecting the early universe. The simulations will generate
comparable data sizes to observation.
Futures: Data sizes are Dark Energy Survey (DES) 4 PB in 2015; Zwicky Transient
Factory (ZTF) 1 PB/year in 2015; Large Synoptic Sky Survey (LSST see CRTS description) 7
45

PB/year in 2019; Simulations > 10 PB in 2017. Huge amounts of supercomputer time
(over 200M hours) will be used.
32. Particle Physics: Analysis of LHC Large Hadron Collider Data: Discovery
of Higgs particle; Geoffrey Fox, Indiana University; Eli Dart, LBNL
Application: Study CERN LHC (Large Hadron Collider) Accelerator and Monte Carlo
producing events describing particle-apparatus interaction. Processed information
defines physics properties of events (lists of particles with type and momenta). These
events are analyzed to find new effects; both new particles (Higgs) and present
evidence that conjectured particles (Supersymmetry) have not been detected. LHC has
a few major experiments including ATLAS and CMS. These experiments have global
participants (for example CMS has 3600 participants from 183 institutions in 38
countries), and so the data at all levels is transported and accessed across continents.
Current Approach: The LHC experiments are pioneers of a distributed Big Data
science infrastructure, and several aspects of the LHC experiments’ workflow highlight
issues that other disciplines will need to solve. These include automation of data
distribution, high performance data transfer, and large-scale high-throughput
computing. Grid analysis with 260,000 cores running “continuously” arranged in 3 tiers
(CERN, “Continents/Countries”. “Universities”). Uses “High Throughput
Computing” (Pleasing parallel) architecture. 15 Petabytes data gathered each year from
Accelerator data and Analysis. Specifically in 2012 ATLAS had at Brookhaven National
Laboratory (BNL) 8PB Tier1 tape; BNL over 10PB Tier1 disk and US Tier2 centers 12PB
disk cache. CMS has similar data sizes.
Futures: Analysis system set up before clouds. Clouds have been shown to be effective
for this type of problem but successful operation of current Grid argues against
architecture change. Object databases (Objectivity) were explored for this use case but
not adopted.
Earth, Environmental and Polar Science
33. EISCAT 3D incoherent scatter radar system; Yin Chen, Cardiff
University; Ingemar Häggström, Ingrid Mann, Craig Heinselman,
EISCAT Science Association
Application: EISCAT, the European Incoherent Scatter Scientific Association, conducts
research on the lower, middle and upper atmosphere and ionosphere using the
incoherent scatter radar technique. This technique is the most powerful ground-based
tool for these research applications. EISCAT studies instabilities in the ionosphere, as
well as investigating the structure and dynamics of the middle atmosphere. It is also a
diagnostic instrument in ionospheric modification experiments with addition of a
separate Heating facility. Currently EISCAT operates 3 of the 10 major incoherent radar
46

scattering instruments worldwide with its facilities in in the Scandinavian sector, north
of the Arctic Circle.
Current Approach: The current terabytes per year rates do not present special
challenges.
Futures: EISCAT_3D will consist of a core site with a transmitting and receiving radar
arrays and four sites with receiving antenna arrays at some 100 km from the core. The
fully operational 5-site system will generate several thousand times data of current
system with 40 PB/year in 2022 and is expected to operate for 30 years. EISCAT 3D
data e-Infrastructure plans to use the high performance computers for central site data
processing and high throughput computers for mirror sites data processing.
Downloading the full data is not time critical, but operations require real-time
information about certain pre-defined events to be sent from the sites to the operation
center and a real-time link from the operation center to the sites to set the mode of
radar operation on with immediate action.
34. ENVRI, Common Operations of Environmental Research
Infrastructure; Yin Chen, Cardiff University
Application: ENVRI Research Infrastructures (ENV RIs) addresses European distributed,
long-term, remote controlled observational networks focused on understanding
processes, trends, thresholds, interactions and feedbacks and increasing the predictive
power to address future environmental challenges. ENVRI includes ICOS is a European
distributed infrastructure dedicated to the monitoring of greenhouse gases (GHG)
through its atmospheric, ecosystem and ocean networks; EURO-Argo is the European
contribution to Argo, which is a global ocean observing system; EISCAT-3D separately
described is a European new-generation incoherent-scatter research radar for upper
atmospheric science; LifeWatch separately described is an e-science Infrastructure for
biodiversity and ecosystem research; EPOS is a European Research Infrastructure on
earthquakes, volcanoes, surface dynamics and tectonics; EMSO is a European network
of seafloor observatories for the long-term monitoring of environmental processes
related to ecosystems, climate change and geo-hazards; IAGOS Aircraft for global
observing system; SIOS Svalbard arctic Earth observing system
Current Approach: ENVRI develops a Reference Model (ENVRI RM) as a common
ontological framework and standard for the description and characterization of
computational and storage infrastructures in order to achieve seamless interoperability
between the heterogeneous resources of different infrastructures. The ENVRI RM serves
as a common language for community communication, providing a uniform framework
into which the infrastructure’s components can be classified and compared, also
serving to identify common solutions to common problems. Note data sizes in a given
infrastructure vary from gigabytes to petabytes per year.
47

Futures: ENVRI’s common environment will empower the users of the collaborating
environmental research infrastructures and enable multidisciplinary scientists to
access, study and correlate data from multiple domains for "system level" research. It
provides Bigdata requirements coming from interdisciplinary research.
35. Radar Data Analysis for CReSIS Remote Sensing of Ice Sheets;
Geoffrey Fox, Indiana University
Application: This data feeds into intergovernmental Panel on Climate Change (IPCC)
and uses custom radars to measures ice sheet bed depths and (annual) snow layers at
the North and South poles and mountainous regions.
Current Approach: The initial analysis is currently Matlab signal processing that
produces a set of radar images. These cannot be transported from field over Internet
and are typically copied to removable few TB disks in the field and flown “home” for
detailed analysis. Image understanding tools with some human oversight find the
image features (layers) that are stored in a database front-ended by a Geographical
Information System. The ice sheet bed depths are used in simulations of glacier flow.
The data is taken in “field trips” that each currently gather 50-100 TB of data over a
few week period.
Futures: An order of magnitude more data (petabyte per mission) is projected with
improved instrumentation. Demands of processing increasing field data in an
environment with more data but still constrained power budget, suggests low power/
performance architectures such as GPU systems.
36. UAVSAR Data Processing, Data Product Delivery, and Data
Services; Andrea Donnellan and Jay Parker, NASA JPL
Application: Synthetic Aperture Radar (SAR) can identify landscape changes caused by
seismic activity, landslides, deforestation, vegetation changes and flooding. This is for
example used to support earthquake science as well as disaster management. This use
case supports the storage, application of image processing and visualization of this
geo-located data with angular specification.
Current Approach: Data from planes and satellites is processed on NASA computers
before being stored after substantial data communication. The data is made public as
soon as processed and requires significant curation due to instrumental glitches. The
current data size is ~150TB.
Futures: The data size would increase dramatically if Earth Radar Mission launched.
Clouds are suitable hosts but are not used today in production.
48

37. NASA LARC/GSFC iRODS Federation Testbed; Brandi Quam, NASA
Langley Research Center
Application: NASA Center for Climate Simulation (NCCS) and NASA Atmospheric
Science Data Center (ASDC) have complementary data sets, each containing vast
amounts of data that is not easily shared and queried. Climate researchers, weather
forecasters, instrument teams, and other scientists need to access data from across
multiple datasets in order to compare sensor measurements from various instruments,
compare sensor measurements to model outputs, calibrate instruments, look for
correlations across multiple parameters, etc.
Current Approach: The data includes MERRA (described separately) and NASA Clouds
and Earth's Radiant Energy System (CERES) EBAF(Energy Balanced And Filled)-TOA(Top
of Atmosphere) Product which is about 420MB and Data from the EBAF-Surface Product
which is about 690MB. Data grows with each version update (about every six months).
To analyze, visualize and otherwise process data from heterogeneous datasets is
currently a time consuming effort that requires scientists to separately access, search
for, and download data from multiple servers and often the data is duplicated without
an understanding of the authoritative source. Often time accessing data is greater than
scientific analysis time. Current datasets are hosted on modest size (144 to 576 cores)
Infiniband clusters.
Futures: The improved access will be enabled through the use of iRODS that enables
parallel downloads of datasets from selected replica servers that can be geographically
dispersed, but still accessible by users worldwide. iRODS operation will be enhanced
with semantically enhanced metadata, managed via a highly precise Earth Science
ontology. Cloud solutions will be explored.
38. MERRA Analytic Services MERRA/AS; John L. Schnase & Daniel Q.
Duffy , NASA Goddard Space Flight Center
Application: This application produces global temporally and spatially consistent
synthesis of 26 key climate variables by combining numerical simulations with
observational data. Three-dimensional results are produced every 6-hours extending
from 1979-2012. This supports important applications like the intergovernmental
Panel on Climate Change (IPCC) research and the NASA/Department of Interior
RECOVER wild land fire decision support system; these applications typically involve
integration of MERRA with other datasets.
Current Approach: MapReduce is used to process a current total of 480TB. The
current system is hosted on a 36 node Infiniband cluster
Futures: Clouds are being investigated. The data is growing by one TB a month.
49

39. Atmospheric Turbulence - Event Discovery and Predictive
Analytics; Michael Seablom, NASA HQ
Application: This builds datamining on top of reanalysis products including the North
American Regional Reanalysis (NARR) and the Modern-Era Retrospective-Analysis for
Research (MERRA) from NASA where latter described earlier. The analytics correlate
aircraft reports of turbulence (either from pilot reports or from automated aircraft
measurements of eddy dissipation rates) with recently completed atmospheric re-
analyses. This is of value to aviation industry and to weather forecasters. There are no
standards for re-analysis products complicating system where MapReduce is being
investigated. The reanalysis data is hundreds of terabytes and slowly updated whereas
turbulence is smaller in size and implemented as a streaming service.
Current Approach: Current 200TB dataset can be analyzed with MapReduce or the like using SciDB or
other scientific database.
Futures: The dataset will reach 500TB in 5 years. The initial turbulence case can be
extended to other ocean/atmosphere phenomena but the analytics would be different
in each case.
40. Climate Studies using the Community Earth System Model at
DOE’s NERSC center; Warren Washington, NCAR
Application: We need to understand and quantify contributions of natural and
anthropogenic-induced patterns of climate variability and change in the 20th and 21st
centuries by means of simulations with the Community Earth System Model (CESM).
The results of supercomputer simulations across the world need to be stored and
compared.
Current Approach: The Earth Systems Grid (ESG) enables world wide access to Peta/
Exa-scale climate science data with multiple petabytes of data at dozens of federated
sites worldwide. The ESG is recognized as the leading infrastructure for the
management and access of large distributed data volumes for climate change research.
It supports the Coupled Model Intercomparison Project (CMIP), whose protocols enable
the periodic assessments carried out by the Intergovernmental Panel on Climate
Change (IPCC).
Futures: Rapid growth of data with 30 PB produced at NERSC (assuming 15 end-to-end climate
change experiments) in 2017 and many times more this worldwide.
41. DOE-BER Subsurface Biogeochemistry Scientific Focus Area; Deb
Agarwal, LBNL
Application: Development of a Genome-Enabled Watershed Simulation Capability
(GEWaSC) that will provide a predictive framework for understanding how genomic
information stored in a subsurface microbiome, affects biogeochemical watershed
50

functioning; how watershed-scale processes affect microbial functioning; and how
these interactions co-evolve.
Current Approach: Current modeling capabilities can represent processes occurring
over an impressive range of scales (ranging from a single bacterial cell to that of a
contaminant plume). Data crosses all scales from genomics of the microbes in the soil
to watershed hydro-biogeochemistry. Data are generated by the different research
areas and include simulation data, field data (hydrological, geochemical, geophysical),
‘omics data, and observations from laboratory experiments.
Futures: Little effort to date has been devoted to developing a framework for
systematically connecting scales, as is needed to identify key controls and to simulate
important feedbacks. GEWaSC will develop a simulation framework that formally scales
from genomes to watersheds and will synthesize diverse and disparate field,
laboratory, and simulation datasets across different semantic, spatial, and temporal
scales.
42.DOE-BER AmeriFlux and FLUXNET Networks; Deb Agarwal, LBNL
Application: AmeriFlux and FLUXNET are US and world collections respectively of
sensors that observe trace gas fluxes (CO2, water vapor) across a broad spectrum of
times (hours, days, seasons, years, and decades) and space. Moreover, such datasets
provide the crucial linkages among organisms, ecosystems, and process-scale studies
—at climate-relevant scales of landscapes, regions, and continents—for incorporation
into biogeochemical and climate models.
Current Approach: Software includes EddyPro, Custom analysis software, R, python,
neural networks, Matlab. There are ~150 towers in AmeriFlux and over 500 towers
distributed globally collecting flux measurements.
Futures: Field experiment data taking would be improved by access to existing data
and automated entry of new data via mobile devices. Need to support interdisciplinary
studies integrating diverse data sources.
Appendix C. Actors and Roles
From http://www.smartplanet.com/blog/bulletin/7-new-types-of-jobs-created-by-big-data/
682 The job roles are mapped to elements of the Reference Architecture in red
“Here are 7 new types of jobs being created by Big Data:
1. Data scientists: This emerging role is taking the lead in processing raw data and
determining what types of analysis would deliver the best results. Typical
51

backgrounds, as cited by Harbert, include math and statistics, as well as artificial
intelligence and natural language processing. (Analytics)
2. Data architects: Organizations managing Big Data need professionals who will
be able to build a data model, and plan out a roadmap of how and when various
data sources and analytical tools will come online, and how they will all fit
together. (Design, Develop, Deploy Tools)
3. Data visualizers: These days, a lot of decision-makers rely on information that is
presented to them in a highly visual format — either on dashboards with colorful
alerts and “dials,” or in quick-to-understand charts and graphs. Organizations
need professionals who can “harness the data and put it in context, in layman’s
language, exploring what the data means and how it will impact the company,”
says Harbert.(Applications)
4. Data change agents: Every forward-thinking organization needs “change
agents” — usually an informal role — who can evangelize and marshal the
necessary resources for new innovation and ways of doing business. Harbert
predicts that “data change agents” may be more of a formal job title in the years
to come, driving “changes in internal operations and processes based on data
analytics.” They need to be good communicators, and a Six Sigma background
— meaning they know how to apply statistics to improve quality on a continuous
basis — also helps. (Not applicable to Reference Architecture)
5. Data engineer/operators: These are the people that make the Big Data
infrastructure hum on a day-to-day basis. “They develop the architecture that
helps analyze and supply data in the way the business needs, and make sure
systems are performing smoothly,” says Harbert. (Data Processing and Data
Stores)
6. Data stewards: Not mentioned in Harbert’s list, but essential to any analytics-
driven organization, is the emerging role of data steward. Every bit and byte of
data across the enterprise should be owned by someone — ideally, a line of
business. Data stewards ensure that data sources are properly accounted for,
and may also maintain a centralized repository as part of a Master Data
Management approach, in which there is one “gold copy” of enterprise data to be
referenced. (Data Resource Management)
7. Data virtualization/cloud specialists: Databases themselves are no longer as
unique as they use to be. What matters now is the ability to build and maintain a
virtualized data service layer that can draw data from any source and make it
available across organizations in a consistent, easy-to-access manner.
Sometimes, this is called “Database-as-a-Service.” No matter what it’s called,
organizations need professionals that can also build and support these
virtualized layers or clouds.” (Infrastructure)
52

Appendix D. Big Data Questions from a Customer Perspective
for the NBD-WG and Draft Answers
These questions are meant to be from the perspective of a target customer for the
September deliverables e.g. an IT executive considering a Big Data project.
The starting point is the initial NIST Big Data definition.
------------------------------------------------------------------------
“Big Data refers to digital data volume, velocity and/or variety [,veracity] that:
· enable novel approaches to frontier questions previously
· inaccessible or impractical using current or conventional
· methods; and/or
· exceed the capacity or capability of current or conventional
· methods and systems”
------------------------------------------------------------------------
This definition implies that Big Data solutions must use new developments beyond
“current or conventional methods and systems”. Previous data architectures,
technologies, requirements, taxonomies, etc. can be a starting point but need to
be expanded for Big Data solutions. A possible user of Big Data solutions will be very
interested in answering the following questions:
General Questions
Q1. What are the new developments that are included in Big Data solutions?
A1. The essential new development in Big Data technology was the adoption of a scale-
out approach to data storage and data processing (e.g. HDFS, Map-Reduce). This
development was necessitated by the volume and velocity requirements that could not
be efficiently handled by scale-up approaches. Large scale-out deployments introduce
new capabilities (e.g. in analytics) and challenges (e.g. frequent fault tolerance) that
have been labelled “Big Data”.
NoSQL databases to handle data variety are also often considered part of Big Data
technology However there have been earlier non-SQL data stores (e.g. hierarchical,
network, object). The key new element is once again the requirement to scale-out data
storage and processing.
Q2. How do the new developments address the issues of needed capacity and
capability?
A2. Scale-out approaches using commodity components can be expanded easily from a
hardware perspective. In general, scale-out architectures will have higher capacity and
53

throughput with reduced performance and component reliability than scale-up systems.
To achieve the higher throughput, it will be necessary to reduce distributed data and
process dependencies across components (e.g. avoiding SQL joins and tightly coupled
tasks). New Big Data scalable algorithms will often be needed for analytic and machine
learning applications.
Q3. What are the strengths and weaknesses of these new developments?
A3. The key strength of robust Big Data technology is the ability to handle very large
scale data storage and processing. The technology originated in Internet-scale
companies (e.g. Google, Facebook, Yahoo) and was past on to the open source
community (e.g. Apache) and then commercialized by multiple startup companies (e.g.
Cloudera, Hortonworks, MapR) who have partnered with larger companies (e.g. IBM,
Teradata) for incorporation in enterprise deployments. The open source packages had
the basic functionality but lacked important capabilities(e.g. security) for enterprise
deployments. These missing capabilities are being addressed by commercial vendors.
Another weakness of the first generation packages (e.g. Hadoop 1.0) was the need to
run in batch mode and to write intermediate results during iterations to disk drives. This
weakness is being addressed by the open source community (e.g. Apache Yarn),
research community(e.g. UC Berkeley Spark) and vendors (e.g. Hortonworks Tez)
Q4. What are the new applications that are enabled by Big Data solutions?
A4. Some applications that are enabled include very large-scale stream processing,
data ingestion and storage, ETL, and analytics. In general, Big Data technology does
not replace existing enterprise technology but supplements it. Specifically Big Data
stores can be used as a data source for analytic databases and visualization
applications.
Q5. Are there any best practices or standards for the use of Big Data solutions?
A5. Several industry groups (e.g. Open Data Center Alliance, Cloud Security Alliance)
are working on Big Data Best Practices. There are also articles from individual vendors
and analysts but there is no common consensus. It would be useful to develop a set of
consensus Best Practices based on experience. The development of de jure Big Data
specific standards is at an early stage. Apache’s Hadoop ecosystem is a generally
accepted industry standard. It would be useful to have a standards life cycle model that
could describe the movement of technology from open source implementations to
industry-standard production to de jure standards compliance.
Questions for Subgroups
Q6. What new definitions are needed to describe elements of new Big Data
solutions?
54

A6. There are many existing data technology terms that do not need to be redefined.
The essential new definitions are needed to characterize the components of a Big Data
Reference architecture. Some key elements are;
* Scale-out and scale-up
* Horizontally scaled file system (e.g. Hadoop File System)
* Horizontally scaled processing frameworks (e.g. Map-Reduce)
* NoSQL databases (Document, Key-Value, Column Oriented, Graph)
* NewSQL, In memory, and SSD-based databases
* Stream analytics and complex event processing
* Deployment models (e.g. Cloud, Enterprise)
* Consistency, Availability, Partition Tolerant (CAP) Theorem
* Batch vs Interactive Analytics and Queries
* ETL and ELT processing
* Structured, Unstructured, and Semi-Structured Data
* Map-Reduce and Batch Synchronous Parallel Processing
* Machine to Machine (M2M) data sources
Q7. How can the best Big Data solution be chosen based on use case
requirements?
A7. Technology requirements should be extracted from the use cases. The first
question is whether these requirements can be handled by conventional data
technology. If the data volume, velocity. or variety is beyond the capability of existing
system hen Big Data solution should be evaluated.
Choosing a Big Data solution will depend on the specific Use Case requirements. These
can include:
* Read-intensive vs. Write-intensive vs. Mixed
* Updatable vs. Non-updatable
* Immediate vs. Eventual Consistency and/or Availability
* Short vs Long Latency for Responses
* Predictable vs. Unpredictable Access Patterns
* Volume, Velocity, and Variety of Data
It would be useful to create a consensus matrix mapping rows of requirements to
columns of Big Data solutions to aid users in selection of the appropriate technology.
Q8. What new Security and Privacy challenge arise from new Big Data solutions?
A8. Much of the industry-standard open source Big Data technology (e.g. Hadoop
Ecosystem) was developed with only limited concern for security and privacy. Basic
authentication is supported (e.g Kerberos) but more robust safeguards are lacking. This
deficit is being addressed by the commercial suppliers of Big Data technology to
enterprises and the government.
55

Q9. How are the new Big Data developments captured in new Reference
Architectures?
A9. At a minimum, a technical Reference Architecture should show the key components
of Big Data technology in a supplier independent representation. A evaluation
benchmark could be the ability to map key elements of the industry-standard Hadoop
ecosystem (e.g. HDFS, Map-Reduce, HBase, Pig, Hive, Drill, S4, Sqoop) to
components of the supplier independent Reference Architecture.
Q10. How will systems and methods evolve to remove Big Data solution
weaknesses?
A10. The existing Big Data technologies will be strengthened by commercial suppliers
and eventually merged into overall enterprise data architectures. There are also newer
Big Data technologies (e.g. Apache Yarn, UC Berkeley Spark) under development in
academic and open source communities to address some of the weaknesses of the first
generation tools. New tools to simplify the creation, deployment, and use of Big Data
applications should also be available in the near future. The emerging technologies will
also eventually be incorporated in enterprise data architectures.
The tougher question will be developing new skill sets, organization structures,
processes, and methodologies for exploiting the capabilities of Big Data technology.
For example, applications such as knowledge discovery, predictive analytics, ad hoc
queries, and complex event detection will usually require different processes and skill
sets. Also Big Data governance questions such as the list below must be answered.
* What data sources to ingest?
* Which data to store?
* How long should data be stored?
* Who has access to data?
* Which processing. monitoring, and management tools are necessary?
* What are the output interfaces to external systems?
* How can the benefits of Big Data collection and applications be measured?
Appendix E. Solutions Glossary
Descriptions and links are listed here to provide references for technology capabilities.
Accumulo - (Database, NoSQL, Key-Value) from Apache
http://accumulo.apache.org/
Acunu Analytics - (Analytics Tool) on top of Aster Data Platform based on Cassandra
http://www.acunu.com/acunu-analytics.html
Aerospike - (Database NoSQL Key-Value) uses index in memory and data on SSDs
56

http://www.aerospike.com/
Akka - Platform for concurrent, distributed, and fault tolerant event-driven JVM applications
http://akka.io/
Alteryx - (Analytics Tool)
http://www.alteryx.com/
Ambari - (Hadoop Cluster Management) from Apache
http://incubator.apache.org/ambari/
Analytica - (Analytics Tool) from Lumina
http://www.lumina.com/why-analytica/
ArangoDB - (Database, NoSQL, Multi-model) Open source from Europe
http://www.arangodb.org/2012/03/07/avocadodbs-design-objectives
Aster - (Analytics) Combines SQL and Hadoop on top of Aster MPP Database
http://www.asterdata.com/
Asterix - (Database for unstructured data) built on top of Hyracks from UCI
http://asterix.ics.uci.edu/
Avro - (Data Serialization) from Apache
http://en.wikipedia.org/wiki/Apache_Avro
Ayasdi Iris - (Machine Learning) uses Topological Data Analysis
http://www.ayasdi.com/product/
Azkaban - (Process Scheduler) for Hadoop
http://bigdata.globant.com/?p=441
Azure Table Storage - (Database, NoSQL, Columnar) from Microsoft
http://msdn.microsoft.com/en-us/library/windowsazure/jj553018.aspx
Behemoth - (Large-scale document processing platform) from Apache
http://www.findbestopensource.com/product/behemoth
Berkeley DB - (Database)
http://www.oracle.com/technetwork/products/berkeleydb/overview/index.html
BigData Appliance - (Integrated Hardware and Software Architecture) from Oracle
includes Cloudera, Oracle NoSQL ,Oracle R and Sun Servers
http://nosql.mypopescu.com/post/15729871938/comparing-hadoop-appliances-oracles-big-data
BigML - (Analytics tool) for business end-users
57

https://bigml.com/
BigQuery - (Query Tool) on top of Google Storage
https://cloud.google.com/products/big-query
BigSheets - (BI Tool) from IBM
http://www-01.ibm.com/software/ebusiness/jstart/downloads/BigSheetsOverview.pdf
BigTable - (Database, NOSQL. Column oriented) from Google
http://en.wikipedia.org/wiki/BigTable
Caffeine - (Search Engine) from Google use BigTable Indexing
http://googleblog.blogspot.com/2010/06/our-new-search-index-caffeine.html
Cascading - (Processing) Java APIs on top of Hadoop from Concurrent
http://www.cascading.org/
Cascalog - (Query Tool using Clojure) on top of Hadoop
http://nathanmarz.com/blog/introducing-cascalog-a-clojure-based-query-language-for-
hado.html
Cassandra - (Database, NoSQL, Column oriented)
http://cassandra.apache.org/
Chukwa - (Monitoring Hadoop Clusters) from Apache
http://incubator.apache.org/chukwa/
Clojure - (Lisp-based Programming Language) compiles to JVM byte code
http://clojure.org/
Cloudant - (Distributed Database as a Service)
https://cloudant.com/
Cloudera - (Hadoop Distribution) including real-time queries
http://www.cloudera.com/content/cloudera/en/home.html
Clustrix - (NewSQL DB) runs on AWS
http://www.clustrix.com/
Coherence - (Data Grid/Caching) from Oracle
http://www.oracle.com/technetwork/middleware/coherence/overview/index.html
Colossus - (File System) Next Generation Google File System
http://www.highlyscalablesystems.com/3202/colossus-successor-to-google-file-system-gfs/
58

Continuity - (Data fabric layer) Interfaces to Hadoop Processing and data stores
http://www.continuuity.com/
Corona - (Hadoop Processing tool) used internally by Facebook and now open sourced
http://gigaom.com/2012/11/08/facebook-open-sources-corona-a-better-way-to-do-webscale-hadoop/
Couchbase - (Database, NoSQL, Document) with CouchDB and Membase capabilities
http://www.couchbase.com/
CouchDB - (Database, NoSQL, Document)
http://couchdb.apache.org/
Crunch - (Java framework for creating Map-Reduce Pipelines) from Apache
http://crunch.apache.org/
Customer Experience Digital Data Acquisition - Possible future standards from W3C
http://www.w3.org/community/custexpdata/wiki/images/9/93/W3C_CustomerExperienceDigitalDataAcquisition_Draft_v0.5.pdf
Data Tamer - (Data integration and curation tools) from MIT
http://www.cidrdb.org/cidr2013/Papers/CIDR13_Paper28.pdf
Datameer - (Analytics) built on top of Hadoop
http://www.datameer.com/
Datastax - (Integration) Built on Cassandra, Solr, Hadoop
http://www.datastax.com/
DeepDB - (Database) Supports SQL and NoSQL APIs
http://deep.is/a-new-approach-to-information-theory/
Dremel - (Query Tool) interactive for columnar DBs from Google
http://research.google.com/pubs/pub36632.html
Drill - (Query Tool) interactive for columnar DBs from Apache
http://en.wikipedia.org/wiki/Apache_Drill
Dryad - (Parallel execution environment) from Microsoft Research
http://research.microsoft.com/en-us/projects/dryad/default.aspx
Dynamo DB - (Database, NoSQL, Key-Value)
http://aws.amazon.com/dynamodb/
Elastic MapReduce - (Cloud-based MapReduce) from Amazon
http://aws.amazon.com/elasticmapreduce/
59

ElasticSearch - (Search Engine) on top of Apache Lucerne
http://www.elasticsearch.org/
Enterprise Control Language (ECL) - (Data Processing Language) from HPPC
http://hpccsystems.com/download/docs/ecl-language-reference
EUDAT(European Data Infrastructure) - European Collaborative Initiative
http://www.eudat.eu/
eXtreme Scale - (Data Grid/Caching) from IBM
http://www-03.ibm.com/software/products/us/en/websphere-extreme-scale/
F1 - (Combines relational and Hadoop processing) from Google built on Google
Spanner http://research.google.com/pubs/pub38125.html
Falcon - (Data processing and management platform) from Apache
http://wiki.apache.org/incubator/FalconProposal
Flume - (Data Collection, Aggregation, Movement)
http://flume.apache.org/
FlumeJava - (Java Library) Supports development and running data parallel pipelines
http://pages.cs.wisc.edu/~akella/CS838/F12/838-CloudPapers/FlumeJava.pdf
Fusion-io - (SSD Storage Platform) can be used to enhance HBase performance
http://www.fusionio.com/company/
GemFire - (Data Grid/Caching) from VMware
https://www.vmware.com/products/application-platform/vfabric-gemfire/overview.html
Genie - (Hadoop job allocation in Amazon Cloud) open sourced by Netflix
http://techblog.netflix.com/2013/06/genie-is-out-of-bottle.html
Gensonix - (NoSQL database) from Scientel
http://scientel.com/platform.html
Gephi - (Visualization Tool) for Graphs
https://gephi.org/features/
Gigaspaces - (Data Grid/Caching)
http://www.gigaspaces.com/
Giraph - (Graph Processing) from Apache
http://giraph.apache.org/
60

Google Refine - (Data Cleansing)
http://code.google.com/p/google-refine/
Google Storage - (Database, NoSQL, Key-Value)
https://developers.google.com/storage/
GraphLab - (Machine Learning on Graphs)
http://graphlab.org/
Graphbase - (Database, NoSQL, Graphical)
http://graphbase.net/
Greenplum - ( MPP Database. Analytic Tools, Hadoop )
http://www.greenplum.com/
GridGain - (In Memory data processing) Streaming, database, HPC
http://www.gridgain.com/products/
Grunt Shell - (Interactive Shell for Apache Pig)
http://pig.apache.org/docs/r0.7.0/setup.html#Grunt+Shell
Guavas - (Stream Analytics)
http://www.guavus-new.com/solutions/
H20 - (Math toolkit) runs on topof Hadoop
http://0xdata.com/h2o/
HBase - (Database, NoSQL, Column oriented)
http://en.wikipedia.org/wiki/HBase
Hadapt - (Combined SQL Layer and Hadoop)
http://hadapt.com/
Hadoop Distributed File System - (Distributed File System) from Apache
http://hadoop.apache.org/docs/stable/hdfs_design.html
Hama - (Processing Framework) Uses BSP model on top of HDFS
http://hama.apache.org/
Hana - (Database, NewSQL) from SAP
http://en.wikipedia.org/wiki/SAP_HANA
Haven - (Analytics Package) from HP
http://www.itworldcanada.com/news/hp-unveils-haven-for-big-data/147217
61

HAWQ - (SQL Interface to Hadoop) from Greenplum and Pivotal
http://www.greenplum.com/blog/dive-in/hawq-the-new-benchmark-for-sql-on-hadoop
HCatalog - (Table and Storage Management) for Hadoop data
http://incubator.apache.org/hcatalog/
HDF5- (A data model, library, and file format for storing/managing large complex data)
http://www.hdfgroup.org/HDF5/
Helix - (Cluster service operation, state management, fault handling) from Apache
http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn
High Performance Computing Cluster (HPCC) - (Big Data Processing Platform)
http://hpccsystems.com/why-hpcc
Hive - (Data warehouse structure on top of Hadoop)
http://en.wikipedia.org/wiki/Apache_Hive
HiveQL - (SQL-like interface on Hadoop File System)
https://www.inkling.com/read/hadoop-definitive-guide-tom-white-3rd/chapter-12/hiveql
Hortonworks - (Extensions of Hadoop)
http://hortonworks.com/
HStreaming - (Real time analytics on top of Hadoop)
http://www.hstreaming.com/
Hue - (UI and Web applications for Cloudera Hadoop) from Cloudera
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/Hue-2-User-Guide/hue2.html
Hunk - (Analytics for Hadoop File System) from Splunk
http://www.splunk.com/view/hunk/SP-CAAAH2E
Hypertable - (Database, NoSQL, Key-Value) open source runs on multiple file systems
http://hypertable.org/
Hyracks - (Parallel Data Processing) from UCI runs Pregelix (Pregel API), Hiivesterix
(Hive API), Algebrix (algebra), Iterative Map-Reduce Update (IMRU) and Hyracks jobs
www.ics.uci.edu/~rares/pub/icde11-borkar.pdf
Impala - (Ad hoc query capability for Hadoop) from Cloudera
http://blog.cloudera.com/blog/2012/10/cloudera-impala-real-time-queries-in-apache-hadoop-for-real/
Indexed Database API - (Possible future standards for NoSqL queries) from W3C
http://www.w3.org/TR/2013/CR-IndexedDB-20130704/
62

InfiniDB - (Scale-up analytic database)
http://infinidb.org/
Infochimps - (Big Data Storage and Analytics in the Cloud)
http://www.infochimps.com/
Infosphere Big Insights - (Analytic) from IBM
http://www-01.ibm.com/software/data/infosphere/biginsights/
InnoDB - (Default storage engine for MYSQL)
http://en.wikipedia.org/wiki/InnoDB
iRODS - (integrated Rule-Oriented Data System)open source from iRODS Consortium
http://www.irods.org
Isilon OneFS - (Parallel Distributed File System) from EMC
http://gigaom.com/2012/01/31/emc-delivers-on-isilon-hadoop-bundle/
Jaql -(Query Language for Hadoop) from IBM
http://www-01.ibm.com/software/data/infosphere/hadoop/jaql/
Jena - (Java framework for building semantic web applications) from Apache
http://jena.apache.org/
JSONIQ - (Query language for JSON)
http://www.jsoniq.org/
JustOneDB - (Relational DB) with flexible capabilities for Big Data
http://www.justonedb.com/
Kafka - (Publish-and-subscribe for data) from Apache
http://kafka.apache.org/
Karmasphere - (Analytics)
http://www.karmasphere.com/
Knowledge Graph - (Graphical data store) from Google
http://en.wikipedia.org/wiki/Knowledge_Graph
Knox - (Secure gateway to Hadoop) from Apache
http://knox.incubator.apache.org/
LightFlow - (Data fabric middleware) from Lightwolf Technologies
http://www.lightwolftech.com/index.php?page=product-overview
63

Lingual - (SQL queries to Hadoop)
http://www.cascading.org/lingual/
Lipstick - (Visual Monitoring of Pig workflows) open sourced by Netflix
http://techblog.netflix.com/2013/06/introducing-lipstick-on-apache-pig.html
Lucidworks - (Search built on Solr/Lucene) and an associated Big Data Platform
http://www.lucidworks.com/
Lustre - (Parallel Distributed File System) Open source from multiple vendors
http://en.wikipedia.org/wiki/Lustre_%28file_system%29
Mahout - (Machine Learning Toolkit) built on Apache Hadoop
http://en.wikipedia.org/wiki/Knowledge_Graph
MapD - (Massive Parallel Database) Open Source on top of GPUs
http://istc-bigdata.org/index.php/mapd-a-way-to-map-big-data-faster/
MapReduce - (Processing algorithm)
http://en.wikipedia.org/wiki/MapReduce
MapR - (Enhanced Hadoop distribution) replaces HDFS with an enhanced file system
http://en.wikipedia.org/wiki/MapR
MarkLogic - (Database, NoSQL, Document) interfaced with Hadoop
http://www.marklogic.com/
Memcached - (Data Grid/Caching)
http://en.wikipedia.org/wiki/Memcached
MemSQL - (In memory analytics database)
http://www.memsql.com/
MillWheel - (High performance, fault tolerant stream processing) from Google
http://db.disi.unitn.eu/pages/VLDBProgram/pdf/industry/p734-akidau.pdf
MongoDB - (Database, NoSQL, Document) from 10gen
http://www.mongodb.org/
mrjob - (Workflow) for Hadoop from Yelp
http://bighadoop.wordpress.com/2012/04/15/yelps-mrjob-a-python-package-for-hadoop-jobs/
MRQL - (Query Language) supports Map-Reduce and BSP processing
http://code.google.com/p/mrql/
MQTT- (Messaging Protocol for M2M) originally from IBM being standardized by OASIS
64

http://en.wikipedia.org/wiki/MQ_Telemetry_Transport
Muppet - (Stream Processing) MapUpdate implementation
http://arxiv.org/pdf/1208.4175.pdf
MySql - (Database Relational)
http://www.mysql.com/
Namenode - Directory service for Hadoop
http://wiki.apache.org/hadoop/NameNode
NCDS (National Consortium for Data Sciences) - US Collaborative Data Initiative
http://data2discovery.org/
Neo4j - (Database, NoSQL, Graphical)
http://www.neo4j.org/
Netezza - (Database Appliance)
http://www-01.ibm.com/software/data/netezza/
NuoDB - (MPP Database)
http://www.nuodb.com/
Oozie - (Workflow Scheduler for Hadoop) from Apache
http://oozie.apache.org/
Oracle NoSQL - (Database, Key-Value)
http://www.oracle.com/technetwork/products/nosqldb/overview/index.html
ORC (Optimized Row Columnar) Files - File Format for Hive data in HDFS
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/orcfile.html
ParAccel - (Analytics Platform) with built-in analytics functions from Actian
http://www.actian.com/platform/paraccel-platform
Parquet - (Columnar file format for Hadoop) from Cloudera
http://blog.cloudera.com/blog/2013/03/introducing-parquet-columnar-storage-for-apache-hadoop/
ParStream - (Real-Time Processing) claims high-performance
http://www.parstream.com/product/technology/
Pattern - (Machine Learning on top of Cascading)
http://www.cascading.org/pattern/
Pentaho - (Analytic tools)
65

http://www.pentaho.com/
Percolater - (Data Processing) from Google
http://research.google.com/pubs/pub36726.html
Pig - (Procedural framework on top of Hadoop)
http://pig.apache.org/
Pig Latin - (Interface language for Pig procedures)
http://pig.apache.org/docs/r0.7.0/piglatin_ref1.html
Pivotal - (New company utilizing VMware and EMC technologies)
http://www.gopivotal.com/
Platfora - (In memory caching for BI on top of Hadoop)
http://www.platfora.com/
Postgres - (Database Relational)
http://www.postgresql.org/
Precog - (Analytics Tool) for JSON data
http://precog.com/
Pregel - (Graph Processing) used by Google
http://kowshik.github.io/JPregel/pregel_paper.pdf
Presto - (SQL Query for HDFS) from Facebook
http://www.datanami.com/datanami/2013-06-07/big_data_big_five.html
Protocol Buffers - (Serialization) from Google )
http://en.wikipedia.org/wiki/Protocol_Buffers
Protovis - (Visualization)
http://mbostock.github.io/protovis/
PureData - (Database Products) from IBM
http://www-01.ibm.com/software/data/puredata/
R - (Data Analysis Tool)
http://en.wikipedia.org/wiki/R_%28programming_language%29
Rainstor - (Combines Hadoop and Relational Processing)
http://rainstor.com/
RCFile (Record Columnar File) - File format optimized for HDFS data warehouses
66

http://en.wikipedia.org/wiki/RCFile
Redis - (Database, NoSQL, Key-Value)
http://redis.io/
Redshift - (Database Relational) Amazon
http://aws.amazon.com/redshift/
REEF (Retainable Evaluator Execution Framework) - From Microsoft on top of Yarn
http://gigaom.com/2013/08/12/microsoft-to-open-source-a-big-data-framework-called-reef/
Resilient Distributed Datasets - (Fauklt tolerant in memory data sharing)
http://www.cs.berkeley.edu/~matei/papers/2011/tr_spark.pdf
Riak - (Database, NoSQL,Key-Value with built-in MapReduce) from Basho
http://basho.com/riak/
Roxie - (Query processing cluster) from HPCC
http://hpccsystems.com/FAQ/what-roxie
RushAnalytics - (Analytics) from Pervasive
http://bigdata.pervasive.com/Products/Big-Data-Analytics-RushAnalytics.aspx
S3 - (Database, NoSQL,Key-Value)
http://en.wikipedia.org/wiki/R_%28programming_language%29
S4 - (Stream Processing) from Apache
http://incubator.apache.org/s4/
Sawzall - (Query Language for Map-Reduce) from Google
http://en.wikipedia.org/wiki/Sawzall_%28programming_language%29
Scala - (Programming Language) Combines functional and imperative programming
http://www.scala-lang.org/
Scalding - (Scala layer on top of Cascading)
http://polyglotprogramming.com/papers/ScaldingForHadoop.pdf
Scalebase - (Scalable Front-end to distributed Relational Databases)
http://www.scalebase.com/
Scaleout HServer - (In memory Hadoop MapReduce)
http://www.scaleoutsoftware.com/products/scaleout-hserver/
67

Scaleout Stateserver - (In memory data grid) from Scaleout Software
http://www.scaleoutsoftware.com/
Scaleout hServer - (In memory data grid integrated with Hadoop) from Scaleout
http://www.scaleoutsoftware.com/products/scaleout-hserver
SciDB - ( NoSQL Database with Array data type) Targeted at scientific applications
http://www.scidb.org/
scikit learn - (Machine Learning Toolkit) in Python
http://scikit-learn.org/stable/
Scribe - (Server for Aggregating Log Data) originally from Facebook may be inactive
http://en.wikipedia.org/wiki/Scribe_%28log_server%29
SequenceFiles - (File format) Binary key-value pairs
http://wiki.apache.org/hadoop/SequenceFile
Sentry - (Find Grained Role-based Authorization for Hadoop) from Apache
https://incubator.apache.org/projects/sentry.html
Serengeti - (Virtual Platform for Hadoop) from VMware
http://www.vmware.com/hadoop/serengeti.html
Shark - (Complex Analytics Platform) on top of Spark
https://amplab.cs.berkeley.edu/projects/shark-making-apache-hive-run-at-interactive-speeds/
Simba - (ODBC SQL Driver for Hive)
http://www.simba.com/Apache-Hadoop-Hive-ODBC-Driver-SQL-Connector.htm
SimpleDB - (Database, NoSQL, Document) from Amazon
http://aws.amazon.com/simpledb/
Skytree - (Analytics Server) provides machine learning capabilities
http://www.skytree.net/
Solr/Lucene - (Search Engine) from Apache
http://lucene.apache.org/solr/
Spotfire - (Stream processing tool) from TIBCO
http://spotfire.tibco.com/
Spanner - (Database, NewSQL) from Google
http://en.wikipedia.org/wiki/Spanner_%28database%29
68

Spark - (In memory cluster computing system)
http://spark-project.org/
Splunk - (Machine Data Analytics)
http://www.splunk.com/
Spring Data - (Data access tool for Hadoop and NoSQL) in Spring Framework
http://www.springsource.org/spring-data
SQLite - (Software library supporting server-less relational database)
http://www.sqlite.org/
SQLstream - (Streaming data analysis products)
http://www.sqlstream.com/
Sqoop - (Data movement) from Apache
http://en.wikipedia.org/wiki/Sqoop
Sqrrl - (Security and Analytics on top of Apache Accumulo)
http://www.sqrrl.com/
Stinger - (Optimized Hive Queries) from Hortonworks
http://hortonworks.com/blog/100x-faster-hive/
StorageHandler - (Storage driver) for Apache Hive to external data stores
https://cwiki.apache.org/confluence/display/Hive/StorageHandlers
Storm - (Stream Processing)
http://www.drdobbs.com/open-source/easy-real-time-big-data-analysis-using-s/240143874
Summingbird - (Streaming MapReduce in realtime, batch or hybrid) from Twitter
https://blog.twitter.com/2013/streaming-mapreduce-with-summingbird
Sumo Logic - (Log Analytics)
http://www.sumologic.com/
SUPERCONDUCTOR - (Visualization language for Big Data) from Berkeley
http://www.eecs.berkeley.edu/~lmeyerov/projects/pbrowser/pubfiles/lashc2012.pdf
Tableau - (Visualization)
http://www.tableausoftware.com/
Tachyon - (File system) from Berkeley
http://strata.oreilly.com/2013/04/tachyon-open-source-distributed-fault-tolerant-in-memory-file-system.html
69

Talend - (Data Integration Product)
http://www.talend.com
Templeton - (REST interface to HCatalog and Hadoop interfaces) from Apache
http://people.apache.org/~thejas/templeton_doc_v1/
TempoDB - (Database, NoSQL, Time Series)
https://tempo-db.com/
Teradata Active EDW - (Database, Relational)
http://www.teradata.com/Active-Enterprise-Data-Warehouse/
Terracotta - (In memory data management)
http://terracotta.org/
Terraswarm - (Data Acquisition) Sensor Integration
http://www.terraswarm.org/
Tervela Data Fabric - (Data Capture and Distribution)
http://www.tervela.com/fact-sheet
Tez - (Execution engine on top of Yarn ) from Apache
http://hortonworks.com/blog/introducing-tez-faster-hadoop-processing/
Thor - (Filesystem and Processing Cluster) from HPCC Systems
http://hpccsystems.com/FAQ/what-thor
Thrift - (Framework for cross-language development)
http://thrift.apache.org/
Tika - (Toolkit for extracting metadata and content from documents ) from Apache
http://tika.apache.org/
Tinkerpop - (Graph Database and Toolkit)
http://thrift.apache.org/
UIMA - (Unstructured Information Management Architecture) from OASIS, and Apache
http://en.wikipedia.org/wiki/UIMA
Vertica - (Database Relational)
http://www.vertica.com/
Voldemort - (Database, NoSQL, Key- Value)
http://www.project-voldemort.com/voldemort/
70

VoltDB - (Database NewSQL)
http://voltdb.com/
vSMP (Virtual SMP on a Cluster)- Versatile SMP from ScaleMP
http://www.scalemp.com/media-hub/resources/white-papers/
VXOA (Virtual Acceleration Open Architecture) - Data networking accelerator from Silver-Peak.
http://www.silver-peak.com/products-solutions/vxoa
Watson from IBM - (Analytic Framework)
http://www-03.ibm.com/innovation/us/watson/
WebHDFS - (REST API for Hadoop)
http://hadoop.apache.org/docs/r1.0.4/webhdfs.html
WEKA - (Machine Learning Toolkit) in Java
http://en.wikipedia.org/wiki/Weka_%28machine_learning%29
Wibidata - (Components for building Big Data applications)
http://www.wibidata.com/
Xtremedata - (MPP Database) for data ingestion and SQL-based analytics
http://www.xtremedata.com/technology/scale-out-scale-up
YarcData Urika - (In-memory Graph Analytics for Big Data)
http://www.yarcdata.com/
Yarn - (Next Generation Hadoop) from Apache
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
Yottamine - (Machine Learning Toolkit) Cloud-based
http://yottamine.com/
Zettaset Orchestrator - (Management and Security for Hadoop)
http://www.zettaset.com/platform.php
ZooKeeper - (Distributed Computing Management)
http://zookeeper.apache.org/
71

NIST Big Data Working Group.pdf

More Related Content

Similar to NIST Big Data Working Group.pdf (20)

Recently uploaded (20)

NIST Big Data Working Group.pdf