![The MADlib Analytics Library
or MAD Skills, the SQL
Joseph M. Hellerstein Christoper Re
´ Florian Schoppmann Daisy Zhe Wang Eugene Fratkin
U.C. Berkeley U. Wisconsin Greenplum U. Florida Greenplum
Aleksander Gorajek Kee Siong Ng Caleb Welton Xixuan Feng Kun Li Arun Kumar
Greenplum Greenplum Greenplum U. Wisconsin U. Florida U. Wisconsin
ABSTRACT of potentially noisy data to support predictive analytics, provided
MADlib is a free, open source library of in-database analytic meth- via statistical models and algorithms. Data Science is a name that is
ods. It provides an evolving suite of SQL-based algorithms for gaining currency for the industry practices evolving around these
machine learning, data mining and statistics that run at scale within workloads.
a database engine, with no need for data import/export to other Data scientists make use of database engines in a very different
tools. The goal is for MADlib to eventually serve a role for scalable way than traditional data warehousing professionals. Rather than
database systems that is similar to the CRAN library for R: a com- carefully designing global schemas and “repelling” data until it is
munity repository of statistical methods, this time written with scale integrated, they load data into private schemas in whatever form is
and parallelism in mind. convenient. Rather than focusing on simple OLAP-style drill-down
In this paper we introduce the MADlib project, including the reports, they implement rich statistical models and algorithms in the
background that led to its beginnings, and the motivation for its open database, using extensible SQL as a language for orchestrating data
source nature. We provide an overview of the library’s architecture movement between disk, memory, and multiple parallel machines.
and design patterns, and provide a description of various statistical In short, for data scientists a DBMS is a scalable analytics runtime—
methods in that context. We include performance and speedup one that is conveniently compatible with the database systems widely
results of a core design pattern from one of those methods over the used for transactions and accounting.
Greenplum parallel DBMS on a modest-sized test cluster. We then In 2008, a group of us from the database industry, consultancy,
report on two initial efforts at incorporating academic research into academia, and end-user analytics got together to describe this usage
MADlib, which is one of the project’s goals. pattern as we observed it in the field. We dubbed it MAD, an
MADlib is freely available at http://madlib.net, and the acronym for the Magnetic (as opposed to repellent) aspect of the
project is open for contributions of both new methods, and ports to platform, the Agile design patterns used for modeling, loading and
additional database platforms. iterating on data, and the Deep statistical models and algorithms
being used. The “MAD Skills” paper that resulted described this
pattern, and included a number of non-trivial analytics techniques
1. INTRODUCTION: implemented as simple SQL scripts [11].
FROM WAREHOUSING TO SCIENCE After the publication of the paper, significant interest emerged
Until fairly recently, large databases were used mainly for account- not only in its design aspects, but also in the actual SQL imple-
ing purposes in enterprises, supporting financial record-keeping and mentations of statistical methods. This interest came from many
reporting at various levels of granularity. Data Warehousing was directions: customers were requesting it of consultants and vendors,
the name given to industry practices for these database workloads. and academics were increasingly publishing papers on the topic.
Accounting, by definition, involves significant care and attention to What was missing was a software framework to focus the energy of
detail. Data Warehousing practices followed suit by encouraging the community, and connect the various interested constituencies.
careful and comprehensive database design, and by following exact- This led to the design of MADlib, the subject of this paper.
ing policies regarding the quality of data loaded into the database.
Attitudes toward large databases have been changing quickly in Introducing MADlib
the past decade, as the focus of large database usage has shifted
from accountancy to analytics. The need for correct accounting and MADlib is a library of analytic methods that can be installed and
data warehousing practice has not gone away, but it is becoming a executed within a relational database engine that supports extensi-
shrinking fraction of the volume—and the value—of large-scale data ble SQL. A snapshot of the current contents of MADlib including
management. The emerging trend focuses on the use of a wide range methods and ports is provided in Table 1. This set of methods and
ports is intended to grow over time.
The methods in MADlib are designed both for in- or out-of-
core execution, and for the shared-nothing, “scale-out” parallelism
offered by modern parallel database engines, ensuring that compu-
tation is done close to the data. The core functionality is written
in declarative SQL statements, which orchestrate data movement
to and from disk, and across networked machines. Single-node
inner loops take advantage of SQL extensibility to call out to high-
performance math libraries in user-defined scalar and aggregate
functions. At the highest level, tasks that require iteration and/or](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-3-320.jpg)
![Category Method today requires extracting advantages in the long tail of “special in-
Supervised Learning Linear Regression terests”, a practice known as “microtargeting”, “hypertargeting” or
Logistic Regression “narrowcasting”. In that context, the first step of SEMMA essentially
Naive Bayes Classification defeats the remaining four steps, leading to simplistic, generalized
Decision Trees (C4.5) decision-making that may not translate well to small populations in
Support Vector Machines the tail of the distribution. In the era of “Big Data”, this argument
Unsupervised Learning k-Means Clustering for enhanced attention to long tails applies to an increasing range of
SVD Matrix Factorization use cases.
Latent Dirichlet Allocation Driven in part by this observation, momentum has been gathering
Association Rules around efforts to develop scalable full-dataset analytics. One pop-
Decriptive Statistics Count-Min Sketch ular alternative is to push the statistical methods directly into new
Flajolet-Martin Sketch parallel processing platforms—notably, Apache Hadoop. For ex-
Data Profiling ample, the open source Mahout project aims to implement machine
Quantiles learning tools within Hadoop, harnessing interest in both academia
Support Modules Sparse Vectors and industry [10, 3]. This is certainly a plausible path to a solution,
Array Operations and Hadoop is being advocated as a promising approach even by
Conjugate Gradient Optimization major database players, including EMC/Greenplum, Oracle and
Microsoft.
Table 1: Methods provided in MADlib v0.3. This version has At the same time that the Hadoop ecosystem has been evolving,
been tested on two DBMS platforms: PostgreSQL (single-node the SQL-based analytics ecosystem has grown rapidly as well, and
open source) and Greenplum Database (massively parallel com- large volumes of valuable data are likely to pour into SQL systems
mercial system, free and fully-functional for research use.) for many years to come. There is a rich ecosystem of tools, know-
how, and organizational requirements that encourage this. For these
cases, it would be helpful to push statistical methods into the DBMS.
structure definition are coded in Python driver routines, which are And as we will see, massively parallel databases form a surprisingly
used only to kick off the data-rich computations that happen within useful platform for sophisticated analytics. MADlib currently targets
the database engine. this environment of in-database analytics.
MADlib is hosted publicly at github, and readers are encouraged
to browse the code and documentation via the MADlib website 2.2 Why Open Source?
http://madlib.net. The initial MADlib codebase reflects contri- From the beginning, MADlib was designed as an open source
butions from both industry (Greenplum) and academia (UC Berke- project with corporate backing, rather than a closed-source corporate
ley, the University of Wisconsin, and the University of Florida). effort with academic consulting. This decision was motivated by a
Code management and Quality Assurance efforts have been con- number of factors, including the following:
tributed by Greenplum. At this time, the project is ready to consider
• The benefits of customization: Statistical methods are rarely
contributions from additional parties, including both new methods
used as turnkey solutions. As a result, it is common for data
and ports to new platforms.
scientists to want to modify and adapt canonical models and
methods (e.g., regression, classification, clustering) to their
2. GOALS OF THE PROJECT own purposes. This is a very tangible benefit of open source
The primary goal of the MADlib open source project is to ac- libraries over traditional closed-source packages. Moreover,
celerate innovation and technology transfer in the Data Science in an open source community there is a process and a set of
community via a shared library of scalable in-database analytics, positive incentives for useful modifications to be shared back
much as the CRAN library serves the R community [32]. Unlike to the benefit of the entire community.
CRAN, which is customized to the R analytics engine, we hope that
• Valuable data vs. valuable software: In many emerging
MADlib’s grounding in standard SQL can lead to community ports
business sectors, the corporate value is captured in the data
to a variety of parallel database engines.
itself, not in the software used to analyze that data. Indeed, it
2.1 Why Databases? is in the interest of these companies to have the open source
community adopt and improve their software. open source
For decades, statistical packages like SAS, Matlab and R have
efforts can also be synergistic for vendors that sell commercial
been the key tools for deep analytics, and the practices surrounding
software, as evidenced by companies like EMC/Greenplum,
these tools have been elevated into widely-used traditional method-
Oracle, Microsoft and others beginning to provide Apache
ologies. One standard analytics methodology advocated in this do-
Hadoop alongside their commercial databases. Most IT shops
main is called SEMMA: Sample, Explore, Modify, Model, Assess.
today run a mix of open source and proprietary software,
The “EMMA” portion of this cycle identifies a set of fundamental
and many software vendors are finding it wise to position
tasks that an analyst needs to perform, but the first, “S” step makes
themselves intelligently in that context. Meanwhile, for most
less and less sense in many settings today. The costs of computation
database system vendors, their core competency is not in
and storage are increasingly cheap, and entire data sets can often be
statistical methods, but rather in the engines that support
processed efficiently by a cluster of computers. Meanwhile, compe-
those methods, and the service industry that evolves around
tition for extracting value from data has become increasingly refined.
them. For these vendors, involvement and expertise with an
Consider fiercely competitive application domains like online adver-
open source library like MADlib is an opportunity to expand
tising or politics. It is of course important to target “typical” people
their functionality and service offerings.
(customers, voters) that would be captured by sampling the database.
But the fact that SEMMA is standard practice means that optimiz- • Closing the research-to-adoption loop: Very few traditional
ing for a sample provides no real competitive advantage. Winning database customers have the capacity to develop significant](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-4-320.jpg)
![in-house research into computing or data science. On the can recruit high-quality, experienced software engineers with the
other hand, it is hard for academics doing computing research attraction of well-compensated, long-term career paths. Equally
to understand and influence the way that analytic processes important, software shops can offer an entire software engineering
are done in the field. An open source project like MADlib pipeline that cannot be replicated on campus: this includes QA
has the potential to connect academic researchers not only to processes and QA engineering staff. The hope is that the corporate
industrial software vendors, but also directly to the end-users staffing of research projects like MADlib can enable more impactful
of analytics software. This can improve technology transfer academic open source research, and speed technology transfer to
from academia into practice without requiring database soft- industry.
ware vendors to serve as middlemen. It can similarly enable
end-users in specific application domains to influence the 2.4 MADlib Status
research agenda in academia. MADlib is still young, currently (as of March, 2012) at Version
0.3. The initial versions have focused on establishing a baseline of
• Leveling the playing field, encouraging innovation: Over useful functionality, while laying the groundwork for future evolu-
the past two decades, database software vendors have devel- tion. Initial development began with the non-trivial work of building
oped proprietary data mining toolkits consisting of textbook the general-purpose framework described in Section 3. Addition-
algorithms. It is hard to assess their relative merits. Mean- ally, we wanted robust implementations of textbook methods that
while, other communities in machine learning and internet were most frequently requested from customers we met through
advertising have also been busily innovating, but their code contacts at Greenplum. Finally, we wanted to validate MADlib as a
is typically not well packaged for reuse, and the code that is research vehicle, by fostering a small number of university groups
available was not written to run in a database system. None working in the area to experiment with the platform and get their
of these projects has demonstrated the vibrancy and breadth code disseminated (Section 5).
we see in the open source community surrounding R and
its CRAN package. The goal of MADlib is to fill this gap:
bring the database community up to a baseline level of compe- 3. MADLIB ARCHITECTURE
tence on standard statistical algorithms, remove opportunities The core of traditional SQL—SELECT... FROM... WHERE...
for proprietary FUD, and help focus a large community on GROUP BY—is quite a powerful harness for orchestrating bulk data
innovation and technology transfer. processing across one or many processors and disks. It is also a
portable, native language supported by a range of widely-deployed
2.3 A Model for Open Source Collaboration open source and commercial database engines. This makes SQL an
The design of MADlib comes at a time when the connections attractive framework for writing data-intensive programs. Ideally,
between open source software and academic research seem particu- we would like MADlib methods to be written entirely in straightfor-
larly frayed. MADlib is designed in part as an experiment in binding ward and portable SQL. Unfortunately, the portable core of “vanilla”
these communities more tightly, to face current realities in software SQL is often not quite enough to express the kinds of algorithms
development. needed for advanced analytics.
In previous decades, many important open source packages origi- Many statistical methods boil down to linear algebra expressions
nated in universities and evolved into significant commercial prod- over matrices. For relational databases to operate over very large
ucts. Examples include the Ingres and Postgres database systems, matrices, this presents challenges at two scales. At a macroscopic
the BSD UNIX and Mach operating systems, the X Window user scale, the matrices must be intelligently partitioned into chunks that
interfaces and the Kerberos authentication suite. These projects can fit in memory on a single node. Once partitioned, the pieces
were characterized by aggressive application of new research ideas, can be keyed in such a way that SQL constructs can be used to
captured in workable but fairly raw public releases that matured orchestrate the movement of these chunks into and out of memory
slowly with the help of communities outside the university. While across one or many machines. At a microscopic scale, the database
all of the above examples were incorporated into commercial prod- engine must invoke efficient linear algebra routines on the pieces
ucts, many of those efforts emerged years or decades after the initial of data it gets in core. To this end it has to have the ability to very
open source releases, and often with significant changes. quickly invoke well-tuned linear algebra methods.
Today, we expect successful open source projects to be quite We proceed to discuss issues involved at both of these levels in a
mature, often comparable to commercial products. To achieve this bit more detail, and solutions we chose to implement in MADlib.
level of maturity, most successful open source projects have one
or more major corporate backers who pay some number of com- 3.1 Macro-Programming (Orchestration)
mitters and provide professional support for Quality Assurance A scalable method for linear algebra depends upon divide-and-
(QA). This kind of investment is typically made in familiar soft- conquer techniques: intelligent partitioning of the matrix, and a
ware packages, not academic research projects. Many of the most pattern to process the pieces and merge results back together. This
popular examples—Hadoop, Linux, OpenOffice—began as efforts partitioning and dataflow is currently outside the scope of a tradi-
to produce open source alternatives to well-identified, pre-existing tional query optimizer or database design tool. But there is a rich
commercial efforts. literature from scientific computing on these issues (e.g., [9]) that
MADlib is making an explicit effort to explore a new model database programmers can use to craft efficient in-database imple-
for industry support of academic research via open source. Many mentations. Once data is properly partitioned, database engines
academic research projects are generously supported by financial shine at orchestrating the resulting data movement of partitions and
grants and gifts from companies. In MADlib, the corporate do- the piecewise results of computation.
nation has largely consisted of a commitment to allocate signifi-
cant professional software engineering time to bootstrap an open 3.1.1 User-Defined Aggregation
source sandbox for academic research and tech transfer to practice. The most basic building block in the macro-programming of
This leverages a strength of industry that is not easily replicated MADlib is the use of user-defined aggregates (UDAs). In general,
by government and other non-profit funding sources. Companies aggregates—and the related window functions—are the natural way](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-5-320.jpg)
![in SQL to implement mathematical functions that take as input the conditions—this was used by Wang et al. to implement Viterbi infer-
values of an arbitrary number of rows (tuples). DBMSs typically ence [42] (Section 5.2). Unfortunately, like windowed aggregates,
implement aggregates as data-parallel streaming algorithms. And recursion support in SQL varies across database products, and does
there is a large body of recent work on online learning algorithms not form a reliable basis for portability.
and model-averaging techniques that fit the computational model of Driver Functions. None of the above methods provides both gener-
aggregates well (see, e.g., [45]). ality and portability. As a result, in MADlib we chose to implement
Unfortunately, extension interfaces for user-defined aggregates complex iterative methods by writing a driver UDF in Python to
vary widely across vendors and open source systems. Nonethe- control iteration, which passes state across iterations intelligently. A
less, the aggregation paradigm (or in functional programming terms, standard pitfall in this style of programming is for the driver code to
“fold” or “reduce”) is natural and ubiquitous, and we expect the basic pull a large amount of data out of the database; this becomes a scala-
algorithmic patterns for user-defined aggregates to be very portable. bility bottleneck since the driver code typically does not parallelize
In most widely-used DBMSs (e.g., in PostgreSQL, MySQL, Green- and hence pulls all data to a single node. We avoid this via a design
plum, Oracle, SQL Server, Teradata), a user-defined aggregate con- pattern in which the driver UDF kicks off each iteration and stages
sists of a well-known pattern of two or three user-defined functions: any inter-iteration output into a temporary table via CREATE TEMP
TABLE... AS SELECT... It then reuses the resulting temp table
1. A transition function that takes the current transition state and in subsequent iterations as needed. Final outputs are also often
a new data point. It combines both into into a new transition stored in temp tables unless they are small, and can be interrogated
state. using small aggregate queries as needed. As a result, all large-data
2. An optional merge function that takes two transition states movement is done within the database engine and its buffer pool.
and computes a new combined transition state. This function Database engines typically provide efficient parallelism as well as
is only needed for parallel execution. buffering and spill files on disk for large temp tables, so this pattern
is quite efficient in practice. Sections 4.2 and 4.3 provide discussion
3. A final function that takes a transition state and transforms it of this pattern in the context of specific algorithms.
into the output value.
3.1.3 Templated Queries
Clearly, a user-defined aggregate is inherently data-parallel if the
transition function is associative and the merge function returns the A second problem is a limitation of SQL’s roots in first-order
same result as if the transition function was called repeatedly for logic, which requires that queries be cognizant of the schema of
every individual element in the second state. their input tables, and produce output tables with a fixed schema.
Unfortunately, user-defined aggregates are not enough. In design- In many cases we want to write “templated” queries that work over
ing the high-level orchestration of data movement for analytics, we arbitrary schemas, with the details of arity, column names and types
ran across two main limitations in standard SQL that we describe to be filled in later.
next. We addressed both these limitations using driver code written For example, the MADlib profile module takes an arbitrary
in simple script-based user-defined functions (UDFs), which in turn table as input, producing univariate summary statistics for each of
kick off more involved SQL queries. When implemented correctly, its columns. The input schema to this module is not fixed, and the
the performance of the scripting language code is not critical, since output schema is a function of the input schema (a certain number
its logic is invoked only occasionally to kick off much larger bulk of output columns for each input column). To address this issue, we
tasks that are executed by the core database engine. use Python UDFs to interrogate the database catalog for details of
input tables, and then synthesize customized SQL queries based on
3.1.2 Driver Functions for Multipass Iteration templates to produce outputs. Simpler versions of this issue arise in
The first problem we faced is the prevalence of “iterative” algo- most of our iterative algorithms.
rithms for many methods in statistics, which make many passes Unfortunately, templated SQL relies on identifiers or expressions
over a data set. Common examples include optimization methods passed as strings to represent database objects like tables. As such,
like Gradient Descent and Markov Chain Monte Carlo (MCMC) the DBMS backend will discover syntactical errors only when the
simulation in which the number of iterations is determined by a generated SQL is executed, often leading to error messages that are
data-dependent stopping condition at the end of each round. There enigmatic to the user. As a result, templated SQL necessitates that
are multiple SQL-based workarounds for this problem, whose appli- MADlib code perform additional validation and error handling up
cability depends on the context. front, in order to not compromise usability. In the future we plan to
Counted Iteration via Virtual Tables. In order to drive a fixed support this pattern as a Python library that ships with MADlib and
number n of independent iterations, it is often simplest (and very ef- provides useful programmer APIs and user feedback.
ficient) to declare a virtual table with n rows (e.g., via PostgreSQL’s
generate series table function), and join it with a view repre- 3.2 Micro-Programming: Data Representa-
senting a single iteration. This approach was used to implement tions and Inner Loops
m-of-n Bootstrap sampling in the original MAD Skills paper [11]. In addition to doing the coarse-grained orchestration of chunks,
It is supported in some fashion in a variety of DBMSs, sometimes the database engine must very efficiently invoke the single-node code
by writing a simple table function consisting of a few lines of code. that performs arithmetic on those chunks. For UDFs that operate at
Window Aggregates for Stateful Iteration. For settings where the row level (perhaps called multiple times per row), the standard
the current iteration depends on previous iterations, SQL’s win- practice is to implement them in C or C++. When computing dense
dowed aggregate feature can be used to carry state across iterations. matrix operations, these functions would make native calls to an
Wang et al. took this approach to implement in-database MCMC open source library like LAPACK [2] or Eigen [17].
inference [41] (Section 5.2). Unfortunately the level of support for Sparse matrices are not as well-handled by standard math libraries,
window aggregates varies across SQL engines. and require more customization for efficient representations both
Recursive Queries. Most generally, it is possible to use the recur- on disk and in memory. We chose to write our own sparse matrix
sion features of SQL to perform iteration with arbitrary stopping library in C for MADlib, which implements a run-length encoding](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-6-320.jpg)
![scheme. Both of these solutions require careful low-level coding, 4.1 Single-Pass: Ordinary Least Squares
and formed part of the overhead of getting MADlib started. In ordinary-least-squares (OLS) linear regression the goal is to
The specifics of a given method’s linear algebra can be coded in fit a linear function to a set of points, with the objective of mini-
a low-level way using loops of basic arithmetic in a language like C, mizing the sum of squared residuals. Formally, we are given points
but it is nicer if they can be expressed in a higher-level syntax that (x1 , y1 ), . . . , (xn , yn ), where xi ∈ Rd and yi ∈ R, and our goal is to
captures the semantics of the linear algebra at the level of matrices find the vector b that minimizes n (yi − b, xi )2 . OLS is one of
i=1
and arrays. We turn to this issue next. the most fundamental methods in statistics. Typically, each yi is
assumed to be an (independent) noisy measurement of b, xi , where
3.3 A C++ Abstraction Layer for UDFs b is an unknown but fixed vector and the noise is uncorrelated with
There are a number of complexities involved in writing C or C++- mean 0 and unknown but fixed variance. Under these assumptions,
based user-defined functions over a legacy DBMS like PostgreSQL, b is the best linear unbiased estimate of b (Gauss-Markov). Un-
all of which can get in the way of maintainable, portable application der additional assumptions (normality, independence), b is also the
logic. This complexity can be especially frustrating for routines maximum-likelihood estimate. Letting X denote the matrix whose
whose pseudocode amounts to a short linear algebra expression rows are xT , and defining y := (y1 , . . . , yn )T , it is well-known that
i
that should result in a compact implementation. MADlib provides
the sum of squared residuals is minimized by b = (X T X)−1 X T y (for
a C++ abstraction layer both to ease the burden of writing high-
exposition purposes we assume the full-rank case here, though this
performance UDFs, and to encapsulate DBMS-specific logic inside
is not a requirement for MADlib).
the abstraction layer, rather than spreading the cost of porting across
all the UDFs in the library. In brief, the MADlib C++ abstraction It has been observed before that computing b lends itself well to
provides three classes of functionality: type bridging, resource data-parallel implementations [10]—in extensible database terms,
management shims, and math library integration. it can be done with a simple user-defined aggregate. The principal
Type bridging is provided via an encapsulated mapping of C++ observation is this: X T X = n xi xT and X T y = n xi yi are
i=1 i i=1
types and methods to database types and functions. UDFs can be just sums of transformations of each data point. Summation is
written with standard C++ atomic types, as well as the vector and associative, so data parallelism virtually comes for free—we can
matrix types that are native to a high performance linear algebra compute the per-process subsums of the previous expressions locally
library. (We have successfully layered multiple alternative libraries in each process, and then sum up all subsums during a second-
under this interface, and are currently using Eigen [17]). The trans- phase aggregation. As a final non-parallelized step, we compute the
lation to and from database types (including composite types like inverse of X T X and then multiply with X T y. These final operations
double precision[] for vectors) is handled by the abstraction are comparatively cheap, since the number of independent variables
layer. Similarly, higher-order templated functions in C++ can be (and thus the dimensions of X T X and X T y) is typically “small”.
mapped to the appropriate object IDs of UDFs in the database, with 4.1.1 MADlib Implementation
the abstraction layer taking care of looking up the function in the
We assume that data points are stored as (x DOUBLE PRECISION[],
database catalog, verifying argument lists, ensuring type-safety, etc.
y DOUBLE PRECISION) tuples. Linear regression is then imple-
The second aspect of the C++ abstraction layer is to provide a
mented as a user-defined aggregate with a transition and final func-
safe and robust standard runtime interface to DBMS-managed re-
tion roughly as in Listings 1 and 2, respectively. (For compactness,
sources. This includes layering C++ object allocation/deallocation
we omitted finiteness checks and several output statistics in the ex-
over DBMS-managed memory interfaces, providing shims between
ample here.) The merge function, which is not shown, just adds all
C++ exception handling and DBMS handlers, and correctly propa-
values in the transition states together. Running the code produces
gating system signals to and from the DBMS.
the following:
Finally, by incorporating proven third-party libraries, the C++
abstraction layer makes it easy for MADlib developers to write psql# SELECT (linregr(y, x)).* FROM data;
correct and performant code. For example, the Eigen linear algebra -[ RECORD 1 ]+--------------------------------------------
coef | {1.7307,2.2428}
library contains well-tested and well-tuned code that makes use r2 | 0.9475
of the SIMD instruction sets (like SSE) found in today’s CPUs. std_err | {0.3258,0.0533}
Likewise, the abstraction layer itself has been tuned for efficient t_stats | {5.3127,42.0640}
p_values | {6.7681e-07,4.4409e-16}
value marshalling, and code based on it will automatically benefit condition_no | 169.5093
from future improvements. By virtue of being a template library,
the runtime and abstraction overhead is reduced to a minimum. Note that the linregr Python UDF produces a composite record
As an illustration of the high-level code one can write over our type in the output, which is a feature of PostgreSQL and Greenplum.
abstraction layer, Listings 1 and 2 show reduced, but fully functional This would be easy to flatten into a string in a strictly relational
code snippets that implement multiple linear regression (as discussed implementation.
further in Section 4.1).
4.2 Multi-Pass: (Binary) Logistic Regression
In (binary) logistic regression, we are given points (x1 , y1 ), . . . ,
4. EXAMPLES (xn , yn ), where xi ∈ Rd and yi ∈ {0, 1}, and our goal is to find the vec-
1
To illustrate the above points, we look at three different algorith- tor b that maximizes n σ((−1)yi +1 · b, xi ). Here, σ(z) = 1+exp(z)
i=1
mic scenarios. The first is Linear Regression using Ordinary Least denotes the logistic function. Statistically, this is the maximum-
Squares (OLS), which is an example of a widely useful, simple likelihood estimate for an unknown vector b under the assumption
single-pass aggregation technique. The second is binary Logistic that each yi is a random variate with Pr[yi = 1 | xi ] = σ( b, xi ) and
Regression, another widely used technique, but one that employs that all observations are independent.
an iterative algorithm. Finally, we look at k-means Clustering, an It is well-known that, in general, no closed-formula expression
iterative algorithm with large intermediate states spread across ma- for b exists. Instead, b can be computed as the solution of a con-
chines. vex program via standard iterative methods. Arguably, the most](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-7-320.jpg)
![1 AnyType 1 AnyType
2 linregr_transition::run(AnyType &args) { 2 linregr_final::run(AnyType &args) {
3 // Transition state is a class that wraps an array. 3 // Immutable array: Array will never be copied
4 // We expect a mutable array. If DBMS allows 4 LinRegrTransitionState<
5 // modifications, copying will be avoided. 5 ArrayHandle<double> > state = args[0];
6 LinRegrTransitionState<
7 MutableArrayHandle<double> > state = args[0]; 7 // The following is a MADlib class that wraps Eigen’s
8 // Dependent variable is a double-precision float 8 // solver for self-adjoint matrices.
9 double y = args[1].getAs<double>(); 9 SymmetricPositiveDefiniteEigenDecomposition<Matrix>
10 // Vector of independent variables wraps an immutable 10 decomposition(state.X_transp_X, EigenvaluesOnly,
11 // array (again, no unnecessary copying). This maps 11 ComputePseudoInverse);
12 // to an Eigen type
13 HandleMap<const ColumnVector> x 13 Matrix inverse_of_X_transp_X
14 = args[2].getAs<ArrayHandle<double> >(); 14 = decomposition.pseudoInverse();
15 // Let backend allocate array for coefficients so to
16 if (state.numRows == 0) 16 // avoid copying on return (if supported by DBMS).
17 // The first row determines the number 17 HandleMap<ColumnVector> coef(
18 // of independent variables 18 allocateArray<double>(state.widthOfX));
19 state.initialize(*this, x.size()); 19 coef.noalias() = inverse_of_X_transp_X
20 state.numRows++; 20 * state.X_transp_Y;
21 state.y_sum += y;
22 state.y_square_sum += y * y; 22 // Return a composite value.
23 // noalias informs Eigen to multiply in-place 23 AnyType tuple;
24 state.X_transp_Y.noalias() += x * y; 24 tuple << coef << decomposition.conditionNo();
25 // Since XˆT X is symmetric, we only need to 25 return tuple;
26 // compute a triangular part 26 }
27 triangularView<Lower>(state.X_transp_X)
28 += x * trans(x);
Figure 2: Linear-regression final function
30 return state;
31 }
language. In many cases there are multiple equivalent patterns
Figure 1: Linear-regression transition function for constructing simple interfaces, but no well-accepted, uniform
design patterns for the kind of algorithmic expressions we tend to
implement in MADlib. We are refining these design patterns as we
common method is to maximize the logarithm of the likelihood evolve the library.
using Newton’s method. In the case of logistic regression this re-
duces to iteratively reweighted least squares with iteration rule 4.3 Large-State Iteration: k-Means
βm+1 = (X T Dm X)−1 X T Dm zm . Here, the diagonal matrix Dm and the In k-means clustering, we are given n points x1 , . . . , xn ∈ Rd , and
vector zm are transformations of X and βm . our goal is to position k centroids c1 , . . . , ck ∈ Rd so that the sum
of squared distances between each point and its closest centroid is
4.2.1 MADlib Implementation minimized. Formally, we wish to minimize n minkj=1 xi − c j 2 .
i=1
Each individual iteration can be implemented via a user-defined Solving this problem exactly is usually prohibitively expensive (for
aggregate using linear regression as a blueprint. However, the hand- theoretical hardness results see, e.g., [1, 23]). However, the local-
ling of iterations and checking for convergence require a further search heuristic proposed by Lloyd [21] performs reasonably well
outer loop. We therefore implement a driver UDF in Python. The both in theory and in practice [5, 4]. At a high level, it works as
control flow follows the high-level outline from Section 3.1.2 and follows:
is illustrated as an activity diagram in Figure 3. Here, the shaded
shapes are executions of generated SQL, where current iteration 1. Seeding phase: Find initial positions for k centroids c1 , . . . , ck .
is a template parameter that is substituted with the corresponding 2. Assign each point x1 , . . . , xn to its closest centroid.
Python variable.
Specifically, the UDF first creates a temporary table for storing 3. Reposition each centroid to the barycenter (mean) of all points
the inter-iteration states. Then, the Python code iteratively calls the assigned to it.
UDA for updating the iteration state, each time adding a new row
to the temporary table. Once the convergence criterion has been 4. If no (or only very few) points got reassigned, stop. Otherwise,
reached, the state is converted into the return value. The important goto (2).
point to note is that there is no data movement between the driver
function and the database engine—all heavy lifting is done within 4.3.1 MADlib implementation
the database engine. k-means has a natural implementation in SQL [29]. Based on the
Unfortunately, implementing logistic regression using a driver assumption that we can always comfortably store k centroids in main
function leads to a different interface than the one we provided for memory, we can implement k-means similarly to logistic regression:
linear regression: Using a driver function that iteratively calls a user-defined aggregate.
In the following, we take a closer look at this implementation. It
SELECT * FROM logregr(’y’, ’x’, ’data’);
is important to make a clear distinction between the inter-iteration
A problem with this implementation is that the logregr UDF is state (the output of the UDA’s final function) and intra-iteration
not an aggregate function and cannot be used in grouping constructs. state (as maintained by the UDA’s transition and merge functions).
To perform multiple logistic regressions at once, one needs to use During aggregation, the transition state contains both inter- and
a join construct instead. We intend to address this non-uniformity intra-iteration state, but only modifies the intra-iteration state. We
in interface in a future version of MADlib. We highlight the issue only store k centroids in both the inter- and intra-iteration states, and
here in part to point out that SQL can be a somewhat “over-rich” consider the assignments of points to centroids as implicitly given.](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-8-320.jpg)
![for updates that touch many rows it is typically faster to copy the
CREATE TEMORARY TABLE iterative_algorithm AS updated data into a new table (i.e., CREATE TABLE AS SELECT
SELECT 0 AS iteration, NULL AS state
and DROP TABLE) rather than issue an UPDATE. These kinds of
DBMS-specific performance tricks may merit encapsulation in an
current_iteration = 0
abstraction layer for SQL portability.
INSERT INTO iterative_algorithm(iteration, state)
SELECT iteration + 1, logregr_irls_step(y, x, state)
4.4 Infrastructure Performance Trends
FROM source, iterative_algorithm
WHERE iteration = current_iteration
In its current beta version, MADlib has been tuned a fair bit over
PostgreSQL and Greenplum, though much remains to be done. Here
current_iteration ++
we report on some results for a basic scenario that exercises our core
functionality, including the C++ abstraction layer, our ability to call
SELECT internal_logregr_irls_did_converte(state) out to linear algebra packages, and parallel speedup validation. We
FROM iterative_algorithm defer macro-benchmarking of MADlib’s current methods to future
WHERE iteration = current_iteration
work, that will focus on specific algorithm implementations.
[False] The basic building block of MADlib is a user-defined aggregate,
[True] typically one that calls out to a linear algebra library. In order to
SELECT internal_logregr_irls_result(state) evaluate the scalability of this construct, we ran linear regression
FROM iterative_algorithm
WHERE iteration = current_iteration over Greenplum’s parallel DBMS on various data sizes, using a 24-
core test cluster we had available, which was outfitted with 144 GB
of RAM over 51 TB of raw storage.2 This is obviously a relatively
modest-sized cluster by today’s standards, but it is sufficient to
illuminate (a) our efforts at minimizing performance overheads, and
Figure 3: Sequence Diagram for Logistic Regression (b) our ability to achieve appropriate parallel speedup.
For running linear regression as outlined in Section 4.1, we expect
runtime O(k3 + (n · k2 )/p) where k is the number of independent
In the transition function, we first compute the centroid that the
variables, n is the number of observations, and p is the number of
current point was closest to at the beginning of the iteration using the
query processes. The k3 time is needed for the matrix inversion, and
inter-iteration state. We then update the barycenter of this centroid
the k2 is needed for computing each outer product xi xT and adding
in the intra-iteration state. Only as the final step of the aggregate, i
it to the running sum. It turns out that our runtime measurements fit
the intra-iteration state becomes the new inter-iteration state.
these expectations quite well, and the constant factors are relatively
Unfortunately, in order to check the convergence criterion that no
small. See Figure 4 and 5. In particular we note:
or only few points got reassigned, we have to do two closest-centroid
computations per point and iteration: First, we need to compute the • The overhead for a single query is very low and only a frac-
closest centroid in the previous iteration and then the closest one in tion of a second. This also implies that we lose little in
the current iteration. If we stored the closest points explicitly, we implementating iterative algorithms using driver functions
could avoid half of the closest-centroid calculations. that run multiple SQL queries.
We can store points in a table called points that has a coords
attribute containing the points’ coordinates and has a second attribute • Given the previous points, the Greenplum database achieves
for the current centroid id for the point. The iteration state stores perfect linear speedup in the example shown.
the centroids’ positions in an array of points called centroids.
MADlib provides a UDF closest point(a,b) that determines In the example, all data was essentially in the buffer cache, and
the point in array a that is closest to b. Thus, we can make the disk I/O was not a limiting factor. This is quite typical in large
point-to-centroid assignments explicit using the following SQL: parallel installations, since an analyst often runs a machine-learning
UPDATE points algorithm several times with different parameters. Given massive
SET centroid_id = closest_point(centroids, coords) amounts of main memory on each node in a well-provisioned cluster,
much of the data should moreover remain in the cache. Finally, the
Ideally, we would like to perform the point reassignment and the
computational cost per row grows at least quadratically, and thus
repositioning with a single pass over the data. Unfortunately, this
will easily surpass I/O cost for complex models. As we compared
cannot be expressed in standard SQL.1
various cluster sizes, numbers of independent variables and the re-
Therefore, while we can reduce the number of closest-centroid
spective execution times for previous versions of MADlib, the lesson
calculations by one half, PostgreSQL processes queries one-by-one
we learned is that even though we anticipate non-trivial overhead
(and does not perform cross-statement optimization), so it will need
by the DBMS, careful performance tuning—e.g., by making use
to make two passes over the data per one k-means iteration. In
of instruction-accurate profiling using Valgrind [27]—still makes
general, the performance benefit of explicitly storing points depends
significant differences:
on the DBMS, the data, and the operating environment.
The pattern of updating temporary state is made a bit more awk- 2
Our cluster is made up of four SuperMicro X8DTT-H server mod-
ward in PostgreSQL due to its legacy of versioned storage. Post- ules, each equipped with one six-core Intel Xeon X5670 processor,
greSQL performs an update by first inserting a new row and then clocked at 2.93 GHz. While hyperthreading is enabled, we only run
marking the old row as invisible [37, Section 23.1.2]. As a result, a single Greenplum “segment” (query process) per physical core.
Each machine has 24 GB of RAM, an LSI MegaRAID 2108 “Raid
1
While PostgreSQL and Greenplum provide an optional On a Chip” controller with six attached 360 GB solid-state drives,
RETURNING clause for UPDATE commands, this returns only one and a Brocade 1020 converged network adapter. The operating
row for each row affected by the UPDATE, and aggregates cannot system is Red Hat Enterprise Linux Server release 5.5 (Tikanga).
be used within the RETURNING clause. Moreover, an UPDATE ... On that we are running Greenplum Database 4.2.0, compiled with
RETURNING cannot be used as a subquery. gcc 4.4.2.](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-9-320.jpg)
![200 5.1 Wisconsin Contributions:
6 Segments Convex Optimization
execution time (s)
150 12 Segments
The MADlib framework goes a long way toward making in-
18 Segments database analytic tools easier to deploy inside an RDBMS. Never-
100 24 Segments theless, to implement an algorithm within the MADlib framework, a
developer must undertake several steps: they must specify a model,
50 select the algorithm used to implement that model, optimize the
algorithm, test the model, and finally ensure that the resulting al-
0 gorithm and model are robust. This is a time consuming process
0 50 100 150 200 250 300 350 that creates code that must be tested and maintained for each data
analysis technique.
# independent variables A focus of the MADlib work at the University of Wisconsin has
been to explore techniques to reduce this burden. Our approach is
to expose a mathematical abstraction on top of MADlib that allows
Figure 5: Linear-regression execution times using MADlib v0.3 a developer to specify a smaller amount of code, which we hope
on Greenplum Database 4.2.0, 1 million rows
10 will lower the development time to add new techniques in many
cases. We discuss the challenge that we faced implementing this
abstraction. To demonstrate our ideas, we have implemented all
• Version 0.1alpha is an implementation in C that computes the of the models shown in Table 2 within the single abstraction (built
outer-vector products xi xT as a simple nested loop.
i within MADlib) that we describe below.
• Version 0.2.1beta introduced an implementation in C++ that
used the Armadillo [35] linear algebra library as a frontend The Key Abstraction. An ideal abstraction would allow us to
for LAPACK/BLAS. It turns out that this version was much decouple the specification of the model from the algorithm used
slower for two reasons: The BLAS library used was the de- to solve the specification. Fortunately, there is a beautiful, power-
fault one shipped with CentOS 5, which is built from the ful abstraction called convex optimization that has been developed
untuned reference BLAS. Profiling and examining the critial for the last few decades [34, 8] that allows one to perform this de-
code paths revealed that computing yT y for a row vector y coupling. More precisely, in convex optimization, we minimize a
is about three to four times slower than computing xxT for convex function over a convex set. The archetype convex function
a column vector x of the same dimension (and the MADlib is f (x) = x2 and is shown in Figure 6. Like all convex functions,
implementation unfortunately used to do the former). Inter- any local minimum of f is a global minimum of f . Many differ-
estingly, the same holds for Apple’s Accelerate framework on ent popular statistical models are defined by convex optimization
Mac OS X, which Apple promises to be a tuned library. The problems, e.g., linear regression, support vector machines, logistic
second reason for the speed disadvantage is runtime overhead regression, conditional random fields. Not every data analysis prob-
in the first incarnation of the C++ abstraction layer (mostly lem is convex—notable exceptions include the a priori algorithm
due to locking and calls into the DBMS backend). and graph mining algorithms—but many are convex. Table 2 lists
models that we have implemented in our abstraction over MADlib,
• Version 0.3 has an updated linear-regression implementation all of which are convex. (Note that the previous built-in MADlib
that relies on the Eigen C++ linear algebra library and takes examples of linear and logistic regression fall into this category!)
advantage of the fact that the matrix X T X is symmetric posi- In spite of the expressive power of convex optimization, even
tive definite. Runtime overhead has been reduced, but some simple algorithms converge at provable rates to the true solution.
calls into the database backend still need better caching. For intuition, examine the archetypical function f (x) = x2 shown
in Figure 6. The graph of this function is like all convex sets: bowl
Other noteworthy results during our performance studies included
shaped. To minimize the function, we just need to get to the bottom
that there are no measurable performance differences between Post-
of the bowl. As a result, even greedy schemes that decrease the
greSQL 9.1.1 (both in single and multi-user mode) and GP 4.1 in
function at each step will converge to an optimal solution. One
running the aggregate function on a single core. Moreover, while
such popular greedy method is called a gradient method. The idea
testing linear/logistic-regression execution times, single-core per-
is to find the steepest descent direction. In 1-d, this direction is
formance of even laptop CPUs (like the Core i5 540M) did not
the opposite direction of the derivative; in higher dimensions, it’s
differ much from today’s server CPUs (like the Xeon family). Typ-
called the gradient of f . Using the gradient, we iteratively move
ically the differences were even less than what the difference in
toward a solution. This process can be described by the following
clock speeds might have suggested, perhaps due to compiler and/or
pseudocode:
architecture issues that we have yet to unpack.
x ← x − α · G(x)
5. UNIVERSITY RESEARCH AND MADLIB where G(x) is the gradient of f (x) and α is a positive number called
An initial goal of the MADlib project was to engage closely the stepsize that goes to zero with more iterations. For example, it
with academic researchers, and provide a platform and distribution suffices to set α = 1/k where k is the number of iterations. In the
channel for their work. To that end, we report on two collaborations f (x) = x2 example, the gradient is the derivative, G(x) = 2x. Since
over the past years. We first discuss a more recent collaboration x = 0 is the minimum value, we have that for x > 0, G(x) < 0
with researchers at the University of Wisconsin. We then report while for x < 0 we have that G(x) > 0. For a convex function, the
on a collaboration with researchers at Florida and Berkeley, which gradient always tells us which direction to go to find a minimum
co-evolved with MADlib and explored similar issues (as part of the value, and the process described above is guaranteed to converge at
BayesStore project [42, 41]) but was actually integrated with the a known rate. One can provide a provable rate of convergence to
MADlib infrastructure only recently. the minimum value, which is in sharp contrast to a typical greedy](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-10-320.jpg)
![# segments # variables # rows v0.3 v0.2.1beta v0.1alpha
(million) (s) (s) (s)
6 10 10 4.447 9.501 1.337
6 20 10 4.688 11.60 1.874
6 40 10 6.843 17.96 3.828
6 80 10 13.28 52.94 12.98
6 160 10 35.66 181.4 51.20
6 320 10 186.2 683.8 333.4
12 10 10 2.115 4.756 0.9600
12 20 10 2.432 5.760 1.212
12 40 10 3.420 9.010 2.046
12 80 10 6.797 26.48 6.469
12 160 10 17.71 90.95 25.67
12 320 10 92.41 341.5 166.6
18 10 10 1.418 3.206 0.6197
18 20 10 1.648 3.805 1.003
18 40 10 2.335 5.994 1.183
18 80 10 4.461 17.73 4.314
18 160 10 11.90 60.58 17.14
18 320 10 61.66 227.7 111.4
24 10 10 1.197 2.383 0.3904
24 20 10 1.276 2.869 0.4769
24 40 10 1.698 4.475 1.151
24 80 10 3.363 13.35 3.263
24 160 10 8.840 45.48 13.10
24 320 10 46.18 171.7 84.59
Figure 4: Linear-regression execution times
search. In our prototype implementation in MADlib, we picked up
one such simple greedy algorithm, called stochastic (or sometimes,
“incremental”) gradient descent (SGD) [33, 6], that goes back to the
1960s. SGD is an approximation of gradient methods that is useful
when the convex function we are considering, f (x), has the form:
N
f (x) = fi (x)
i=1
If each of the fi is convex, then so is f [8, pg. 38]. Notice that Figure 6: The Archetypical Convex Function f (x) = x2 .
all problems in Table 2 are of this form: intuitively each of these
models is finding some model (i.e., a vector w) that is scored on
Application Objective
many different training examples. SGD leverages the above form to
Least Squares T u − y)2
construct a rough estimate of the gradient of f using the gradient of a (u,y)∈Ω (x
Lasso [38] T u − y)2 + µ x
single term: for example, the estimate if we select i is the gradient of (u,y)∈Ω (x 1
Logisitic Regression (u,y)∈Ω log(1 + exp(−yxt u))
fi (that we denote Gi (x)). The resulting algorithm is then described
Classification (SVM) (u,y)∈Ω (1 − yx u)+
T
as: 2 2
Recommendation (i, j)∈Ω (Li R j − Mi j ) + µ L, R
T
F
x ← x − αN · Gi (x) (1) Labeling (CRF) [40] k j x j F j (yk , zk ) − log Z(zk )
This approximation is guaranteed to converge to an optimal solu- Table 2: Models currently Implemented in MADlib using the
tion [26]. SGD-based approach.
Using the MADlib framework. In our setting, each tuple in
the input table for an analysis task encodes a single fi . We use the
micro-programming interfaces of Sections 3.2 and 3.3 to perform macro programming layer helps us test for convergence (which is
the mapping from the tuples to the vector representation that is used implemented with either a python combination or C driver.) Using
in Eq. 1. Then, we observe Eq. 1 is simply an expression over each this approach, we were able to add in implementations of all the
tuple (to compute Gi (x)) which is then averaged together. Instead of models in Table 2 in a matter of days.
averaging a single number, we average a vector of numbers. Here, In an upcoming paper we report initial experiments showing that
we use the macro-programming provided by MADlib to handle our SGD based approach achieves higher performance than prior
all data access, spills to disk, parallelized scans, etc. Finally, the data mining tools for some datasets [13].](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-11-320.jpg)
![Statistical Methods POS NER ER grams and compute a similarity as the approximate matching score.
Text Feature Extraction This functionality comes packaged with PostgreSQL.
Viterbi Inference Once we have the features, the next step is to perform inference
MCMC Inference
Approximate String Matching on the model. We also implemented two types of statistical inference
within the database: Viterbi (when we only want the most likely
Table 3: Statistical Text Analysis Methods answer from the model) and MCMC (when we want the probabilities
or confidence of an answer as well).
Viterbi Inference: The Viterbi dynamic programming algorithm [14]
is the popular algorithm to find the top-k most likely labelings of a
5.2 Florida/Berkeley Contributions: document for (linear chain) CRF models.
Statistical Text Analytics Like any dynamic programming algorithm, the Viterbi algorithm
The focus of the MADlib work at Florida and Berkeley has been is recursive. We experimented with two different implementations
to integrate statistical text analytics into a DBMS. In many domains, of macro-coordination over time. First, we chose to implement it
structured data and unstructured text are both important assets for using a combination of recursive SQL and window aggregate func-
data analysis. The increasing use of text analysis in enterprise tions. We discussed this implementation at some length in earlier
applications has increased the expectation of customers and the op- work [42]. Our initial recursive SQL implementation only runs over
portunities for processing big data. The state-of-the-art text analysis PostgreSQL versions 8.4 and later; it does not run in Greenplum.
tools are based on statistical models and algorithms [18, 12]. With Second, we chose to implement a Python UDF that uses iterations
the goal to become a framework for statistical methods for data anal- to drive the recursion in Viterbi. This iterative implementation
ysis at scale, it is important for MADlib to include basic statistical runs over both PostgreSQL and Greenplum. In Greenplum, Viterbi
methods to implement text analysis tasks. can be run in parallel over different subsets of the document on a
Basic text analysis tasks include part-of-speech (POS) tagging, multi-core machine.
named entity extraction (NER), and entity resolution (ER) [12]. MCMC Inference: Markov chain Monte Carlo (MCMC) meth-
Different statistical models and algorithms are implemented for ods are classical sampling algorithms that can be used to estimate
each of these tasks with different runtime-accuracy tradeoffs. For probability distributions. We implemented two MCMC methods in
example, an entity resolution task could be to find all mentions in MADlib: Gibbs sampling and Metropolis-Hastings (MCMC-MH).
a text corpus that refer to a real-world entity X. Such a task can be The MCMC algorithms involve iterative procedures where the
done efficiently by approximate string matching [25] techniques to current iteration depends on previous iterations. We used SQL win-
find all mentions in text that approximately match the name of entity dow aggregates for macro-coordination in this case, to carry “state”
X. However, such a method is not as accurate as the state-of-the-art across iterations to perform the Markov-chain process. This window
collective entity resolution algorithms based on statistical models, function based implementation runs over PostgreSQL version 8.4
such as Conditional Random Fields (CRFs) [19]. and later. We discussed this implementation at some length in recent
work [41]. We are currently working on integrating MCMC algo-
rithms into Greenplum DBMS. We also plan to implement MCMC
Pushing Statistical Text Analysis into MADlib. Based on using Python UDF macro-coordination analogous to Section 4.3,
the MADlib framework, our group set out to implement statistical
and compare the performance between the two implementations.
methods in SQL to support various text analysis tasks. We use CRFs
as the basic statistical model to perform more advanced text analysis.
Similar to Hidden Markov Models (HMM) cite, CRFs are a leading Using the MADlib Framework. Because this work predated
probabilistic model for solving many text analysis tasks, including the release of MADlib, it diverged from some of the macro-coordination
POS, NER and ER [19]. To support sophisticated text analysis, we patterns of Section 3.1, and took advantage of PostgreSQL features
implement four key methods: text feature extraction, most-likely that are less portable. These details require refactoring to fit into the
inference over a CRF (Viterbi), MCMC inference, and approximate current MADlib design style, which we are finding manageable. In
string matching (Table 3). addition to the work reported above, there are a host of other features
Text Feature Extraction: Text feature extraction is a step in of both PostgreSQL and MADlib that are valuable for text analyt-
most statistical text analysis methods, and it can be an expensive ics. Extension libraries for PostgreSQL and Greenplum provides
operation. To achieve high quality, CRF methods often assign text processing features such as inverted indexes, trigram indexes
hundreds of features to each token in the document. Examples of for approximate string matching, and array data types for model
such features include: (1) dictionary features: does this token exist parameters. Existing modules in MADlib, such as Naive Bayes and
in a provided dictionary? (2) regex features: does this token match Sparse/Dense Matrix manipulations, are building blocks to imple-
a provided regular expression? (3) edge features: is the label of ment statistical text analysis methods. Leveraging this diverse set
a token correlated with the label of a previous token? (4) word of tools and techniques that are already within the database allowed
features: does this the token appear in the training data? and (5) us to build a sophisticated text analysis engine that has comparable
position features: is this token the first or last in the token sequence? raw performance to off-the-shelf implementations, but runs natively
The right combination of features depends on the application, and in a DBMS close to the data [42, 41].
so our support for feature extraction heavily leverages the micro-
programming interface provided by MADlib.
Approximate String Matching: A recurring primitive operation 6. RELATED WORK
in text processing applications is the ability to match strings approx- The space of approaches to marry analytics with data is large
imately. The technique we use is based on qgrams [16]. We used and seems to be growing rapidly. At a high-level there are two
the trigram module in PostgreSQL to create and index 3-grams over approaches, (1) bring a statistical language to a data processing
text. Given a string “Tim Tebow” we can create a 3-gram by using substrate, or (2) provide a framework to express statistical tech-
a sliding window of 3 characters over this text string. Given two niques on top of a data processing substrate. Specifically, we
strings we can compare the overlap of two sets of corresponding 3- split the approaches to achieve this marriage into two groups: (1)](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-12-320.jpg)
![top-down approaches begin with a high-level statistical program- Two recent efforts to provide Scala-based domain-specific lan-
ming language, e.g., R or Matlab. The technical goal is to build a guages (DSLs) and parallel execution frameworks blur distinctions
parallel data processing infrastructure that is able to support run- between frameworks and language-based approaches. Spark [44] is
ning this high-level language in parallel. Examples of approach a Scala DSL targeted at Machine Learning, providing access to the
include System ML from IBM [15], Revolution Analytics [31], fault-tolerant, main-memory resilient distributed datasets: which
and SNOW [39]. (2) We call the second group framework-based are read-only collections of data that can be partitioned across a
approaches whose goal is to provide a set of building blocks (in- cluster. ScalOps [43] provides a Scala DSL for Machine Learning
dividual machine-learning algorithms) along with library support that is translated to Datalog, which is then optimized to run in par-
for micro- and macro-programming to write the algorithms. Recent allel on the Hyracks infrastructure [7]. Given its roots in Datalog
examples are Mahout [3], Graphlab [22], SciDB [36] and MADlib. and parallel relational algebra, ScalOps bears more similarity to
MADlib than any of the other toolkits mentioned here. It would be
Top-Down, Language-based Approaches. Within the ap- interesting to try and compile its DSL to the MADlib runtime.
proaches of type (1), we again divide the space into imperative and
declarative approaches. In the imperative approach, an analyst takes Other Data Processing Systems. There are a host of data
the responsibility of expressing how to parallelize the data access, processing techniques that can be used to solve elements of the
e.g., SNOW and Parallel R packages provide an MPI-interface along underlying data analysis problem. For example, Pregel [24] from
with calls for data partitioning within the framework of R. In con- Google is designed for data analysis over graphs. In Pregel, the ab-
trast in a declarative approach, the analyst declares their analysis straction is to write ones code using a graph-based abstraction: each
problem and it is the responsibility of the system to achieve this goal function can be viewed as a node that sends and receives messages
(in analogy with a standard RDBMS). Examples of the declarative to its neighbors in the graph. Pregel distributes the computation and
approach include SystemML [15], which is an effort from IBM to provides fault tolerance.
provide an R-like language to specify machine learning algorithms.
In SystemML, these high-level tasks are then compiled down to 7. CONCLUSION AND DISCUSSION
a Hadoop-based infrastructure. The essence of SystemML is its Scalable analytics are a clear priority for the research and in-
compilation techniques: they view R as a declarative language, and dustrial communities. MADlib was designed to fill a vacuum for
the goal of their system is to compile this language into an analog of scalable analytics in SQL DBMSs, and connect database research to
the relational algebra. Similar approaches are taken in Revolution market needs. In our experience, a parallel DBMS provides a very
Analytics [31] and Oracle’s new parallel R offering [28] in that these efficient and flexible dataflow substrate for implementing statistical
approaches attempt to automatically parallelize code written in R. and analytic methods at scale. Standardized support for SQL exten-
sions across DBMSs could be better—robust and portable support
Framework-based Approaches. In framework-based approaches, for recursion, window aggregates and linear algebra packages would
the goal is to provide the analyst with low-level primitives and co- simplify certain tasks. Since this is unlikely to occur across vendors
ordination primitives built on top of a data processing substrate. in the short term, we believe that MADlib can evolve reasonable
Typically, framework-based approaches offer a template whose goal workarounds at the library and “design pattern” level.
is to automate the common aspects of deploying an analytic task. The popular alternative to a DBMS infrastructure today is Hadoop
Framework-based approaches differ in what their data-processing MapReduce, which provides much lower-level programming APIs
substrates offer. For example, MADlib is in in this category as it than SQL. We have not yet undertaken performance comparisons
provides a library of functions over an RDBMS. The macro- and with Hadoop-based analytics projects like Mahout. Performance
micro-programming described earlier are examples of design tem- comparisons between MADlib and Mahout today would likely boil
plates. The goal of Apache Mahout [3] is to provide an open source down to (a) variations in algorithm implementations, and (b) well-
machine learning library over Apache Hadoop. Currently, Mahout known (and likely temporary) tradeoffs between the current states
provides a library of machine learning algorithms and a template to of DBMSs and Hadoop: C vs. Java, pipelining vs. checkpointing,
extend this library. At a lower level, the DryadLINQ large vector etc. [30] None of these variations and tradeoffs seem endemic, and
library provides the necessary data types to build analysis tasks (e.g., they may well converge over time. Moreover, from a marketplace
vectors) that are commonly used by machine learning techniques. perspective, such comparisons are not urgent: many users deploy
SciDB advocates a completely rewritten DBMS engine for numeri- both platforms and desire analytics libraries for each. So a rea-
cal computation, arguing that RDBMSs have “the wrong data model, sonable strategy for the community is to foster analytics work in
the wrong operators, and are missing required capabilities” [36]. both SQL and Hadoop environments, and explore new architectures
MADlib may be seen as a partial refutation of this claim. Indeed, (GraphLab, SciDB, ScalOps, etc.) at the same time.
the assertion in [36] that “RDBMSs, such as GreenPlum [sic]...must MADlib has room for growth in multiple dimensions. The library
convert a table to an array inside user-defined functions” is incor- infrastructure itself is still in beta, and has room to mature. There
rect: the MADlib C++ library hides representation from the UDF is room for enhancements in its core treatment of mathematical
developer and only pays for copies when modifying immutable kernels (e.g. linear algebra over both sparse and dense matrices)
structures. GraphLab is a framework to simplify the design of pro- especially in out-of-core settings. And of course there will always
gramming parallel machine learning tasks. The core computational be an appetite for additional statistical models and algorithmic meth-
abstraction is a graph of processing nodes that allows one to define ods, both textbook techniques and cutting-edge research. Finally,
asynchronous updates. Graphlab initially focused on providing a there is the challenge of porting MADlib to DBMSs other than
framework for easy access to multicore parallelism [22]. However, PostgreSQL and Greenplum. As discussed in Section 3, MADlib’s
the core computational abstraction is general and is now also de- macro-coordination logic is written in largely standard Python and
ployed in a cluster setting. Another popular toolkit in this space SQL, but its finer grained “micro-programming” layer exploits
is Vowpal Wabbit [20] that is extensively used in the academic proprietary DBMS extension interfaces. Porting MADlib across
machine learning community and provides high performance. DBMSs is a mechanical but non-trivial software development effort.
At the macro level, porting will involve the package infrastructure](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-13-320.jpg)
![(e.g. a cross-platform installer) and software engineering frame- [19] J. D. Lafferty, A. McCallum, and F. C. N. Pereira. Conditional random
work (e.g. testing scripts for additional database engines). At the fields: Probabilistic models for segmenting and labeling sequence data.
micro-programming logic level, inner loops of various methods will In ICML, pages 282–289, 2001.
need to be revisited (particularly user-defined functions). Finally, [20] J. Langford. http://hunch.net/ vw/.
since Greenplum retains most of the extension interfaces exposed [21] S. Lloyd. Least squares quantization in pcm. IEEE Transactions on
Information Theory, 28(2):129–137, Mar. 1982. Technical Report
by PostgreSQL, the current MADlib portability interfaces (e.g., the appeared much earlier in: Bell Telephone Laboratories Paper (1957).
C++ abstraction of Section 3.3) will likely require revisions when [22] Y. Low, J. Gonzalez, A. Kyrola, et al. Graphlab: A new framework for
porting to a system without PostgreSQL roots. parallel machine learning. In P. Grnwald and P. Spirtes, editors, UAI,
Compared to most highly scalable analytics packages today, pages 340–349, 2010.
MADlib v0.3 provides a relatively large number of widely-used [23] M. Mahajan, P. Nimbhorkar, and K. Varadarajan. The planar k-means
statistical and analytic methods. It is still in its early stages of de- problem is NP-hard. Theoretical Computer Science, June 2010. In
velopment, but is already in use both at research universities and at Press.
customer sites. As the current software matures, we hope to foster [24] G. Malewicz, M. H. Austern, A. J. Bik, et al. Pregel: a system for
large-scale graph processing. In SIGMOD, pages 135–146, 2010.
more partnerships with academic institutions, database vendors, and [25] G. Navarro. A guided tour to approximate string matching. ACM
customers. In doing so, we plan to pursue additional analytic meth- Comput. Surv., 33(1):31–88, Mar. 2001.
ods prioritized by both research “push” and customer “pull”. We [26] A. Nedic and D. P. Bertsekas. Convergence rate of incremental
also look forward to ports across DBMSs. subgradient algorithms. In S. Uryasev and P. M. Pardalos, editors,
Stochastic Optimization: Algorithms and Applications, pages 263–304.
Kluwer Academic Publishers, 2000.
8. ACKNOWLEDGMENTS [27] N. Nethercote and J. Seward. Valgrind: a framework for heavyweight
In addition to the authors, the following people have contributed dynamic binary instrumentation. In PLDI, pages 89–100, 2007.
to MADlib and its development: Jianwang Ao, Joel Cardoza, Brian [28] Oracle R Enterprise.
Dolan, Hlya Emir-Farinas, Christan Grant, Hitoshi Harada, Steven http://www.oracle.com/technetwork/database/options/advanced-
Hillion, Luke Lonergan, Gavin Sherry, Noelle Sio, Kelvin So, Gavin analytics/r-enterprise/index.html.
Yang, Ren Yi, Jin Yu, Kazi Zaman, Huanming Zhang. [29] C. Ordonez. Integrating k-means clustering with a relational dbms
using sql. IEEE Trans. Knowl. Data Eng., 18(2):188–201, 2006.
[30] A. Pavlo, E. Paulson, A. Rasin, et al. A comparison of approaches to
large-scale data analysis. In SIGMOD, pages 165–178. ACM, 2009.
9. REFERENCES [31] Revloution Analytics. http://www.revolutionanalytics.com/.
[1] D. Aloise, A. Deshpande, P. Hansen, et al. NP-hardness of Euclidean [32] B. Ripley. The r project in statistical computing. MSOR Connections,
sum-of-squares clustering. Machine Learning, 75:245–248, 2009. 1(1):23–25, 2001.
[2] E. Anderson, Z. Bai, C. Bischof, et al. LAPACK Users’ Guide. Society [33] H. Robbins and S. Monro. A stochastic approximation method. Ann.
for Industrial and Applied Mathematics, Philadelphia, PA, third Math. Statistics, 22, 1951.
edition, 1999.
[34] R. T. Rockafellar. Convex Analysis (Princeton Landmarks in
[3] Apache Mahout. http://mahout.apache.org/. Mathematics and Physics). Princeton University Press, 1996.
[4] D. Arthur, B. Manthey, and H. R¨ glin. k-means has polynomial
o [35] C. Sanderson. Armadillo: An open source C++ linear algebra library
smoothed complexity. arXiv, 2009. for fast prototyping and computationally intensive experiments.
[5] D. Arthur and S. Vassilvitskii. k-means++: the advantages of careful Technical report, NICTA, 2010.
seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on [36] M. Stonebraker, P. Brown, A. Poliakov, et al. The architecture of scidb.
Discrete Algorithms (SODA’07), pages 1027–1035, 2007. In Scientific and Statistical Database Management, pages 1–16.
[6] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, Springer, 2011.
MA, 2nd edition, 1999. [37] The PostgreSQL Global Development Group. PostgreSQL 9.1.3
[7] V. Borkar, M. Carey, R. Grover, et al. Hyracks: A flexible and Documentation, 2011.
extensible foundation for data-intensive computing. In ICDE, pages [38] R. Tibshirani. Regression shrinkage and selection via the lasso.
1151–1162, 2011. Journal of the Royal Statistical Society, Series B, 58:267–288, 1994.
[8] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge [39] L. Tierney, A. J. Rossini, and N. Li. Snow: a parallel computing
University Press, New York, NY, USA, 2004. framework for the r system. Int. J. Parallel Program., 37(1):78–90,
[9] J. Choi, J. Demmel, I. Dhillon, et al. ScaLAPACK: A portable linear Feb. 2009.
algebra library for distributed memory computers – design issues and [40] H. M. Wallach. Conditional random fields: An introduction. Technical
performance. Computer Physics Communications, 97(1):1–15, 1996. report, Dept. of CIS, Univ. of Pennsylvania, 2004.
[10] C.-T. Chu, S. K. Kim, Y.-A. Lin, et al. Map-reduce for machine [41] D. Wang, M. Franklin, M. Garofalakis, et al. Hybrid in-database
learning on multicore. In NIPS, pages 281–288, 2006. inference for declarative information extraction. In SIGMOD, 2011.
[11] J. Cohen, B. Dolan, M. Dunlap, et al. Mad skills: New analysis [42] D. Z. Wang, M. J. Franklin, M. N. Garofalakis, et al. Querying
practices for big data. PVLDB, 2(2), 2009. probabilistic information extraction. PVLDB, 3(1), 2010.
[12] R. Feldman and J. Sanger. The text mining handbook: advanced [43] M. Weimer, T. Condie, R. Ramakrishnan, et al. Machine learning in
approaches in analyzing unstructured data. Cambridge Univ Pr, 2007. scalops, a higher order cloud computing language. In NIPS 2011
[13] X. Feng, A. Kumar, B. Recht, et al. Towards a unified architecture for Workshop on parallel and large-scale machine learning (BigLearn),
in-rdbms analytics. In SIGMOD, 2012. volume 9, pages 389–396, 2011.
[14] G. Forney Jr. The viterbi algorithm. Proceedings of the IEEE, [44] M. Zaharia, M. Chowdhury, T. Das, et al. Resilient distributed datasets:
61(3):268–278, 1973. A fault-tolerant abstraction for in-memory cluster computing.
[15] A. Ghoting, R. Krishnamurthy, E. Pednault, et al. SystemML: Technical Report UCB/EECS-2011-82, EECS Department, University
Declarative Machine Learning on MapReduce. In ICDE, 2011. of California, Berkeley, Jul 2011.
[16] L. Gravano, P. Ipeirotis, H. Jagadish, et al. Using q-grams in a dbms [45] M. Zinkevich, M. Weimer, A. Smola, et al. Parallelized stochastic
for approximate string processing. IEEE Data Engineering Bulletin, gradient descent. In NIPS, 2010.
24(4):28–34, 2001.
[17] G. Guennebaud, B. Jacob, et al. Eigen v3. http://eigen.tuxfamily.org,
2010.
[18] D. Jurafsky and M. J. H. Speech and Language Processing. Pearson
Prentice Hall, 2008.](https://image.slidesharecdn.com/madlibwhitepaper-120705152603-phpapp01/85/The-MADlib-Analytics-Library-14-320.jpg)
The MADlib Analytics Library
- 1. The MADlib Analytics Library or MAD Skills, the SQL Joseph M. Hellerstein Christopher Ré Florian Schoppmann Zhe Daisy Wang Eugene Fratkin Aleksander Gorajek Kee Siong Ng Caleb Welton Xixuan Feng Kun Li Arun Kumar Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2012-38 http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-38.html April 3, 2012
- 2. Copyright © 2012, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission.
- 3. The MADlib Analytics Library or MAD Skills, the SQL Joseph M. Hellerstein Christoper Re ´ Florian Schoppmann Daisy Zhe Wang Eugene Fratkin U.C. Berkeley U. Wisconsin Greenplum U. Florida Greenplum Aleksander Gorajek Kee Siong Ng Caleb Welton Xixuan Feng Kun Li Arun Kumar Greenplum Greenplum Greenplum U. Wisconsin U. Florida U. Wisconsin ABSTRACT of potentially noisy data to support predictive analytics, provided MADlib is a free, open source library of in-database analytic meth- via statistical models and algorithms. Data Science is a name that is ods. It provides an evolving suite of SQL-based algorithms for gaining currency for the industry practices evolving around these machine learning, data mining and statistics that run at scale within workloads. a database engine, with no need for data import/export to other Data scientists make use of database engines in a very different tools. The goal is for MADlib to eventually serve a role for scalable way than traditional data warehousing professionals. Rather than database systems that is similar to the CRAN library for R: a com- carefully designing global schemas and “repelling” data until it is munity repository of statistical methods, this time written with scale integrated, they load data into private schemas in whatever form is and parallelism in mind. convenient. Rather than focusing on simple OLAP-style drill-down In this paper we introduce the MADlib project, including the reports, they implement rich statistical models and algorithms in the background that led to its beginnings, and the motivation for its open database, using extensible SQL as a language for orchestrating data source nature. We provide an overview of the library’s architecture movement between disk, memory, and multiple parallel machines. and design patterns, and provide a description of various statistical In short, for data scientists a DBMS is a scalable analytics runtime— methods in that context. We include performance and speedup one that is conveniently compatible with the database systems widely results of a core design pattern from one of those methods over the used for transactions and accounting. Greenplum parallel DBMS on a modest-sized test cluster. We then In 2008, a group of us from the database industry, consultancy, report on two initial efforts at incorporating academic research into academia, and end-user analytics got together to describe this usage MADlib, which is one of the project’s goals. pattern as we observed it in the field. We dubbed it MAD, an MADlib is freely available at http://madlib.net, and the acronym for the Magnetic (as opposed to repellent) aspect of the project is open for contributions of both new methods, and ports to platform, the Agile design patterns used for modeling, loading and additional database platforms. iterating on data, and the Deep statistical models and algorithms being used. The “MAD Skills” paper that resulted described this pattern, and included a number of non-trivial analytics techniques 1. INTRODUCTION: implemented as simple SQL scripts [11]. FROM WAREHOUSING TO SCIENCE After the publication of the paper, significant interest emerged Until fairly recently, large databases were used mainly for account- not only in its design aspects, but also in the actual SQL imple- ing purposes in enterprises, supporting financial record-keeping and mentations of statistical methods. This interest came from many reporting at various levels of granularity. Data Warehousing was directions: customers were requesting it of consultants and vendors, the name given to industry practices for these database workloads. and academics were increasingly publishing papers on the topic. Accounting, by definition, involves significant care and attention to What was missing was a software framework to focus the energy of detail. Data Warehousing practices followed suit by encouraging the community, and connect the various interested constituencies. careful and comprehensive database design, and by following exact- This led to the design of MADlib, the subject of this paper. ing policies regarding the quality of data loaded into the database. Attitudes toward large databases have been changing quickly in Introducing MADlib the past decade, as the focus of large database usage has shifted from accountancy to analytics. The need for correct accounting and MADlib is a library of analytic methods that can be installed and data warehousing practice has not gone away, but it is becoming a executed within a relational database engine that supports extensi- shrinking fraction of the volume—and the value—of large-scale data ble SQL. A snapshot of the current contents of MADlib including management. The emerging trend focuses on the use of a wide range methods and ports is provided in Table 1. This set of methods and ports is intended to grow over time. The methods in MADlib are designed both for in- or out-of- core execution, and for the shared-nothing, “scale-out” parallelism offered by modern parallel database engines, ensuring that compu- tation is done close to the data. The core functionality is written in declarative SQL statements, which orchestrate data movement to and from disk, and across networked machines. Single-node inner loops take advantage of SQL extensibility to call out to high- performance math libraries in user-defined scalar and aggregate functions. At the highest level, tasks that require iteration and/or
- 4. Category Method today requires extracting advantages in the long tail of “special in- Supervised Learning Linear Regression terests”, a practice known as “microtargeting”, “hypertargeting” or Logistic Regression “narrowcasting”. In that context, the first step of SEMMA essentially Naive Bayes Classification defeats the remaining four steps, leading to simplistic, generalized Decision Trees (C4.5) decision-making that may not translate well to small populations in Support Vector Machines the tail of the distribution. In the era of “Big Data”, this argument Unsupervised Learning k-Means Clustering for enhanced attention to long tails applies to an increasing range of SVD Matrix Factorization use cases. Latent Dirichlet Allocation Driven in part by this observation, momentum has been gathering Association Rules around efforts to develop scalable full-dataset analytics. One pop- Decriptive Statistics Count-Min Sketch ular alternative is to push the statistical methods directly into new Flajolet-Martin Sketch parallel processing platforms—notably, Apache Hadoop. For ex- Data Profiling ample, the open source Mahout project aims to implement machine Quantiles learning tools within Hadoop, harnessing interest in both academia Support Modules Sparse Vectors and industry [10, 3]. This is certainly a plausible path to a solution, Array Operations and Hadoop is being advocated as a promising approach even by Conjugate Gradient Optimization major database players, including EMC/Greenplum, Oracle and Microsoft. Table 1: Methods provided in MADlib v0.3. This version has At the same time that the Hadoop ecosystem has been evolving, been tested on two DBMS platforms: PostgreSQL (single-node the SQL-based analytics ecosystem has grown rapidly as well, and open source) and Greenplum Database (massively parallel com- large volumes of valuable data are likely to pour into SQL systems mercial system, free and fully-functional for research use.) for many years to come. There is a rich ecosystem of tools, know- how, and organizational requirements that encourage this. For these cases, it would be helpful to push statistical methods into the DBMS. structure definition are coded in Python driver routines, which are And as we will see, massively parallel databases form a surprisingly used only to kick off the data-rich computations that happen within useful platform for sophisticated analytics. MADlib currently targets the database engine. this environment of in-database analytics. MADlib is hosted publicly at github, and readers are encouraged to browse the code and documentation via the MADlib website 2.2 Why Open Source? http://madlib.net. The initial MADlib codebase reflects contri- From the beginning, MADlib was designed as an open source butions from both industry (Greenplum) and academia (UC Berke- project with corporate backing, rather than a closed-source corporate ley, the University of Wisconsin, and the University of Florida). effort with academic consulting. This decision was motivated by a Code management and Quality Assurance efforts have been con- number of factors, including the following: tributed by Greenplum. At this time, the project is ready to consider • The benefits of customization: Statistical methods are rarely contributions from additional parties, including both new methods used as turnkey solutions. As a result, it is common for data and ports to new platforms. scientists to want to modify and adapt canonical models and methods (e.g., regression, classification, clustering) to their 2. GOALS OF THE PROJECT own purposes. This is a very tangible benefit of open source The primary goal of the MADlib open source project is to ac- libraries over traditional closed-source packages. Moreover, celerate innovation and technology transfer in the Data Science in an open source community there is a process and a set of community via a shared library of scalable in-database analytics, positive incentives for useful modifications to be shared back much as the CRAN library serves the R community [32]. Unlike to the benefit of the entire community. CRAN, which is customized to the R analytics engine, we hope that • Valuable data vs. valuable software: In many emerging MADlib’s grounding in standard SQL can lead to community ports business sectors, the corporate value is captured in the data to a variety of parallel database engines. itself, not in the software used to analyze that data. Indeed, it 2.1 Why Databases? is in the interest of these companies to have the open source community adopt and improve their software. open source For decades, statistical packages like SAS, Matlab and R have efforts can also be synergistic for vendors that sell commercial been the key tools for deep analytics, and the practices surrounding software, as evidenced by companies like EMC/Greenplum, these tools have been elevated into widely-used traditional method- Oracle, Microsoft and others beginning to provide Apache ologies. One standard analytics methodology advocated in this do- Hadoop alongside their commercial databases. Most IT shops main is called SEMMA: Sample, Explore, Modify, Model, Assess. today run a mix of open source and proprietary software, The “EMMA” portion of this cycle identifies a set of fundamental and many software vendors are finding it wise to position tasks that an analyst needs to perform, but the first, “S” step makes themselves intelligently in that context. Meanwhile, for most less and less sense in many settings today. The costs of computation database system vendors, their core competency is not in and storage are increasingly cheap, and entire data sets can often be statistical methods, but rather in the engines that support processed efficiently by a cluster of computers. Meanwhile, compe- those methods, and the service industry that evolves around tition for extracting value from data has become increasingly refined. them. For these vendors, involvement and expertise with an Consider fiercely competitive application domains like online adver- open source library like MADlib is an opportunity to expand tising or politics. It is of course important to target “typical” people their functionality and service offerings. (customers, voters) that would be captured by sampling the database. But the fact that SEMMA is standard practice means that optimiz- • Closing the research-to-adoption loop: Very few traditional ing for a sample provides no real competitive advantage. Winning database customers have the capacity to develop significant
- 5. in-house research into computing or data science. On the can recruit high-quality, experienced software engineers with the other hand, it is hard for academics doing computing research attraction of well-compensated, long-term career paths. Equally to understand and influence the way that analytic processes important, software shops can offer an entire software engineering are done in the field. An open source project like MADlib pipeline that cannot be replicated on campus: this includes QA has the potential to connect academic researchers not only to processes and QA engineering staff. The hope is that the corporate industrial software vendors, but also directly to the end-users staffing of research projects like MADlib can enable more impactful of analytics software. This can improve technology transfer academic open source research, and speed technology transfer to from academia into practice without requiring database soft- industry. ware vendors to serve as middlemen. It can similarly enable end-users in specific application domains to influence the 2.4 MADlib Status research agenda in academia. MADlib is still young, currently (as of March, 2012) at Version 0.3. The initial versions have focused on establishing a baseline of • Leveling the playing field, encouraging innovation: Over useful functionality, while laying the groundwork for future evolu- the past two decades, database software vendors have devel- tion. Initial development began with the non-trivial work of building oped proprietary data mining toolkits consisting of textbook the general-purpose framework described in Section 3. Addition- algorithms. It is hard to assess their relative merits. Mean- ally, we wanted robust implementations of textbook methods that while, other communities in machine learning and internet were most frequently requested from customers we met through advertising have also been busily innovating, but their code contacts at Greenplum. Finally, we wanted to validate MADlib as a is typically not well packaged for reuse, and the code that is research vehicle, by fostering a small number of university groups available was not written to run in a database system. None working in the area to experiment with the platform and get their of these projects has demonstrated the vibrancy and breadth code disseminated (Section 5). we see in the open source community surrounding R and its CRAN package. The goal of MADlib is to fill this gap: bring the database community up to a baseline level of compe- 3. MADLIB ARCHITECTURE tence on standard statistical algorithms, remove opportunities The core of traditional SQL—SELECT... FROM... WHERE... for proprietary FUD, and help focus a large community on GROUP BY—is quite a powerful harness for orchestrating bulk data innovation and technology transfer. processing across one or many processors and disks. It is also a portable, native language supported by a range of widely-deployed 2.3 A Model for Open Source Collaboration open source and commercial database engines. This makes SQL an The design of MADlib comes at a time when the connections attractive framework for writing data-intensive programs. Ideally, between open source software and academic research seem particu- we would like MADlib methods to be written entirely in straightfor- larly frayed. MADlib is designed in part as an experiment in binding ward and portable SQL. Unfortunately, the portable core of “vanilla” these communities more tightly, to face current realities in software SQL is often not quite enough to express the kinds of algorithms development. needed for advanced analytics. In previous decades, many important open source packages origi- Many statistical methods boil down to linear algebra expressions nated in universities and evolved into significant commercial prod- over matrices. For relational databases to operate over very large ucts. Examples include the Ingres and Postgres database systems, matrices, this presents challenges at two scales. At a macroscopic the BSD UNIX and Mach operating systems, the X Window user scale, the matrices must be intelligently partitioned into chunks that interfaces and the Kerberos authentication suite. These projects can fit in memory on a single node. Once partitioned, the pieces were characterized by aggressive application of new research ideas, can be keyed in such a way that SQL constructs can be used to captured in workable but fairly raw public releases that matured orchestrate the movement of these chunks into and out of memory slowly with the help of communities outside the university. While across one or many machines. At a microscopic scale, the database all of the above examples were incorporated into commercial prod- engine must invoke efficient linear algebra routines on the pieces ucts, many of those efforts emerged years or decades after the initial of data it gets in core. To this end it has to have the ability to very open source releases, and often with significant changes. quickly invoke well-tuned linear algebra methods. Today, we expect successful open source projects to be quite We proceed to discuss issues involved at both of these levels in a mature, often comparable to commercial products. To achieve this bit more detail, and solutions we chose to implement in MADlib. level of maturity, most successful open source projects have one or more major corporate backers who pay some number of com- 3.1 Macro-Programming (Orchestration) mitters and provide professional support for Quality Assurance A scalable method for linear algebra depends upon divide-and- (QA). This kind of investment is typically made in familiar soft- conquer techniques: intelligent partitioning of the matrix, and a ware packages, not academic research projects. Many of the most pattern to process the pieces and merge results back together. This popular examples—Hadoop, Linux, OpenOffice—began as efforts partitioning and dataflow is currently outside the scope of a tradi- to produce open source alternatives to well-identified, pre-existing tional query optimizer or database design tool. But there is a rich commercial efforts. literature from scientific computing on these issues (e.g., [9]) that MADlib is making an explicit effort to explore a new model database programmers can use to craft efficient in-database imple- for industry support of academic research via open source. Many mentations. Once data is properly partitioned, database engines academic research projects are generously supported by financial shine at orchestrating the resulting data movement of partitions and grants and gifts from companies. In MADlib, the corporate do- the piecewise results of computation. nation has largely consisted of a commitment to allocate signifi- cant professional software engineering time to bootstrap an open 3.1.1 User-Defined Aggregation source sandbox for academic research and tech transfer to practice. The most basic building block in the macro-programming of This leverages a strength of industry that is not easily replicated MADlib is the use of user-defined aggregates (UDAs). In general, by government and other non-profit funding sources. Companies aggregates—and the related window functions—are the natural way
- 6. in SQL to implement mathematical functions that take as input the conditions—this was used by Wang et al. to implement Viterbi infer- values of an arbitrary number of rows (tuples). DBMSs typically ence [42] (Section 5.2). Unfortunately, like windowed aggregates, implement aggregates as data-parallel streaming algorithms. And recursion support in SQL varies across database products, and does there is a large body of recent work on online learning algorithms not form a reliable basis for portability. and model-averaging techniques that fit the computational model of Driver Functions. None of the above methods provides both gener- aggregates well (see, e.g., [45]). ality and portability. As a result, in MADlib we chose to implement Unfortunately, extension interfaces for user-defined aggregates complex iterative methods by writing a driver UDF in Python to vary widely across vendors and open source systems. Nonethe- control iteration, which passes state across iterations intelligently. A less, the aggregation paradigm (or in functional programming terms, standard pitfall in this style of programming is for the driver code to “fold” or “reduce”) is natural and ubiquitous, and we expect the basic pull a large amount of data out of the database; this becomes a scala- algorithmic patterns for user-defined aggregates to be very portable. bility bottleneck since the driver code typically does not parallelize In most widely-used DBMSs (e.g., in PostgreSQL, MySQL, Green- and hence pulls all data to a single node. We avoid this via a design plum, Oracle, SQL Server, Teradata), a user-defined aggregate con- pattern in which the driver UDF kicks off each iteration and stages sists of a well-known pattern of two or three user-defined functions: any inter-iteration output into a temporary table via CREATE TEMP TABLE... AS SELECT... It then reuses the resulting temp table 1. A transition function that takes the current transition state and in subsequent iterations as needed. Final outputs are also often a new data point. It combines both into into a new transition stored in temp tables unless they are small, and can be interrogated state. using small aggregate queries as needed. As a result, all large-data 2. An optional merge function that takes two transition states movement is done within the database engine and its buffer pool. and computes a new combined transition state. This function Database engines typically provide efficient parallelism as well as is only needed for parallel execution. buffering and spill files on disk for large temp tables, so this pattern is quite efficient in practice. Sections 4.2 and 4.3 provide discussion 3. A final function that takes a transition state and transforms it of this pattern in the context of specific algorithms. into the output value. 3.1.3 Templated Queries Clearly, a user-defined aggregate is inherently data-parallel if the transition function is associative and the merge function returns the A second problem is a limitation of SQL’s roots in first-order same result as if the transition function was called repeatedly for logic, which requires that queries be cognizant of the schema of every individual element in the second state. their input tables, and produce output tables with a fixed schema. Unfortunately, user-defined aggregates are not enough. In design- In many cases we want to write “templated” queries that work over ing the high-level orchestration of data movement for analytics, we arbitrary schemas, with the details of arity, column names and types ran across two main limitations in standard SQL that we describe to be filled in later. next. We addressed both these limitations using driver code written For example, the MADlib profile module takes an arbitrary in simple script-based user-defined functions (UDFs), which in turn table as input, producing univariate summary statistics for each of kick off more involved SQL queries. When implemented correctly, its columns. The input schema to this module is not fixed, and the the performance of the scripting language code is not critical, since output schema is a function of the input schema (a certain number its logic is invoked only occasionally to kick off much larger bulk of output columns for each input column). To address this issue, we tasks that are executed by the core database engine. use Python UDFs to interrogate the database catalog for details of input tables, and then synthesize customized SQL queries based on 3.1.2 Driver Functions for Multipass Iteration templates to produce outputs. Simpler versions of this issue arise in The first problem we faced is the prevalence of “iterative” algo- most of our iterative algorithms. rithms for many methods in statistics, which make many passes Unfortunately, templated SQL relies on identifiers or expressions over a data set. Common examples include optimization methods passed as strings to represent database objects like tables. As such, like Gradient Descent and Markov Chain Monte Carlo (MCMC) the DBMS backend will discover syntactical errors only when the simulation in which the number of iterations is determined by a generated SQL is executed, often leading to error messages that are data-dependent stopping condition at the end of each round. There enigmatic to the user. As a result, templated SQL necessitates that are multiple SQL-based workarounds for this problem, whose appli- MADlib code perform additional validation and error handling up cability depends on the context. front, in order to not compromise usability. In the future we plan to Counted Iteration via Virtual Tables. In order to drive a fixed support this pattern as a Python library that ships with MADlib and number n of independent iterations, it is often simplest (and very ef- provides useful programmer APIs and user feedback. ficient) to declare a virtual table with n rows (e.g., via PostgreSQL’s generate series table function), and join it with a view repre- 3.2 Micro-Programming: Data Representa- senting a single iteration. This approach was used to implement tions and Inner Loops m-of-n Bootstrap sampling in the original MAD Skills paper [11]. In addition to doing the coarse-grained orchestration of chunks, It is supported in some fashion in a variety of DBMSs, sometimes the database engine must very efficiently invoke the single-node code by writing a simple table function consisting of a few lines of code. that performs arithmetic on those chunks. For UDFs that operate at Window Aggregates for Stateful Iteration. For settings where the row level (perhaps called multiple times per row), the standard the current iteration depends on previous iterations, SQL’s win- practice is to implement them in C or C++. When computing dense dowed aggregate feature can be used to carry state across iterations. matrix operations, these functions would make native calls to an Wang et al. took this approach to implement in-database MCMC open source library like LAPACK [2] or Eigen [17]. inference [41] (Section 5.2). Unfortunately the level of support for Sparse matrices are not as well-handled by standard math libraries, window aggregates varies across SQL engines. and require more customization for efficient representations both Recursive Queries. Most generally, it is possible to use the recur- on disk and in memory. We chose to write our own sparse matrix sion features of SQL to perform iteration with arbitrary stopping library in C for MADlib, which implements a run-length encoding
- 7. scheme. Both of these solutions require careful low-level coding, 4.1 Single-Pass: Ordinary Least Squares and formed part of the overhead of getting MADlib started. In ordinary-least-squares (OLS) linear regression the goal is to The specifics of a given method’s linear algebra can be coded in fit a linear function to a set of points, with the objective of mini- a low-level way using loops of basic arithmetic in a language like C, mizing the sum of squared residuals. Formally, we are given points but it is nicer if they can be expressed in a higher-level syntax that (x1 , y1 ), . . . , (xn , yn ), where xi ∈ Rd and yi ∈ R, and our goal is to captures the semantics of the linear algebra at the level of matrices find the vector b that minimizes n (yi − b, xi )2 . OLS is one of i=1 and arrays. We turn to this issue next. the most fundamental methods in statistics. Typically, each yi is assumed to be an (independent) noisy measurement of b, xi , where 3.3 A C++ Abstraction Layer for UDFs b is an unknown but fixed vector and the noise is uncorrelated with There are a number of complexities involved in writing C or C++- mean 0 and unknown but fixed variance. Under these assumptions, based user-defined functions over a legacy DBMS like PostgreSQL, b is the best linear unbiased estimate of b (Gauss-Markov). Un- all of which can get in the way of maintainable, portable application der additional assumptions (normality, independence), b is also the logic. This complexity can be especially frustrating for routines maximum-likelihood estimate. Letting X denote the matrix whose whose pseudocode amounts to a short linear algebra expression rows are xT , and defining y := (y1 , . . . , yn )T , it is well-known that i that should result in a compact implementation. MADlib provides the sum of squared residuals is minimized by b = (X T X)−1 X T y (for a C++ abstraction layer both to ease the burden of writing high- exposition purposes we assume the full-rank case here, though this performance UDFs, and to encapsulate DBMS-specific logic inside is not a requirement for MADlib). the abstraction layer, rather than spreading the cost of porting across all the UDFs in the library. In brief, the MADlib C++ abstraction It has been observed before that computing b lends itself well to provides three classes of functionality: type bridging, resource data-parallel implementations [10]—in extensible database terms, management shims, and math library integration. it can be done with a simple user-defined aggregate. The principal Type bridging is provided via an encapsulated mapping of C++ observation is this: X T X = n xi xT and X T y = n xi yi are i=1 i i=1 types and methods to database types and functions. UDFs can be just sums of transformations of each data point. Summation is written with standard C++ atomic types, as well as the vector and associative, so data parallelism virtually comes for free—we can matrix types that are native to a high performance linear algebra compute the per-process subsums of the previous expressions locally library. (We have successfully layered multiple alternative libraries in each process, and then sum up all subsums during a second- under this interface, and are currently using Eigen [17]). The trans- phase aggregation. As a final non-parallelized step, we compute the lation to and from database types (including composite types like inverse of X T X and then multiply with X T y. These final operations double precision[] for vectors) is handled by the abstraction are comparatively cheap, since the number of independent variables layer. Similarly, higher-order templated functions in C++ can be (and thus the dimensions of X T X and X T y) is typically “small”. mapped to the appropriate object IDs of UDFs in the database, with 4.1.1 MADlib Implementation the abstraction layer taking care of looking up the function in the We assume that data points are stored as (x DOUBLE PRECISION[], database catalog, verifying argument lists, ensuring type-safety, etc. y DOUBLE PRECISION) tuples. Linear regression is then imple- The second aspect of the C++ abstraction layer is to provide a mented as a user-defined aggregate with a transition and final func- safe and robust standard runtime interface to DBMS-managed re- tion roughly as in Listings 1 and 2, respectively. (For compactness, sources. This includes layering C++ object allocation/deallocation we omitted finiteness checks and several output statistics in the ex- over DBMS-managed memory interfaces, providing shims between ample here.) The merge function, which is not shown, just adds all C++ exception handling and DBMS handlers, and correctly propa- values in the transition states together. Running the code produces gating system signals to and from the DBMS. the following: Finally, by incorporating proven third-party libraries, the C++ abstraction layer makes it easy for MADlib developers to write psql# SELECT (linregr(y, x)).* FROM data; correct and performant code. For example, the Eigen linear algebra -[ RECORD 1 ]+-------------------------------------------- coef | {1.7307,2.2428} library contains well-tested and well-tuned code that makes use r2 | 0.9475 of the SIMD instruction sets (like SSE) found in today’s CPUs. std_err | {0.3258,0.0533} Likewise, the abstraction layer itself has been tuned for efficient t_stats | {5.3127,42.0640} p_values | {6.7681e-07,4.4409e-16} value marshalling, and code based on it will automatically benefit condition_no | 169.5093 from future improvements. By virtue of being a template library, the runtime and abstraction overhead is reduced to a minimum. Note that the linregr Python UDF produces a composite record As an illustration of the high-level code one can write over our type in the output, which is a feature of PostgreSQL and Greenplum. abstraction layer, Listings 1 and 2 show reduced, but fully functional This would be easy to flatten into a string in a strictly relational code snippets that implement multiple linear regression (as discussed implementation. further in Section 4.1). 4.2 Multi-Pass: (Binary) Logistic Regression In (binary) logistic regression, we are given points (x1 , y1 ), . . . , 4. EXAMPLES (xn , yn ), where xi ∈ Rd and yi ∈ {0, 1}, and our goal is to find the vec- 1 To illustrate the above points, we look at three different algorith- tor b that maximizes n σ((−1)yi +1 · b, xi ). Here, σ(z) = 1+exp(z) i=1 mic scenarios. The first is Linear Regression using Ordinary Least denotes the logistic function. Statistically, this is the maximum- Squares (OLS), which is an example of a widely useful, simple likelihood estimate for an unknown vector b under the assumption single-pass aggregation technique. The second is binary Logistic that each yi is a random variate with Pr[yi = 1 | xi ] = σ( b, xi ) and Regression, another widely used technique, but one that employs that all observations are independent. an iterative algorithm. Finally, we look at k-means Clustering, an It is well-known that, in general, no closed-formula expression iterative algorithm with large intermediate states spread across ma- for b exists. Instead, b can be computed as the solution of a con- chines. vex program via standard iterative methods. Arguably, the most
- 8. 1 AnyType 1 AnyType 2 linregr_transition::run(AnyType &args) { 2 linregr_final::run(AnyType &args) { 3 // Transition state is a class that wraps an array. 3 // Immutable array: Array will never be copied 4 // We expect a mutable array. If DBMS allows 4 LinRegrTransitionState< 5 // modifications, copying will be avoided. 5 ArrayHandle<double> > state = args[0]; 6 LinRegrTransitionState< 7 MutableArrayHandle<double> > state = args[0]; 7 // The following is a MADlib class that wraps Eigen’s 8 // Dependent variable is a double-precision float 8 // solver for self-adjoint matrices. 9 double y = args[1].getAs<double>(); 9 SymmetricPositiveDefiniteEigenDecomposition<Matrix> 10 // Vector of independent variables wraps an immutable 10 decomposition(state.X_transp_X, EigenvaluesOnly, 11 // array (again, no unnecessary copying). This maps 11 ComputePseudoInverse); 12 // to an Eigen type 13 HandleMap<const ColumnVector> x 13 Matrix inverse_of_X_transp_X 14 = args[2].getAs<ArrayHandle<double> >(); 14 = decomposition.pseudoInverse(); 15 // Let backend allocate array for coefficients so to 16 if (state.numRows == 0) 16 // avoid copying on return (if supported by DBMS). 17 // The first row determines the number 17 HandleMap<ColumnVector> coef( 18 // of independent variables 18 allocateArray<double>(state.widthOfX)); 19 state.initialize(*this, x.size()); 19 coef.noalias() = inverse_of_X_transp_X 20 state.numRows++; 20 * state.X_transp_Y; 21 state.y_sum += y; 22 state.y_square_sum += y * y; 22 // Return a composite value. 23 // noalias informs Eigen to multiply in-place 23 AnyType tuple; 24 state.X_transp_Y.noalias() += x * y; 24 tuple << coef << decomposition.conditionNo(); 25 // Since XˆT X is symmetric, we only need to 25 return tuple; 26 // compute a triangular part 26 } 27 triangularView<Lower>(state.X_transp_X) 28 += x * trans(x); Figure 2: Linear-regression final function 30 return state; 31 } language. In many cases there are multiple equivalent patterns Figure 1: Linear-regression transition function for constructing simple interfaces, but no well-accepted, uniform design patterns for the kind of algorithmic expressions we tend to implement in MADlib. We are refining these design patterns as we common method is to maximize the logarithm of the likelihood evolve the library. using Newton’s method. In the case of logistic regression this re- duces to iteratively reweighted least squares with iteration rule 4.3 Large-State Iteration: k-Means βm+1 = (X T Dm X)−1 X T Dm zm . Here, the diagonal matrix Dm and the In k-means clustering, we are given n points x1 , . . . , xn ∈ Rd , and vector zm are transformations of X and βm . our goal is to position k centroids c1 , . . . , ck ∈ Rd so that the sum of squared distances between each point and its closest centroid is 4.2.1 MADlib Implementation minimized. Formally, we wish to minimize n minkj=1 xi − c j 2 . i=1 Each individual iteration can be implemented via a user-defined Solving this problem exactly is usually prohibitively expensive (for aggregate using linear regression as a blueprint. However, the hand- theoretical hardness results see, e.g., [1, 23]). However, the local- ling of iterations and checking for convergence require a further search heuristic proposed by Lloyd [21] performs reasonably well outer loop. We therefore implement a driver UDF in Python. The both in theory and in practice [5, 4]. At a high level, it works as control flow follows the high-level outline from Section 3.1.2 and follows: is illustrated as an activity diagram in Figure 3. Here, the shaded shapes are executions of generated SQL, where current iteration 1. Seeding phase: Find initial positions for k centroids c1 , . . . , ck . is a template parameter that is substituted with the corresponding 2. Assign each point x1 , . . . , xn to its closest centroid. Python variable. Specifically, the UDF first creates a temporary table for storing 3. Reposition each centroid to the barycenter (mean) of all points the inter-iteration states. Then, the Python code iteratively calls the assigned to it. UDA for updating the iteration state, each time adding a new row to the temporary table. Once the convergence criterion has been 4. If no (or only very few) points got reassigned, stop. Otherwise, reached, the state is converted into the return value. The important goto (2). point to note is that there is no data movement between the driver function and the database engine—all heavy lifting is done within 4.3.1 MADlib implementation the database engine. k-means has a natural implementation in SQL [29]. Based on the Unfortunately, implementing logistic regression using a driver assumption that we can always comfortably store k centroids in main function leads to a different interface than the one we provided for memory, we can implement k-means similarly to logistic regression: linear regression: Using a driver function that iteratively calls a user-defined aggregate. In the following, we take a closer look at this implementation. It SELECT * FROM logregr(’y’, ’x’, ’data’); is important to make a clear distinction between the inter-iteration A problem with this implementation is that the logregr UDF is state (the output of the UDA’s final function) and intra-iteration not an aggregate function and cannot be used in grouping constructs. state (as maintained by the UDA’s transition and merge functions). To perform multiple logistic regressions at once, one needs to use During aggregation, the transition state contains both inter- and a join construct instead. We intend to address this non-uniformity intra-iteration state, but only modifies the intra-iteration state. We in interface in a future version of MADlib. We highlight the issue only store k centroids in both the inter- and intra-iteration states, and here in part to point out that SQL can be a somewhat “over-rich” consider the assignments of points to centroids as implicitly given.
- 9. for updates that touch many rows it is typically faster to copy the CREATE TEMORARY TABLE iterative_algorithm AS updated data into a new table (i.e., CREATE TABLE AS SELECT SELECT 0 AS iteration, NULL AS state and DROP TABLE) rather than issue an UPDATE. These kinds of DBMS-specific performance tricks may merit encapsulation in an current_iteration = 0 abstraction layer for SQL portability. INSERT INTO iterative_algorithm(iteration, state) SELECT iteration + 1, logregr_irls_step(y, x, state) 4.4 Infrastructure Performance Trends FROM source, iterative_algorithm WHERE iteration = current_iteration In its current beta version, MADlib has been tuned a fair bit over PostgreSQL and Greenplum, though much remains to be done. Here current_iteration ++ we report on some results for a basic scenario that exercises our core functionality, including the C++ abstraction layer, our ability to call SELECT internal_logregr_irls_did_converte(state) out to linear algebra packages, and parallel speedup validation. We FROM iterative_algorithm defer macro-benchmarking of MADlib’s current methods to future WHERE iteration = current_iteration work, that will focus on specific algorithm implementations. [False] The basic building block of MADlib is a user-defined aggregate, [True] typically one that calls out to a linear algebra library. In order to SELECT internal_logregr_irls_result(state) evaluate the scalability of this construct, we ran linear regression FROM iterative_algorithm WHERE iteration = current_iteration over Greenplum’s parallel DBMS on various data sizes, using a 24- core test cluster we had available, which was outfitted with 144 GB of RAM over 51 TB of raw storage.2 This is obviously a relatively modest-sized cluster by today’s standards, but it is sufficient to illuminate (a) our efforts at minimizing performance overheads, and Figure 3: Sequence Diagram for Logistic Regression (b) our ability to achieve appropriate parallel speedup. For running linear regression as outlined in Section 4.1, we expect runtime O(k3 + (n · k2 )/p) where k is the number of independent In the transition function, we first compute the centroid that the variables, n is the number of observations, and p is the number of current point was closest to at the beginning of the iteration using the query processes. The k3 time is needed for the matrix inversion, and inter-iteration state. We then update the barycenter of this centroid the k2 is needed for computing each outer product xi xT and adding in the intra-iteration state. Only as the final step of the aggregate, i it to the running sum. It turns out that our runtime measurements fit the intra-iteration state becomes the new inter-iteration state. these expectations quite well, and the constant factors are relatively Unfortunately, in order to check the convergence criterion that no small. See Figure 4 and 5. In particular we note: or only few points got reassigned, we have to do two closest-centroid computations per point and iteration: First, we need to compute the • The overhead for a single query is very low and only a frac- closest centroid in the previous iteration and then the closest one in tion of a second. This also implies that we lose little in the current iteration. If we stored the closest points explicitly, we implementating iterative algorithms using driver functions could avoid half of the closest-centroid calculations. that run multiple SQL queries. We can store points in a table called points that has a coords attribute containing the points’ coordinates and has a second attribute • Given the previous points, the Greenplum database achieves for the current centroid id for the point. The iteration state stores perfect linear speedup in the example shown. the centroids’ positions in an array of points called centroids. MADlib provides a UDF closest point(a,b) that determines In the example, all data was essentially in the buffer cache, and the point in array a that is closest to b. Thus, we can make the disk I/O was not a limiting factor. This is quite typical in large point-to-centroid assignments explicit using the following SQL: parallel installations, since an analyst often runs a machine-learning UPDATE points algorithm several times with different parameters. Given massive SET centroid_id = closest_point(centroids, coords) amounts of main memory on each node in a well-provisioned cluster, much of the data should moreover remain in the cache. Finally, the Ideally, we would like to perform the point reassignment and the computational cost per row grows at least quadratically, and thus repositioning with a single pass over the data. Unfortunately, this will easily surpass I/O cost for complex models. As we compared cannot be expressed in standard SQL.1 various cluster sizes, numbers of independent variables and the re- Therefore, while we can reduce the number of closest-centroid spective execution times for previous versions of MADlib, the lesson calculations by one half, PostgreSQL processes queries one-by-one we learned is that even though we anticipate non-trivial overhead (and does not perform cross-statement optimization), so it will need by the DBMS, careful performance tuning—e.g., by making use to make two passes over the data per one k-means iteration. In of instruction-accurate profiling using Valgrind [27]—still makes general, the performance benefit of explicitly storing points depends significant differences: on the DBMS, the data, and the operating environment. The pattern of updating temporary state is made a bit more awk- 2 Our cluster is made up of four SuperMicro X8DTT-H server mod- ward in PostgreSQL due to its legacy of versioned storage. Post- ules, each equipped with one six-core Intel Xeon X5670 processor, greSQL performs an update by first inserting a new row and then clocked at 2.93 GHz. While hyperthreading is enabled, we only run marking the old row as invisible [37, Section 23.1.2]. As a result, a single Greenplum “segment” (query process) per physical core. Each machine has 24 GB of RAM, an LSI MegaRAID 2108 “Raid 1 While PostgreSQL and Greenplum provide an optional On a Chip” controller with six attached 360 GB solid-state drives, RETURNING clause for UPDATE commands, this returns only one and a Brocade 1020 converged network adapter. The operating row for each row affected by the UPDATE, and aggregates cannot system is Red Hat Enterprise Linux Server release 5.5 (Tikanga). be used within the RETURNING clause. Moreover, an UPDATE ... On that we are running Greenplum Database 4.2.0, compiled with RETURNING cannot be used as a subquery. gcc 4.4.2.
- 10. 200 5.1 Wisconsin Contributions: 6 Segments Convex Optimization execution time (s) 150 12 Segments The MADlib framework goes a long way toward making in- 18 Segments database analytic tools easier to deploy inside an RDBMS. Never- 100 24 Segments theless, to implement an algorithm within the MADlib framework, a developer must undertake several steps: they must specify a model, 50 select the algorithm used to implement that model, optimize the algorithm, test the model, and finally ensure that the resulting al- 0 gorithm and model are robust. This is a time consuming process 0 50 100 150 200 250 300 350 that creates code that must be tested and maintained for each data analysis technique. # independent variables A focus of the MADlib work at the University of Wisconsin has been to explore techniques to reduce this burden. Our approach is to expose a mathematical abstraction on top of MADlib that allows Figure 5: Linear-regression execution times using MADlib v0.3 a developer to specify a smaller amount of code, which we hope on Greenplum Database 4.2.0, 1 million rows 10 will lower the development time to add new techniques in many cases. We discuss the challenge that we faced implementing this abstraction. To demonstrate our ideas, we have implemented all • Version 0.1alpha is an implementation in C that computes the of the models shown in Table 2 within the single abstraction (built outer-vector products xi xT as a simple nested loop. i within MADlib) that we describe below. • Version 0.2.1beta introduced an implementation in C++ that used the Armadillo [35] linear algebra library as a frontend The Key Abstraction. An ideal abstraction would allow us to for LAPACK/BLAS. It turns out that this version was much decouple the specification of the model from the algorithm used slower for two reasons: The BLAS library used was the de- to solve the specification. Fortunately, there is a beautiful, power- fault one shipped with CentOS 5, which is built from the ful abstraction called convex optimization that has been developed untuned reference BLAS. Profiling and examining the critial for the last few decades [34, 8] that allows one to perform this de- code paths revealed that computing yT y for a row vector y coupling. More precisely, in convex optimization, we minimize a is about three to four times slower than computing xxT for convex function over a convex set. The archetype convex function a column vector x of the same dimension (and the MADlib is f (x) = x2 and is shown in Figure 6. Like all convex functions, implementation unfortunately used to do the former). Inter- any local minimum of f is a global minimum of f . Many differ- estingly, the same holds for Apple’s Accelerate framework on ent popular statistical models are defined by convex optimization Mac OS X, which Apple promises to be a tuned library. The problems, e.g., linear regression, support vector machines, logistic second reason for the speed disadvantage is runtime overhead regression, conditional random fields. Not every data analysis prob- in the first incarnation of the C++ abstraction layer (mostly lem is convex—notable exceptions include the a priori algorithm due to locking and calls into the DBMS backend). and graph mining algorithms—but many are convex. Table 2 lists models that we have implemented in our abstraction over MADlib, • Version 0.3 has an updated linear-regression implementation all of which are convex. (Note that the previous built-in MADlib that relies on the Eigen C++ linear algebra library and takes examples of linear and logistic regression fall into this category!) advantage of the fact that the matrix X T X is symmetric posi- In spite of the expressive power of convex optimization, even tive definite. Runtime overhead has been reduced, but some simple algorithms converge at provable rates to the true solution. calls into the database backend still need better caching. For intuition, examine the archetypical function f (x) = x2 shown in Figure 6. The graph of this function is like all convex sets: bowl Other noteworthy results during our performance studies included shaped. To minimize the function, we just need to get to the bottom that there are no measurable performance differences between Post- of the bowl. As a result, even greedy schemes that decrease the greSQL 9.1.1 (both in single and multi-user mode) and GP 4.1 in function at each step will converge to an optimal solution. One running the aggregate function on a single core. Moreover, while such popular greedy method is called a gradient method. The idea testing linear/logistic-regression execution times, single-core per- is to find the steepest descent direction. In 1-d, this direction is formance of even laptop CPUs (like the Core i5 540M) did not the opposite direction of the derivative; in higher dimensions, it’s differ much from today’s server CPUs (like the Xeon family). Typ- called the gradient of f . Using the gradient, we iteratively move ically the differences were even less than what the difference in toward a solution. This process can be described by the following clock speeds might have suggested, perhaps due to compiler and/or pseudocode: architecture issues that we have yet to unpack. x ← x − α · G(x) 5. UNIVERSITY RESEARCH AND MADLIB where G(x) is the gradient of f (x) and α is a positive number called An initial goal of the MADlib project was to engage closely the stepsize that goes to zero with more iterations. For example, it with academic researchers, and provide a platform and distribution suffices to set α = 1/k where k is the number of iterations. In the channel for their work. To that end, we report on two collaborations f (x) = x2 example, the gradient is the derivative, G(x) = 2x. Since over the past years. We first discuss a more recent collaboration x = 0 is the minimum value, we have that for x > 0, G(x) < 0 with researchers at the University of Wisconsin. We then report while for x < 0 we have that G(x) > 0. For a convex function, the on a collaboration with researchers at Florida and Berkeley, which gradient always tells us which direction to go to find a minimum co-evolved with MADlib and explored similar issues (as part of the value, and the process described above is guaranteed to converge at BayesStore project [42, 41]) but was actually integrated with the a known rate. One can provide a provable rate of convergence to MADlib infrastructure only recently. the minimum value, which is in sharp contrast to a typical greedy
- 11. # segments # variables # rows v0.3 v0.2.1beta v0.1alpha (million) (s) (s) (s) 6 10 10 4.447 9.501 1.337 6 20 10 4.688 11.60 1.874 6 40 10 6.843 17.96 3.828 6 80 10 13.28 52.94 12.98 6 160 10 35.66 181.4 51.20 6 320 10 186.2 683.8 333.4 12 10 10 2.115 4.756 0.9600 12 20 10 2.432 5.760 1.212 12 40 10 3.420 9.010 2.046 12 80 10 6.797 26.48 6.469 12 160 10 17.71 90.95 25.67 12 320 10 92.41 341.5 166.6 18 10 10 1.418 3.206 0.6197 18 20 10 1.648 3.805 1.003 18 40 10 2.335 5.994 1.183 18 80 10 4.461 17.73 4.314 18 160 10 11.90 60.58 17.14 18 320 10 61.66 227.7 111.4 24 10 10 1.197 2.383 0.3904 24 20 10 1.276 2.869 0.4769 24 40 10 1.698 4.475 1.151 24 80 10 3.363 13.35 3.263 24 160 10 8.840 45.48 13.10 24 320 10 46.18 171.7 84.59 Figure 4: Linear-regression execution times search. In our prototype implementation in MADlib, we picked up one such simple greedy algorithm, called stochastic (or sometimes, “incremental”) gradient descent (SGD) [33, 6], that goes back to the 1960s. SGD is an approximation of gradient methods that is useful when the convex function we are considering, f (x), has the form: N f (x) = fi (x) i=1 If each of the fi is convex, then so is f [8, pg. 38]. Notice that Figure 6: The Archetypical Convex Function f (x) = x2 . all problems in Table 2 are of this form: intuitively each of these models is finding some model (i.e., a vector w) that is scored on Application Objective many different training examples. SGD leverages the above form to Least Squares T u − y)2 construct a rough estimate of the gradient of f using the gradient of a (u,y)∈Ω (x Lasso [38] T u − y)2 + µ x single term: for example, the estimate if we select i is the gradient of (u,y)∈Ω (x 1 Logisitic Regression (u,y)∈Ω log(1 + exp(−yxt u)) fi (that we denote Gi (x)). The resulting algorithm is then described Classification (SVM) (u,y)∈Ω (1 − yx u)+ T as: 2 2 Recommendation (i, j)∈Ω (Li R j − Mi j ) + µ L, R T F x ← x − αN · Gi (x) (1) Labeling (CRF) [40] k j x j F j (yk , zk ) − log Z(zk ) This approximation is guaranteed to converge to an optimal solu- Table 2: Models currently Implemented in MADlib using the tion [26]. SGD-based approach. Using the MADlib framework. In our setting, each tuple in the input table for an analysis task encodes a single fi . We use the micro-programming interfaces of Sections 3.2 and 3.3 to perform macro programming layer helps us test for convergence (which is the mapping from the tuples to the vector representation that is used implemented with either a python combination or C driver.) Using in Eq. 1. Then, we observe Eq. 1 is simply an expression over each this approach, we were able to add in implementations of all the tuple (to compute Gi (x)) which is then averaged together. Instead of models in Table 2 in a matter of days. averaging a single number, we average a vector of numbers. Here, In an upcoming paper we report initial experiments showing that we use the macro-programming provided by MADlib to handle our SGD based approach achieves higher performance than prior all data access, spills to disk, parallelized scans, etc. Finally, the data mining tools for some datasets [13].
- 12. Statistical Methods POS NER ER grams and compute a similarity as the approximate matching score. Text Feature Extraction This functionality comes packaged with PostgreSQL. Viterbi Inference Once we have the features, the next step is to perform inference MCMC Inference Approximate String Matching on the model. We also implemented two types of statistical inference within the database: Viterbi (when we only want the most likely Table 3: Statistical Text Analysis Methods answer from the model) and MCMC (when we want the probabilities or confidence of an answer as well). Viterbi Inference: The Viterbi dynamic programming algorithm [14] is the popular algorithm to find the top-k most likely labelings of a 5.2 Florida/Berkeley Contributions: document for (linear chain) CRF models. Statistical Text Analytics Like any dynamic programming algorithm, the Viterbi algorithm The focus of the MADlib work at Florida and Berkeley has been is recursive. We experimented with two different implementations to integrate statistical text analytics into a DBMS. In many domains, of macro-coordination over time. First, we chose to implement it structured data and unstructured text are both important assets for using a combination of recursive SQL and window aggregate func- data analysis. The increasing use of text analysis in enterprise tions. We discussed this implementation at some length in earlier applications has increased the expectation of customers and the op- work [42]. Our initial recursive SQL implementation only runs over portunities for processing big data. The state-of-the-art text analysis PostgreSQL versions 8.4 and later; it does not run in Greenplum. tools are based on statistical models and algorithms [18, 12]. With Second, we chose to implement a Python UDF that uses iterations the goal to become a framework for statistical methods for data anal- to drive the recursion in Viterbi. This iterative implementation ysis at scale, it is important for MADlib to include basic statistical runs over both PostgreSQL and Greenplum. In Greenplum, Viterbi methods to implement text analysis tasks. can be run in parallel over different subsets of the document on a Basic text analysis tasks include part-of-speech (POS) tagging, multi-core machine. named entity extraction (NER), and entity resolution (ER) [12]. MCMC Inference: Markov chain Monte Carlo (MCMC) meth- Different statistical models and algorithms are implemented for ods are classical sampling algorithms that can be used to estimate each of these tasks with different runtime-accuracy tradeoffs. For probability distributions. We implemented two MCMC methods in example, an entity resolution task could be to find all mentions in MADlib: Gibbs sampling and Metropolis-Hastings (MCMC-MH). a text corpus that refer to a real-world entity X. Such a task can be The MCMC algorithms involve iterative procedures where the done efficiently by approximate string matching [25] techniques to current iteration depends on previous iterations. We used SQL win- find all mentions in text that approximately match the name of entity dow aggregates for macro-coordination in this case, to carry “state” X. However, such a method is not as accurate as the state-of-the-art across iterations to perform the Markov-chain process. This window collective entity resolution algorithms based on statistical models, function based implementation runs over PostgreSQL version 8.4 such as Conditional Random Fields (CRFs) [19]. and later. We discussed this implementation at some length in recent work [41]. We are currently working on integrating MCMC algo- rithms into Greenplum DBMS. We also plan to implement MCMC Pushing Statistical Text Analysis into MADlib. Based on using Python UDF macro-coordination analogous to Section 4.3, the MADlib framework, our group set out to implement statistical and compare the performance between the two implementations. methods in SQL to support various text analysis tasks. We use CRFs as the basic statistical model to perform more advanced text analysis. Similar to Hidden Markov Models (HMM) cite, CRFs are a leading Using the MADlib Framework. Because this work predated probabilistic model for solving many text analysis tasks, including the release of MADlib, it diverged from some of the macro-coordination POS, NER and ER [19]. To support sophisticated text analysis, we patterns of Section 3.1, and took advantage of PostgreSQL features implement four key methods: text feature extraction, most-likely that are less portable. These details require refactoring to fit into the inference over a CRF (Viterbi), MCMC inference, and approximate current MADlib design style, which we are finding manageable. In string matching (Table 3). addition to the work reported above, there are a host of other features Text Feature Extraction: Text feature extraction is a step in of both PostgreSQL and MADlib that are valuable for text analyt- most statistical text analysis methods, and it can be an expensive ics. Extension libraries for PostgreSQL and Greenplum provides operation. To achieve high quality, CRF methods often assign text processing features such as inverted indexes, trigram indexes hundreds of features to each token in the document. Examples of for approximate string matching, and array data types for model such features include: (1) dictionary features: does this token exist parameters. Existing modules in MADlib, such as Naive Bayes and in a provided dictionary? (2) regex features: does this token match Sparse/Dense Matrix manipulations, are building blocks to imple- a provided regular expression? (3) edge features: is the label of ment statistical text analysis methods. Leveraging this diverse set a token correlated with the label of a previous token? (4) word of tools and techniques that are already within the database allowed features: does this the token appear in the training data? and (5) us to build a sophisticated text analysis engine that has comparable position features: is this token the first or last in the token sequence? raw performance to off-the-shelf implementations, but runs natively The right combination of features depends on the application, and in a DBMS close to the data [42, 41]. so our support for feature extraction heavily leverages the micro- programming interface provided by MADlib. Approximate String Matching: A recurring primitive operation 6. RELATED WORK in text processing applications is the ability to match strings approx- The space of approaches to marry analytics with data is large imately. The technique we use is based on qgrams [16]. We used and seems to be growing rapidly. At a high-level there are two the trigram module in PostgreSQL to create and index 3-grams over approaches, (1) bring a statistical language to a data processing text. Given a string “Tim Tebow” we can create a 3-gram by using substrate, or (2) provide a framework to express statistical tech- a sliding window of 3 characters over this text string. Given two niques on top of a data processing substrate. Specifically, we strings we can compare the overlap of two sets of corresponding 3- split the approaches to achieve this marriage into two groups: (1)
- 13. top-down approaches begin with a high-level statistical program- Two recent efforts to provide Scala-based domain-specific lan- ming language, e.g., R or Matlab. The technical goal is to build a guages (DSLs) and parallel execution frameworks blur distinctions parallel data processing infrastructure that is able to support run- between frameworks and language-based approaches. Spark [44] is ning this high-level language in parallel. Examples of approach a Scala DSL targeted at Machine Learning, providing access to the include System ML from IBM [15], Revolution Analytics [31], fault-tolerant, main-memory resilient distributed datasets: which and SNOW [39]. (2) We call the second group framework-based are read-only collections of data that can be partitioned across a approaches whose goal is to provide a set of building blocks (in- cluster. ScalOps [43] provides a Scala DSL for Machine Learning dividual machine-learning algorithms) along with library support that is translated to Datalog, which is then optimized to run in par- for micro- and macro-programming to write the algorithms. Recent allel on the Hyracks infrastructure [7]. Given its roots in Datalog examples are Mahout [3], Graphlab [22], SciDB [36] and MADlib. and parallel relational algebra, ScalOps bears more similarity to MADlib than any of the other toolkits mentioned here. It would be Top-Down, Language-based Approaches. Within the ap- interesting to try and compile its DSL to the MADlib runtime. proaches of type (1), we again divide the space into imperative and declarative approaches. In the imperative approach, an analyst takes Other Data Processing Systems. There are a host of data the responsibility of expressing how to parallelize the data access, processing techniques that can be used to solve elements of the e.g., SNOW and Parallel R packages provide an MPI-interface along underlying data analysis problem. For example, Pregel [24] from with calls for data partitioning within the framework of R. In con- Google is designed for data analysis over graphs. In Pregel, the ab- trast in a declarative approach, the analyst declares their analysis straction is to write ones code using a graph-based abstraction: each problem and it is the responsibility of the system to achieve this goal function can be viewed as a node that sends and receives messages (in analogy with a standard RDBMS). Examples of the declarative to its neighbors in the graph. Pregel distributes the computation and approach include SystemML [15], which is an effort from IBM to provides fault tolerance. provide an R-like language to specify machine learning algorithms. In SystemML, these high-level tasks are then compiled down to 7. CONCLUSION AND DISCUSSION a Hadoop-based infrastructure. The essence of SystemML is its Scalable analytics are a clear priority for the research and in- compilation techniques: they view R as a declarative language, and dustrial communities. MADlib was designed to fill a vacuum for the goal of their system is to compile this language into an analog of scalable analytics in SQL DBMSs, and connect database research to the relational algebra. Similar approaches are taken in Revolution market needs. In our experience, a parallel DBMS provides a very Analytics [31] and Oracle’s new parallel R offering [28] in that these efficient and flexible dataflow substrate for implementing statistical approaches attempt to automatically parallelize code written in R. and analytic methods at scale. Standardized support for SQL exten- sions across DBMSs could be better—robust and portable support Framework-based Approaches. In framework-based approaches, for recursion, window aggregates and linear algebra packages would the goal is to provide the analyst with low-level primitives and co- simplify certain tasks. Since this is unlikely to occur across vendors ordination primitives built on top of a data processing substrate. in the short term, we believe that MADlib can evolve reasonable Typically, framework-based approaches offer a template whose goal workarounds at the library and “design pattern” level. is to automate the common aspects of deploying an analytic task. The popular alternative to a DBMS infrastructure today is Hadoop Framework-based approaches differ in what their data-processing MapReduce, which provides much lower-level programming APIs substrates offer. For example, MADlib is in in this category as it than SQL. We have not yet undertaken performance comparisons provides a library of functions over an RDBMS. The macro- and with Hadoop-based analytics projects like Mahout. Performance micro-programming described earlier are examples of design tem- comparisons between MADlib and Mahout today would likely boil plates. The goal of Apache Mahout [3] is to provide an open source down to (a) variations in algorithm implementations, and (b) well- machine learning library over Apache Hadoop. Currently, Mahout known (and likely temporary) tradeoffs between the current states provides a library of machine learning algorithms and a template to of DBMSs and Hadoop: C vs. Java, pipelining vs. checkpointing, extend this library. At a lower level, the DryadLINQ large vector etc. [30] None of these variations and tradeoffs seem endemic, and library provides the necessary data types to build analysis tasks (e.g., they may well converge over time. Moreover, from a marketplace vectors) that are commonly used by machine learning techniques. perspective, such comparisons are not urgent: many users deploy SciDB advocates a completely rewritten DBMS engine for numeri- both platforms and desire analytics libraries for each. So a rea- cal computation, arguing that RDBMSs have “the wrong data model, sonable strategy for the community is to foster analytics work in the wrong operators, and are missing required capabilities” [36]. both SQL and Hadoop environments, and explore new architectures MADlib may be seen as a partial refutation of this claim. Indeed, (GraphLab, SciDB, ScalOps, etc.) at the same time. the assertion in [36] that “RDBMSs, such as GreenPlum [sic]...must MADlib has room for growth in multiple dimensions. The library convert a table to an array inside user-defined functions” is incor- infrastructure itself is still in beta, and has room to mature. There rect: the MADlib C++ library hides representation from the UDF is room for enhancements in its core treatment of mathematical developer and only pays for copies when modifying immutable kernels (e.g. linear algebra over both sparse and dense matrices) structures. GraphLab is a framework to simplify the design of pro- especially in out-of-core settings. And of course there will always gramming parallel machine learning tasks. The core computational be an appetite for additional statistical models and algorithmic meth- abstraction is a graph of processing nodes that allows one to define ods, both textbook techniques and cutting-edge research. Finally, asynchronous updates. Graphlab initially focused on providing a there is the challenge of porting MADlib to DBMSs other than framework for easy access to multicore parallelism [22]. However, PostgreSQL and Greenplum. As discussed in Section 3, MADlib’s the core computational abstraction is general and is now also de- macro-coordination logic is written in largely standard Python and ployed in a cluster setting. Another popular toolkit in this space SQL, but its finer grained “micro-programming” layer exploits is Vowpal Wabbit [20] that is extensively used in the academic proprietary DBMS extension interfaces. Porting MADlib across machine learning community and provides high performance. DBMSs is a mechanical but non-trivial software development effort. At the macro level, porting will involve the package infrastructure
- 14. (e.g. a cross-platform installer) and software engineering frame- [19] J. D. Lafferty, A. McCallum, and F. C. N. Pereira. Conditional random work (e.g. testing scripts for additional database engines). At the fields: Probabilistic models for segmenting and labeling sequence data. micro-programming logic level, inner loops of various methods will In ICML, pages 282–289, 2001. need to be revisited (particularly user-defined functions). Finally, [20] J. Langford. http://hunch.net/ vw/. since Greenplum retains most of the extension interfaces exposed [21] S. Lloyd. Least squares quantization in pcm. IEEE Transactions on Information Theory, 28(2):129–137, Mar. 1982. Technical Report by PostgreSQL, the current MADlib portability interfaces (e.g., the appeared much earlier in: Bell Telephone Laboratories Paper (1957). C++ abstraction of Section 3.3) will likely require revisions when [22] Y. Low, J. Gonzalez, A. Kyrola, et al. Graphlab: A new framework for porting to a system without PostgreSQL roots. parallel machine learning. In P. Grnwald and P. Spirtes, editors, UAI, Compared to most highly scalable analytics packages today, pages 340–349, 2010. MADlib v0.3 provides a relatively large number of widely-used [23] M. Mahajan, P. Nimbhorkar, and K. Varadarajan. The planar k-means statistical and analytic methods. It is still in its early stages of de- problem is NP-hard. Theoretical Computer Science, June 2010. In velopment, but is already in use both at research universities and at Press. customer sites. As the current software matures, we hope to foster [24] G. Malewicz, M. H. Austern, A. J. Bik, et al. Pregel: a system for large-scale graph processing. In SIGMOD, pages 135–146, 2010. more partnerships with academic institutions, database vendors, and [25] G. Navarro. A guided tour to approximate string matching. ACM customers. In doing so, we plan to pursue additional analytic meth- Comput. Surv., 33(1):31–88, Mar. 2001. ods prioritized by both research “push” and customer “pull”. We [26] A. Nedic and D. P. Bertsekas. Convergence rate of incremental also look forward to ports across DBMSs. subgradient algorithms. In S. Uryasev and P. M. Pardalos, editors, Stochastic Optimization: Algorithms and Applications, pages 263–304. Kluwer Academic Publishers, 2000. 8. ACKNOWLEDGMENTS [27] N. Nethercote and J. Seward. Valgrind: a framework for heavyweight In addition to the authors, the following people have contributed dynamic binary instrumentation. In PLDI, pages 89–100, 2007. to MADlib and its development: Jianwang Ao, Joel Cardoza, Brian [28] Oracle R Enterprise. Dolan, Hlya Emir-Farinas, Christan Grant, Hitoshi Harada, Steven http://www.oracle.com/technetwork/database/options/advanced- Hillion, Luke Lonergan, Gavin Sherry, Noelle Sio, Kelvin So, Gavin analytics/r-enterprise/index.html. Yang, Ren Yi, Jin Yu, Kazi Zaman, Huanming Zhang. [29] C. Ordonez. Integrating k-means clustering with a relational dbms using sql. IEEE Trans. Knowl. Data Eng., 18(2):188–201, 2006. [30] A. Pavlo, E. Paulson, A. Rasin, et al. A comparison of approaches to large-scale data analysis. In SIGMOD, pages 165–178. ACM, 2009. 9. REFERENCES [31] Revloution Analytics. http://www.revolutionanalytics.com/. [1] D. Aloise, A. Deshpande, P. Hansen, et al. NP-hardness of Euclidean [32] B. Ripley. The r project in statistical computing. MSOR Connections, sum-of-squares clustering. Machine Learning, 75:245–248, 2009. 1(1):23–25, 2001. [2] E. Anderson, Z. Bai, C. Bischof, et al. LAPACK Users’ Guide. Society [33] H. Robbins and S. Monro. A stochastic approximation method. Ann. for Industrial and Applied Mathematics, Philadelphia, PA, third Math. Statistics, 22, 1951. edition, 1999. [34] R. T. Rockafellar. Convex Analysis (Princeton Landmarks in [3] Apache Mahout. http://mahout.apache.org/. Mathematics and Physics). Princeton University Press, 1996. [4] D. Arthur, B. Manthey, and H. R¨ glin. k-means has polynomial o [35] C. Sanderson. Armadillo: An open source C++ linear algebra library smoothed complexity. arXiv, 2009. for fast prototyping and computationally intensive experiments. [5] D. Arthur and S. Vassilvitskii. k-means++: the advantages of careful Technical report, NICTA, 2010. seeding. In Proceedings of the 18th Annual ACM-SIAM Symposium on [36] M. Stonebraker, P. Brown, A. Poliakov, et al. The architecture of scidb. Discrete Algorithms (SODA’07), pages 1027–1035, 2007. In Scientific and Statistical Database Management, pages 1–16. [6] D. P. Bertsekas. Nonlinear Programming. Athena Scientific, Belmont, Springer, 2011. MA, 2nd edition, 1999. [37] The PostgreSQL Global Development Group. PostgreSQL 9.1.3 [7] V. Borkar, M. Carey, R. Grover, et al. Hyracks: A flexible and Documentation, 2011. extensible foundation for data-intensive computing. In ICDE, pages [38] R. Tibshirani. Regression shrinkage and selection via the lasso. 1151–1162, 2011. Journal of the Royal Statistical Society, Series B, 58:267–288, 1994. [8] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge [39] L. Tierney, A. J. Rossini, and N. Li. Snow: a parallel computing University Press, New York, NY, USA, 2004. framework for the r system. Int. J. Parallel Program., 37(1):78–90, [9] J. Choi, J. Demmel, I. Dhillon, et al. ScaLAPACK: A portable linear Feb. 2009. algebra library for distributed memory computers – design issues and [40] H. M. Wallach. Conditional random fields: An introduction. Technical performance. Computer Physics Communications, 97(1):1–15, 1996. report, Dept. of CIS, Univ. of Pennsylvania, 2004. [10] C.-T. Chu, S. K. Kim, Y.-A. Lin, et al. Map-reduce for machine [41] D. Wang, M. Franklin, M. Garofalakis, et al. Hybrid in-database learning on multicore. In NIPS, pages 281–288, 2006. inference for declarative information extraction. In SIGMOD, 2011. [11] J. Cohen, B. Dolan, M. Dunlap, et al. Mad skills: New analysis [42] D. Z. Wang, M. J. Franklin, M. N. Garofalakis, et al. Querying practices for big data. PVLDB, 2(2), 2009. probabilistic information extraction. PVLDB, 3(1), 2010. [12] R. Feldman and J. Sanger. The text mining handbook: advanced [43] M. Weimer, T. Condie, R. Ramakrishnan, et al. Machine learning in approaches in analyzing unstructured data. Cambridge Univ Pr, 2007. scalops, a higher order cloud computing language. In NIPS 2011 [13] X. Feng, A. Kumar, B. Recht, et al. Towards a unified architecture for Workshop on parallel and large-scale machine learning (BigLearn), in-rdbms analytics. In SIGMOD, 2012. volume 9, pages 389–396, 2011. [14] G. Forney Jr. The viterbi algorithm. Proceedings of the IEEE, [44] M. Zaharia, M. Chowdhury, T. Das, et al. Resilient distributed datasets: 61(3):268–278, 1973. A fault-tolerant abstraction for in-memory cluster computing. [15] A. Ghoting, R. Krishnamurthy, E. Pednault, et al. SystemML: Technical Report UCB/EECS-2011-82, EECS Department, University Declarative Machine Learning on MapReduce. In ICDE, 2011. of California, Berkeley, Jul 2011. [16] L. Gravano, P. Ipeirotis, H. Jagadish, et al. Using q-grams in a dbms [45] M. Zinkevich, M. Weimer, A. Smola, et al. Parallelized stochastic for approximate string processing. IEEE Data Engineering Bulletin, gradient descent. In NIPS, 2010. 24(4):28–34, 2001. [17] G. Guennebaud, B. Jacob, et al. Eigen v3. http://eigen.tuxfamily.org, 2010. [18] D. Jurafsky and M. J. H. Speech and Language Processing. Pearson Prentice Hall, 2008.