Building an analytical platform

Building an Analytical Platform

I was recently asked to build an analyti- 3. Provide a way to automate the An incidental but equally useful
cal platform for a project. But what is an running of the statistical data models, consequence of using a column-store
analytical platform? The client, a retailer, once developed, so that they can be database such as SAP Sybase IQ is that
described it as a database where it could run without engaging the statistical there is no advantage in creating a star
store data and as a front end where it development resources. schema as a data model. Instead, hold-
could do statistical work. This work Of course, time was of the essence ing all the data in one large wide table is
would range from simple means and and costs had to be as low as possible – -
standard deviations through to more but we’ve come to expect that with ing each column with a key means that
complex predictive analytics that could the underlying storage of data is a star
be used, for example, to analyze past Step 1: The database schema. Creating a star schema in a
performance of a customer to assess the Our chosen solution for the database column-store database rather than a
likelihood that the customer will exhibit a was an SAP® Sybase® IQ database, a large single table would mean incurring
future behavior. Or it might involve using technology our client was already famil- unnecessary additional join and process-
models to classify customers into groups iar with. SAP Sybase IQ is a column-store ing overhead.
and ultimately to bring the two processes database. This means that instead of As a result of choosing SAP
together into an area known as decision storing all the data in its rows, as many Sybase IQ’s column-store database
models. The customer had also come up other databases do, the data is organized we are able to have a data model that
with an innovative way to resource the on disk by the columns. For example if a consists of a number of simple single
table data sets (one table for each
work placements to master’s degree have the text of each country (for exam-
students studying statistics at the local ple, “United Kingdom”) stored many that is quick to load and to query.
university and arranged for them to work times. In a column-store database the It should be noted that this type of
with the customer insight team to text is stored only once and given a
describe and develop the advanced unique ID. This is repeated for each online transaction processing (OLTP)
models. All the customer needed was a column and therefore the “row” of data applications because of the cost of doing
platform to work with. consists of a list of IDs linked to the data small inserts and updates. However, this
From a systems architecture and held for each column. is not relevant for this particular
development perspective, we could
describe the requirements in three rela- reporting and analytical databases. The solution can be deployed only on
tively simple statements: a Linux platform. We use Linux for three
1. Build a database with a very simple reasons. First, RStudio Server Edition is
data model that could be easily used. In our example, “United Kingdom” not yet available for Microsoft Windows.
loaded, that was capable of support- would occupy 14 bytes, while the ID Second, precompiled packages for all
ing high-performance queries, and might occupy only 1 byte – reducing the elements of the solution on Linux reduce
that did not consume a massive storage for that one value in that one
amount of disk space. It would also column by a ratio of 14:1 – and this environments are normally cheaper than
ideally be capable of being placed in Windows environments due to the cost
the cloud. the data. Furthermore, because there is of the operating system license. We
2. Create a Web-based interface that less data on the disk, the time taken to chose CentOS because it is a Red Hat
would allow users to securely log on, read the data from disk and to process derivative that is free.
to write statistical programs that One additional advantage of this solu-
could use the database as a source of which massively speeds up the queries tion for some organizations is the ability
data, and to output reports and graph- too. Finally, each column is already to deploy it in the cloud. Since the solu-
ics and well as to populate other indexed, which again helps the overall -
tables (for example, target lists) as a query speed. ered, and since all querying is done via a
result of statistical models. Web interface, it is possible to use any

SAP White Paper – Building an Analytical Platform 3

colocation or cloud-based hosting your environment, but these are well At this point data has to be loaded and
provider. Colocation or cloud deploy- documented on the source Web sites the statisticians can get to work.
and in general automatically download if Obviously this is more time consuming
systems management overhead, and you are using a tool such as yum. than the build, and over the days and
access for both data delivery and data The next step was to get access to the weeks the analysts created their models
access. The system requires SSH access data held in our SAP Sybase IQ server. and produced the results.
for management; FTP, SFTP, or SCP for This proved to also be very straightfor- For this exercise we used our in-house
ward. There is a SAP Sybase white paper extract, transform, and load (ETL) tool to
port open. The RStudio server uses the create a repeatable data extraction and
server login accounts for security but load process, but it would have been
can also be tied to existing LDAP describes the process that can be simply possible to use any of a wide range of
infrastructure. stated as: tools that are available for this process
Step 2: Statistical tools and Web Install the R JDBC package Step 3: Automatically running the
interface Set up a JDBC connection statistical models
There are a number of statistical tools Establish your connection Eventually a number of models for
in the market. Most are very expensive, Query the table analyzing the data had been created and
prohibitively so in this case, and the We now have an R object that contains we were ready to move into a production
associated skills are hard to come by data sourced from SAP Sybase IQ that environment. We automated the load of
and expensive. However, since 1993 an we can work with. And what is amazing is the data into the agreed single-table
open-source programming language that it took me less than half a day to structure and wanted to also run the
called R (www.r-project.org) for statisti- build the platform from scratch. data models.
cal computing and graphics has been
under development. It is now widely used
among statisticians for developing statis-
tical software and data analysis, is used
by many universities, and is predicted to Analytical Platform Server
become the most widely used statistical
package by 2015. The R project provides
R Studio
a command line and graphical interface R
Server Edition
as well as a large open-source library of
useful routines (http:/ /cran.r-project. R/JDBC
org) and it is available as packaged soft- Connection
ware for most platforms including Linux.
In addition, a second open-source proj- SAP
ect called RStudio (http:/ /rstudio.org) Sybase
provides a single integrated development R/JDBC Connection
IQ
environment for R and can be deployed (S) FTP/SCP
on a local machine or as a Web-based File Delivery Write to
service using the server’s security model. Database Any Network
In this case, we implemented the server Connected Computer
Read File ETL with a Browser Accessing
edition in order to make the entire envi- the R Studio Server Edition
Engine
ronment Web based.
So in two simple steps (download and
install R, followed by download and
install RStudio) we can install a full CentOS
Web-based statistical environment. Note
©2012 Data Management & Warehousing

packages may be required depending on

SAP Sybase IQ has the functionality ConCluSionS ABout the Author

David Walker has been involved with business
These C++ programs “talk” to a process Business intelligence requirements are
intelligence and data warehousing for over
known as Rserve, which in turn executes changing and business users are moving
the R program and returns the results more and more from historical reporting
to SAP Sybase IQ. This allows R func- into predictive analytics in an attempt to
tions to be embedded directly into SAP get both a better and deeper under- Data Management & Warehousing (http://
datamgmt.com) in 1995.
Sybase IQ SQL commands. While setting standing of their data.
this up requires a little more program- Traditionally, building an analytical David and his team have worked around
ming experience, it does mean that all platform has required an expensive infra- the world on projects designed to deliver
processing can be done within SAP structure and a considerable amount of
Sybase IQ. time for setup and deployment. converting data into information and by
Conversely, it is possible to run R from By combining the high performance,
exploit that information.
the command line and call the program low footprint of SAP Sybase IQ with the
that in turn uses the RJDBC connection open-source R and RStudio statistical David’s project work has given him experi-
to read and write data to the database. packages, it is possible to quickly deploy ence in a wide variety of industries including
Having a choice of methods is very an analytical platform in the cloud for -
facturing, transportation, and public sector
helpful as it means that it can be inte- which there are readily available skills.
as well as a broad and deep knowledge of
grated with the ETL environment in the This infrastructure can be used both business intelligence and data warehousing
most appropriate way. If the ETL tool for rapid prototyping on analytical technologies.
models and for running completed
function (UDF) route is the most attrac- models on new data sets to deliver
tive. However, if the ETL tool supports greater insight into the data.
host callouts (as ours does) then running
R programs from a command line callout
is quicker than developing the UDF.

SAP White Paper – Building an Analytical Platform 5

www.sap.com/contactsap

12/08 ©2012 SAP AG. All rights reserved.
SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign,
SAP BusinessObjects Explorer, StreamWork, SAP HANA, and
other SAP products and services mentioned herein as well as
their respective logos are trademarks or registered trademarks
of SAP AG in Germany and other countries.

Business Objects and the Business Objects logo, BusinessObjects,
Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other
Business Objects products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of Business
Objects Software Ltd. Business Objects is an SAP company.

Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and
other Sybase products and services mentioned herein as well as their
respective logos are trademarks or registered trademarks of Sybase Inc.
Sybase is an SAP company.

Crossgate, m@gic EDDY, B2B 360°, and B2B 360° Services are registered
trademarks of Crossgate AG in Germany and other countries. Crossgate
is an SAP company.

All other product and service names mentioned are the trademarks of
their respective companies. Data contained in this document serves

These materials are subject to change without notice. These materials

for informational purposes only, without representation or warranty of
any kind, and SAP Group shall not be liable for errors or omissions with
respect to the materials. The only warranties for SAP Group products and
services are those that are set forth in the express warranty statements
accompanying such products and services, if any. Nothing herein should
be construed as constituting an additional warranty.

Building an analytical platform

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Building an analytical platform (20)

More from David Walker (18)

Recently uploaded (20)

Building an analytical platform