ACG_Rackspace.pdf

Analytical Compute Grid
(ACG)

Elastic “Big Data” Infrastructure
Rackspace® Private Cloud powered by OpenStack® Use Case

by Natasha Gajic
October 17, 2012

Rackspace’s EBI Environment

Current Environment “Big Data” Problem
 Windows and Linux  Cost of purchasing
operating systems additional licenses
 Oracle and Microsoft  Time required to set up
databases solutions new hardware
 Microsoft and Oracle  Increased demand for DBA
replication technology resources
 SSIS  System performance

 Informatica  System scalability

 Dedicated servers  Capacity

 Rapid data set growth

RACKSPACE® HOSTING | WWW.RACKSPACE.COM
2

Analytical Compute Grid (ACG) Features

• Host ever growing set of data
• Quick data collection and retrieval
• Rapid scalability
• Ease of maintenance
• Provide standard data access API

3

Analytical Compute Grid (ACG) Features
• Ability to provide variety of storage types:
• Columnar
• Relational
• HDFS
• Enable users to select optimal storage
type for information collected
• Leverage Rackspace® Private Cloud
powered by OpenStack® and open
source technology

4

Analytical Compute Grid (ACG) Quality Attributes

5

ACG on Rackspace® Private
Cloud powered by OpenStack®
High Level Architecture

6

ACG on Rackspace® Private Cloud powered by OpenStack®

8 Hypervisor Servers each:
Dual Socket Six Core 2.4GHz Processors
96GB RAM
Terabytes of Storage

*The environment will grow significantly next year

7

Image

8

Database Engine Selection

Columnar Cassandra
Relational PostgreSQL
HDFS Hadoop

9

Node

10

Node

11

Node

12

Node

13

Controller

14

Controller

15

Controller

16

API

17

Indexing Structure

18

Indexing Structure
• ACG Indexing Structure:
• Resides on a set of Rackspace® Private Cloud powered by
OpenStack® instances

19

Indexing Structure
• It is a set of pointers ultimately addressing database entities

20

Indexing Structure
• ACG Controller manages Indexing Structure

21

Indexing Structure
• ACG Controller manages Indexing Structure
• Dynamically expands vertically and horizontally to address a
growing data set

22

Indexing Structure
• ACG Indexing Structure Enables:
• Distribution of data bases across many instances

23

Indexing Structure
• Splitting large data sets across many instances

24

Indexing Structure
• Parallelization of large data set queries

25

Indexing Structure
• Deploying data stores with optimal configuration, minimizing
maintenance

26

Indexing Structure
• Deploying data stores with optimal configuration, minimizing
maintenance
• Accessing data residing in
variety of storage types via
uniform interface

27

Sorter & Aggregator
• ACG Sorter & Aggregator Enables:
• Joining the results from multiple ACG nodes

28

Sorter & Aggregator
• Result sorting and aggregation

29

Sorter & Aggregator
• Result sorting and aggregation
• Together with temporary segment it will support
joining heterogeneous data sets

30

Quality Attributes

31

Quality Attributes - Performance

Rackspace® Private Cloud ACG
powered by OpenStack®

32


Creates ACG node in 30 seconds

33


Creates ACG node in 30 seconds
Creates ACG nodes concurrently

34


powered by OpenStack® Controlled data set size resulting
Creates ACG node in 30 seconds in:

35


Creates ACG node in 30 seconds in: Quick data distribution

36


Creates ACG nodes concurrently Query parallelization

37


Creates ACG nodes concurrently Query parallelization
Fast data retrieval

38

Quality Attributes – Scalability


39


Quick and concurrent ACG node
creation

40


creation
Ability to re-size existing nodes

41


creation
Ability to re-size existing nodes
Ability to remove nodes

42


powered by OpenStack® Indexing structure and controlled
Quick and concurrent ACG node data set size allow ACG to
creation stabilize quickly as it expands or
Ability to re-size existing nodes contracts
Ability to remove nodes

43

Quality Attributes – Availability


44


Rapidly replace failed ACG nodes

45


powered by OpenStack® Deploys data store native
Rapidly replace failed ACG nodes availability mechanisms
(replication, data distribution…)

46

Quality Attributes – Maintainability


47


Adding ACG nodes expands:

48


Storage capacity

49


Storage capacity
CPU power

50


Storage capacity
CPU power
RAM

51


Storage capacity
CPU power
RAM
No DBA or system administrators
activity required

52


powered by OpenStack® Controlled data set size enables:
Storage capacity
CPU power
RAM
activity required

53


Adding ACG nodes expands: Optimal and stable data store
Storage capacity configuration
CPU power
RAM
activity required

54


CPU power Reducing demand for managing
RAM data store objects
activity required

55


CPU power Reducing demand for managing
RAM data store objects
No DBA or system administrators Stable query execution plans
activity required

56

Quality Attributes – Flexibility

ACG

57


ACG
Variety of storage types:

58


ACG
Columnar – Cassandra : time series data

59


ACG
Relational – PostgreSQL : relational data

60


ACG
HDFS – Hadoop : un-structured data

61


ACG
Variety of storage types: Ability to select optimal storage
Columnar – Cassandra : time series data type for individual use case
HDFS – Hadoop : un-structured data

62

Quality Attributes – Usability

ACG

63


ACG
Standard interfaces:

64


ACG
SQL language

65


ACG
SQL language
JDBC API

66


ACG
SQL language
JDBC API
Data store native calls

67


ACG
Standard interfaces: Native bulk loader utility
SQL language
JDBC API
Data store native calls

68


ACG
Standard interfaces: Native bulk loader utility
SQL language
JDBC API ACG will support joining
Data store native calls heterogeneous data sets

69

Rackspace Use Case

70

Rackspace Use Case
• Subject:
• Complex availability calculation sourcing 3
months of monitoring data and creating 1 billion
records in initial calculation

71

Rackspace Use Case
• Environment 1
• Data Warehouse Microsoft SQL server database
• SSIS data loading
• SQL server with 24 CPUs and 250GB RAM was
dedicated to the initial calculation
• SQL server stored procedure performed the
calculation
• Source and result are stored in traditional data
warehouse structure

72

Rackspace Use Case
• Environment 2
• In 30 seconds, ACG Node Manager instantiated
new columnar data store consisting of 4
Cassandra nodes, and registered it in ACG
Indexing Structure
• Each ACG node has 2CPUs and 8GB RAM
• Informatica data loading
• Calculation developed in Java
• Source and result are stored in columnar
structure suitable for time series data
73

Rackspace Use Case - Result
• Calculation Duration
•Microsoft SQL Server lasted 5 days
•ACG calculation completed in 3.5 hours
• Storage Size
• Microsoft SQL server 500GB
•ACG 20 GB
• Complexity of the calculation
•Columnar data store is optimal for time series data.
Sourcing from columnar data store resulted in relatively
simple Java calculation process comparing to SQL
server stored procedure
74

Rackspace Use Case - Conclusion
• Selecting optimal data store for use case resulted in:
• Substantial performance improvement

75

• Reduced storage demand

76

•Simplified processes

77

•Ability to process terabytes of data per day close to
real-time and on-demand

78

•Ability to process terabytes of data per day close to
•Improved trending and reporting:
• enhances support capabilities
• improved Rackspace customer experience

79

• Simplified processes
• Ability to process terabytes of data per day close to
• Improved trending and reporting:
• enhances support capabilities
• improved Rackspace customer experience
• Significant cost reduction
80

ACG_Rackspace.pdf

More Related Content

What's hot (19)

Viewers also liked (9)

Similar to ACG_Rackspace.pdf (20)

More from OpenStack Foundation (20)

ACG_Rackspace.pdf