Column Oriented Databases

Public
Column Oriented Databases
Arundhati Kanungo, Developer Associate, SAP Labs India Pvt. Ltd.
June 25, 2017

© 2016 SAP SE or an SAP affiliate company. All rights reserved. 2Public
Agenda
• Definition of Column Oriented Databases
• History of Column Oriented Databases
• Working of Column Oriented Databases
• Top 3 Market Selling Column Oriented Databases
• Advantages of Column Oriented Databases
• Disadvantages of Column Oriented Databases
• Row vs Column Oriented Databases
• Industries to benefit from Column Oriented Databases
• Conclusion
• Future Work

Column Oriented Databases -
Definition

• Database management systems that store table data as columns of data rather than as rows
of data
• Have advantages for data warehousing, customer relationship management (CRM), and
library card catalogs, and other ad hoc enquiry where aggregates are computed over large
numbers of similar data items
• Refer to both the ease of expressing a column oriented structure and the focus on
optimizations for column oriented workloads
• Common examples include Greenplum Database, Calpont InfiniDB, Accumulo, Teradata,
SenSage, EXASOL, MonetDB, RCFile, Sqrrl, etc.

History

• 1969
• With focus on information-retrieval in biology, the first application of a column oriented database storage
system called TAXIR was developed.
• 1976
• Statistics Canada implemented the RAPID system for processing and retrieval of the Canadian Census of
Population and Housing as well as several other statistical applications.
• 1977 - 1990
• The RAPID system was shared with other statistical organizations throughout the world for widespread use in
the 1980s. It was used by Statistics Canada until the 1990s.
• 1993
• KDB was launched as the first commercially available column oriented database.
• 1995
• Sybase IQ gained prominence as the second column oriented database.
• 2005
• The traditional column oriented databases have changed rapidly since 2005 with many open source and
commercial implementations.

Working

EmpId LastName FirstName Salary
10 Smith Joe 40000
12 Jones Mary 50000
11 Johnson Cathy 44000
22 Jones Bob 55000
Ensuring minimum seeks, a column oriented database serializes all values of a column
together, then the values of the next column, and so on. For our example table, the data would
be stored as below.
10:001,12:002,11:003,22:004;Smith:001,Jones:002,Johnson:003,Jones:004;Joe:001,Mary:002,
Cathy:003,Bob:004;40000:001,50000:002,44000:003,55000:004;
A column-oriented store "is really just" a row-store with an index on every column.

Top 3 Column Oriented
Databases

• Sybase
• Aims to deliver high end performance for critical analytics, business intelligence and data warehousing solutions
leveraging highly optimized server dedicated for analytics.
• Is column oriented with grid-based architecture, patented data compression, and advanced query optimizer, and
henceforth, delivers high performance, flexibility, and economy in challenging reporting and analytics
environments.
• Infobright
• Combination of a column oriented database with their Knowledge Grid architecture delivers a self-managed,
scalable, high performance analytics query platform.
• Industry-leading data compression (10:1 up to 40:1) considerably lowers storage requirements and expensive
hardware infrastructures.
• Vertica
• Purpose-built platform to enable data values having high performance real time analytics needs.
• Highly scalable with a seamless integration ecosystem leveraging capabilities like data loading, queries, columnar
storage, MPP architecture, and data compression features.

Column Oriented Databases –
Advantages

• Scalability and fast data loading for Big Data
• Accessible by many third-party BI analytic tools
• Simple systems administration
• High performance on aggregation queries (like COUNT, SUM, AVG, MIN, MAX)
• Highly efficient data compression and/or partitioning

Disadvantages

• Record updates and deletes reduce storage efficiency
• Effective partitioning/indexing schemes can be difficult to design
• Transactions are to be avoided or just not supported
• Queries with table joins can reduce high performance

Row vs Column Oriented
Databases

Comparative Analysis
Row Oriented Databases
• Relatively less efficient in terms of aggregate computation,
multiple updates, multiple select, single insert
• Well-suited for OLTP (On-Line Transaction Processing) like
workloads
Column Oriented Databases
• Highly efficient in terms of aggregate computation, multiple
updates, multiple select, single insert
• Well-suited for OLAP (On-Line Analytical Processing) like
workloads

Benefited Industries

• Telecommunications: Helps in improving response time to the customer by reducing input
and output
• Financial Services: Supports high performance, millisecond response time to queries
required for inbound market
• Retail: Reads the data only referenced in question driving higher performance and lowering
processing costs compared to reading all the columns in the table

Column oriented database is a key technology that delivers high business value by helping
enterprises adapt their information infrastructure to the evolving demands for timely, reliable
intelligence to run the business. In addition, it has far-reaching implications for the design of
systems, and offers major cost savings affecting higher power and cooling requirements.

Columnar databases can be very helpful in big data project. Big data, today is one of the
biggest problem ever faced. When we have volume and variety of random real-time data, we
might want to use a columnar database to exploit its flexibility, performance and scalability. Till
date, HBase is the only column oriented database that is used with big data. I look forward to
carry out a comparative analysis of HBase performance and other columnar database
performance when fed with big data.

References
1. Stonebraker et al., “C-Store: A Column-oriented DBMS”, Proceedings of the 31st VLDB Conference, Trondheim, Norway, 2005
2. Copeland, George P. and Khoshafian, Setrag N., “A Decomposition Storage Model”, SIGMOD '85, 1985
3. Daniel Abadi and Samuel Madden, "Debunking Another Myth: Column-Stores vs. Vertical Partitioning", The Database Column, 31 July 2008
4. Stavros Harizopoulos, Daniel Abadi and Peter Boncz, "Column-Oriented Database Systems", VLDB 2009 Tutorial, p. 5
5. Pat & Betty O’Neil, Xuedong Chen and Stephen Revilak, “The Star Schema Benchmark and Augmented Fact Table Indexing”, TPC Technology
Conference 8/24/09
6. D. J. Abadi, S. R. Madden, N. Hachem, “Column-stores vs. Row-stores: How Different are They Really?”, in: SIGMOD’08, 2008, pp. 967–980.
7. N. Bruno, “Teaching an old elephant new tricks”, in: CIDR ’09, 2009.
8. Daniel Lemire, Owen Kaser, Kamel Aouiche, "Sorting Improves Word-Aligned Bitmap Indexes", Data & Knowledge Engineering, Volume 69,
Issue 1 (2010), pp. 3-28.
9. Daniel Lemire and Owen Kaser, “Reordering Columns for Smaller Indexes”, Information Sciences 181 (12), 2011
10. Slezak et al., Brighthouse: “An Analytic Data Warehouse for Ad hoc Queries”, Proceedings of the 34th VLDB Conference, Auckland, New
Zealand, 2008
11. Estabrook, Brill, “The Theory of the TAXIR Accessioned”, Mathematical Biosciences, Volume 5, Issues 3–4, Elsevier B.V.
12. Turner, Hammond, Cotton, “A DBMS for Large Statistical Databases”, Proceedings of VLDB 1979, Rio de Janeiro, Brazil

Column Oriented Databases

More Related Content

What's hot (20)

Similar to Column Oriented Databases (20)

Recently uploaded (20)

Column Oriented Databases