データベースMeetup vol2
3 likes525 views
HTAP (Hybrid Transactional/Analytical Processing) databases allow both online transaction processing (OLTP) and online analytical processing (OLAP) to run simultaneously on the same data for real-time analytics. SAP HANA is an in-memory HTAP database that utilizes techniques like columnar storage and SIMD instructions to provide fast transactional processing and analytical queries on the same data. SAP HANA's lightweight locking mechanism and use of hardware transactional memory further improves the performance of OLTP workloads.
![7PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ
( ) SIGMOD09 (‘09) Hasso Plattner
SIGMOD09 (‘09) Hasso Plattner ’06 Hasso SAP (Hasso Plattner
Institute) OLTP OLAP Column Store
SAP HANA SAP HANA
( )
SAP HANA
http://www.sigmod09.org/images/sigmod1ktp-plattner.pdf
“A Common Database Approach for OLTP and OLAP
Using an In-Memory Column Database”
https://en.wikipedia.org/wiki/Hasso_Plattner
A Common Database Approach for OLTP and OLAP Using
an In-Memory Column Database
Hasso Plattner
Hasso Plattner Institute for IT Systems Engineering
University of Potsdam
Prof.-Dr.-Helmert-Str. 2-3
14482 Potsdam, Germany
hasso.plattner@hpi.uni-potsdam.de
Categories and Subject Descriptors
H.2.0 [Information Systems]: DATABASE MANAGE-
MENT—General
General Terms
Design, Performance
1. INTRODUCTION
Relational database systems have been the backbone of
business applications for more than 20 years. We promised
to provide companies with a management information sys-
tem that covers the core applications, including financials,
sales, order fulfillment, manufacturing, as well as human re-
sources, which run from planning through business processes
to individually defined analytics. However, we fell short of
achieving this goal. The more complex business require-
ments became, the more we focused on the so-called trans-
actional processing part and designed the database struc-
tures accordingly. These systems are called OLTP (Online
Transactional Processing) system. Analytical and financial
planning applications were increasingly moved out to sep-
arate systems for more flexibility and better performance.
These systems are called OLAP (Online Analytical Process-
ing) systems. In reality, parts of the planning process were
even moved o↵ to specialized applications mainly around
spreadsheets.
Both systems, OLTP and OLAP, are based on the rela-
tional theory but using di↵erent technical approaches [13].
For OLTP systems, tuples are arranged in rows which are
stored in blocks. The blocks reside on disk and are cached
in main memory in the database server. Sophisticated in-
dexing allows fast access to single tuples, however access
get increasingly slower as the number of requested tuples
increases. For OLAP systems, in contrast, data is often or-
ganized in star schemas, where a popular optimization is to
compress attributes (columns) with the help of dictionaries.
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
SIGMOD’09, June 29–July 2, 2009, Providence, Rhode Island, USA.
Copyright 2009 ACM 978-1-60558-551-2/09/06 ...$10.00.
After the conversion of attributes into integers, processing
becomes faster. More recently, the use of column store data-
bases for analytics has become quite popular. Dictionary
compression on the database level and reading only those
columns necessary to process a query speed up query pro-
cessing significantly in the column store case.
I always believed the introduction of so-called data ware-
houses was a compromise. The flexibility and speed we
gained had to be paid for with the additional management
of extracting, and loading data, as well as controlling the
redundancy. For many years, the discussion seemed to be
closed and enterprise data was split into OLTP and OLAP
[9]. OLTP is the necessary prerequisite for OLAP, however
only with OLAP companies are able to understand their
business and come to conclusions about how to steer and
change course. When planned data and actual data are
matched, business becomes transparent and decisions can be
made. While centralized warehouses also handle the integra-
tion of data from many sources, it is still desirable to have
OLTP and OLAP capabilities in one system which could
make both components more valuable to their users.
Over the last 20 years, Moore’s law enabled us to let the
enterprise system grow both in functionality and volume
[16]. When the processor clock speed hit the 3 GHz level
(2002) and further progress seemed to be distant, two devel-
opments helped out: unprecedented growth of main mem-
ory and massive parallelism through blade computing and
multi-core CPUs [14]. While main memory was always wel-
come for e.g. caching and a large number of CPUs could be
used for application servers, the database systems for OLTP
where not ideally suited for massive parallelism and stayed
on SMP (symmetric multi processing) servers. The reasons
were temporary locking of data storage segments for updates
and the potential of deadlocks while updating multiple ta-
bles in parallel transactions. This is the main reason why
for example R/3 from SAP ran all update transactions in a
single thread and relied heavily on row level locking and su-
per fast communication between parallel database processes
on SMP machines. Some of the shortcomings could be over-
come later by a better application design, but the separation
of OLTP and OLAP remained unchallenged.
Early tests at SAP and HPI with in-memory databases of
the relational type based on row storage did not show sig-
nificant advantages over leading RDBMSs with equivalent
memory for caching. Here, the alternative idea to inves-
tigate the advantages of using column store databases for
OLTP was born. Column storage was successfully used for](https://image.slidesharecdn.com/meetupvol2-180210105259/85/Meetup-vol2-6-320.jpg)
データベースMeetup vol2
- 1. PUBLIC ” ” SAP ( ) - Vol.2 One One -
- 3. 4PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ – HTAP Translytical Database HTAP SAP HANA – / HTAP SAP HANA – OLAP – OLTP SAP HANA Agenda
- 5. 6PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ HTAP(‘14) Translytical Database(‘15) “ / (HTAP) ” ” ” ” ” https://en.wikipedia.org/wiki/Hybrid_transactional/analytical_processing_(HTAP) “ Forrester ”Translytical” ” https://www.forrester.com/report/Emerging+Technology+Translytical+Databases+ Deliver+Analytics+At+The+Speed+Of+Transactions/-/E-RES116487 HTAP (Hybrid Transaction/Analytical Processing) Translytical Database
- 6. 7PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ ( ) SIGMOD09 (‘09) Hasso Plattner SIGMOD09 (‘09) Hasso Plattner ’06 Hasso SAP (Hasso Plattner Institute) OLTP OLAP Column Store SAP HANA SAP HANA ( ) SAP HANA http://www.sigmod09.org/images/sigmod1ktp-plattner.pdf “A Common Database Approach for OLTP and OLAP Using an In-Memory Column Database” https://en.wikipedia.org/wiki/Hasso_Plattner A Common Database Approach for OLTP and OLAP Using an In-Memory Column Database Hasso Plattner Hasso Plattner Institute for IT Systems Engineering University of Potsdam Prof.-Dr.-Helmert-Str. 2-3 14482 Potsdam, Germany hasso.plattner@hpi.uni-potsdam.de Categories and Subject Descriptors H.2.0 [Information Systems]: DATABASE MANAGE- MENT—General General Terms Design, Performance 1. INTRODUCTION Relational database systems have been the backbone of business applications for more than 20 years. We promised to provide companies with a management information sys- tem that covers the core applications, including financials, sales, order fulfillment, manufacturing, as well as human re- sources, which run from planning through business processes to individually defined analytics. However, we fell short of achieving this goal. The more complex business require- ments became, the more we focused on the so-called trans- actional processing part and designed the database struc- tures accordingly. These systems are called OLTP (Online Transactional Processing) system. Analytical and financial planning applications were increasingly moved out to sep- arate systems for more flexibility and better performance. These systems are called OLAP (Online Analytical Process- ing) systems. In reality, parts of the planning process were even moved o↵ to specialized applications mainly around spreadsheets. Both systems, OLTP and OLAP, are based on the rela- tional theory but using di↵erent technical approaches [13]. For OLTP systems, tuples are arranged in rows which are stored in blocks. The blocks reside on disk and are cached in main memory in the database server. Sophisticated in- dexing allows fast access to single tuples, however access get increasingly slower as the number of requested tuples increases. For OLAP systems, in contrast, data is often or- ganized in star schemas, where a popular optimization is to compress attributes (columns) with the help of dictionaries. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGMOD’09, June 29–July 2, 2009, Providence, Rhode Island, USA. Copyright 2009 ACM 978-1-60558-551-2/09/06 ...$10.00. After the conversion of attributes into integers, processing becomes faster. More recently, the use of column store data- bases for analytics has become quite popular. Dictionary compression on the database level and reading only those columns necessary to process a query speed up query pro- cessing significantly in the column store case. I always believed the introduction of so-called data ware- houses was a compromise. The flexibility and speed we gained had to be paid for with the additional management of extracting, and loading data, as well as controlling the redundancy. For many years, the discussion seemed to be closed and enterprise data was split into OLTP and OLAP [9]. OLTP is the necessary prerequisite for OLAP, however only with OLAP companies are able to understand their business and come to conclusions about how to steer and change course. When planned data and actual data are matched, business becomes transparent and decisions can be made. While centralized warehouses also handle the integra- tion of data from many sources, it is still desirable to have OLTP and OLAP capabilities in one system which could make both components more valuable to their users. Over the last 20 years, Moore’s law enabled us to let the enterprise system grow both in functionality and volume [16]. When the processor clock speed hit the 3 GHz level (2002) and further progress seemed to be distant, two devel- opments helped out: unprecedented growth of main mem- ory and massive parallelism through blade computing and multi-core CPUs [14]. While main memory was always wel- come for e.g. caching and a large number of CPUs could be used for application servers, the database systems for OLTP where not ideally suited for massive parallelism and stayed on SMP (symmetric multi processing) servers. The reasons were temporary locking of data storage segments for updates and the potential of deadlocks while updating multiple ta- bles in parallel transactions. This is the main reason why for example R/3 from SAP ran all update transactions in a single thread and relied heavily on row level locking and su- per fast communication between parallel database processes on SMP machines. Some of the shortcomings could be over- come later by a better application design, but the separation of OLTP and OLAP remained unchallenged. Early tests at SAP and HPI with in-memory databases of the relational type based on row storage did not show sig- nificant advantages over leading RDBMSs with equivalent memory for caching. Here, the alternative idea to inves- tigate the advantages of using column store databases for OLTP was born. Column storage was successfully used for
- 7. 8PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ ( ) SIGMOD09 (‘09) Hasso Plattner “Figure 8 shows a future managementmeeting with information finally at your fingertips without any restriction.” 2009 2016 SAPPHIRE NOW 2016 (SAP Digital Boardroom)
- 8. 9PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ HTAP / Translytical Database One Fact One Place, Real-Time Ex) BI/ReportEx) ERP OLAPOLTP ETL à IT (ex. ERP) (BI/Report) HTAP One Fact One Place, Real-Time HTAP OLTP & OLAP Ex) ERP/BI/Report/NEW... & In-Memory Computing AnalyticsMobile
- 9. 10PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ HTAP ( ) HTAP Hadoop HTAP / OLTP OLAP Hadoop / Distributed Processing System HTAP CEP / Streaming
- 10. 11PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ SAP The Forrester Wave • In-Memory Database Platforms • Translytical Data Platform • Streaming Analytics • Big Data Warehouse • Enterprise Data Virtualization • Predictive Analytics and Machine Learning Gartner Magic Quadrant Operational Database Management System• Data Management Solutions for Analytics• Data Integration Tools• Data Quality Tools• Challenges Contenders Strong Performers Leaders Strong Weak Current offering Weak StrongStrategy Market presence Challengers Leaders Niche Players Visionaries Abilitytoexecute Completeness of Vision As of November 2017 Figure 1: Magic Quadrant for Operational Database Management Systems, Source: Gartner (November 2017)
- 11. HTAP SAP HANA
- 12. 13PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ HTAP SAP HANA è J
- 13. HTAP SAP HANA
- 14. 15PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ / HTAP SAP HANA OLTP CPU WAL(Write Ahead Log) SSD/Flash Storage OLAP CPU ( -SIMD) NUMA CPU L1/L2 HTAP
- 15. 16PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ OLAP ( ) SIMD (Single Instruction Multiple Data) Pentium SSE(Streaming SIMD Extensions) SIMD 128bit Sandy Bridge Intel AVX (Advanced Vector eXtensions) SIMD 256bit Intel Skylake Xeon SIMD 512bit SIMD 1 2 3 4 4 A( A) 1 2 3 4 4 B( B) 1 2 3 4 1 2 3 4 2 4 6 8 + + + + = = = = 1 2 3 4 1 2 3 4 + (SIMD_Plus) 2 4 6 8 = + 4 4 + ( ) 4 + 1 SIMD 4 1 + ( ) => CPU SIMD ( 128bit) A: B:
- 16. 17PUBLIC© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ OLTP SAP HANA ( ) OLTP - (semaphore mutex) ( ) I/O 16-25% lock manager “Lightweight Locking for Main Memory Database Systems” http://www.vldb.org/pvldb/vol6/p145-ren.pdf
- 17. 18PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ - H/W(CPU) ( ) H/W OS (≒ ) START XBEGIN or XABORT Write-set XEND END Write-set ABORT * TSX RTM(Restricted Transactional Memory) () () TSX OLTP OLTP SAP HANA ( )
- 18. 19PUBLIC2018 SAP SE or an SAP affiliate company. All rights reserved.© ǀ ( ) SAP/Intel ( ) Intel Xeon processor E7 v2 family Intel* TXS Intel Xeon processor E7 v3 family Intel TXS Intel Xeon processor E7 v3 family Intel TXS Intel Xeon processor E7 v4 family Intel TXS SAP HANA* SPS 09 SAP HANA SPS 09 SAP HANA SPS 09 SAP HANA SPS 12 2.7 1 6.0 1 6.3 1TPM : (Transaction Per Minute) 1 1 2 3 ® Xeon® E7 v4 ® TSX SAP HANA* 1 Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products . Up to 6x performance improvement for transactional workloads with new Intel® Transactional Synchronization Extensions (TSX) claim based on SAP* OLTP internal insert and select tests measuring transactions per minute (tpm) on SuSE* LINUX Enterprise Server 11 SP3. Configurations: a. Baseline 1.0: 4S Intel Xeon processor E7-4890 v2, 512 GB memory, SuSE* LINUX Enterprise Server 11 SP3, SAP HANA* 1 SP8 scoring 14,327 tpm.b. Up to 1.8x more tpm: 4S Intel Xeon processor E7-4890 v2, 512 GB memory, SuSE* LINUX Enterprise Server 11 SP3, SAP HANA* 1 SP9 scoring 26,139 tpm.c. Up to 2.7x more tpm: 4S Intel Xeon processor E7-8890 v3, 512 GB memory, SuSE* LINUX Enterprise Server 11 SP3, SAP HANA* 1 SP9 – Intel TSX disabled scoring 39,330 tpm.d. Up to 6x more tpm: 4S Intel Xeon processor E7-8890 v3, 512 GB memory, SuSE* LINUX Enterprise Server 11 SP3, SAP HANA* 1 SP9 – Intel TSX enabled scoring 89,619 tpm. For more complete information visit http://www.intel.com/performance/datacenter.
- 19. 20PUBLIC© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ ) SAP Smart Traffic (co-innovation with ) 800
- 20. 21PUBLIC© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ usecase video
- 21. 22PUBLIC© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ SAP Smart Traffic IoT SAP HANA/ (RFID) 30 + / (GPS) 80 + / 24 + / ~10 / ~10 / (GPS) 60 + / SAP HANA What-if ( ) / /
- 23. 24PUBLIC© 2018 SAP SE or an SAP affiliate company. All rights reserved. ǀ SAP HANA OLTP OLAP ( ) ( ) ( ) (CPU) SAP HANA OLTP OLAP CPU (NVM) (RDMA) OLTP OLAP SAP HANA HTAP
- 24. Thank you.
- 25. No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice. Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary. These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty. In particular, SAP SE or its affiliated companies have no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation, and SAP SE’s or its affiliated companies’ strategy and possible future developments, products, and/or platforms, directions, and functionality are all subject to change and may be changed by SAP SE or its affiliated companies at any time for any reason without notice. The information in this document is not a commitment, promise, or legal obligation to deliver any material, code, or functionality. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, and they should not be relied upon in making purchasing decisions. SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. See www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices. © 2018 SAP SE or an SAP affiliate company. All rights reserved.