Networking Israeli Day 2013 - Hecatonchire: Transparent Memory Aggregation
- 1. Hecatonchire: Transparent Resource Aggregation Israeli Networking Day 2013 Aidan Shribman; Sr. Researcher; SAP Research Israel With the contribution of Benoit Hudzia, Roei Tell, Steve Walsh, Peter Izsak
- 2. In the talk today Motivation: the SAP HANA in- memory database. Intro: the Hecatonchire project Capabilities: mainly memory aggregation. Numbers: quicksort & SAP HANA TCP-H benchmarks. © 2012 SAP AG. All rights reserved. 2
- 3. Back story: SAP HANA In-Memory Real-Time Analytics SAP ERP R/3 (1980s) SAP HANA In-Memory Database (2010) Moderate hardware requirements; Is optimized for slow Big multi-core large-memory systems; high storage devices. memory/core ratio: 128 GB; 256 GB; … 1024 GB. OLTP (OnLine Transactioanl Processing) : mainly OLAP (OnLine Analytical Processing): on-the-fly storing and retrieving transactions analytics on real-time data Single-record access: Single-scan aggregation: SELECT * FROM Sales_Orders SELECT Country, SUM(Sales) FROM Sales_Orders WHERE Order = ‘457’ WHERE Product=‘corn’ GROUP BY Country Order Country Product Sales 456 France corn 1000 Order Country Product Sales 456 France corn 1000 457 Italy wheat 900 457 Italy wheat 900 458 Italy corn 600 458 Italy G corn 600 459 Spain rice 800 459 Spain rice 800 Column order organization Row order organization © 2012 SAP AG. All rights reserved. 3
- 4. A “gap” in Current Cloud Offerings Guest are constrained by host sizes Host resources are pooled together (future?) Limited by physical constraints (AWS high- No limitation: Better scalability; Better economics; memory is 244 GB guest atop a 256 GB host). Better performance. Guests Guests VM VM VM VM VM App VM App App App App OS App OS OS OS VM App OS OS H/W H/W H/W H/W OS H/W H/W H/W Server #1 Server #2 Server #n Server #1 Server #2 Server #n CPUs CPUs CPUs CPUs CPUs CPUs Memory Memory Memory Memory Memory Memory I/O I/O I/O I/O I/O I/O Fast RDMA Communication Fast RDMA Communication © 2012 SAP AG. All rights reserved. 4
- 5. The Hecatonchire Project Hecatonchires in Greek mythology means “Hundred Handed One”. The project addresses the transparent aggregation of compute resources such as memory, CPU and storage Hecatonchire aims at extending (not replacing) existing cloud and virtualization stacks (Linux/KVM, QEMU, OpenStack) while using of commodity hardware and modern interconnects (10 GbE or InfiniBand) Hecatonchire is an open source project hosted on github developed by SAP Research © 2012 SAP AG. All rights reserved. 5
- 6. Faster to Scale-Out … Author: Chaim Bendalac © 2012 SAP AG. All rights reserved. 6
- 7. Challenges of Resources Aggregation CPU (future) State is MBs in size (can be migrated sub-second) Can be virtualized using hypervisor vCPU abstraction Difficult challenge: Implementing efficient fine-grained cache-coherency (difficult) Memory (in-progress) GBs to TBs is size (accessed in page granularity 4KB – 4MB) Can be virtualized by extending the MMU (Main Memory Unit) system Challenge: achieving near-native performance while not degrading availability Storage (mostly-done) TBs to PBs in size (but accessed in blocks MBs in size) Already virtualized using VFS; BLK; SCSI; iSCSI; virtio/vhost ;iSER; LIO. Continued challenge: high bandwidth and high IOPS (partially achieved) © 2012 SAP AG. All rights reserved. 7
- 8. VM Memory Pooling Fast Remote Page Access Low latency RDMA (Remote Direct Memory Access) page transfer protocol Supports best price/performance hardware: InfiniBand, iWARP, RoCE Demand pre-paging (pre-fetching) mechanism Built-in RAID-1 style Memory Mirroring No extra single points for failure Minimal overhead : 1st slave to answer synchronous; 2nd - asynchronous Complimentary to VM-based High-Availability (e.g. Kemari) Transparent for guest VMs Integration by adding static hooks into Linux MMU (page fault, swap-out, etc.). QEMU Binds Remote Memory to Guest Linux/KVM Virtual Machine. © 2012 SAP AG. All rights reserved. 8
- 9. Multithreaded 2 GB Quicksort Benchmark 110.00% Completion Time / Native Time 108.00% 106.00% 104.00% 102.00% Native 100.00% Remote Mirrored 98.00% 96.00% 94.00% 75% 50% 33% 25% 20% Local Memory Percentage © 2012 SAP AG. All rights reserved. 9
- 10. SAP HANA TPC-H like OLAP Benchmark (128 GB / 40 vCPU) 102.00% Completion Time / Native Time 101.50% 101.00% 100.50% Native 100.00% Mirrored 99.50% 99.00% 50% 33% Local Memory Percentage © 2012 SAP AG. All rights reserved. 10
- 11. Application Memory Pooling Required if virtualizaiton can„t be used due to performance of usage of exotic hardware. When using virtualization we „remotify“ the entire guest VM address space. Here we need to „remofity“ all processes associated with the application. © 2012 SAP AG. All rights reserved. 11
- 12. Hecatonchire key takeaways Already provides transparent VM / application memory aggregation Extends existing cloud & virtualization solutions Released as open source / hosted on GitHub Developed by SAP Research Technology Infrastructure (TI) Practice © 2012 SAP AG. All rights reserved. 12
- 13. Thank you! Aidan Shribman SAP Research Israel aidan.shribman@sap.com The Hecatonchire Project http://www.hecatonchire.com hecatonchire@liberlist.com
- 14. Legal Disclaimer The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. This presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this document or any related presentation, or to develop or release any functionality mentioned therein. This document, or any related presentation and SAP's strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information on this document is not a commitment, promise or legal obligation to deliver any material, code or functionality. This document is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This document is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this document, except if such damages were caused by SAP intentionally or grossly negligent. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions. © 2012 SAP AG. All rights reserved. 14