2012 11 Openstack China
2 likes653 views
This document summarizes a presentation about evaluating the performance of the OpenStack Swift object storage system using the COSBench benchmark. Key points include: - COSBench is an open source benchmark developed by Intel to measure the performance of cloud object storage services like OpenStack Swift and Amazon S3. - A case study was conducted using COSBench to evaluate Swift performance on different hardware configurations and identify bottlenecks. - The results showed that for small object operations the proxy CPU was the bottleneck, while for large object operations disk and network bandwidth were limiting factors. - Going forward the presenter plans to support additional storage services with COSBench and make it more openly available to help optimize
2012 11 Openstack China
- 1. Measuring Swift Performance with COSBench Yaguang Wang 2012.11 Updated June 2012
- 2. Agenda • COSBench Introduction • Case Study to evaluate OpenStack* swift performance with COSBench • Next Step plan and Summary
- 3. OpenStack Key Components Nova Compute Flexible Workloads Virtualized Storage Infrastructure Swift Open Quantu Platforms Network m Cinder Common Fabrics
- 4. 工欲善其事，必先利其器 Business DB Performance Performance Requirement Random small (OLTP, OLAP) Storage TPC-C/E/H IOMeter (Requests per second) YCSB, Large analytics HiBench (e.g NoSQL, Hadoop) Sequential Large Capacity Storage COSBench Cloud Object storage (e.g. photos/videos) Gigabytes Terabytes Petabytes Exabytes Capacity Requirement fferent Usage model requires different benchmark to
- 5. COSBench Introduction • COSBench is an Intel developed Benchmark to measure Cloud Object Storage Service performance – For S3, OpenStack Swift like Object Storage – Not for File system (NFS e.g) and Block Device system (EBS e.g.) COSBench • Benefits: – Compare public Cloud Object Storage services (User) – Evaluate different Hardware/Software Stacks (Provider) – Identify bottleneck and make optimization (Provider) The IOMeter for Cloud Storage Services.
- 6. Key Components • Driver (Load Web Generator): Console Controller – generate load and stats. • Controller: Config.xml Benchmark – Control all drivers Cloud – Collect & report stats. Driver Driver • Config.xml: – define workloads Proxy • Web Console: Node – Manage Controller Storage – Browse real-time stats Cloud Storage Node
- 7. Web Console Driver list Workload List History list Intuitive UI to get overview.
- 8. Workload Configuration Flexible load control object size distribution Read/Write Operations Workflow for complex stages Flexible configuration parameters.
- 9. Performance Metrics • Throughput (Operations/s): the operations completed in one second • Response Time (in ms): the duration between operation initiation and completion. • Bandwidth (KB/s): the total data in KiB transferred in one second • Success Ratio (%): the ratio of successful operations
- 10. Performance Reporting Id Op RT TH BW Succ% Timeline Throughput W1-s1- 500 160 write 24,8390.16 172 100% 400 140 1 300 120 100 summa timelin 80 200 60 ry e 40 100 20 0 0 histogra loadlin Time (in 5s) m e Response Time Histogram Performance Loadline (100% Read) 80,000 120% 6,000 400 Response Time (ms) Throughput (Op/s) Throughput (Op/s) 362 70,000 100% 5,000 350 Avg Response Time 60,000 300 80% (ms) 50,000 4,000 250 read 40,000 60% 3,000 200 CDF (%) 30,000 40% 150 20,000 2,000 20% 91 100 10,000 1,000 45 50 0 0% 15 15 19 26 0 0 0~10 60~70 120~130 180~190 240~250 16 32 64 128 256 512 2048 Workers
- 11. Extensible API • Easily extend for new storage system: AuthAPI Auth • Support – OpenStack Swift Context PUT – Amplistor* GET – Adding More StorageAPI DELETE Extensible API is able to support more storages.
- 12. Setup-SATA has higher CPU System Configurationpower Setup-SAS has faster disks Client Client Client Client Client Node Node Node Node Node 2GbE 2GbE 2GbE 2GbE 2GbE Setup-SATA Both Setup - CPU: 2 * Client - CPU: 2 * 2.93GHz (4C/8T) Ethernet 2.7GHz (8C/16T) Network - MEM: 12GB DDR3 1333MHz 10GbE - NIC: 2 * Intel 82579 1GbE - MEM: 32GB DDR3 1333MHz - NIC: Intel 82599 10GbE bonding (mode=rr) Proxy Setup-SAS Node - CPU: 2 * Setup-SAS 2.3GHz (8C/16T) 10GbE - 12 * 70GB SAS (15000 rpm) - MEM: 64GB DDR3 1333MHz Storage - NIC: Intel 82599 10GbE Ethernet Network Setup-SATA - 14 * 1T SATA (5,400 rpm) 2GbE 2GbE 2GbE 2GbE 2GbE Storage Storage Storage Storage Storage Node Node Node Node Node
- 13. 128KB-Read # 95%- Throughpu Setup-SAS 128KB-Read Worker ResTime t ms op/s Throughput Avg-ResTime 95%-ResTime 5 20.00 369.49 response time (ms) 6.0k 1000 10 20.00 711.24 throughput (op/s) 5.02k5.00k4.95k4.84k 4.69k 20 20.00 1383.30 4.5k 800 3.66k 40 30.00 2517.94 600 3.0k 2.52k 80 46.67 3662.71 400 160 56.67 4693.97 1.38k 1.5k 0.71k 200 320 106.67 5019.85 0.37k 640 230.00 4998.13 0.0k 0 5 20 80 320 1280 1280 470.00 4947.15 2560 923.33 4840.19 Total Number of Worker bottleneck The was identified to be the proxy’s CPU -- The CPU utilization at that node was ~100%! For more complete information about setup-SATA was 5576 results in higher -- The peak throughput for Better CPU op/s (640 performance and benchmark results, visit workers) throughput
- 14. 128KB-Write • SLA: 200ms + 128KB/1MBps = 325ms # 95%- Throughpu Setup-SAS 128KB-Write Worker ResTime t ms op/s Throughput Avg-ResTime 95%-ResTime 5 40.00 219.73 1.87k1.89k 4000 response time (ms) 2.0k 1.77k1.77k 10 40.00 391.14 throughput (op/s) 1.59k 3200 20 50.00 668.19 1.5k 1.33k 40 70.00 1022.07 1.02k 2400 1.0k 80 100.00 1333.34 0.67k 1600 160 143.33 1594.12 0.5k 0.22k0.39k 800 320 370.00 1769.55 640 1223.33 1773.12 0.0k 0 5 20 80 320 1280 1280 1690.00 1871.58 2560 3160.00 1886.81 Total Number of Worker The Disks at storage nodes had significant impact on overall throughput -- The peak throughput for setup-SATA was 1208 op/s (320 Workers) For more complete information about -- still gaps even after we had put account/container DB files performance and benchmark results, visit
- 15. 10MB-Read • SLA: 200ms + 10MB/1MBps = 1200ms # 95%- Throughpu Setup-SAS10MB -Read Worker ResTime t ms op/s Throughput Avg-ResTime 95%-ResTime 5 270.00 34.69 response time (ms) 80 70 71 73 60,000 10 320.00 51.87 throughput (op/s) 67 69 70 65 60 50,000 20 480.00 59.91 60 52 40,000 40 900.00 65.48 40 35 30,000 80 1636.67 67.37 20,000 160 3093.33 68.69 20 320 5950.00 69.58 10,000 640 11906.67 70.18 0 0 5 20 80 320 1280 1280 24090.00 71.41 2560 52090.00 72.90 Total Number of Worker bottleneck was identified to be the clients’ The NIC BandWidth Double client receive bandwidth can double the throughput For more complete information about performance and benchmark results, visit
- 16. 10MB-Write • SLA: 200ms + 10MB/1MBps = 1200ms # 95%- Throughpu Setup-SAS 10MB-Write Worker ResTime t ms op/s Throughput Avg-ResTime 95%-ResTime 5 536.67 13.12 24 response time (ms) 25 23 23 250,000 22 10 936.67 17.50 throughput (op/s) 21 21 22 20 20 17 200,000 20 1596.67 20.28 15 13 150,000 40 2786.67 21.30 80 5133.33 21.38 10 100,000 160 9800.00 22.21 5 50,000 320 18623.33 23.19 640 41576.67 23.23 0 0 5 20 80 320 1280 1280 102090.00 21.55 2560 200306.67 23.71 Total Number of Worker bottleneck The might be the storage nodes’ NICs -- in setup-SATA, the peak throughput was 23.23 op/s (10 clients) For more complete information about performance and benchmark results, visit -- in both setups, the write performance was 1/3 of the read
- 17. Next Step • Support more storage services like Amazon S3, Google Cloud Storage, and Microsoft Azure… • Make COSBench available (so far under NDA, will open source finally) for industry and community to build efficient cloud storage service. • Continue to use COSBench to analyze and optimize OpenStack* Swift performance on Intel Platform and share our findings back to community.
- 18. Disclaimers • INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. • A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. • Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. • The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. • Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. • Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm%20 • This document contains information on products in the design phase of development. • Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries. • *Other names and brands may be claimed as the property of others. • Copyright © 2012 Intel Corporation. All rights reserved.