DUG'20: 03 - Online compression with QAT in DAOS
0 likes107 views
This document discusses compression capabilities added to DAOS (Distributed Asynchronous Object Storage). It provides an overview of the DAOS compression framework and algorithms including LZ4 and deflate. It evaluates the performance of these algorithms with and without using Intel QuickAssist Technology hardware acceleration. Testing shows QAT achieves the best compression throughput and ratio while LZ4 has the best decompression throughput. Future work is outlined to improve compression functionality in DAOS.
![DAOS Compressor: Abstraction layer in DAOS
compress_ft
daos_compressor
dc_ctx
lz4_algo
init()
compress()
decompress()
“lz4”
COMPRESS_TYPE_LZ4
liblz4
deflate_algo
init()
compress()
decompress()
“deflate”
COMPRESS_TYPE_DEFLATE
libisal
qat_deflate_algo
init()
compress()
decompress()
“deflate”
COMPRESS_TYPE_DEFLATE
libqat
isal_compress_algo_table[ ] qat_compress_algo_table[ ]
daos_compress_type2algo(COMPRESS_TYPE, qat_prefered)
choose algorithm
functions
daos_compressor_init
daos_compressor_compress
daos_compressor_decompress
daos_compressor_compress_async
daos_compressor_decompress_async
daos_compressor_destroy](https://image.slidesharecdn.com/3onlinecompressionwithqatindaos-201223000939/85/DUG-20-03-Online-compression-with-QAT-in-DAOS-4-320.jpg)
DUG'20: 03 - Online compression with QAT in DAOS
- 1. Online Compression with Intel® QAT in DAOS Weigang Li, Intel Johann Lombardi, Intel November, 2020 DAOS User Group (DUG’20)
- 2. DAOS Node Compute Nodes DAOS compression Compute Nodes DAOS library Application Compressor DAOS Nodes DAOS service Compressor Storage ▪ Compress on Server • Optimize storage bandwidth/usage and not network bandwidth • Able to use hardware acceleration ▪ Compress on Client • Reduce amount of data transferred over the network • Consume compute CPU cycles Note: Only data will be compressed for now and not metadata Compressed Data Compressed Data
- 3. Current Status ▪ DAOS-5605 common: add basic infrastructure for QAT #3402 (Merged) ▪ DAOS-5605 build: add lz4 dependency #3403 (Merged) ▪ DAOS-5719 compression: add framework and lz4/deflate support #3561 (Merged) ▪ DAOS-5719 compression: add qat support for deflate #3621 (Merged) ▪ QAT async support and compress_timing test (WIP) • All the infrastructure to enable compression and encryption has been landed. • It is set at container creation time via properties DAOS_PROP_CO_COMPRESS and DAOS_PROP_CO_ENCRYPT. # daos cont get-prop --path /mnt/lustre/mycontainer --svc $SVC Container properties for d28d9df6-5ee7-4030-b20f- bf06c13e2226 : label: container label not set layout type: POSIX (1) layout version: 1 checksum type: off checksum chunk-size: 32768 cksum verif. on server: off deduplication: off dedup threshold: 4096 redundancy factor: rf1 redundancy level: rack max snapshots: 0 compression type: off encryption type: off owner: root@ owner-group: root@
- 4. DAOS Compressor: Abstraction layer in DAOS compress_ft daos_compressor dc_ctx lz4_algo init() compress() decompress() “lz4” COMPRESS_TYPE_LZ4 liblz4 deflate_algo init() compress() decompress() “deflate” COMPRESS_TYPE_DEFLATE libisal qat_deflate_algo init() compress() decompress() “deflate” COMPRESS_TYPE_DEFLATE libqat isal_compress_algo_table[ ] qat_compress_algo_table[ ] daos_compress_type2algo(COMPRESS_TYPE, qat_prefered) choose algorithm functions daos_compressor_init daos_compressor_compress daos_compressor_decompress daos_compressor_compress_async daos_compressor_decompress_async daos_compressor_destroy
- 5. Compression algorithms ▪ LZ4 • Fast, low compression ratio ▪ Deflate • Slow, high compression ratio • Implementation: ZLIB, GZIP ▪ zStandard • New algorithm • Fast, high compression ratio • May be added to DAOS in future ▪ Performance › Throughput › Compression Ratio ▪ Cost Hardware acceleration
- 6. Intel® QAT Software Stack Intel® QuickAssist Technology (QAT) • Intel® QuickAssist Technology (QAT) integrates hardware acceleration for compute intensive workloads: ✓ Bulk Cryptography ✓ Public Key Exchange ✓ Compression • Formfactors: Chipset, PCIE card, SoC • Integrated with many mainstream software framework Intel® QuickAssist Technology Device OpenSSL libcrypto EVP API OpenSSL libssl OpenSSL “engine” API Intel QuickAssist Technology Kernel Driver User Space Kernel Space nginx* SSL API Intel QuickAssist Technology User Space Library Intel QuickAssist Technology API QAT Engine zlib zlib patch zlib API App Other Apps Hadoop QAT codec QATzip LKCF LKCF “transform” API (acomp) btrfs LKCF “algorithm” API ZFS https://01.org/intel-quickassist-technology Intel® C620 Series Chipsets Intel® QuickAssist Adapter 89xx Rangeley, Denverton
- 7. QAT Compression Sync Mode QAT Driver QAT HW cpaDcCompressData2 qat_dc_compress: do { icp_sal_DcPollInstance } while (!complete); callback function: complete = 1; Poll response Send request trigger callback (s_buf, d_buf, &complete) allocate pinned memory and copy source buffer copy d_buf to virtual memory return; Main Thread block waiting in main thread daos_compressor_compress
- 8. QAT Compression Async Mode QAT Driver QAT HW qat_dc_compress_async do { qat_dc_poll_response } while (!all_complete); callback function: memcpy d_buf to virtual memory complete_x = 1; Poll response Send request trigger callback s_buf, d_buf, complete_1 Main Thread qat_dc_compress_async qat_dc_compress_async Polling Thread s_buf, d_buf, complete_2 s_buf, d_buf, complete_3 Option1: polling in main thread Option2: polling in separate thread do { qat_dc_poll_response // wait for all responses } while (!all_complete); daos_compressor_compress_async Compress multiple blocks: Wait for all responses:
- 9. Compress_timing For BS = 4KB, 8KB, 16KB, … 512KB: BS Divide 3.2MB Calgary Corpus buffer to block size: BS BS BS BS…… daos_compressor_compress(_async) Iterations = 1000 Calculate performance: throughput, ratio Decompress and verify result Repeat every block size Single Thread
- 10. Compress_timing results @64KB block size ▪ Tested with QAT Gen-2 DH8950 plugin card ▪ Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz ▪ Ratio = compressed_size / origin_size ▪ Single Thread 0 100 200 300 400 500 600 lz4 isal-1 isal-2 qat-1 qat-2 221.1 259.4 227.4 555.8 513 Compression throughput (MB/s) @64KB 0 100 200 300 400 500 600 700 800 lz4 isal-1 isal-2 qat-1 qat-2 785.4 332.2 374.9 746.8 746.3 Decompression throughput (MB/s) @64KB 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% lz4 isal-1 isal-2 qat-1 qat-2 51.81% 47.45% 38.24% 41.38% 38.48% Compression ratio @64KB (less is better)
- 11. Compress_timing results for all block sizes • QAT ✓Best compression throughput ✓Best compression ratio • LZ4 ✓Best decompression throughput 248.9 240 231.9 226 221.1 251.6 249 247.5240.5 250.1 255.2 257.9 259.4 259.5 259.3 259.1 375.2 469.5 511.5 541.2 556 510.3 437.8 359.3 4k 8k 16k 32k 64k 128k 256k 512k lz4 isal-1 qat-1 872.1 850.9 825.2 799.1 785.4 986 985 983.8 264.5 308.9 308.9 325 332.6 341.3 349.1 352.6 395.6 570 665 715.8 746.5 760.5 709.1 619.9 4k 8k 16k 32k 64k 128k 256k 512k lz4 isal-1 qat-1 62.15% 58.54% 55.67% 53.38% 51.81% 52.42% 52.12% 51.93% 56.35% 52.97% 50.41% 48.54% 47.45% 46.90% 46.67% 46.53% 44.74% 43.16% 42.20% 41.56% 41.38% 41.62% 42.55% 43.99% 4 K 8 K 1 6 K 3 2 K 6 4 K 1 2 8 K 2 5 6 K 5 1 2 K COMPRESSION RATIO LESS BETTER lz4 isal-1 qat-1
- 12. Future Work • Implement actual extents/records compression on both client and server • Change aggregation to support compression • Support partial read in the middle of a compressed extent • Similar to end-to-end checksum support where we have to read the whole extent to validate the checksum. We will just reuse this code. • Automatically discover QAT at build time and runtime, intelligently switch between software and hardware acceleration API • Chaining of compression and hash in one function call
- 13. ▪ INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. ▪ A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS. ▪ Intel may make changes to specifications and product descriptions at any time, without notice. Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined". Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them. The information here is subject to change without notice. Do not finalize a design with this information. ▪ The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request. ▪ Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order. ▪ Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to: http://www.intel.com/design/literature.htm%20 ▪ All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. ▪ Intel, Intel logo, Look Inside, Intel Inside, Intel Inside logo, the Look Inside Logo, Intel Atom, and Intel Xeon are trademarks of Intel Corporation in the U.S. and other countries. ▪ *Other names and brands may be claimed as the property of others. ▪ Other vendors are listed by Intel as a convenience to Intel's general customer base, but Intel does not make any representations or warranties whatsoever regarding quality, reliability, functionality, or compatibility of these devices. This list and/or these devices may be subject to change without notice. ▪ Copyright © 2020, Intel Corporation. All rights reserved. Legal Disclaimer 13
- 14. 14