A27 Vectorwise Performance Considerations_implementation_best_practices
0 likes899 views
This document provides best practices for implementing Vectorwise, a high performance database. It discusses hardware recommendations including using CPUs with high clock rates and memory, with at least 8GB per core. It recommends 64-bit operating systems like Red Hat, SUSE, or Windows. Database configuration defaults are generally good. Data loading is best done through bulk load or incremental insert, and high availability can be achieved through hardware redundancy and backups. Monitoring includes OS metrics, vwinfo data, and third party tools.
A27 Vectorwise Performance Considerations_implementation_best_practices
- 1. Vectorwise Implementation best practices Mark Van de Wiel Director Product Management, Vectorwise Thursday, November 01, 2012 1 of 9 1 of 9 Confidential © 2012 Actian Corporation
- 2. Agenda Hardware Operating system Database configuration Database design Data loading High availability Monitoring Confidential © 2012 Actian Corporation 2
- 3. 100x (+) Performance Difference – 2003 Custom C versus Relational Database TPC-H 1 GB query 1 (runtime in s) 30 28.1 26.2 25 20 MySQL 15 DBMS 'X' C program 10 Vectorwise 5 0.2 0.6 0 MySQL DBMS 'X' C program Vectorwise Confidential © 2012 Actian Corporation 3
- 4. Some Numbers Traditional RDBMS: <200 MB/s per core Even these use MPP to I/O challenges Vectorwise (lab environment): >1.5 GB/s per core Maximum throughput requirement is extremely high Realistically (cost-effectively) only RAM can serve data quick enough Confidential © 2012 Actian Corporation 4
- 5. What Hardware to Use CPU Memory Storage I/O and capacity Requirements Budget Confidential © 2012 Actian Corporation 5
- 6. Hardware Considerations – MEMORY Ideally frequently-accessed data should fit in memory May be all data May be a small portion of the data Note: data is compressed in memory buffer • 3x – 5x compression ratios are common Query execution should all take place in memory Operations against larger data sets require more memory Consider query concurrency “Spill to disk” is supported but should be a last resort Confidential © 2012 Actian Corporation 6
- 7. Hardware Recommendation CPUs Use CPUs with higher clock rate for better raw throughput Use more cores for higher throughput Higher power CPUs are faster Memory At least 8 GB per core (more is always better) Storage Use as many drives as possible Ensure sufficient capacity Use the fastest drives available • SAS over SATA, ideally 15k RPM • SSDs are often not cost-effective relative to more memory Confidential © 2012 Actian Corporation 7
- 8. Examples Small configuration (1 TB) Dell R620 Lenovo RD430 Medium configuration (single digit TBs) Dell R720 HP DL380 IBM x3650 Lenovo RD630 High-end configuration Dell R910 HP DL580 or DL980 IBM x3750 Confidential © 2012 Actian Corporation 8
- 9. Operating System Considerations 64-bit Redhat Windows 7 (or higher) SuSE xfs, ext3, ext4 Windows 2008 (or higher) Ubuntu Confidential © 2012 Actian Corporation 9
- 10. Database Configuration Installation defaults are generally good May want to adjust column buffer size (default 25% of RAM) May want to adjust processing memory (default 50% of RAM) Confidential © 2012 Actian Corporation 10
- 11. Database Design Schema – no particular preference Single demormalized table, star schema, snowflake schema, 3rd normal form Constraints Only on empty tables today… (to be addressed in Vectorwise 3.0) Consider data loading order and impact Indexes Note: clustered index-only today (“index-organized table”) One per table Consider incremental load Confidential © 2012 Actian Corporation 11
- 12. Data Loading Initial load File-based bulk load through vwload or copy Conversion into UTF8 Use tools Pentaho Informatica Talend HVR Attunity Confidential © 2012 Actian Corporation 12
- 13. Data Loading Incremental load INSERT, UPDATE and/or DELETE Append if possible Batch if possible Use COMBINE Positional Delta Trees Memory considerations Propagation to disk Use tools Confidential © 2012 Actian Corporation 13
- 14. Moving Window of Data Considerations COMBINE on a large table can be expensive Mostly relevant for updates and deletes Alternative: manual partitioning One table per period Single view across all tables Confidential © 2012 Actian Corporation 14
- 15. High Availability Hardware and OS best practices UPS, RAID Vectorwise backup Only read-only, full backup Consider periodic full backup and file incremental loads Disaster recovery Dual load Active/active possibility Confidential © 2012 Actian Corporation 15
- 16. Monitoring OS monitoring CPU, memory utilization, I/O statistics vwinfo data Actian Director DBA tools Confidential © 2012 Actian Corporation 16
- 17. Agenda Hardware Operating system Database configuration Database design Data loading High availability Monitoring More information in the Vectorwise Developer Guide: http://www.actian.com/images/white_papers/vw_developers_v2.5.pdf Confidential © 2012 Actian Corporation 17
- 18. Confidential © 2012 Actian Corporation