Hi Speed Datawarehousing
9 likes1,095 views
The document discusses strategies for building high-speed data warehouses, including upgrading hardware, extending existing warehouses with additional data marts or appliances, and migrating to new high-speed solutions. It provides examples of new solutions that use massively parallel processing and column-based storage to improve query performance. Benchmark results are presented showing significant performance gains from these approaches for analyzing very large datasets.
Hi Speed Datawarehousing
- 1. Hi-Speed DataWarehousing Jos van Dongen, Tholis Consulting
- 2. Agenda Introduction Why Hi-Speed DWH? Where do we ‘Hi-speed’ the DWH? Part 1: Hi-Speed Strategies Upgrade Extend Migrate Part 2: New Hi-Speed DWH solutions What’s new? Which products? How fast are they? What does it cost? Database Systems 2008 THOLIS CONSULTING
- 3. Hi-Speed Why? Growing data volume: Gartner group: 2007: 50% of DWH’s > 10 TB 2011: 50% of DWH’s > 50 TB < 5 TB is considered small (!) Increasing workload: Operational BI Pervasive BI Advanced Analytics & Mining Database Systems 2008 THOLIS CONSULTING
- 4. Hi-Speed Where? Datawarehousing: Development ETL Query & Analysis Maintenance (Index, aggregate, backup, restore, authorization, etc.) Presentation focus: Query & Analysis Database Systems 2008 THOLIS CONSULTING
- 5. Hi-Speed How? Database Systems 2008 THOLIS CONSULTING Upgrade 2-5* Extend 5-100* Migrate 10-400* Hardware Processing power Memory Disk Software 64bit New OS versions New RDBMS versions Add datamarts OLAP engines (Datamart) Appliances ‘ Buddy’ system Datastore replacement DWH Appliances HW/SW packages SW (roll your own) Outsource
- 6. Upgrade Hardware: Cost ‘no issue’ Database Systems 2008 THOLIS CONSULTING <2000: Solve performance problems in software (tuning, optimization) 2008: hardware is cheaper than time!
- 7. Upgrade Hardware: Memory Database Systems 2008 THOLIS CONSULTING Feb 2008: 2 GB €41,- (PC) 4 GB €160,- (Server) Entry level server: 32 GB à €1.280,-
- 8. Upgrade Hardware: CPU Database Systems 2008 THOLIS CONSULTING 2004 2008
- 9. Upgrade Hardware: disk Database Systems 2008 THOLIS CONSULTING 2008: SSD disks + Access time 0,1 ms (vs. 4-5 ms SAS disk) + I/O *2-3 + Power consumption 10-20% of HDD + Noise level 0 db - Still very expensive: €2.300,- for 128 GB EMC Symmetrix: SSD as ‘ultra high performance’ option
- 10. Upgrade Software 64 bit OS is mandatory (32 bit max 4 GB) Oracle 11g: Cube Organized Materialized View (auto) Partitioning options Data Compression Information LifeCycle Management Hot Standby DB for real-time reporting SQL Server 2008: Partitioning Data Compression Win2008/SQL2008 doubles Win2003/SQL2005 performance! Database Systems 2008 THOLIS CONSULTING
- 11. Extend: Add datamart(s) Default in Inmon (CIF) architecture. Dimensional: Often DM’s as views on DWH Add OLAP engine (or replace RDMBS) for datamarts Add Appliance for datamarts Netezza started here (but is scaling up to EDW level) TeraData scales ‘down’ to this level as well Competitive ‘sweet spot’ for Appliance Vendors Use alternative solution (see also ‘Roll your own’) Database Systems 2008 THOLIS CONSULTING
- 12. Extend: ‘Buddy’ system ParAccel ‘Amigo’: Database Systems 2008 THOLIS CONSULTING Q-Router handles all requests: OLTP is executed on Database of Record Analytical query is executed on ParAccel MPP Grid
- 13. Extend: Datastore replacement DatAupia ‘Satori’: Database Systems 2008 THOLIS CONSULTING
- 14. Migrate: Appliances ‘ Traditional’ TeraData HP NeoView Kognitio DATAllegro GreenPlum Netezza Characteristics: Plug and play Combination of HW, SW, Support & Services Database Systems 2008 THOLIS CONSULTING
- 15. Migrate: ‘Roll your own’ Mostly column based, MPP, Shared Nothing architectures 1 Established Vendor: Sybase IQ, since 1993 Wide choice of closed and open source products: Open Source: LucidDB, MonetDB Software only: Vertica*, ParAccel*, Brighthouse, ExaSol, Valentina, VectorStar, Tenbase, Sand, etc # . Soft/hardware: Dataupia ‘ Lab’ware: Calpont Database Systems 2008 THOLIS CONSULTING *Also available as DWH Appliance #Mostly special purpose solutions, e.g. BigTable
- 16. Part 2: New Hi-Speed Solutions Since 2005, 4 new vendors on the market: Vertica (Michael Stonebraker, $25Mln funding) ParAccel (Barry Zane*, $20Mln funding) DatAupia (Foster Hinshaw*, $16Mln funding) InfoBright (Warsaw University, $8 Mln funding) Database Systems 2008 THOLIS CONSULTING * Netezza founders
- 17. What’s different? Massive Parallel Processing (MPP) Throw lots of commodity hardware at it (see ‘Upgrade’) Column based data organization Limit I/O by ‘pruning’ (compare horizontal partitioning) 1 datatype per column allows for heavy compression Data compression CPU is not the bottleneck, I/O is Read optimization In memory operation Database Systems 2008 THOLIS CONSULTING
- 18. SMP vs MPP Database Systems 2008 THOLIS CONSULTING Different storage approaches: Shared Disk (clustering) Shared Nothing All DWH appliance & new software vendors use Shared Nothing architecture
- 19. Rows vs Columns Nothing new about column storage: Taxir, 1969 Conceptual view: Database Systems 2008 THOLIS CONSULTING Rows Columns
- 20. Products: Vertica (1) Database Systems 2008 THOLIS CONSULTING Architecture: WOS & ROS Architecture: Columns & projections MPP Shared Nothing Column Storage Compression Read Optimized
- 21. Products: Vertica (2) Database Systems 2008 THOLIS CONSULTING
- 22. Products: ParAccel (1) Database Systems 2008 THOLIS CONSULTING Two implementation modes: Amigo* & Maverick Two versions: in memory & disk based (no hybrid solution yet) MPP Shared Nothing Compression Parallel loader *SQL Server only; Oracle version in Beta
- 23. Products: ParAccel (2) Database Systems 2008 THOLIS CONSULTING High availability built in: Shattered TPC-H benchmark: Appliance partnership with Sun: Phoenix all in memory DWH appliance Sedona disk based VLDB
- 24. Products: ExaSol Database Systems 2008 THOLIS CONSULTING MPP Column based Auto tuning In-Memory based In-Memory Compression ExaCluster OS
- 25. Products: BrightHouse (1) Uses MySQL as DBMS Not columns but 64K Data Packs Knowledge Grid and DP nodes replace traditional indexes Heavy Compression (10:1) Database Systems 2008 THOLIS CONSULTING
- 26. Products: BrightHouse (2) Database Systems 2008 THOLIS CONSULTING
- 27. Products: DatAupia Database Systems 2008 THOLIS CONSULTING Database Appliance Adds MPP capability to DB/2, Oracle & SQL ‘ Invisible’ appliance Lowest cost solution on the market Plug and play:
- 28. How Fast: TPC/H Benchmark Database Systems 2008 THOLIS CONSULTING Typical BI queries, e.g. Top 10 of non-shipped orders on date x Annual growth of marketshare Profit per producttype, year and country Profit share local suppliers etc.
- 29. Query Example: TPC-H Q9 -- $ID$ -- TPC-H/TPC-R Product Type Profit Measure Query (Q9) -- Functional Query Definition -- Approved February 1998 Select nation, o_year, sum(amount) as sum_profit from ( select n_name as nation, extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount from part, supplier, lineitem, partsupp, orders, nation where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey = l_orderke y and s_nationkey = n_nationkey and p_name like '%green%' ) as profit group by nation, o_year order by nation, o_year desc Database Systems 2008 THOLIS CONSULTING
- 30. Remember last year? Database Systems 2008 THOLIS CONSULTING Qry 1-10 on SF2 (2 GB data) Single CPU, Single disk, 2GB Ram, Windows 2003
- 31. How Fast: TPC-H 100 & 300GB Database Systems 2008 THOLIS CONSULTING
- 32. How Fast: TPC-H 1 TB Database Systems 2008 THOLIS CONSULTING But: So: Always verify your own workload against your own data!
- 33. Hi-Speed, what now? Upgrading hardware might be the most time- and cost effective short term solution to performance problems OK, this software is fast, but what about ETL/ELT ? (physical) Design? Maintenance ? Support ? When you hit the limits of your traditional DWH: Evaluate & Proof of Value When will Oracle, Microsoft and IBM enter this arena? Database Systems 2008 THOLIS CONSULTING
- 34. Database Systems 2008 THOLIS CONSULTING ?