Greenplum feature
3 likes1,667 views
This document provides an overview of Greenplum Database, a massively parallel processing (MPP) database developed by EMC. It discusses Greenplum's performance capabilities such as its ability to scale linearly through parallel query processing. Additional sections cover Greenplum's components, including its parallel query optimizer and gNet interconnect. The document also summarizes Greenplum's data loading, storage, analytics, manageability and high availability features.
Greenplum feature
- 1. Greenplum Database Overview Michael Crutcher Greenplum Product Management © Copyright 2012 EMC Corporation. All rights reserved. 1
- 2. © Copyright 2012 EMC Corporation. All rights reserved. 2
- 3. © Copyright 2012 EMC Corporation. All rights reserved. 3
- 4. © Copyright 2012 EMC Corporation. All rights reserved. 4
- 5. Greenplum Unified Analytic Platform © Copyright 2012 EMC Corporation. All rights reserved. 5
- 6. GREENPLUM DATABASE Industry Leading Database with Massively Parallel Performance To Empower your Analytics © Copyright 2012 EMC Corporation. All rights reserved. 6
- 7. GREENPLUM DATABASE Extreme Performance for Analytics Optimized for BI and analytics – Deep integration with statistical packages – High performance parallel implementations • Simple and automatic – Just load and query like any database – Tables are automatically distributed across nodes • Extremely scalable – MPP shared-nothing architecture – All nodes can scan and process in parallel – Linear scalability by adding nodes © Copyright 2012 EMC Corporation. All rights reserved. 7
- 8. GREENPLUM DATABASE Performance Through Parallelism Master Servers ... ... Query planning & dispatch Network Interconnect Segment Servers ... ... Query processing & data storage External Sources Loading, streaming, etc. © Copyright 2012 EMC Corporation. All rights reserved. 8
- 9. GREENPLUM DATABASE Greenplum Delivers Choice & Flexibility Greenplum Data Computing Appliance Greenplum Software Solutions Choose Greenplum Database and/or Hadoop modules in ¼ rack increments Greenplum Database, Hadoop, & Chorus on your x86 hardware Scale up by adding your choice of additional modules Flexibility for any workload or environment Minimal time to value Perpetual or subscription licenses © Copyright 2012 EMC Corporation. All rights reserved. 9
- 10. Core Functionality GREENPLUM DATABASE © Copyright 2012 EMC Corporation. All rights reserved. 10
- 11. GREENPLUM DATABASE Component Overview CLIENT ACCESS CLIENT ACCESS & TOOLS 3rd PARTY TOOLS ADMIN TOOLS ODBC, JDBC, OLEDB, BI Tools, ETL Tools Greenplum Command Center MapReduce, etc. Data Mining, etc Greenplum Package Manager LOADING & EXT. ACCESS LANGUAGE SUPPORT Petabyte-Scale Loading PRODUCT FEATURES STORAGE & DATA ACCESS Hybrid Storage & Execution (Row- & Column-Oriented) Comprehensive SQL Trickle Micro-Batching In-Database Compression Anywhere Data Access Native MapReduce SQL 2003 OLAP Extensions Multi-Level Partitioning Indexes – Btree, Bitmap, etc. Programmable Analytics External Table Support GREENPLUM DATABASE ADAPTIVE SERVICES CORE MPP ARCHITECTURE Multi-Level Fault Tolerance (RAID, Mirroring, DR with Data Domain Boost) Analytics Extensions (GeoSpatial, PR/R, PL/Java, PL/Python, PL/Perl) Online System Expansion Workload Management Shared-Nothing MPP Parallel Dataflow Engine Parallel Query Optimizer gNet™ Software Interconnect Polymorphic Data Storage™ Scatter/Gather Streaming™ Data Loading © Copyright 2012 EMC Corporation. All rights reserved. 11
- 12. GREENPLUM DATABASE Most Powerful Data Loading Capabilities Industry leading performance at 10+TB per-hour per-rack SINGLE RACK COMPARISON Scatter-Gather Streaming™ provides true linear scaling Support for both large-batch and continuous real-time loading strategies Enable complex data transformations ―in-flight‖ Transparent interfaces to loading via support files, application, and services © Copyright 2012 EMC Corporation. All rights reserved. Greenplum Oracle Exadata Netezza Teradata Greenplum load rates scale linearly with the number of racks, others do not. For example, two racks = >20TB/H 12
- 13. GREENPLUM DATABASE Polymorphic Table StorageTM TABLE ‗CUSTOMER‘ Mar ‗11 Apr ‗11 May ‗11 Jun ‗11 Jul ‗11 Aug ‗11 Column-oriented for COLD DATA Sept ‗11 Oct ‗11 Nov ‗11 Row-oriented for HOT DATA • Storage types can be mixed within a table or database – Four table types: heap, row-oriented AO, column-oriented AO, external • Rich compression functionality, definable column by column – Block compression: Gzip (levels 1-9), QuickLZ – Stream compression: RLE (levels 1-4) • Flexible indexing, partitioning, and more © Copyright 2012 EMC Corporation. All rights reserved. 13
- 14. GREENPLUM DATABASE gNet Software Interconnect A supercomputing-based ―soft-switch‖ responsible for – Efficiently pumping streams of data between motion nodes during query-plan execution – Delivers messages, moves data, collects results, and coordinates work among the segments in the system gNet Software Interconnect © Copyright 2012 EMC Corporation. All rights reserved. 14
- 15. GREENPLUM DATABASE Parallel Query Optimizer PHYSICAL EXECUTION PLAN FROM SQL OR MAPREDUCE Cost-based optimization looks for the most efficient plan Gather Motion 4:1(Slice 3) Sort Physical plan contains scans, joins, sorts, aggregations, etc. Global planning avoids sub-optimal ‘SQL pushing’ to segments Directly inserts ‘motion’ nodes for inter-segment communication © Copyright 2012 EMC Corporation. All rights reserved. HashAggregate HashJoin Redistribute Motion 4:4(Slice 1) Hash HashJoin HashJoin Seq Scan on lineitem Hash Seq Scan on orders Seq Scan on customer Hash Broadcast Motion 4:4(Slice 2) Seq Scan on motion 15
- 16. Analytics Overview GREENPLUM DATABASE © Copyright 2012 EMC Corporation. All rights reserved. 16
- 17. GREENPLUM DATABASE Analytical Capabilities Overview Data Access & Query Layer ODBC JDBC SQL Stored Procedures SQL 2003 OLAP MapReduce In-Database Analytics Polymorphic Storage GREENPLUM HD GREENPLUM DATABASE Greenplum gNet © Copyright 2012 EMC Corporation. All rights reserved. 17
- 18. GREENPLUM DATABASE In-Database Analytics: Categories Data Access & Query Layer ODBC JDBC SQL In-Database Analytics Embedded Partner Open-Source GPDB Embedded Analytics SAS Scoring Accelerator SAS/HPA High Performance Analytics Open Source Extensions User-Written Analytical Algorithms User-written GREENPLUM DATABASE © Copyright 2012 EMC Corporation. All rights reserved. 18
- 19. GREENPLUM DATABASE Analytics Highlight: MADlib Scalable in-database analytics Data-parallel – – – – Mathematical Algorithms Statistical Algorithms Machine learning Algorithms Supports structured and unstructured data. Open-source software – Source Accessibility – Converge business, academic, and open-source communities © Copyright 2012 EMC Corporation. All rights reserved. 19
- 20. Manageability, Extensions GREENPLUM DATABASE © Copyright 2012 EMC Corporation. All rights reserved. 20
- 21. GREENPLUM DATABASE Easy Manageability for Big Data Single console for both Database and Hadoop Administration – Start, Stop Database – Recover, Rebalance Segments Interactive view of System Metrics – Real-time – Historic (Configurable by time period) In-depth view for System Health – Hardware health – Software (Database, Hadoop) Query Monitoring – Search, Prioritize, Cancel Queries – View Query‘s Execution Plan Workload Management – Configure Resource Queues – Prioritize Users © Copyright 2012 EMC Corporation. All rights reserved. 21
- 22. GREENPLUM DATABASE Easy Extension Installation Greenplum Package Manager Greenplum supports easy deployment of numerous extensions like Madlib, PL/Perl, PL/Java, PostGIS, etc. Master Servers Segment Servers ... © Copyright 2012 EMC Corporation. All rights reserved. ... 22
- 23. GREENPLUM DATABASE High Performance gNet for Hadoop Parallel Query Access Connect any data set in Hadoop to GP DB‘s SQL Engine Process Hadoop data in place Parallelize import/export data from/to Hadoop thanks to GP DB‘s market leading data sharing performance gNet for Hadoop Text Binary UserDefined Supported formats: – Text (compressed and uncompressed) – binary – proprietary/user-defined GP HD 1.x, GP MR 1.x, CDH3u2 © Copyright 2012 EMC Corporation. All rights reserved. 23
- 24. High Availability, Back up, Support GREENPLUM DATABASE © Copyright 2012 EMC Corporation. All rights reserved. 24
- 25. GREENPLUM DATABASE High Availability GPDB cluster – 2 Master servers – Multiple Segment servers Segment servers support multiple database instances – Primary instances that actively process queries – Standby mirror instances Block level mirroring – Low resource consumption – Differential resynch capable for fast recovery © Copyright 2012 EMC Corporation. All rights reserved. Set of Active Segment Instances 25
- 26. GREENPLUM DATABASE Backup/Restore with EMC Data Domain Integration options Full Appliance + Data Domain Boost or NFS 2 X 10GBit IP – NFS: Data Domain device mounted as NFS storage – DD Boost: Native, client-side deduplication. Supported in GPDB 4.2 and higher Drastic reduction in backup storage requirement Backup all segment servers in parallel directly to Data Domain Data Domain Integrates seamlessly into standard Greenplum full backup data export and data restore procedures © Copyright 2012 EMC Corporation. All rights reserved. 26
- 27. GREENPLUM DATABASE Backup/Restore with EMC Data Domain Backup and restore between remote and primary sites Greenplum DCA Greenplum DCA Data Domain Data Domain LAN/WAN Data Domain Replication Ideal for configurations with RPO and RTO requirements that can be specified in hours Supports: – Collection Replication for DD Boost backup – Directory-level replication for NFS backup – Encryption over the WAN © Copyright 2012 EMC Corporation. All rights reserved. 27
- 28. GREENPLUM DATABASE Customer Support Services • Remote Technical Support – 24x7 technical support and remote troubleshooting – Customer-managed case severity level – Four-hour response objective • Onsite Support (DCA Only) – Installation of replacement parts – Replacement parts shipped for next business day arrival – GP SW upgrade included • Proactive Service – Secure remote monitoring for hardware (DCA) – Notification of engineering technical advisories – Built-in tools maximize stability and performance • Secure Self-Help – © Copyright 2012 EMC Corporation. All rights reserved. 24x7 access to eService support tools including knowledgebase, forums, and appropriately licensed software updates 28
- 29. GREENPLUM DATABASE Other Relevant Greenplum Sessions Session Presenter Times Unified Analytics Platform Introduction Brian Wilson Tues 10:00-11:00 Thurs 1:00-2:00 Greenplum Hadoop Overview Susheel Kaushik Mon 10:00-11:00 Wed 4:15-5:15 Greenplum DCA Overview Hanxi Chen Mon 4:00-5:00 Thurs 10:00-11:00 Greenplum Analytics Workbench Apurva Desai Wed 8:30-9:30 Thurs 10:00-11:00 Analytics on Hadoop Don Miner Tues 11:30-12:30 Thurs 8:30-9:30 Big Data Driven Businesses in Action: Creating Real Business Value Using Greenplum UAP (Panel w/4 Customers) Mike Maxey Wed 4:15-5:15 Thurs 11:30-12:30 Analytics for Business Value: Collaboration Josh Klahr Mon 10:00-11:00 Wed 2:45-3:45 Disruptive Data Science — How Data Science and Big Data are Transforming Business, IT and People Annika Jimenez David Dietrich Tues 4:15-5:15 Thurs 11:30-12:30 © Copyright 2012 EMC Corporation. All rights reserved. 29
- 30. Thank You © Copyright 2012 EMC Corporation. All rights reserved. 30