Learn How Dell Improved Postgres/Greenplum Performance 20x with a Database Proxy - Greenplum Summit 2019
3 likes634 views
The document discusses how Dell enhanced PostgreSQL/Greenplum performance by 20x using Heimdall data proxy to batch database operations and optimize workflows. It also details the evolution of Greenplum and Heimdall, highlighting their capabilities in managing complex datasets and hybrid transaction/analytical processing (HTAP). Additionally, it provides case studies on improvements in report generation and MVP product development by leveraging caching and batch processing solutions.
Learn How Dell Improved Postgres/Greenplum Performance 20x with a Database Proxy - Greenplum Summit 2019
- 1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved. Zack Odom - Field Engineer, Pivotal Erik Brandsberg - CTO, Heimdall Data Inc Learn How Dell Improved Postgres/Greenplum Performance 20x with a Database Proxy
- 2. The Fundamental Theorem of Software Engineering "We can solve any problem by introducing an extra level of indirection." originated by Andrew Koenig to describe a remark by Butler Lampson attributed to the late David J. Wheeler Question: What provides indirection for Databases?
- 3. Backgrounds Greenplum History ● Exists since PostgreSQL 7.4 - went live in 2005 ● Merged with PostgreSQL until 8.2, then forked ● Product evolved, company acquired ● Open Source in 2015 ● Greenplum getting closer to latest PostgreSQL with every new merged version ○ PG 8.3 is merged (GPDB Version V5) ○ PG 9.4 is WIP (GBDB Version 6) Heimdall History ● Founded in 2014 ● Advanced Partner with Pivotal ● AWS Competency Partner ● Database Vendor Neutral: ○ Postgres, SQL Server, MySQL, JDBC data sources
- 4. Pivotal Greenplum Powerful, Postgres-based MPP and multi-cloud analytics on petabyte-scale data Challenges • Legacy scale-up DBs are expensive to operate • Hadoop doesn’t fit low-latency, iterative analytics with high user concurrency • Multiple environments with messy, disjointed structured and unstructured data Greenplum Delivers • Multi-cloud, Open-source, analytics data platform • Massively parallel processing with machine learning and ANSI SQL compliance • Unify and query structured and unstructured data from native, HDFS, and cloud storage - including text, spatial, and graph data Benefits • Scales linearly with hardware for optimal cost and performance • Faster workflow; train models in parallel, publish to DB for rapid parallel scoring • Analyze more types of data more quickly for faster, deeper insights
- 5. Hadoop Data Lakes Massively Parallel Data Warehouse Public Cloud Data Lakes Predefined Libraries Programmatic GPText Massively Parallel Analytical Processing High Speed of Ingestion Pivotal Greenplum Massively Parallel Data Load from External Sources In-DB Predictive Analytics High Speed of Processing Massively Parallel Postgres Architecture <Postgres in Parallel>
- 6. Application Server Application Heimdall Data Driver/Proxy Application Servers SQL Auto-Caching Auto-invalidation Auto-Cache Refresh Automated Failover Load Balancing Read/Write Split Batch Processing OLTP/OLAP Routing Query Triggers Query Analytics, Transformation, & Firewall Connection Pooling Heimdall Architecture Application Server Application Heimdall Data Driver/Proxy Application Server Application Heimdall Database Proxy
- 7. OLAP VS OLTP Analytics-based (OLAP) ● High latency, many reads, less writes ● Bulk ETL operations, complex queries ● Calculate Results across very large datasets ● Most are purposely built to scale (and expand) to many nodes with replication and HA built in. ● Optimizer should evaluate best plan using statistics for more complex analytical queries ● SLA’s are not sub-second/minute ● Caching or materialized views typically is not leveraged due to inherent nature of deep/wide analytical queries Transactional-based (OLTP) ● Low Latency, memory intensive operations ● Singleton ETL operations including DML ● Typically targeted data retrieval ● Scale has limitations and expensive – single node for OTLP purposes (Postgres) ● Optimizer does not need to be intelligent as most queries are single threaded ● SLA’s are typically sub-second ● Caching utilized heavily for SLA
- 8. OLTP happens - on Analytical shared-nothing systems Many applications ported from Oracle, etc ● Greenplum will open and spawn many threads based on query type ● Singleton ops take up unnecessary threads exhausting finite RAM/CPU resources. ● Pooling agents can alleviate pressure on the Master – but throughput will be affected by number of resources used and operation type. ● Small, quick queries are not cached resulting in re-reads (lookup / dim tables, etc). ● Historically, applications need to be re-written to utilize batch loading operations – expensive! ● When combined, referred to as HTAP “Hybrid Transaction Analytical Processing”
- 9. HTAP Use Case: Dell, Inc. Problem: Legacy Apps with Singleton DML (Insert/Update/Delete) ● Existing infrastructure supported applications performing single inserts/update/deletes in volume ● Greenplum’s MPP Design has slow commit times for Singleton Inserts ● Customer desired to support DML without a redesign Solution: Heimdall Auto-Batching into Greenplum ● DML operations are isolated and batched by Heimdall ● Commits are performed over many operations, reducing overhead ● Exceptions are tracked by Heimdall for later analysis Result: DML Performance Increased by 20x, Meeting Requirements
- 10. 4 3 2 1 Application DML Request 6 5 4 3 2 1 Queue Batch Size 4 78 START TRANSACTION; DML 1; DML 2; DML 3; DML 4; COMMIT; Exceptions are logged, removed from batch, and transaction restarted Benefits: •Lower CPU overhead due to fewer commits •Improved application response time •Improved DML scale #1 #2 #3 #4 Asynchronous Batch Processing
- 11. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.© Copyright 2019 Pivotal Software, Inc. All rights Reserved. DEMO
- 12. Customer Example: CATL Problem: Slow Report Generation in Tableau ● Each report contained up to 30 queries, taking 30 seconds each ● Data was updated every two hours ● Reports viewed at random intervals by management Solution: Heimdall Auto-Refresh Caching into Gemfire ● Redundant queries were learned by Heimdall ● Via Stored Procedure after data load, Heimdall invalidates modified tables ● Query cache was refreshed from Greenplum into Gemfire by Heimdall Result: Average Report Generation Went From 17s to 3s
- 13. Auto-Refresh Caching Bulk Data Upload Invalidation SP (Or Trigger) Invalidation Event Initial Request & Response Initial Request & Response Cache Populated Query Tracker Application Caches (L1+L2) Data Source Cached Result Later Request & Cached Result Cache Invalidated Queries Reissued Cache Re- Populated Refresh Request & Response Auto-Refresh targets finite query pattern environments, i.e reporting and dashboard interfaces
- 14. Customer Example: Questis w/ Aurora for Postgres Problem: Productizing MVP (Minimally Viable Product) ● Development had focus on features, not performance ● No cache layer had been implemented during MVP development ● In use, many redundant queries were being performed Solution: Heimdall Caching Logic for Amazon Elasticache ● Reduced Database load by 90% ● Improved page generation time ● Auto-Invalidation gave peak cache efficiency without stale data Result: MVP Code was put into production without rewrites for caching and met customer SLA’s
- 15. Pivotal Greenplum: Learn More ● Find out more about Pivotal Greenplum and Heimdall at ○ https://pivotal.io/pivotal-greenplum ○ https://heimdalldata.com ● OR learn more about the open source Greenplum at ○ http://greenplum.org/ ● OR give a try: ○ Amazon AWS, Azure, Google GCP or Heimdall website ● Check for the Heimdall Q/A Deep Dive (Date TBD)
- 16. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.© Copyright 2019 Pivotal Software, Inc. All rights Reserved. Q & A