GPCloud ( GP on PKS)
0 likes556 views
The document is a presentation about running Greenplum on Pivotal Container Service (PKS). It discusses how PKS provides an enterprise-grade Kubernetes platform using BOSH for deployment, lifecycle management, and monitoring. It then outlines use cases for running Greenplum on PKS such as flexible sizing, automated testing, and advanced security/high availability in production. Finally, it discusses the roadmap for tighter integration between Greenplum and PKS capabilities like command center, backup/restore, and ecosystem partnerships.
![On Demand Greenplum Data
>$ ./gppks create-cluster 100 sales-dept
+ SEGMENT_COUNT=100
+ CLUSTER_NAME=gp-pks-sales-dept
+ ./generate_cluster_configuration.bash
+ DEST_TEMPLATE_DIR=./localbuild/templates
+ POD_FILE_PATH=./templates/pod.yml
+ GREENPLUM_SERVICE_PATH=./templates/service.yml
…
+ helm template -f Values-common.yml --set
KUBE_ENV=pks . -x ./templates/storageclass.yml
+ generateSimpleTemplate limit-range.yml
...
>$ kubectl exec -it master -- /bin/bash
gpadmin@master:~/tools$ ./initialize_cluster.py
...
gpinitsystem:master:gpadmin-[INFO]:-Greenplum Database
instance successfully created
1
2
3](https://image.slidesharecdn.com/5gponpks-public-seoul-180326084410/85/GPCloud-GP-on-PKS-15-320.jpg)
GPCloud ( GP on PKS)
- 1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Greenplum on Pivotal Container Service Oz Basarir - Product Manager
- 2. Safe Harbor This presentation contains statements relating to Pivotal’s expectations, projections, beliefs and prospects which are "forward-looking statements” about Pivotal’s future which by their nature are uncertain. Such forward-looking statements are not guarantees of future performance, and you are cautioned not to place undue reliance on these forward-looking statements. Actual results could differ materially from those projected in the forward-looking statements as a result of many factors, including but not limited to: (i) adverse changes in general economic or market conditions; (ii) delays or reductions in information technology spending; (iii) risks associated with managing the growth of Pivotal’s business, including operating costs; (iv) changes to Pivotal’s software business model; (v) competitive factors, including pricing pressures and new product introductions; (vi) Pivotal’s customers' ability to transition to new products and computing strategies such as cloud computing, the uncertainty of customer acceptance of emerging technologies, and rapid technological and market changes; (vii) Pivotal's ability to protect its proprietary technology; (viii) Pivotal’s ability to attract and retain highly qualified employees; (ix) Pivotal’s ability to execute on its plans and strategy; and (x) risks related to data and information security vulnerabilities. All information set forth in this presentation is current as of the date of this presentation. These forward-looking statements are based on current expectations and are subject to uncertainties and changes in condition, significance, value and effect as well as other risks disclosed previously and from time to time in documents filed by Dell Technologies Inc., the parent company of Pivotal, with the U.S. Securities and Exchange Commission. Dell and Pivotal assume no obligation to, and do not currently intend to, update any such forward-looking statements after the date of this presentation. The following is intended to outline the general direction of Pivotal's offerings. It is intended for information purposes only and may not be incorporated into any contract. Any information regarding pre-release of Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal and is subject to change. This information is provided without warranty or any kind, express or implied, and is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on features currently available. The development, release, and timing of any features or functionality described for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to update forward-looking information in this presentation.
- 3. Agenda ● IT Deployments in the new world of containers ○ Kubernetes ○ BOSH ○ PKS ● Greenplum on Pivotal Container Service (PKS) ○ Use Cases ○ Operational Benefits ○ Road Ahead
- 4. Run your analytics anywhere you need it • Infrastructure Agnostic: A portable, 100% software solution • Same platform, no switching/migration cost
- 5. Economics for the Customer “As we’ve transitioned from physical hardware to virtual machines to containers, we’ve reduced the cost per unit value of deployments, and that’s enabled us to create software platforms with increasing capability to deliver on the promise of rapid evolution of software. Containers have certainly played a huge role in the current evolution of platform building.” - Matt Stine, Author of Migrating to Cloud-Native Application Architectures
- 6. Patches Patching platform components with thousands of apps running should feel normal. Scaling Seamlessly scale platform components to accommodate changing demand. Upgrades. How do you roll out new versions of the platform with the lights on? Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits? Multi-cloud Provide a reliable and smooth experience for any cloud. Open APIs Allow platform operations from different toolsets and the creation of CD pipelines. Consistency Provide a consistent setup experience, across different cloud environment configurations. Setup time How long does it take to setup a real world working environment? Think hours, not weeks. Operational Goals Day 1 - Build Day 2 - Operate
- 7. “Run this containerized app for me. Let me tell you how.” Containerized workloads. Custom and ISV packaged apps and services delivered as containers Stateful services. Services using persistent storage such as Greenplum, MongoDB, Spark, Elastic Search, PostgreSQL Customization. Specify how your app is deployed and operated to optimize performance and reliability Ideal for packaged apps, apps/services exposing multiple ports and where finer grained control is needed Container Orchestrator
- 8. Kubernetes - especially hard to operationalize High Availability. No out-of-the-box fault- tolerance for the cluster components themselves (masters and etcd nodes). Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn’t provide a mechanism to scale Masters & etcd VMs. Health checks and healing. The Kubernetes cluster does routine health checks for the health of Nodes only. Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on? BOSH
- 9. PIVOTAL CLOUD FOUNDRY OPS Powered by BOSH BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. BOSH Packaging w/ embedded OS Server provisioning on any IaaS Software deployment across availability zones Health monitoring (server AND processes) Self-healing w/ Resurrector Storage management Rolling upgrades via canaries Easy scaling of clusters
- 10. Pivotal Container Service (PKS) A turnkey solution to provision, operate and manage enterprise grade Kubernetes clusters Kubernetes Dial Tone: • Health management • Aggregated Metrics and Logging • Autoscaling • Persistence interface Control Plane: • Provisioning Engine • T-shirt sized clusters • Self-service Clusters • Software Update Automation • Load balancing • Networking • Multi-tenancy + +
- 11. BOSH GCP Service Broker Harbor NSX-T Kubernetes K8s Cluster K8s Cluster K8s Cluster Built with open-source Kubernetes — Constant compatibility with the current stable release of Kubernetes, operated by BOSH. No proprietary extensions. Production-ready — Highly available from apps to infrastructure, no single points of failure. Built-in health checks, scaling, auto-healing and rolling upgrades. Multicloud — BOSH provides a reliable and consistent operational experience. For any cloud. Network management and security out-of-the-box with VMware NSX-T. Multi-cloud, multi-hypervisor. GCP APIs access — The GCP Service Broker allows apps to transparently access Google Cloud APIs, from anywhere. Easily move workloads to/from Google Container Engine (GKE). Fully automated Ops — Fully automated deploy, scale, patch, upgrade. No downtime. Use CD pipelines to deploy your platform, too. VMware GCP Azure Openstack AWS PKSController
- 13. Apps are constantly compatible between PKS and Google Container Engine (GKE) Google Container Engine (GKE) Pivotal Container Service (PKS) Private and Public Cloud Public Cloud (on GCP)
- 14. GPCloud
- 15. On Demand Greenplum Data >$ ./gppks create-cluster 100 sales-dept + SEGMENT_COUNT=100 + CLUSTER_NAME=gp-pks-sales-dept + ./generate_cluster_configuration.bash + DEST_TEMPLATE_DIR=./localbuild/templates + POD_FILE_PATH=./templates/pod.yml + GREENPLUM_SERVICE_PATH=./templates/service.yml … + helm template -f Values-common.yml --set KUBE_ENV=pks . -x ./templates/storageclass.yml + generateSimpleTemplate limit-range.yml ... >$ kubectl exec -it master -- /bin/bash gpadmin@master:~/tools$ ./initialize_cluster.py ... gpinitsystem:master:gpadmin-[INFO]:-Greenplum Database instance successfully created 1 2 3
- 16. Cloud Native Apps & Greenplum BOSH Other Broker Services Platform Services Logging Metrics Monitoring Elastic Runtime (ERT) VMware GCP Azure Openstack AWS Spring Boot App PKSController GCP Service Broker Harbor NSX-T Kubernetes Spring Boot App Greenplum ML, GeoSpatial, Graph, GPText, PXF PL/Python, PL/R SQL
- 17. Use Cases & Operational Benefits DEV Flexible Repeatable Cost-effective sizes TEST Automated Testing (CI) Manual Testing Quick Spin Up / Tear Down PROD DEPLOYED ON PIVOTAL CONTAINER SERVICE Advanced Security / High Availability / Operational Efficiency On Demand Clusters on Public or Private Clouds Data-Throughput-Tuned Recovery with Persistent Disks Compute & Storage separated HA & No Degraded Mode Pre-Hardened
- 18. Road Ahead ● Enterprise Greenplum on PKS ○ GP Command Center ○ GPText ○ Backup / Restore ○ PXF ● No-downtime Patches ● Ecosystem: Tighter integration with BI, ETL, Big Data
- 20. Appendix
- 21. Successful GPDB deployments on K8s/PKS sdw1 Container Pod mdw Container Pod sdwN Container Pod smdw Container Pod sdw1 Container Pod sdwN Container Pod Mirrors Primaries Mirror becomes primary And no need for rebalancing. ● Storage runs on persistent disks ● gpinitsystem -I to set up this cluster configuration. ● gprecoverseg works! Failed primary becomes mirror when recovered. ● No degraded mode Standby Master mdw Container Pod sdw1 sdwN Sandbox
- 22. Security (disable user - >gpadmin) Alan makes PKS available Alan creates a cluster Patrick manages Greenplum Alan / Patrick deploys Greenplum Personas and Workflows Patrick manages Greenplum gpinitsystem gpactivatestandby gprecoverseg gpexpand Set unsafe sysctl Upgrade minor version of GPDB Upgrade major version of GPDB Support Utility Custom host names Gpcheckperf /benchmark Deployment Plans (S,M,L) enabled: planned: Backup / Restore with PVs Recommended OS settings other than unsafe sysctl Scale Compute and Storage independently Anti-affinity for storage PL/* (and PL/Container), GPText, Madlib, PXF, GPCC/WLM/Resource Groups, Utilities, …
- 24. NSX-T On GCP / GKE, there are a number of options to handle Kubernetes NetworkPolicy. On vSphere, there is nothing. NSX-T is the solution. Without NSX-T, Kubernetes assigns IPs to each node in the cluster. No concept of subnets, no way to create true multi-tenancy. NSX-T enables this multi-cloud creation. Disclaimer: If you want true isolation, you need to build separate hardware clusters. But, NSX-T comes close and makes it easy for large enterprises to build this abstraction layer to create software-defined clusters at will.
- 25. What is K8s? WorkerK WorkerK WorkerK API K8s Cluster Services Deployment Pod1 - contimg1 - contimg2 - replicas: 3 Pod2 - contimg3 - replicas: 2 App1.yml K8s enforces “Desired State Mgmt”, meaning that we can feed configuration and K8s Cluster Services will be responsible for running it. P1R1 P1R2 P2R1 P1R2 P1R3 P2R2 If Worker dies, P1R2 gets moved Kubelet process responsible for communicating with Cluster Services Container Host K8s Architecture
- 26. CONTAINERS EVENT-DRIVEN FUNCTIONS DATA SERVICES MICROSERVICES Batches MONOLITHIC APPLICATIONS Companies have many ways to package and run their workloads in the cloud
- 27. Their goal: pick the right runtime for each workload CONTAINERS EVENT-DRIVEN FUNCTIONS DATA SERVICESMICROSERVICES Batches MONOLITHIC APPLICATIONS IaaS Container Orchestrator (CaaS) Application Platform (PaaS) Serverless Functions (FaaS)
- 28. Container Orchestrator Container Scheduling Primitives for Network, Routing, Logs & Metrics CONTAINER Choose the right tool for the job Developer Provides Tool Provides Application Platform APPLICATION Container Orchestrator Serverless Functions FUNCTION Application Platform IaaS Container Image & build L7 Network & Routing Logs, Metrics, Monitoring Services Marketplace Team, Quotas & Usage Function scheduling Function exec services Container Orchestrator
- 29. Container Orchestrator Container Scheduling Primitives for Network, Routing, Logs & Metrics CONTAINER Choose the right tool for the job Developer Provides Tool Provides Application Platform APPLICATION Container Orchestrator Serverless Functions FUNCTION IaaS Container Image & build L7 Network & Routing Logs, Metrics, Monitoring Services Marketplace Team, Quotas & Usage Application Platform Container Orchestrator Function scheduling Function exec services Lower complexity, higher efficiency Higher flexibility, lower standardization
- 30. Hardware IaaS Container Orchestrator Application Platform Serverless Functions Strategic goal: Push as many workloads as technically feasible to the top of the platform hierarchy Higher flexibility and less enforcement of standards Lower development complexity and higher operational efficiency
- 31. VMware PKS Analytics Automation SecurityOperations MonitoringLogging Physical Infrastructure Container Registry vSphere vSAN Kubernetes on BOSH (Kubo) NSX BOSH GCP Service Broker masteretcd workermasteretcd worker