As a Service: Cloud Foundry on OpenStack - Lessons Learnt
6 likes2,590 views
The document discusses the deployment of Cloud Foundry on OpenStack, highlighting lessons learned from various challenges faced, such as instability, API compliance, and resource capacity. It emphasizes the need for automation in setup, deployment, and maintenance to address common issues users encounter in integrating these platforms. Additionally, it advocates for offering Cloud Foundry and OpenStack as services to streamline updates and simplify operational complexity.
As a Service: Cloud Foundry on OpenStack - Lessons Learnt
- 1. As a Service: Cloud Foundry on OpenStack - Lessons Learnt Apps @AnimeshSingh OpenStack Summit Barcelona October 2016 @blueboxjesse
- 2. A polyglot “platform for the people” • The de facto open PaaS platform • Foundation established Dec. 2014, under Linux Foundation umbrella Cloud Foundry
- 3. Cloud Foundry Community 60+ member companies driving Cloud Foundry.
- 4. PaaS Cloud Foundry on OpenStack IaaS UAA Router DEA Pool Service Gateway Apps Service Connector Health Manager Messaging Cloud Controller Build Packs Service Nodes Cloud Foundry BOSH Cloud Provider Interface
- 5. BOSH Deployment Process Deployment Manifest • Release name/version • # VMs, job params • Stemcells to use Stemcell • Base OS • BOSH agent Release • Name • Software packages • Config templates • Scripts BOSH Cloud Foundry Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software Packages Virtual Machine • Configuration • Software packages
- 6. © IBM Corporation 6 Problems: Faced, by us and by the community!
- 7. Problems we faced As CF and OpenStack deployers we experienced various issues deploying CF and OpenStack, and CF on OpenStack § Instability: Instability of OpenStack environments from various distros § APIs: Lack of compliance in APIs and new releases. Some of the API behavior also differs based on the plugins used § Capacity: Right from cpu/mem/disk etc. to HA, persistent disk, floating ips etc. – sizing needs to be precise § Network: Should CF co reside with other management components in a network? § OpenStack for Enterprise Software: Enterprise software lags, with most variations still available only for VMware. § Generic Deployment: Consistency when deploying CF on OpenStack, VMware or any other IaaS § Combined OpenStack and Cloud Foundry Usage: How to allow seamless usage and consumption of OpenStack services along with CF services at the same time? § CF HA: What works, what doesn’t? § Proxy: Environments behind customer firewalls with restricted outbound access § Constant Release Cycles: Both CF and OpenStack have frequent releases, patches, updates etc. How do we reconcile?7
- 8. 50% experience significant issues deploying CF onto OS 45% are intermediate OS users Problems Community Faces: Survey Results
- 9. Close to 50% of users had to customize their environment for CF Problems Community Faces: Survey Results
- 10. Most users on Juno/Kilo Close to 50% of users run their own local OpenStack Problems Community Faces: Survey Results
- 11. © IBM Corporation 11 Our OpenStack and Cloud Foundry Offerings
- 12. APACHEHAPROXY APACHEHAPROXY HA MYSQL ARBITER (PERCONA) HA NEUTRON! HA KEYSTONE! HA GLANCE! HA HORIZON! NOVA! CINDER API & ISCSI! HA NEUTRON! HA KEYSTONE! HA GLANCE! HA HORIZON! NOVA! CINDER API & ISCSI! HA MYSQL (PERCONA) RABBITMQ HA MYSQL (PERCONA) RABBITMQ Our OpenStack Offering - Bluemix Private Cloud (IaaS) NOVA CINDER API & ISCSI NOVA CINDER API & ISCSI
- 13. § IBM Platform as a Services offering • Three deployments styles: public, dedicated, private • 1M+ registered users (20K+/month) • 100K+ running apps and 500+ services Bluemix Bare Metal (public and dedicated) and OpenStack/VMware (private) Our Cloud Foundry Offering - Bluemix (PaaS) Services Lifecycle Management IDS Application Runtime Runtimes & Frameworks Middleware Application Operational Mobile ExternalData Node Java Ruby Worklight WebSphere Liberty Eclipse IDE Application Composition Environment Create & Manage Services Test/Run Test/Run Explore Services Explore Services IBM Bluemix Check In Code Check In Code Web IDE (Eclipse Orion)
- 14. © IBM Corporation 14 Lessons Learnt deploying Cloud Foundry on OpenStack
- 15. Test if your OpenStack is fit for Cloud Foundry deployment Manually, you can run the validations here to see if your OpenStack is a good fit: https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html § Can you access the OpenStack APIs for your instance of OpenStack? § Can you access OpenStack metadata service from a virtual machine? § Can you ping one virtual machine from another? § Can you invoke large numbers of API calls? § Can you create and mount a large volume? § Can you upload and deploy an Ubuntu Server Cloud Image? § Can networking be configured for both external and internal IPs? § Can you access the Internet from within instances?
- 16. Test if your OpenStack is fit for Cloud Foundry deployment To check in automated fashion if your OpenStack is ready to run BOSH and Cloud Foundry, you can run this CF Incubator project : https://github.com/cloudfoundry-incubator/cf-openstack-validator 16 Testing the CPI API Upload stemcell Create VM Find VM Create disk Find disk Attach disk to VM Detach disk from VM Create disk snapshot Delete disk snapshot Delete disk Delete VM Delete stemcell Other OpenStack tests Check API rate limit Check required versions of OpenStack projects CPI requires API version 1 for glance and cinder Security group settings Check if security group rules allow necessary incoming/outgoing ports Outbound internet access Timeservers can be reached Attach a floating IP Set VM metadata tags Access a VM over ssh from the outside Access one VM from another VM Create a large volume
- 17. Get the right sizing for Cloud Foundry • Sample sizing are available online as references e.g https://docs.cloudfoundry.org/deploying/openstack/required-flavors.html https://www.mirantis.com/blog/full-stack-devops-with-pivotal-cloud-foundry-on-mirantis-openstack https://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html#versions • Increase default quota of the tenant to meet minimum requirement e.g number of ports always gets hit first • Recommended ‘OpenStack flavors’ disk space should be go in ephemeral disk. Keep root disk to 10GB minimum
- 18. Optimize OpenStack scheduling and communication Optimize Internal Communication: • Configure OpenStack for high volume API calls e.g Increase OpenStack API rate limits (/etc/nova/api-paste.ini) • Avoid name based security groups with nova-network. Name based security groups require message bus activity and database updates proportional to the number of existing VMs • If Neutron is configured with VXLAN via the Open vSwitch mechanism, the MTU should be 1400. For GRE, the recommended number is 1460. Use the right Scheduler Use an OpenStack scheduler which evenly distributes load e.g compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
- 19. Account for different OpenStack configurations • If your OpenStack setup requires you to use disks from block storage instead, that will work with Cloud Foundry as well. properties. openstack.boot_from_volume: true • By default, the VMs created try to receive data from OpenStack's HTTP metadata service. If your OpenStack installation doesn't provide metadata and userdata over HTTP, but requires you to use config-drive instead of metadata, you need to specify this in the property properties.openstack.config_drive: cdrom
- 20. Optimize Cloud Foundry / BOSH bandwidth and communication Optimize Internal Communication: • Increase BOSH NATS time out for high concurrency e.g Configure NATS messaging bus to increase ping interval Optimized routing and bandwidth allocation • Isolate Cloud Foundry components using multiple networks e.g DEAs in their own network, brokered services in their own network Optimize log forwarding • Supporting services like logging, report generation etc should go in their own tenant network. Also any communication between the VM(s) sending logs, and log receiving component(s) should go over the private network. Don’t use floating ips as destination, else you are paying double the cost.
- 21. Optimize for Security and Network • Only open ports which are needed. Use the most limited permissions required to complete the job. • If OpenStack is using a self-signed certificate, configure properties.openstack.connection_options to include the property ca_cert • Use tenant credentials: Do not use full admin credentials in your BOSH manifest • Minimize floating ips: Except for the incoming Gateway device(HA Proxy, Datapower, F5 etc), none of the fabric VMs should be on public network or need a floating ip
- 22. Account for customer boundary firewall or proxy • BOSH will retrieve CF release components via the URL provided in deployment manifest. Also services and CF apps may need outbound access. In environments behind firewall or proxy with restricted access, this could be a major issue. • If the firewall or proxy requires both destination and source ips to be enlisted, prepare a list of destination IPs/URLs you need to reach out and hand it to datacenter admin • If your OpenStack VMs don’t have a floating ip from external network, the source ip presented to firewall will be Neutron gateway ip • Cloud Foundry doesn’t support out of the box SSL packet inspection where the SSL certificate for a given sight is replaced by your own self signed certificate. Inspection is only supported using an Internet Authority signed certificate.
- 23. Work on seamless OpenStack update/upgrade OpenStack separates the availability of workloads (Data Place) from the availability of the API services (Control Plane) OpenStack Code Major Upgrade Maintenance: Moving from one OpenStack release to another, e.g. from Kilo to Mitaka with data migration • Infrequent, twice a year at most. New OpenStack code is put in place for all of the services that make up a cloud (e.g. Nova, Neutron, Glance, Heat, Cinder, Ceilometer, Aodh, Swift, etc..). • ~10 minutes for each service OpenStack Code Minor Upgrade Maintenance: Upgrading OpenStack service, without data migration • Typically only bug fixes. New OpenStack code is put in place and the services associated with that code are simply restarted. • ~5 seconds for each service OpenStack Config Maintenance: Changing the configuration for one or more of the OpenStack services, needed to correct bugs in the service, or to tune the service due to consumer requests. • Require the restart of a particular service. The disruption to the API control plane is isolated to the services receiving configuration updates. • ~ seconds for each service
- 25. OpenStack Installation: • Leverage the open source Ursula Ansible and Rally Cloud infrastructure Automation framework • Requires information about hardware, network environment and software repositories. Bluemix Private Cloud Install Automation Setup Storage Setup OpenStack Setup Network Run validation Setup Hardware Ursula, Rally-OpenStack
- 26. OpenStack Discovery: • Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner • Require OpenStack credentials and discover OpenStack compute and network information. Cloud Foundry on OpenStack deployment Automation Discover VM Configuration Sizes Discover Network Subnets Discover Network Security Rules Discover DHCP , DNS Gateway and floating IPs Discover Security Credentials
- 27. Cloud Foundry Pre-req setup on OpenStack: • Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner • Setup according to best practices and guidelines – still giving users the flexibility to change if desired Create Security Credentials Create VM configs for Router, DEAs, Cloud Controller, Service Nodes Create network Security Rules Setup tenant quota Cloud Foundry on OpenStack deployment Automation
- 28. Cloud Foundry Deployment Automation • Automate stemcell modification • Automate Cloud Foundry deployment manifest file genration using Ruby ERB • Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud Foundry deployment Stemcell Creation and Upload Generate BOSH and Cloud Foundry Manifest Upload Cloud Foundry core, Services and runtime Deploy Cloud Foundry Deploy bosh director RUBY BOSH Cloud Foundry on OpenStack deployment Automation
- 29. © IBM Corporation 29 As a Service: As a provider, look into offering Cloud Foundry and OpenStack as a Service
- 30. Why as a Service? Cloud Foundry: – New release every 2-3 weeks – Bluemix PaaS is a combination of CF and 150+ Services – Older versions will lead to huge version mismatches, and lead to version sprawl – Keeps Public/Dedicated and Local Bluemix in sync OpenStack: – Twice annual releases that touch the entire code base. – Upgrading sequentially is important: stay up to date! – OpenStack’s complexity requires expertise in many operational areas – Focus on higher business value. Work with OpenStack, not on OpenSTack
- 31. Private Cloud Hardware Bluemix Private Cloud: IaaS Relay: Box Panel Box Panel Site Controller (Software) Bluemix Private Cloud OpenStack Box Panel Formations Central Authentication Customer Relationship Management Service Catalog and Metering Billing and Invoicing Object Storage Block Storage Core Networking Inventory Management Network Management Reporting and Analytics Support Ticking, Chat and Email
- 32. IaaS Relay: Box Panel
- 33. IaaS Relay: Site Controller Box Panel Site Controller Customer Cloud Resides on-premises adjacent to customer clouds, providing real-time administrative control of cloud environments. – Network Automation – Power Distribution Unit Automation – System and Network: • Monitoring • Telemetry • Logging – Secure Remote Control and Access – Bare Metal Provisioning – Package and Container Repo
- 34. SDN router The Internet Customer Network IBM Urban Code Deploy Softlayer Server Bluemix Platform Stemcells Releases Manifests BOSH CLI Automated Management Processes (Deploy, Upgrade etc.) IBM Urban Code Deploy Relay Customer Hardware & Infrastructure Bluemix Core Services • Monitoring & Logging • Cache • Cloudant (Data Store) • Qradar EPS • IEM Relay Configuration Store (per customer) Bluemix Code & Automation Repository Opensource Code IBM Test & Staging Validation IBM Production Deployment &Validation Bluemix Local Inception VM UCD Agent • Secure connection • Connection originated from customer premise • Restricted access (agent- only) Cloud Foundry ACE UI Enterprise ITSM Customer Services Customer Premises Premises On-premise data store for logs, monitoring data etc. Bluemix Ops Console IBM Cloud Security Services Qradar Console IEM Server Bluemix Ops Directory Server Privileged ID Governance IPSec Tunnel DataPower • Customer’s Service traffic • Syndicated & 3rd party service traffic • App staging artifacts • Inbound & outbound user to app traffic • LDAP • Enterprise services • Other SaaS services • vCenter Network IsolationCustomer Services Bluemix Private Cloud PaaS Relay IBMPremises
- 35. PaaS Relay: Operations Console
- 36. PaaS Relay: Operations Console
- 37. VPN Tunnel Inception VM Stemcells Releases Manifests BOSH CLI DataPower ACE UI Metering Admin UI NATS BM DB Backup Login server UAA CC Blobsto re HM CCDB Loggre gator Go router DEAs UAADB Logging UCD AgentOps Center Agent VPN Tunnel Bluemix Private Cloud: PaaS and IaaS As a Service! Bluemix Relay Bluemix Private Cloud Relay Site Controller Repo / Deploy Server Monitoring Server (Sensu) Logging (ELK) Bastion (Access)
- 38. Cloud Foundry on OpenStack – A Great Fit! • 100% Open PaaS and IaaS solutions – No vendor lock-ins • Strong and growing community of contributors and sponsors on both sides • Power of Open Source community can be leveraged to automate the deployment and lifecycle management of Cloud Foundry on OpenStack • OpenStack meets Cloud Foundry integration requirements, and is totally configurable and adaptable to handle the scale of a PaaS solution like Cloud Foundry • Bottom Line: It’s a match made in Heaven !