SlideShare a Scribd company logo
Migration of an Enterprise UI Microservice System
from Cloud Foundry to Kubernetes
Tony Erwin, IBM
Jonathan Schweikhart, IBM
Agenda
• Overview of IBM Cloud Console Architecture
• What is Cloud Foundry? What is Kubernetes? Why Switch?
• Experiences And Lessons Learned During Migration
• Conclusion
Overview of IBM Cloud
Console Architecture
IBM Cloud Console
• Large UI serving as front-end to the IBM Cloud
• Lets users create, view, and manage
PaaS/IaaS resources:
– Cloud Foundry apps & services
– Kubernetes clusters
– Virtual servers
– Bare metal
• Provides additional functionality for:
– Registration/onboarding
– Identity and Access Management (IAM)
– Billing/usage
– Docs
IBM Cloud Console Architecture
• Started life about 5
years ago as a
monolithic Java
app
• Now composed of
about 40 Node.js,
cloud-native
microservices +
more than 20
external plugins
• Originally deployed
as apps to Cloud
Foundry
• Currently deployed
as containers on
Kubernetes
uService 1 uService 2 … uService nuService n-1
Backend APIs (CF, Containers, VMs, IAM, Billing/Usage, etc.)
Console Client
Proxy
Core Deployment
Watson IoT
Funcs Clusters
Mobile …
(External Plugins)
What is Cloud Foundry?
What is Kubernetes?
Why Switch?
What is Cloud Foundry*?
• Provides a PaaS with an abstraction at the
application level
– Developers can focus on code rather than
underlying infrastructure
• Leverages the Open Service Broker API to
make it easy to use services from apps
• Manages apps as Diego containers
(internally)
* Technically describing the Cloud Foundry Application Runtime which is one of the two open source components from the CF Foundation.
What is Kubernetes?
• Abstracts at the container level
• Provides many of the benefits of PaaS with
the flexibility of IaaS
– Often referred to as IaaS+
• Orchestrates computing, networking, and
storage infrastructure on behalf of user
workloads
• Enables portability across infrastructure
providers
Why Did We Switch?
• Nothing “wrong” with CF
– Very easy to get apps running, relatively low learning curve, etc.
– Used in some way by at least half of the Fortune 500
• Kubernetes offers several advantages for our use case
– More granular control to better manage our large, complex microservice system
– Dedicated clusters to avoid performance/availability problems from friendly fire
• In fairness, CF can be installed in a dedicated manner as well (even on Kubernetes!)
– Simpler “front door” stack with built-in Ingress proxy to avoid extra network hops
– Private host names
• All apps in CF have public host names, so not possible to have a “private” microservice
– Private networking
• Calls between microservices in CF require going out over the public internet
– Improved memory and CPU usage (dynamic allocation)
– Ability to run our own services (like Redis)
– Integrated monitoring with Prometheus
Experiences And Lessons
Learned During Migration
Need to Dockerize
Node
Application
cf push
Diego
Container
Node
Application
helm install
Docker
Image
CF Flow
Kube Flow
Migrating Manifest to Helm
• Helm - Deployment
– Docker image
– CPU & memory
– Environment variables
• Helm – Service
– Single alias for the deployment
• Helm – Ingress
– Hostname/URL mapping to service
Deployment Configuration
• Cloud Foundry
– Configuration per deployment environment
• Kubernetes
– Helm cli makes hierarchical simple
– Global
– Global-<Environment>
– Cluster
– Cluster-<namespace>
Exposure of Microservices
• Cloud Foundry
– Public URL per microservice
– Each microservice has to protect against direct access
• Security concerns
• Common code repeated
• Kubernetes
– Microservice gets to choose exposure
• Service – Allows an internal only route to the application
• Ingress – Allows external routes to be defined to map to Services
– Protections take place at a higher level to allow
microservices to ignore exposure issues
Common Code Migration Problems
• Cloud Foundry assumptions
– Environment variable assumptions
• VCAP_SERVICES
• PORT
• Invalid OS name characters like hyphens
– URL format for intra-microservice communication
• CF: https://ace-common-production.us-south.bluemix.net
• Kubernetes: http://common
• URL construction vs URL variables
Installing a Local Redis with Stateful Sets
Redis
1
Redis
2
Redis
3
Cloud Foundry Kubernetes
Redis
1
Worker Node 1
Redis
2
Worker Node 2
Redis
3
Worker Node 3
Worker Nodes
Kubernetes Cluster
Worker Nodes
cadvisor
Nginx-logger
Prometheus
Monitoring in Kubernetes
Worker Nodes
Kubernetes Cluster
Worker Nodes
cadvisor
Nginx-logger
Prometheus
Worker Nodes
Kubernetes Cluster
Prometheus
• CPU
• Memory
• Network
• File system
• Status
Filter
Monitoring NGINX Ingress
• Nginx logs contain invaluable metrics about incoming calls
– Timestamp
– HTTP method
– HTTP status codes
– Headers
– URI
– Response time
• Implemented custom solution for accessing those metrics
– Configure nginx to log to syslog
– Create microservice that scrapes the syslog and exposes the data
to Prometheus
– Filter, monitor, and alert
Red/Black Deployments
Live URL Ondeck URL
Red Ingress Black Ingress
Proxy
Ingress
Live URL Ondeck URL
Red Ingress Black Ingress
Proxy
Ingress
Built-in Liveness/Readiness Checks
• /readiness
– ”I am ready to accept traffic”
– One time initialization checks
• Connections to resources (URLs, DBs, etc..)
– Periodic checks
• Circuit breakers
• Current status
• Content Throttling
• /liveness
– “I should keep living“
– Unrecoverable situations/Unexpected Failures
– “Have you tried turning it off and on again?”
Rolling Out Kubernetes
Geo Load Balancing and Failover (CF)
• One global URL
(https://console.bluemix.net)
• Use Dyn geo load balancing to
serve UI from the nearest
healthy region
• If healthcheck in a region shows
a problem, Dyn routes to the
next closest healthy region
• Odds of all regions being down
at the same time much less
than one region being down
• Reduces regional latency
Geo Load Balancing and Failover (Migration)
• Needed to verify stability
of Kube clusters before
turning off CF
deployments in production
• Solution: Add Kube
clusters to Dyn rotation
and run CF deployments
side-by-side with Kube
deployments
Geo Load Balancing and Failover (Final)
• Once satisfied,
removed CF
deployments from
rotation and only
Kube deployments
remained
Conclusion
Conclusion
• CF is a great technology, but Kubernetes better
meets the needs of our microservice system
• Nothing is free, and we had to solve several new
problems along the way
• Allowed us to achieve greater performance,
scalability, reliability, and security than we had
before
Questions?
• Tony Erwin
– Email: aerwin@us.ibm.com
– Twitter: @tonyerwin
• Jonathan Schweikhart
– Email: jschweik@us.ibm.com
The End

More Related Content

PPTX
Migrating Enterprise Microservices From Cloud Foundry to Kubernetes
PPTX
Kube journey 2017-04-19
PPTX
Introduction to container mangement
PPTX
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
PDF
Creating a Kubernetes Operator in Java
PDF
Make Java Microservices Resilient with Istio - Mangesh - IBM - CC18
PDF
[Lakmal] Automate Microservice to API
PPTX
Container orchestration overview
Migrating Enterprise Microservices From Cloud Foundry to Kubernetes
Kube journey 2017-04-19
Introduction to container mangement
Evolution of the IBM Cloud Console: From Monolith to Microservices and Beyond
Creating a Kubernetes Operator in Java
Make Java Microservices Resilient with Istio - Mangesh - IBM - CC18
[Lakmal] Automate Microservice to API
Container orchestration overview

What's hot (20)

PDF
Build your First IoT Application with IBM Watson IoT
PPTX
Developing Enterprise Applications for the Cloud, from Monolith to Microservices
PDF
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
PDF
K8s vs Cloud Foundry
ODP
Using Red Hat JBoss Fuse on OpenShift
PPTX
Introduction to Microservices and Cloud Native Application Architecture
PDF
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
PPTX
Container Patterns
PDF
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber
PPTX
Kubernetes on OpenStack @eBay
PDF
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
PDF
Everything You Need to Know About Docker and Storage by Ryan Wallner, ClusterHQ
PPTX
Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...
KEY
20120317 CloudFoundry #pyfes
PDF
introduction to micro services
PDF
Confluent Operations Training for Apache Kafka
PPTX
PDF
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
PDF
Spring cloud
PPTX
MuleSoft Deployment Strategies (RTF vs Hybrid vs CloudHub)
Build your First IoT Application with IBM Watson IoT
Developing Enterprise Applications for the Cloud, from Monolith to Microservices
Streaming with Spring Cloud Stream and Apache Kafka - Soby Chacko
K8s vs Cloud Foundry
Using Red Hat JBoss Fuse on OpenShift
Introduction to Microservices and Cloud Native Application Architecture
Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...
Container Patterns
WSO2Con USA 2017: Scalable Real-time Complex Event Processing at Uber
Kubernetes on OpenStack @eBay
The Good, the Bad and the Ugly of Migrating Hundreds of Legacy Applications ...
Everything You Need to Know About Docker and Storage by Ryan Wallner, ClusterHQ
Encrypting Kafka messages at rest to secure applications | Robert Barnes, Has...
20120317 CloudFoundry #pyfes
introduction to micro services
Confluent Operations Training for Apache Kafka
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
Spring cloud
MuleSoft Deployment Strategies (RTF vs Hybrid vs CloudHub)
Ad

Similar to Migration of an Enterprise UI Microservice System from Cloud Foundry to Kubernetes (20)

PPTX
20191201 kubernetes managed weblogic revival - part 1
PDF
The Kubernetes WebLogic revival (part 1)
PDF
Building a PaaS Platform like Bluemix on OpenStack
PPT
JDD 2016 - Jacek Bukowski - "Flying To Clouds" - Can It Be Easy?
PPT
Flying to clouds - can it be easy? Cloud Native Applications
PDF
To Kill a Monolith: Slaying the Demons of a Monolith with Node.js Microservic...
PDF
VMworld 2013: Three Advantages of Running Cloud Foundry in a VMware Private C...
PPTX
Netflix0SS Services on Docker
PPTX
Ibm cloud nativenetflixossfinal
PPTX
Introduction to Kubernetes
PPTX
Microservices and Best Practices
PDF
To Kill a Monolith: Slaying the Demons of a Monolith with Node.js Microservic...
PDF
Cloudstack for beginners
PPTX
.NET microservices with Azure Service Fabric
PDF
Cloud stack for_beginners
PPTX
NetflixOSS for Triangle Devops Oct 2013
PPTX
Migrating Java EE applications to IBM Bluemix Platform-as-a-Service
PPTX
Un-clouding the cloud
PDF
Migrating Java EE applications to IBM Bluemix platform as-a-service (CloudFou...
PDF
Presentation citrix cloud platform for infrastructure as a service
20191201 kubernetes managed weblogic revival - part 1
The Kubernetes WebLogic revival (part 1)
Building a PaaS Platform like Bluemix on OpenStack
JDD 2016 - Jacek Bukowski - "Flying To Clouds" - Can It Be Easy?
Flying to clouds - can it be easy? Cloud Native Applications
To Kill a Monolith: Slaying the Demons of a Monolith with Node.js Microservic...
VMworld 2013: Three Advantages of Running Cloud Foundry in a VMware Private C...
Netflix0SS Services on Docker
Ibm cloud nativenetflixossfinal
Introduction to Kubernetes
Microservices and Best Practices
To Kill a Monolith: Slaying the Demons of a Monolith with Node.js Microservic...
Cloudstack for beginners
.NET microservices with Azure Service Fabric
Cloud stack for_beginners
NetflixOSS for Triangle Devops Oct 2013
Migrating Java EE applications to IBM Bluemix Platform-as-a-Service
Un-clouding the cloud
Migrating Java EE applications to IBM Bluemix platform as-a-service (CloudFou...
Presentation citrix cloud platform for infrastructure as a service
Ad

Recently uploaded (20)

PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
PDF
Design an Analysis of Algorithms II-SECS-1021-03
PDF
Upgrade and Innovation Strategies for SAP ERP Customers
PDF
Which alternative to Crystal Reports is best for small or large businesses.pdf
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 41
PDF
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
PDF
Design an Analysis of Algorithms I-SECS-1021-03
DOCX
The Five Best AI Cover Tools in 2025.docx
PDF
Adobe Premiere Pro 2025 (v24.5.0.057) Crack free
PPTX
ManageIQ - Sprint 268 Review - Slide Deck
PDF
Softaken Excel to vCard Converter Software.pdf
PPTX
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
PDF
Adobe Illustrator 28.6 Crack My Vision of Vector Design
PPTX
Online Work Permit System for Fast Permit Processing
PDF
Digital Strategies for Manufacturing Companies
PDF
AI in Product Development-omnex systems
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
PDF
How to Choose the Right IT Partner for Your Business in Malaysia
PDF
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
PPTX
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Design an Analysis of Algorithms II-SECS-1021-03
Upgrade and Innovation Strategies for SAP ERP Customers
Which alternative to Crystal Reports is best for small or large businesses.pdf
Internet Downloader Manager (IDM) Crack 6.42 Build 41
Flood Susceptibility Mapping Using Image-Based 2D-CNN Deep Learnin. Overview ...
Design an Analysis of Algorithms I-SECS-1021-03
The Five Best AI Cover Tools in 2025.docx
Adobe Premiere Pro 2025 (v24.5.0.057) Crack free
ManageIQ - Sprint 268 Review - Slide Deck
Softaken Excel to vCard Converter Software.pdf
Lecture 3: Operating Systems Introduction to Computer Hardware Systems
Adobe Illustrator 28.6 Crack My Vision of Vector Design
Online Work Permit System for Fast Permit Processing
Digital Strategies for Manufacturing Companies
AI in Product Development-omnex systems
VVF-Customer-Presentation2025-Ver1.9.pptx
How to Choose the Right IT Partner for Your Business in Malaysia
T3DD25 TYPO3 Content Blocks - Deep Dive by André Kraus
CHAPTER 12 - CYBER SECURITY AND FUTURE SKILLS (1) (1).pptx

Migration of an Enterprise UI Microservice System from Cloud Foundry to Kubernetes

  • 1. Migration of an Enterprise UI Microservice System from Cloud Foundry to Kubernetes Tony Erwin, IBM Jonathan Schweikhart, IBM
  • 2. Agenda • Overview of IBM Cloud Console Architecture • What is Cloud Foundry? What is Kubernetes? Why Switch? • Experiences And Lessons Learned During Migration • Conclusion
  • 3. Overview of IBM Cloud Console Architecture
  • 4. IBM Cloud Console • Large UI serving as front-end to the IBM Cloud • Lets users create, view, and manage PaaS/IaaS resources: – Cloud Foundry apps & services – Kubernetes clusters – Virtual servers – Bare metal • Provides additional functionality for: – Registration/onboarding – Identity and Access Management (IAM) – Billing/usage – Docs
  • 5. IBM Cloud Console Architecture • Started life about 5 years ago as a monolithic Java app • Now composed of about 40 Node.js, cloud-native microservices + more than 20 external plugins • Originally deployed as apps to Cloud Foundry • Currently deployed as containers on Kubernetes uService 1 uService 2 … uService nuService n-1 Backend APIs (CF, Containers, VMs, IAM, Billing/Usage, etc.) Console Client Proxy Core Deployment Watson IoT Funcs Clusters Mobile … (External Plugins)
  • 6. What is Cloud Foundry? What is Kubernetes? Why Switch?
  • 7. What is Cloud Foundry*? • Provides a PaaS with an abstraction at the application level – Developers can focus on code rather than underlying infrastructure • Leverages the Open Service Broker API to make it easy to use services from apps • Manages apps as Diego containers (internally) * Technically describing the Cloud Foundry Application Runtime which is one of the two open source components from the CF Foundation.
  • 8. What is Kubernetes? • Abstracts at the container level • Provides many of the benefits of PaaS with the flexibility of IaaS – Often referred to as IaaS+ • Orchestrates computing, networking, and storage infrastructure on behalf of user workloads • Enables portability across infrastructure providers
  • 9. Why Did We Switch? • Nothing “wrong” with CF – Very easy to get apps running, relatively low learning curve, etc. – Used in some way by at least half of the Fortune 500 • Kubernetes offers several advantages for our use case – More granular control to better manage our large, complex microservice system – Dedicated clusters to avoid performance/availability problems from friendly fire • In fairness, CF can be installed in a dedicated manner as well (even on Kubernetes!) – Simpler “front door” stack with built-in Ingress proxy to avoid extra network hops – Private host names • All apps in CF have public host names, so not possible to have a “private” microservice – Private networking • Calls between microservices in CF require going out over the public internet – Improved memory and CPU usage (dynamic allocation) – Ability to run our own services (like Redis) – Integrated monitoring with Prometheus
  • 11. Need to Dockerize Node Application cf push Diego Container Node Application helm install Docker Image CF Flow Kube Flow
  • 12. Migrating Manifest to Helm • Helm - Deployment – Docker image – CPU & memory – Environment variables • Helm – Service – Single alias for the deployment • Helm – Ingress – Hostname/URL mapping to service
  • 13. Deployment Configuration • Cloud Foundry – Configuration per deployment environment • Kubernetes – Helm cli makes hierarchical simple – Global – Global-<Environment> – Cluster – Cluster-<namespace>
  • 14. Exposure of Microservices • Cloud Foundry – Public URL per microservice – Each microservice has to protect against direct access • Security concerns • Common code repeated • Kubernetes – Microservice gets to choose exposure • Service – Allows an internal only route to the application • Ingress – Allows external routes to be defined to map to Services – Protections take place at a higher level to allow microservices to ignore exposure issues
  • 15. Common Code Migration Problems • Cloud Foundry assumptions – Environment variable assumptions • VCAP_SERVICES • PORT • Invalid OS name characters like hyphens – URL format for intra-microservice communication • CF: https://ace-common-production.us-south.bluemix.net • Kubernetes: http://common • URL construction vs URL variables
  • 16. Installing a Local Redis with Stateful Sets Redis 1 Redis 2 Redis 3 Cloud Foundry Kubernetes Redis 1 Worker Node 1 Redis 2 Worker Node 2 Redis 3 Worker Node 3
  • 17. Worker Nodes Kubernetes Cluster Worker Nodes cadvisor Nginx-logger Prometheus Monitoring in Kubernetes Worker Nodes Kubernetes Cluster Worker Nodes cadvisor Nginx-logger Prometheus Worker Nodes Kubernetes Cluster Prometheus • CPU • Memory • Network • File system • Status Filter
  • 18. Monitoring NGINX Ingress • Nginx logs contain invaluable metrics about incoming calls – Timestamp – HTTP method – HTTP status codes – Headers – URI – Response time • Implemented custom solution for accessing those metrics – Configure nginx to log to syslog – Create microservice that scrapes the syslog and exposes the data to Prometheus – Filter, monitor, and alert
  • 19. Red/Black Deployments Live URL Ondeck URL Red Ingress Black Ingress Proxy Ingress Live URL Ondeck URL Red Ingress Black Ingress Proxy Ingress
  • 20. Built-in Liveness/Readiness Checks • /readiness – ”I am ready to accept traffic” – One time initialization checks • Connections to resources (URLs, DBs, etc..) – Periodic checks • Circuit breakers • Current status • Content Throttling • /liveness – “I should keep living“ – Unrecoverable situations/Unexpected Failures – “Have you tried turning it off and on again?”
  • 22. Geo Load Balancing and Failover (CF) • One global URL (https://console.bluemix.net) • Use Dyn geo load balancing to serve UI from the nearest healthy region • If healthcheck in a region shows a problem, Dyn routes to the next closest healthy region • Odds of all regions being down at the same time much less than one region being down • Reduces regional latency
  • 23. Geo Load Balancing and Failover (Migration) • Needed to verify stability of Kube clusters before turning off CF deployments in production • Solution: Add Kube clusters to Dyn rotation and run CF deployments side-by-side with Kube deployments
  • 24. Geo Load Balancing and Failover (Final) • Once satisfied, removed CF deployments from rotation and only Kube deployments remained
  • 26. Conclusion • CF is a great technology, but Kubernetes better meets the needs of our microservice system • Nothing is free, and we had to solve several new problems along the way • Allowed us to achieve greater performance, scalability, reliability, and security than we had before
  • 27. Questions? • Tony Erwin – Email: aerwin@us.ibm.com – Twitter: @tonyerwin • Jonathan Schweikhart – Email: jschweik@us.ibm.com

Editor's Notes

  • #5: IBM Cloud Console: https://console.bluemix.net/
  • #8: Cloud Foundry Foundation: https://www.cloudfoundry.org/ CF Application Runtime: https://www.cloudfoundry.org/application-runtime/ Open Service Broker API: https://www.openservicebrokerapi.org/
  • #9: Kubernetes: https://kubernetes.io/
  • #27: IBM Cloud Console: https://console.bluemix.net/