Continuously delivered scalable geoservies
- 1. Continuously Delivered & Scalable Geoservices on Kubernetes Daniel Fenton Esri R&D @dmfenton
- 2. Quick Background • ArcGIS Hub: • Cities Publish Open Data • Work with Citizens to Solve Urban Problems • Product Launched in 2014 • Actively Used by Hundreds of Organizations • Local Gov • State Gov • Federal Gov • NGOs • Private Business • E.g. https://data-stlcogis.opendata.arcgis.com/
- 3. Overview • Kubernetes @ 10k Feet • Spatial Download System Architecture • Jenkins and our CI/CD Pipeline • Key Component: Deployment • Kubeutils: FOSS Deployment CLI • Deployment Rollouts • Networking • Monitoring
- 4. Benefits • Faster time from idea to deployment • Faster and more complete feedback from production • Easier to debug • Easier to add new services • Automated, reliable deployments • Smooth Git Workflow • Scaling on autopilot • Fewer idle resources consumed
- 5. K8s @ 10k: Key Terminology • Cluster • Master & Nodes • Namespace • Pod • Deployment • Horizonal Pod Autoscaler • Service • Ingress
- 6. Download System Architecture • Feature Services => (ND) GeoJSON, KML, CSV, Shapefile, Geohash • 15m per year, 1 TB per month • Spiky workloads • Delayed job system, refresh in background
- 9. Deployment Configuration • Yaml for each service • API • Import Worker • Export Worker • Autoscaling Configuration in same file • Secrets referenced externally (safe to check in) • Changes to this file trigger a deploy
- 11. Rollouts: blue/green style • Each microservice implements “readiness check” • Is this service up and ready to handle traffic? • Scheduler adds new container with updated code • If readiness check passes => remove old container • Repeat until all old pods have been replaced with new pods • Updates sent in Slack Channel
- 12. Networking
- 13. Monitoring • Logging Stack • FluentD Daemonset • 3 Node Elasticsearch Statefulset • Kibana • Exposed on Internal Network • Behind Github Auth Proxy • Metrics/Alerting Stack • Run by Prometheus Operator • Prometheus Statefulset • Alertmanager Statefulset • Pagerduty + Slack Alerts
- 14. Links • Prometheus operator • Kubernetes Guide • ArcGIS Hub • Koop • Kubeutils • Kops • NGINX Ingress Controller • Cluster Autoscaler