DevOpsDays Houston 2019 - Terry Shea - Centralizing Kubernetes Operations
1 like170 views
The document discusses Kublr's approach to centralizing Kubernetes and container operations, highlighting the need for a unified application delivery platform that addresses various operational aspects such as monitoring, governance, and scalability. It covers the complexities involved in deploying and managing Kubernetes clusters across different environments, including multi-cloud and hybrid setups, and emphasizes the importance of automation and centralized control for security and customization. Key technologies mentioned include Prometheus for monitoring, Elasticsearch for logging, and CI/CD practices for application management.
![Centralized Log Collection: Considerations
• Tune Elasticsearch resource usage
• Take into account additional load on API server
• Log index structure normalization
{
"data": {
"elasticsearch": {
"version": "6.x"
}
}
}
{
"flatData": [
{
"key": "elasticsearch.version",
"type": "string",
"key_type": "elasticsearch.version.string",
"value_string": "6.x"
},
...
]
}
@kublr http://smnh.me/indexing-and-searching-arbitrary-json-data-using-elasticsearch/](https://image.slidesharecdn.com/devopsdayshouston2019-terryshea-centralizingkubernetesoperations-190507123349/85/DevOpsDays-Houston-2019-Terry-Shea-Centralizing-Kubernetes-Operations-18-320.jpg)
DevOpsDays Houston 2019 - Terry Shea - Centralizing Kubernetes Operations
- 1. Centralizing Kubernetes and Container Operations Terry Shea| Kublr
- 2. History • Custom software development company • Dozens of projects per year • Varying target environments: clouds, on-prem, hybrid • Recurring need for unified application delivery and ops platform w/ monitoring, logs, security, multiple env, ... @kublr
- 3. Docker and Kubernetes to the Rescue • Docker is great, but local • Kubernetes is great... when it is up and running • Who sets up and operates K8S clusters? • Who takes care of operational aspects at scale? • How do you provide governance and ensure compliance? @kublr
- 4. Applications and Architecture Digital– Web/Mobile Data Science & Machine Learning Video Streaming DigitalTransformation and App Modernization Hybrid Cloud Multi-Cloud Edge Computing Private Cloud Internet of Things
- 5. Some Application Considerations Data Science/AI/ML etc. • Operators • Spark • Kafka • Projects • Kubeflow (note ksonnet deprecated) • Jupyter K8s • Specialized Hardware • GPUs, TPUs, FPGAs (oh my!) NFV/SDN • ONAP • Intel Multus CNI Web/Mobile/Microservice Apps • Service Meshes • Istio, Linkerd • Deployment Strategies • Rolling, Canary, Blue-Green Hybrid Apps • App Scaling re: Back-end Resources IOT/Edge Projects • Eclipse Foundation • KubeEdge (Huawei)
- 6. Enterprise Kubernetes Needs Developers SRE/Ops/DevOps/SecOps • Self-service • Compatible • Conformant • Configurable • Open & Flexible • Org multi-tenancy • Single pane of glass • Operations • Monitoring • Log collection • Image management • Identity management • Security • Reliability • Performance • Portability @kublr
- 7. @kublr Automation Ingress Custom Clusters Infrastructure Logging Monitoring Observability API Usage Reporting RBAC IAM Air Gap TLS Certificate Rotation Audit Storage Networking Container Registry CI / CD App Mgmt Infrastructure Container Runtime Kubernetes OPERATIONS SECURITY & GOVERNANCE
- 8. Central Control Plane: Operations K8S Clusters Cloud(s) Data center API UI Log collection Operations Monitoring Authn and authz, SSO, federation Audit Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev @kublr
- 9. Central Control Plane: Operations @kublr
- 10. Flexible Deployment for Security & Customization Requirements
- 11. Central Control Plane: Logs and Metrics K8S Clusters Cloud(s) Data center API UI Operations Authn and authz, SSO, federation Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev Log collection Monitoring Audit @kublr
- 12. K8S Monitoring with Prometheus • Discover nodes, services, pods via K8S API • Query metrics from discovered endpoints • Endpoint are accessed directly via internal cluster addresses Kubernetes Cluster Prometheus Nodes K8S API Grafana Pods Discovery Srv Metrics @kublr
- 13. Centralized Monitoring Cluster registry PROMETHEUSGrafana K8S Proxy API nodes, pods, service endpoints Ship externally Ship externally Prometheus config Prometheus data Configurator Control plane KUBERNETES CLUSTER Prometheus (collector) Prometheus (collector) @kublr
- 14. Centralized Monitoring: Considerations • Prometheus resource usage tuning • Long-term storage (m3) • Configuration file growth with many clusters • Metrics labeling • Additional load on API server @kublr
- 16. K8S Logging with Elasticsearch • Fluentd runs on nodes • OS, K8S, and container logs collected and shipped to Elasticsearch • Kibana for visualization Kubernetes Cluster Elasticsearch Kibana Pods Logs @kublr
- 17. Prometheus (collector) RabbitMQ Centralized Log Collection Cluster registry K8S Proxy API Port forwarding MQTT Ship externally Messaging config Configurator Control plane RabbitMQ Shovel ElasticsearchLogstash Fluentd KUBERNETES CLUSTER filter filter analyze Ship externally MQTT Forwarder filter @kublr
- 18. Centralized Log Collection: Considerations • Tune Elasticsearch resource usage • Take into account additional load on API server • Log index structure normalization { "data": { "elasticsearch": { "version": "6.x" } } } { "flatData": [ { "key": "elasticsearch.version", "type": "string", "key_type": "elasticsearch.version.string", "value_string": "6.x" }, ... ] } @kublr http://smnh.me/indexing-and-searching-arbitrary-json-data-using-elasticsearch/
- 19. The Rest: Considerations • Identity management Use Identity Broker (e.g. KeyCloak): Users, Authn, Autzn, SSO, RBAC, • Backup and disaster recovery K8s metadata + app data/volumes: full cluster recovery or copy • Docker image management Docker image registry (e.g. Nexus, Artifactory, Docker Hub); image scanning; air-gapped or isolated environment: image registries proxying and caching, “system” images @kublr
- 20. CI/CD projects at Kublr • CircleCI Orb (coming soon) • Keptn ( a Dynatrace open source project) https://keptn.sh • Canary Deployments with Istio and Spinaker https://kublr.com/blog/hands-on-canary-deployments-with-istio-and-kubernetes/
- 21. Q&A @kublr Use Kublr for k8s POCs kublr.com/deploy Free non-production license.
- 22. Terry Shea terry@kublr.com Kublr | kublr.com @kublr