Centralizing Kubernetes and Container Operations
Download as PPTX, PDF1 like2,537 views
The document discusses the importance of centralized operations for Kubernetes and container management, emphasizing Kublr's platform which aids enterprises in adopting these technologies. Key features include self-service governance, reliability, and support for multi-cluster environments, highlighting the operational aspects of Kubernetes such as monitoring, logging, and security. The presentation also mentions best practices for infrastructure automation, centralized monitoring, and log management using tools like Prometheus and Elasticsearch.
![Centralized Log Collection: Considerations
• Tune Elasticsearch resource usage
• Take into account additional load on API server
• Log index structure normalization
{
"data": {
"elasticsearch": {
"version": "6.x"
}
}
}
{
"flatData": [
{
"key": "elasticsearch.version",
"type": "string",
"key_type": "elasticsearch.version.string",
"value_string": "6.x"
},
...
]
}](https://image.slidesharecdn.com/centralizingkubernetesandcontaineroperations-181108223807/85/Centralizing-Kubernetes-and-Container-Operations-20-320.jpg)
Centralizing Kubernetes and Container Operations
- 1. Centralizing Kubernetes and Container Operations Oleg Chunikhin | CTO, Kublr
- 2. Introductions Oleg Chunikhin CTO, Kublr • Nearly 20 years in the field of software architecture and development. • Joined Kublr as the CTO in 2016. • Kublr is an enterprise Kubernetes management and operations platform that helps accelerate Kubernetes adoption and containerized applications management for enterprises.
- 3. History • Custom software development company • Dozens of projects per year • Varying target environments: clouds, on-prem, hybrid • Unified application delivery and ops platform wanted: monitoring, logs, security, multiple env, ...
- 4. Docker and Kubernetes to the Rescue • Docker is great, but local • Kubernetes is great... when it is up and running • Who sets up and operates K8S clusters? • Who takes care of operational aspects at scale? • How do you provide governance and ensure compliance?
- 5. Enterprise Kubernetes Needs Developers SRE/Ops/DevOps/SecOps • Self-service • Compatible • Conformant • Configurable • Open & Flexible • Org multi-tenancy • Single pane of glass • Operations • Monitoring • Log collection • Image management • Identity management • Security • Reliability • Performance • Portability
- 6. Kubernetes Management Platform Wanted • Portability – clouds, on-prem, air-gapped, different OS’ • Centralized multi-cluster operations saves resources – many environments (dev, prod, QA, ...), teams, applications • Self-service and governance for Kubernetes operations • Reliability – cluster self-healing, self-reliance • Limited management profile – cloud and K8S API • Architecture – flexible, open, pluggable, compatible • Sturdy – secure, scalable, modular, HA, DR etc.
- 7. Central Control Plane: Operations K8S Clusters Cloud(s) Data center API UI Log collection Operations Monitoring Authn and authz, SSO, federation Audit Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev
- 8. Central Control Plane: Operations
- 9. Infrastructure Automation Cluster: Self-Sufficiency Central control plane MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store Secrets discovery Simple orchestration and configuration agent
- 10. Cluster: Portability • (Almost) everything runs in containers • Simple (single-binary) management agent • Minimal store requirements • Shared, eventually consistent • Secure: RW files for masters, RO for nodes • Thus the store can be anything: S3, SA, NFS, rsynced dir, provided files, ... • Minimal infra automation requirements • Configure and run configuration agent • Enable access to the store • Can be AWS CF, Azure ARM, BOSH, Ansible, ... • Load balancer is not required for multi-master; each agent can independently fail over to a healthy master Infrastructure Automation MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store Secrets discovery
- 11. Cluster: Reliability • Rely on underlying platform as much as possible • ASG on AWS • IAM on AWS for store access • SA on Azure, S3 on AWS • ARM on Azure, CF on AWS • Minimal infrastructure SLA tolerate temporary failures • Multi-muster API failover on nodes • Resource management, memory requests and limits for OS and k8s components Infrastructure Automation MASTER KUBLR overlay network, discovery, connectivity K8s Master Components: etcd, scheduler, API, controller Docker KUBELET KUBLRKUBELET NODE Docker overlay network, discovery, connectivity Infrastructure and Application containers Orchestration Store
- 12. Central Control Plane: Logs and Metrics K8S Clusters Cloud(s) Data center API UI Operations Authn and authz, SSO, federation Image Repo Infrastructure management Backup & DR Dev K8S API Cloud API Prod PoC Dev Log collection Monitoring Audit
- 13. Centralized Monitoring and Log Collection. Why Bother? • Prometheus and ELK are heavy and not easy to operate; need attention and at least 4-8 Gb RAM... each, per cluster • Cloud/SaaS monitoring is not always permitted or available • Existing monitoring is often not container-aware • No aggregated view and analysis • No alerting governance
- 14. K8S Monitoring with Prometheus • Discover nodes, services, pods via K8S API • Query metrics from discovered endpoints • Endpoint are accessed directly via internal cluster addresses Kubernetes Cluster Prometheus Nodes K8S API Grafana Pods Discovery Srv Metrics
- 15. Centralized Monitoring Cluster registry PROMETHEUSGrafana K8S Proxy API nodes, pods, service endpoints Ship externally Ship externally Prometheus config Prometheus data Configurator Control plane KUBERNETES CLUSTER Prometheus (collector) Prometheus (collector)
- 16. Centralized Monitoring: Considerations • Prometheus resource usage tuning • Long-term storage (m3) • Configuration file growth with many clusters • Metrics labeling • Additional load on API server
- 18. K8S Logging with Elasticsearch • Fluentd runs on nodes • OS, K8s, and container logs collected and shipped to Elasticsearch • Kibana for visualization Kubernetes Cluster Elasticsearch Kibana Pods Logs
- 19. Prometheus (collector) RabbitMQ Centralized Log Collection Cluster registry K8S Proxy API Port forwarding MQTT Ship externally Messaging config Configurator Control plane RabbitMQ Shovel ElasticsearchLogstash Fluentd KUBERNETES CLUSTER filter filter analyze Ship externally MQTT Forwarder filter
- 20. Centralized Log Collection: Considerations • Tune Elasticsearch resource usage • Take into account additional load on API server • Log index structure normalization { "data": { "elasticsearch": { "version": "6.x" } } } { "flatData": [ { "key": "elasticsearch.version", "type": "string", "key_type": "elasticsearch.version.string", "value_string": "6.x" }, ... ] }
- 21. The Rest: Considerations • Identity management Use Identity Broker (e.g. KeyCloak): Users, Authn, Autzn, SSO, RBAC, Federation, ... • Backup and disaster recovery K8s metadata + app data/volumes: full cluster recovery or copy Docker image management Docker image registry (e.g. Nexus, Artifactory, Docker Hub); image scanning; air-gapped or isolated environment: image registries proxying and caching, “system” images
- 22. Q&A
- 23. Oleg Chunikhin Chief Technology Officer oleg@kublr.com @olgch Kublr | kublr.com @kublr Thank you!
Editor's Notes
- #4: Where the project comes from Company overview Kubernetes as a solution – standardized delivery platform Kubernetes is great for managing containers, but who manages Kubernetes? How to streamline monitoring and collection of logs with multiple Kubernetes clusters?
- #7: Requirements Portability – support for cloud environments, on prem deployment, and isolated deployments Multi-cluster operations support Centralized log collection and monitoring Reliability – self healing, modularity, cluster self-reliance Limited connectivity profile – do not require many ports Architecture – flexible, open, pluggable Security
- #10: The control plane is only critically involved in the cluster when the cluster is created The control plane uses cloud specific infrastructure management automation frameworks – CF, ARM, BOSH, VMware, etc. After the cluster infrastructure is created and configured, the cluster does not need the control plane Self-coordination via the orchestration store Orchestration store and underlying platform are the only coordination devices the cluster needs to operate and recover failures from Masters and nodes are configured for the orchestration store access Master(s) will try to get secrets and discovery information from the store; if not available – will generate and publish a new set With multiple masters – the latest published package wins Nodes will take the latest published data and use it.
- #14: Prometheus
- #15: Prometheus
- #16: Control plane keeps track of managed clusters Configurator reconfigures Prometheus when cluster list changes Prometheus configuration is in K8S config maps