Ippevent : openshift Introduction
0 likes339 views
This document provides an introduction to Docker and Openshift including discussions around infrastructure, storage, monitoring, metrics, logs, backup, and security considerations. It describes the recommended infrastructure for a 3 node Openshift cluster including masters, etcd, and nodes. It also discusses strategies for storage, monitoring both internal pod status and external infrastructure metrics, collecting and managing logs, backups, and security features within Openshift like limiting resource usage and isolating projects.
![Docker/ Openshift introduction
Logs
Following best practice, pod log go to console/stdout
11
Log can be seen from the pod
oc log mymicser-31-9koc7 -n lovely-project
Don’t forget grep|sort|awk…
Kubetail can be useful
[Sat Jun 17 22:05:13.613952 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx
[Sat Jun 17 22:05:13.613955 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx
[Sat Jun 17 22:05:13.614566 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx
[Sat Jun 17 22:05:13.614575 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx
122.138.20.1 - - [17/Jun/2017:22:05:13 +0200] "GET / HTTP/1.1" 200 29601 xxxx xx xx xx
[Sat Jun 17 22:05:14.272141 2017] [:error] [pid 66435] [client 122.138.19.1:24097] xxxx xx xx xx](https://image.slidesharecdn.com/ippeventopenshiftrex2-170630073728/85/Ippevent-openshift-Introduction-11-320.jpg)
![Docker/ Openshift introduction
Copy an object from project-1 to project-2
oc export xx -n project-1 | oc create -n project-2 -f -
Check container environment variable (first oc get dc) :
for proj in $(oc get project -l type=lamp); do
oc get po -n XXX -l app=mysql, version=5.6
done;
Check container environment variable (first oc get dc) :
oc get dc $DCTOINSPEC -n $DEV_PROJECT_NAME -o json | jq ".spec.template.spec.containers[]?.env"
Check user access within project
for PROJECTID in $PROJECTIDS;
do echo "$PROJECTID" &&
oc describe policyBindings :default -n $PROJECTID | grep -C 1 $USENAME_TO_ADD ;
done
Check image SHA tagged OK
for PROJECTID in $PROJECTIDS;
do echo "$PROJECTID" &&
oc get is $PROJECTID -n $PROJECTID -o json|
jq -r ".status.tags[]|select(.tag == "ok")|.items|
max_by(.created)|.dockerImageReference"
done
Tips
17
Always this scripting](https://image.slidesharecdn.com/ippeventopenshiftrex2-170630073728/85/Ippevent-openshift-Introduction-17-320.jpg)
Ippevent : openshift Introduction
- 1. Docker/ Openshift introduction Openshift Cluster in practice
- 2. Docker/ Openshift introduction Agenda ● Infrastructure ● Storage ● Monitoring ● Metrics ● Logs ● Backup ● Security 2
- 3. Docker/ Openshift introduction Infrastructure 3 Global Infrastructure
- 4. Docker/ Openshift introduction Infrastructure 3x Masters + etcd : • 4 (v)CPUs • 16 GB RAM • 100 GB HDD 4 Masters, etcd cluster and nodes are VMWare ESX Virtual Machines etcd performances issues since 3.4 ! • Separated from Masters • Redhat recommends SSD drives • Upgrade blocked ... nodes : 2 to n per zone : • 16 (v)CPUs • 32 GB RAM • 600 GB HDD Infra nodes : 2 to n : • 4 (v)CPUs • 16 GB RAM • 340 GB HDD Logs : • High memory, CPU and disk consumption !!! • Move to Physical Servers (40 Threads, 30 TB HDD) Global capacity to support a Rolling upgrade and reboot
- 5. Docker/ Openshift introduction Infrastructure : capacity planning 5 oc describe node node001 Name: node001 Role: Labels: kubernetes.io/hostname=node001, …. Taints: <none> CreationTimestamp:Tue, 24 May 2016 16:58:10 +0200 Phase: Conditions: Type Status ... Reason Message ---- ------ ----------------- ------------------ ------ Ready True ... ... KubeletReady kubelet is posting ready status OutOfDisk False ... ... KubeletHasSufficientDisk kubelet has sufficient disk space available MemoryPressure False ... ... KubeletHasSufficientMemory kubelet has sufficient memory available Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits --------- ---- ------------ ---------- --------------- ------------- xxxxxxx zzzz-www-9-123ee 500m (3%) 1 (6%) 256Mi (0%) 1Gi (3%) xxxxxxx zzzz-wcw-9-123ee 500m (3%) 1 (6%) 1024Mi (0%) 2Gi (3%) xxxxxxx zzzz-waw-7-123ee 500m (3%) 1 (6%) 512Mi (0%) 1Gi (3%) xxxxxxx zzzz-wsw-5-123ee 500m (3%) 1 (6%) 256Mi (0%) 4Gi (3%) xxxxxxx zzzz-qww-4-123ee 500m (3%) 1 (6%) 256Mi (0%) 1Gi (3%) Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted. CPU Requests CPU Limits Memory Requests Memory Limits ------------ ---------- --------------- ------------- 2500m (15%) 5 (31%) 2048Mi (6%) 9Gi (44%) Capacity: alpha.kubernetes.io/nvidia-gpu: 0 cpu: 16 memory: 32780448Ki pods: 110 Allocatable: alpha.kubernetes.io/nvidia- gpu: 0 cpu: 14 memory: 30731670Ki pods: 110 System Info: ... Remember to reserve resources for the node itself
- 6. Docker/ Openshift introduction Storage Criteria : • Type : NFS, GlusterFS, EBS, GCE, iSCSI, FC, Azure, … • Access mode : RWO (for one pod), RWX (shared), ROX (read only) • Capacity : volume size, IOPS, throughput. StorageClass : • Storage definition for dynamically provisioned on demand storage • Can be associated to a specific “storage-admin” role • Currently only for “cloud” resources Phase : • Available / Bound / Released / Failed Recycling Policy : • Retain / Recycle 6 Persistent Storage
- 7. Docker/ Openshift introduction Storage 7 No SAN, no NAS, no cloud. So … What ? Build your own Storage Cluster ! Two big servers, Pacemaker, DRBD and XFS : NFS HA Scalable Storage Cluster
- 8. Docker/ Openshift introduction Monitoring Node liveness is done by kubernetes. Two kinds of checks for pod status : • Liveness probe : if not alive, pod will be restarted • Readiness probe : if not ready, traffic won’t be sent to this pod. If back, traffic will be back again. 8 Internal monitoring of pods and nodes status SHAME ! Don’t use openshift to hide your mistakes and incompetence Openshift won’t do functional testings and monitoring for you. Also associated to pod limits (see metrics), if limit is exceeded, pod is killed (and restarted).
- 9. Docker/ Openshift introduction Monitoring • Check the whole infrastructure : memory, cpu, load, storage, ... • But also processus, docker thinpools, pods, persistent volumes, … • Centreon + homemade scripts in shell and python with snmp and oc 9 External monitoring
- 10. Docker/ Openshift introduction Metrics • Integrated realtime metrics : heapster, hawkular and Cassandra • By default : Memory, CPU (millicores) and Network • View for pods, and aggregated view for services 10 Pods and services metrics • Metrics enable one great functionality : horizontal pod autoscaling ! • It can be associated with quotas (resource limits). If limits are exceeded, pod is killed and restarted. Number of Nodes Number of Pods Cassandra Storage growth speed Cassandra storage growth per day Cassandra storage growth per week 210 10500 500 MB per hour 15 GB 75 GB 990 11000 1 GB per hour 30 GB 210 GB
- 11. Docker/ Openshift introduction Logs Following best practice, pod log go to console/stdout 11 Log can be seen from the pod oc log mymicser-31-9koc7 -n lovely-project Don’t forget grep|sort|awk… Kubetail can be useful [Sat Jun 17 22:05:13.613952 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx [Sat Jun 17 22:05:13.613955 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx [Sat Jun 17 22:05:13.614566 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx [Sat Jun 17 22:05:13.614575 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx 122.138.20.1 - - [17/Jun/2017:22:05:13 +0200] "GET / HTTP/1.1" 200 29601 xxxx xx xx xx [Sat Jun 17 22:05:14.272141 2017] [:error] [pid 66435] [client 122.138.19.1:24097] xxxx xx xx xx
- 12. Docker/ Openshift introduction Logs / EFK Aggregation Elasticsearch A highly-available environment requires at least three replicas of Elasticsearch; each on a different host. Elasticsearch replicas require their own storage, but an OpenShift Container Platform deployment configuration shares storage volumes between all its pods. So, when scaled up, the EFK deployer ensures each replica of Elasticsearch has its own deployment configuration. • Dedicate node using Node Selector • Persistent Elasticsearch Storage (local storage) Fluentd Fluentd is deployed as a DaemonSet that deploys replicas according to a node label selector (which you can specify with the deployer parameter fluentd-nodeselector; the default is logging-infra-fluentd). You can configure Fluentd to send a copy of its logs to an external log aggregator, and not the default Elasticsearch, using the secure-forward plug-in Kibana To access the Kibana console from the OpenShift Container Platform web console Curator Curator allows administrators to configure scheduled Elasticsearch maintenance operations to be performed automatically on a per-project basis. Ops Cluster If you set enable-ops-cluster to true for the deployer, Fluentd is configured to split logs between the main ElasticSearch cluster and another cluster reserved for operations logs 12 $ oc label node/xxx logging-infra-fluentd=true
- 13. Docker/ Openshift introduction Logs / EFK Assumptions: 1. Which application: Apache 2. Bytes per line: 256 3. Lines per second load on application: 1 4. Raw text data → JSON Baseline (256 characters per minute → 15KB/min) The implication of keeping all those log need a lot of reflexion on : • What to keep ? • How long ? • How to use them on the long run ? You really need a project reflexion on the matter, because logs grow fast and can rapidly become unmanageable. 13 Sometime size matter Logging Infra Pods Storage Throughput 3 es 1 kibana 1 curator 1 fluentd 6 pods total: 90000 x 86400 = 7,7 GB/day 3 es 1 kibana 1 curator 11 fluentd 16 pods total: 225000 x 86400 = 24,0 GB/day 3 es 1 kibana 1 curator 20 fluentd 25 pods total: 225000 x 86400 = 32,4 GB/day
- 14. Docker/ Openshift introduction • Master / nodes backup →Git / ansible →etcd backup • PV backup →Backula, rsync... • Project backup →Images Docker.io, artifactory, …. →Object oc export …, Project level https://github.com/vbehar/openshift-git →PVC https://github.com/getupcloud/backup Backup 14 3 layers of back-up
- 15. Docker/ Openshift introduction Security 15 Dr. Egon Spengler: Don’t use root in your image Dr. Peter Venkman: Why? In Openshift Pods are launched with an arbitrary / random used ID. Thanks to user namespaces. Root is prevented by default. You can’t escape :) Your images must be adapted. Most docker hub images can’t be reused as such.
- 16. Docker/ Openshift introduction Security 16 Project side : thanks to cgroups and limits Remember : Petazzoni slides • cgroups : limits how much you can use (and gauge) • namespaces : limits what you can see (and therefore use) • Security context constraints : allow administrators to control permissions for pods. • Segregation by namespaces : projects are isolated from each other • Policies and roles at each level (cluster, project, resources, …) • Images : Image vulnerability scanning, (Xray / BlackDuck), Base images • Network : segregation between projects by default (openvswith, tags) : if you need to communicate, use services and routes EVEN in the same project. • Secrets : because you don’t want to spread hidden things in your templates or configuration files (credentials, keys, …), openshift handles them for you. • selinux : RHEL and docker
- 17. Docker/ Openshift introduction Copy an object from project-1 to project-2 oc export xx -n project-1 | oc create -n project-2 -f - Check container environment variable (first oc get dc) : for proj in $(oc get project -l type=lamp); do oc get po -n XXX -l app=mysql, version=5.6 done; Check container environment variable (first oc get dc) : oc get dc $DCTOINSPEC -n $DEV_PROJECT_NAME -o json | jq ".spec.template.spec.containers[]?.env" Check user access within project for PROJECTID in $PROJECTIDS; do echo "$PROJECTID" && oc describe policyBindings :default -n $PROJECTID | grep -C 1 $USENAME_TO_ADD ; done Check image SHA tagged OK for PROJECTID in $PROJECTIDS; do echo "$PROJECTID" && oc get is $PROJECTID -n $PROJECTID -o json| jq -r ".status.tags[]|select(.tag == "ok")|.items| max_by(.created)|.dockerImageReference" done Tips 17 Always this scripting