Deploying on Kubernetes - An intro
0 likes419 views
The document covers deploying applications on Kubernetes, focusing on concepts such as pods, services, and Helm, as well as security considerations and scaling through horizontal pod autoscaling. It includes examples of Kubernetes resources, configuration, and best practices for managing workloads and security. Additionally, it provides an overview of Helm for package management in Kubernetes and demonstrates how to create and manage releases effectively.
![Pod
• Basic building block
• Encapsulates:
• Container(s)
• IP
• Storage
• Runtime config
apiVersion: v1
kind: Pod
metadata:
name: memory-demo
spec:
containers:
- name: memory-demo-ctr
image: polinux/stress
env:
- name: DEBUG
value: "True"
- name: SECRET_KEY
valueFrom:
secretKeyRef:
name: stress-secret
key: secret-key
command: ["stress"]
volumeMounts:
- name: gcp-credentials
mountPath: /secrets/gcp
readOnly: true
volumes:
- name: gcp-credentials
secret:
secretName: gcpcreds](https://image.slidesharecdn.com/kubernetes-pixelscampv3-190327164528/85/Deploying-on-Kubernetes-An-intro-10-320.jpg)
![CD Example (migration)
sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-migration.yaml
kubectl create -f kube/${KUBE_APP}-migration.yaml --namespace=$KUBE_NAMESPACE
while [[ -z `kubectl get pods --selector=app=${KUBE_APP}-migration --namespace=$KUBE_NAMESPACE
--output=jsonpath={.items..metadata.name}` ]]; do
echo "Waiting for pod to be provisioned";
sleep 1;
done;
export POD=$(kubectl get pods --selector=app=${KUBE_APP}-migration --namespace=$KUBE_NAMESPACE
--output=jsonpath={.items..metadata.name})
export CURRENT_STATUS="kubectl get pod -a $POD --namespace=$KUBE_NAMESPACE
--output=jsonpath={.status.phase}"
while [[ `$CURRENT_STATUS` =~ ^(Pending|Running)$ ]]; do
echo "Waiting for pod to start and finish - $POD";
sleep 1;
done;
kubectl describe job ${KUBE_APP}-migration-job --namespace=$KUBE_NAMESPACE
kubectl get pod -a $POD --namespace=$KUBE_NAMESPACE --output=jsonpath="{.status.phase}" || true
kubectl logs $POD --namespace=$KUBE_NAMESPACE || true
kubectl delete -f kube/${KUBE_APP}-migration.yaml --namespace=$KUBE_NAMESPACE;](https://image.slidesharecdn.com/kubernetes-pixelscampv3-190327164528/85/Deploying-on-Kubernetes-An-intro-29-320.jpg)
![Template flow control
{{- range $job := .Values.jobs }}
---
apiVersion: batch/v1beta1
kind: CronJob
spec:
concurrencyPolicy: {{ $job.concurrencyPolicy }}
jobTemplate:
spec:
template:
spec:
containers:
- image: "{{ $job.image.repository }}:{{ $job.image.tag }}"
name: {{ $job.name }}
{{- if $job.command }}
command: {{ $job.command }}
{{- end }}
{{- with $job.args }}
args:
{{ toYaml . | indent 12 }}
restartPolicy: {{ $job.restartPolicy }}
schedule: {{ $job.schedule | quote }}
{{- end }}
jobs:
- name: hello-world
image:
repository: hello-world
tag: latest
schedule: "* * * * *"
concurrencyPolicy: Allow
restartPolicy: OnFailure
- name: hello-ubuntu
image:
repository: ubuntu
tag: latest
schedule: "*/5 * * * *"
command: ["/bin/bash"]
args:
- "-c"
- "echo $(date) - hello from ubuntu"
concurrencyPolicy: Forbid
restartPolicy: OnFailure
Values Template](https://image.slidesharecdn.com/kubernetes-pixelscampv3-190327164528/85/Deploying-on-Kubernetes-An-intro-43-320.jpg)
Deploying on Kubernetes - An intro
- 1. Deploying on Kubernetes André Cruz
- 2. Agenda • Kubernetes concepts - 35% • Security considerations - 15% • Mapping to traditional use cases - 15% • Helm - 35%
- 8. Persistent Volume Claim POD Container Volume SecretConfigMap Config / Storage Persistent Volume
- 9. Namespace apiVersion: v1 kind: Namespace metadata: name: <insert-namespace-name-here>
- 10. Pod • Basic building block • Encapsulates: • Container(s) • IP • Storage • Runtime config apiVersion: v1 kind: Pod metadata: name: memory-demo spec: containers: - name: memory-demo-ctr image: polinux/stress env: - name: DEBUG value: "True" - name: SECRET_KEY valueFrom: secretKeyRef: name: stress-secret key: secret-key command: ["stress"] volumeMounts: - name: gcp-credentials mountPath: /secrets/gcp readOnly: true volumes: - name: gcp-credentials secret: secretName: gcpcreds
- 11. Pod - Resources • Limits • Requests ... resources: limits: cpu: "1500m" memory: "200Mi" requests: cpu: "300m" memory: "100Mi" ...
- 12. Pod - Health Checks • Liveness • Readiness • Probe types: • tcpSocket • httpGet • exec ... readinessProbe: httpGet: path: /healthz port: 8080 httpHeaders: - name: Custom-Header value: Awesome initialDelaySeconds: 3 periodSeconds: 3 livenessProbe: tcpSocket: port: 8080 initialDelaySeconds: 15 periodSeconds: 20 ...
- 13. Service • Service Discovery • TCP/UDP load balancer • Types: • ClusterIP • NodePort • LoadBalancer • ExternalName kind: Service apiVersion: v1 metadata: name: my-service spec: selector: app: MyApp ports: - protocol: TCP port: 80 targetPort: 9376
- 14. Ingress • HTTP(S) load balancer • Requires Ingress Controller • Several controller implementations: • Nginx • GCE • Contour (Envoy) • Traefik apiVersion: extensions/v1beta1 kind: Ingress metadata: name: tls-example-ingress annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - sslexample.foo.com secretName: testsecret-tls rules: - host: sslexample.foo.com http: paths: - path: / backend: serviceName: service1 servicePort: 80
- 15. Horizontal Pod Autoscaler apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: php-apache namespace: default spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: php-apache minReplicas: 1 maxReplicas: 10 targetCPUUtilizationPercentage: 50 • Scales pod replica count • Based on: • Resource usage • Metrics
- 16. Autoscaler - External metrics apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name: pubsub spec: minReplicas: 1 maxReplicas: 5 metrics: - external: metricName: pubsub.googleapis.com|subscription|num_undelivered_messages metricSelector: matchLabels: resource.labels.subscription_id: echo-read targetAverageValue: "2" type: External scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: pubsub
- 17. kubectl • Fetch list of pods •kubectl get pods -n • Apply config •kubectl apply -f file.yaml • Fetch pod logs •kubectl logs POD_NAME
- 18. kubectl demo
- 20. Security considerations • No extra content on docker image (compilers, etc) • Pod security policy • Don't run as root
- 21. Don't run as root # grep Cap /proc/self/status CapInh: 00000000a80425fb CapPrm: 00000000a80425fb CapEff: 00000000a80425fb CapBnd: 00000000a80425fb CapAmb: 0000000000000000 $ grep Cap /proc/self/status CapInh: 0000000000000400 CapPrm: 0000000000000000 CapEff: 0000000000000000 CapBnd: 0000000000000400 CapAmb: 0000000000000000 VS
- 22. Don't run as root apiVersion: v1 kind: Pod metadata: name: run-as-uid-1000 spec: securityContext: runAsUser: 1000 # ... apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: non-root spec: privileged: false allowPrivilegeEscalation: false runAsUser: rule: 'MustRunAsNonRoot' Pod Security PolicyPod Specification
- 23. Security considerations • Role-based access control (RBAC) • Default API credentials on pod • automountServiceAccountToken • Use namespaces • Network policies
- 24. Security considerations • Separate sensitive workloads • Encrypted Secrets • Cloud provider metadata leak • GKE’s metadata concealment feature
- 26. Kubernetes in the wild
- 27. Applying K8S concepts • DB schema migration -> Job • Hourly backups -> CronJob • Long running daemon -> Deployment • Exposing endpoint -> Service • External HTTP(S) endpoint -> Ingress • API Key configuration -> Secret • Log collection agent -> DaemonSet • Database -> StatefulSet + Persistent Volume Claim
- 28. Example dir structure kube/govtech-cron-atokens.yaml kube/govtech-cron-random.yaml kube/govtech-cron-standings.yaml kube/sqlproxy-depl.yaml kube/sqlproxy-svc.yaml kube/govtech-migration.yaml kube/govtech-svc.yaml kube/govtech-ing.yaml kube/govtech-production-ing.yaml kube/govtech-depl.yaml Auxiliary Service DB migration job Actual deployment Periodic jobs Exposing deployment
- 29. CD Example (migration) sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-migration.yaml kubectl create -f kube/${KUBE_APP}-migration.yaml --namespace=$KUBE_NAMESPACE while [[ -z `kubectl get pods --selector=app=${KUBE_APP}-migration --namespace=$KUBE_NAMESPACE --output=jsonpath={.items..metadata.name}` ]]; do echo "Waiting for pod to be provisioned"; sleep 1; done; export POD=$(kubectl get pods --selector=app=${KUBE_APP}-migration --namespace=$KUBE_NAMESPACE --output=jsonpath={.items..metadata.name}) export CURRENT_STATUS="kubectl get pod -a $POD --namespace=$KUBE_NAMESPACE --output=jsonpath={.status.phase}" while [[ `$CURRENT_STATUS` =~ ^(Pending|Running)$ ]]; do echo "Waiting for pod to start and finish - $POD"; sleep 1; done; kubectl describe job ${KUBE_APP}-migration-job --namespace=$KUBE_NAMESPACE kubectl get pod -a $POD --namespace=$KUBE_NAMESPACE --output=jsonpath="{.status.phase}" || true kubectl logs $POD --namespace=$KUBE_NAMESPACE || true kubectl delete -f kube/${KUBE_APP}-migration.yaml --namespace=$KUBE_NAMESPACE;
- 30. CD Example (deployment) sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-cron-standings.yaml sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-cron-random.yaml sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-cron-atokens.yaml sed -i.bak "s#${IMAGE_NAME}:latest#${IMAGE_NAME}:${IMAGE_TAG}#" kube/${KUBE_APP}-depl.yaml kubectl apply -f kube/${KUBE_APP}-cron-standings.yaml --namespace=${KUBE_NAMESPACE} kubectl apply -f kube/${KUBE_APP}-cron-random.yaml --namespace=${KUBE_NAMESPACE} kubectl apply -f kube/${KUBE_APP}-cron-atokens.yaml --namespace=${KUBE_NAMESPACE} kubectl apply -f kube/${KUBE_APP}-depl.yaml --namespace=${KUBE_NAMESPACE}
- 31. Problems with this approach • Complex and brittle code • Duplication • Not all resources are updated • Cannot rollback as a unit
- 32. Helm
- 33. What is it? • Package format (Chart) • Package manager
- 34. Advantages • Official CNCF project • Charts bundle all kubernetes descriptors • Atomic install/upgrade/rollback (--atomic) • Lifecycle hooks • Chart sharing
- 35. Useful charts • NGINX ingress - https://kubernetes.github.io/ingress-nginx/ • cert-manager - https://github.com/jetstack/cert-manager
- 36. Helm Architecture • Helm • Local chart development • Managing repositories • Interacting with Tiller server • Tiller • Listens for incoming requests from the Helm client • Combines a chart and config to build a release • Installs charts into Kubernetes and manages the subsequent release
- 38. Chart directory structure .helmignore Chart.yaml values.yaml charts/ templates/deployment.yaml templates/ingress.yaml templates/service.yaml templates/_helpers.tpl templates/NOTES.txt Patterns to ignore Chart description Default values for chart Dependency charts Usage notes Kubernetes manifests Manifest helpers
- 39. Example helm commands •Create a chart: •helm create chart •Creates new release from a chart: •helm install CHART --name RELEASE -f v.yaml --namespace NS --atomic •List deployed releases: •helm ls •Upgrade deployed release: •helm upgrade RELEASE CHART -f v.yaml --namespace NS --atomic •Rollback release to a previous version: •helm rollback RELEASE VERSION
- 40. Helm demo
- 41. Helm templating • Go templates (https://godoc.org/text/template) • Sprig (https://godoc.org/github.com/Masterminds/sprig) • env, expandenv • include, required
- 42. Template partials {{/* Generate basic labels */}} {{- define "mychart.labels" }} labels: generator: helm date: {{ now | htmlDate }} chart: {{ .Chart.Name }} version: {{ .Chart.Version }} {{- end }} apiVersion: v1 kind: ConfigMap metadata: name: {{ .Release.Name }}-configmap {{- include "mychart.labels" . }}
- 43. Template flow control {{- range $job := .Values.jobs }} --- apiVersion: batch/v1beta1 kind: CronJob spec: concurrencyPolicy: {{ $job.concurrencyPolicy }} jobTemplate: spec: template: spec: containers: - image: "{{ $job.image.repository }}:{{ $job.image.tag }}" name: {{ $job.name }} {{- if $job.command }} command: {{ $job.command }} {{- end }} {{- with $job.args }} args: {{ toYaml . | indent 12 }} restartPolicy: {{ $job.restartPolicy }} schedule: {{ $job.schedule | quote }} {{- end }} jobs: - name: hello-world image: repository: hello-world tag: latest schedule: "* * * * *" concurrencyPolicy: Allow restartPolicy: OnFailure - name: hello-ubuntu image: repository: ubuntu tag: latest schedule: "*/5 * * * *" command: ["/bin/bash"] args: - "-c" - "echo $(date) - hello from ubuntu" concurrencyPolicy: Forbid restartPolicy: OnFailure Values Template
- 44. Problems • Server-side component: Tiller • Error handling • Templates of templates • Whitespace significant templates
- 45. Tips • Debug using "helm template" • Pass only image tag value on command line
- 46. QA