KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline
0 likes769 views
The document discusses the use of ephemeral namespaces in CI/CD pipelines, highlighting their benefits for resource isolation, efficient testing, and quick provisioning. It covers concepts like namespace fundamentals, deployment processes, and specific Kubernetes commands for managing tests across various environments. The presentation also emphasizes best practices for policy management and system configurations to enhance development workflows.
![Running Every Piece in PodsRunning Every Piece in Pods
Nightwatch scripts
kubectl run -i -tty nightwatch --image=canthefason/e2e-tests:$E2E_IMAGE_TAG
--restart=Never --namespace=e2e-$GO_PIPELINE_LABEL
state=$(kubectl get -o template po nightwatch $kubeargs
--template={{.status.phase}})
while [ "$state" == "Running" ]; do
sleep 5
echo "waiting for the state"
state=$(kubectl get -o template po nightwatch $kubeargs
--template={{.status.phase}})
done
echo "State: $state"
if [ "$state" == "Failed" ]; then
exit 1
fi](https://image.slidesharecdn.com/k8s-ephemeral-namespaces-160329025515/85/KubeCon-EU-2016-Leveraging-ephemeral-namespaces-in-a-CI-CD-pipeline-23-320.jpg)
![Health CheckersHealth Checkers
Text
curl -k --retry 10 --retry-delay 5 -v
https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
curl -k --silent --output /dev/stderr --write-out "%{http_code}" -v
https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping
if [ "$STATUSCODE" -ne "200" ]; then
if [ "$rcExist" != "ReplicationController" ]; then
kubectl delete -f scripts/rc.yml $kubeargs
fi
exit 1
fi](https://image.slidesharecdn.com/k8s-ephemeral-namespaces-160329025515/85/KubeCon-EU-2016-Leveraging-ephemeral-namespaces-in-a-CI-CD-pipeline-25-320.jpg)
KubeCon EU 2016: Leveraging ephemeral namespaces in a CI/CD pipeline
- 1. LEVERAGINGLEVERAGING EPHEMERAL NAMESPACESEPHEMERAL NAMESPACES IN A CI/CD PIPELINEIN A CI/CD PIPELINE Can Yücel (@canthefason) Senior Software Engineer KubeCon EU 2016
- 2. HighlightsHighlights Fundamentals of namespaces Breaking the idea of having separate clusters Ephemeral namespaces Talk about some Kubernetes early stage features Running every single piece as Kubernetes components
- 3. NamespacesNamespaces “ A namespace is a mechanism to partition resources created by users into a logically named group. ~ Kubernetes Docs
- 4. Isolation on Different LevelsIsolation on Different Levels Network level isolation Access policies Resource control
- 5. Network Level IsolationNetwork Level Isolation Leveraging subdomainsLeveraging subdomains
- 6. Access PoliciesAccess Policies {"user":"admin"} {"user":"scheduler", "readonly": true, "resource": "pods"} {"user":"scheduler", "resource": "bindings"} {"user":"proxy", "resource": "services"} {"user":"proxy", "resource": "endpoints"} {"user":"kubelet", "resource": "pods"} {"user":"kubelet", "resource": "nodes"} {"user":"kubelet", "readonly": true, "resource": "services"} {"user":"kubelet", "readonly": true, "resource": "endpoints"} {"user":"kubelet", "resource": "events"} {"user":"bob", "readonly": true, "namespace": "prod"} {"user":"alice", "namespace": "prod"} policy.jsonl ABAC provides much more granularity on policy management
- 7. Resource ControlResource Control apiVersion: v1 kind: ResourceQuota metadata: name: quota spec: - hard: memory: "1Gi" cpu: 20 pods: 15 services: 5 replicationcontrollers: 10 resourcequotas: 1 Cluster: 32 GB RAM, and 16 cores Team A: 20 GB RAM, and 10 cores Team B: 10 GB RAM, and 4 cores
- 8. How to Partition?How to Partition? Environment based partitioningEnvironment based partitioning qa, stage, production... System / team based partitioningSystem / team based partitioning kube-system, devops, bots Project based partitioningProject based partitioning example.com, better-example.com
- 9. A Day of a CI/CD PipelineA Day of a CI/CD Pipeline Provision separate machines for every build Run your tests on isolated clusters When all tests are successful tear down the cluster If it fails keep the cluster up for a while for debugging Ephemeral Namespaces!Ephemeral Namespaces! namespaces namespaces namespace
- 10. Ephemeral Namespaces AreEphemeral Namespaces Are Isolated environments that are running different versions of services on top of it The environments where we run our integrations/e2e tests, and gets dumped when we get the end results
- 11. Namespaces with Benefits!Namespaces with Benefits! Time effective provisioning Efficient resource utilization In a CI/CD pipeline, namespaces provide:In a CI/CD pipeline, namespaces provide:
- 12. Time Effective ProvisioningTime Effective Provisioning It takes only a couple of seconds to create all services
- 13. Efficient Resource UtilizationEfficient Resource Utilization Let your scheduler decide on which host you will run your test instances
- 14. Deployment ProcessDeployment Process 1. Run your unit tests 2. Build Docker Image 3. Deploy to sandbox 4. Provision services that you will run your tests against 5. Run your integration/e2e tests 6. Delete namespace 7. Deploy updated services to staging/prod Happy Path!
- 15. Provisioning Test EnvironmentsProvisioning Test Environments Identical environments with different versions!
- 16. Pods From Different NamespacesPods From Different Namespaces ➜ kubectl get po --namespace=e2e-1 NAME READY STATUS RESTARTS AGE mongo-oij3f 1/1 Running 0 10m nginx-44k6p 1/1 Running 0 10m selenium-9bcfc 1/1 Running 0 10m todo-service-phgrb 1/1 Running 0 10m todo-service-rbrjl 1/1 Running 0 10m ➜ kubectl get po --namespace=e2e-2 NAME READY STATUS RESTARTS AGE mongo-p6g8c 1/1 Running 0 5m nginx-mgdzz 1/1 Running 0 5m selenium-9l81p 1/1 Running 0 5m todo-service-mt9gh 1/1 Running 0 5m todo-service-yxo9v 1/1 Running 0 5m ➜ kubectl get po --namespace=e2e-3 NAME READY STATUS RESTARTS AGE mongo-llm3x 1/1 Running 0 1m nginx-vvov6 1/1 Running 0 1m nightwatch 1/1 Running 0 34s selenium-g2g1i 1/1 Running 0 1m todo-service-1k8vc 1/1 Running 0 1m todo-service-ddfjw 1/1 Running 0 1m
- 17. Adding E2E Components as PodsAdding E2E Components as Pods Selenium server Nightwatch.js scripts
- 18. All Tests PassedAll Tests Passed ✓✓ $ kubectl delete namespace e2e-10 It will dump every Kubernetes component within that namespace!
- 19. Test Gets FailedTest Gets Failed 😞 Find a way to connect to the Selenium Server for debugging Expose VNC Port 5900 kubectl port-forward selenium :5900 --namespace=e2e-1
- 20. Live In ActionLive In Action
- 21. How We Use GoCDHow We Use GoCD Idempotent pipeline stages Dependency management is handled with fan-in resolution
- 22. Evaluating DependenciesEvaluating Dependencies Text GO_DEPENDENCY_LABEL_E2E=5.2eedd92 GO_DEPENDENCY_LOCATOR_E2E=e2e-tests/5/buildImage/1 GO_DEPENDENCY_LABEL_TODO=35.86ca86c GO_DEPENDENCY_LOCATOR_TODO=todo-service/35/deployK8s/1 GO_DEPENDENCY_LABEL_NGINX=12.4288a7c GO_DEPENDENCY_LOCATOR_NGINX=nginx/12/deployK8s/1 Each GO_DEPENDENCY variable has dependant pipeline information For Provisioning Test Environments: Create all dependencies For Deployment: Compare versions and call create/rolling update
- 23. Running Every Piece in PodsRunning Every Piece in Pods Nightwatch scripts kubectl run -i -tty nightwatch --image=canthefason/e2e-tests:$E2E_IMAGE_TAG --restart=Never --namespace=e2e-$GO_PIPELINE_LABEL state=$(kubectl get -o template po nightwatch $kubeargs --template={{.status.phase}}) while [ "$state" == "Running" ]; do sleep 5 echo "waiting for the state" state=$(kubectl get -o template po nightwatch $kubeargs --template={{.status.phase}}) done echo "State: $state" if [ "$state" == "Failed" ]; then exit 1 fi
- 24. Running Every Piece in PodsRunning Every Piece in Pods Selenium manifest apiVersion: v1 kind: ReplicationController metadata: name: selenium spec: replicas: 1 selector: app: selenium template: metadata: name: selenium labels: app: selenium spec: volumes: - name: shm hostPath: path: /dev/shm containers: - name: selenium image: selenium/standalone-chrome-debug:2.52.0 ports: - containerPort: 4444 - containerPort: 5900 imagePullPolicy: Always volumeMounts: - name: shm mountPath: /dev/shm
- 25. Health CheckersHealth Checkers Text curl -k --retry 10 --retry-delay 5 -v https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping curl -k --silent --output /dev/stderr --write-out "%{http_code}" -v https://$KUBE_HOST/api/v1/proxy/namespaces/sandbox/services/todo/ping if [ "$STATUSCODE" -ne "200" ]; then if [ "$rcExist" != "ReplicationController" ]; then kubectl delete -f scripts/rc.yml $kubeargs fi exit 1 fi
- 26. Future WorkFuture Work Scale down the pods when the namespace is idle Automatically delete namespaces that are older than certain age Build a Selenium Grid infrastructure and utilize Selenium Agents among the namespaces
- 27. TakeawaysTakeaways Never ever expose your Apiserver 8080 port! Think twice before defining your ssh keys as secrets! Make sure that you properly setup kubelet garbage collectors --maximum-dead-containers=100 --maximum-dead-containers-per-container=2 --minimum-container-ttl-duration=1m0s
- 29. Thanks ToThanks To Kubernetes Team LaunchPad Central Quest Henkart UK Consulate in NY...
- 30. Q & AQ & A Twitter: @canthefason GitHub: /canthefason