2018 04-06 kubernetes ingress in production
- 1. DecompileD 2018 2018-04-06 SANDOR SZÜCS @sszuecs Kubernetes Ingress in production
- 2. 2 WE ARE CONSTANTLY INNOVATING TECHNOLOGY HOME-BREWED, CUTTING-EDGE & SCALABLE technology solutions ~ 2,000 employees from tech locations (HQs in Berlin)7 77 nations help our brand to WIN ONLINE
- 4. 4 FOUR ERAS AT ZALANDO TECH ZOMCATPHP STUPS KUBERNETES 2010 2015 2016 Data center WAR LXC AWS Docker Cloud Formation Low level (AWS API) AWS Docker Cloud Formation Kubernetes manifest Higher abstraction level Data center PHP files
- 7. 7 KUBERNETES
- 8. 8 DEPLOYMENT - PODs A deployment defines A set of PODS
- 9. 9 SERVICE A service is an cluster internal TCP loadbalancer TO PODS
- 10. 10 INGRESS AN EXTERNAL ACCESS POINT TO SERVICES
- 12. 12 Logical Target picture ALB Node Skipper Node Skipper MyApp MyApp MyApp Service Service K8s network EC2 network TLS HTTP
- 13. 13 Technical Target picture ALB Node Skipper Node Skipper MyApp MyApp MyApp K8s network EC2 network TLS HTTP
- 14. 14 ingress
- 15. 15 Skipper • Skipper → HTTP Route $ curl -H “Host: my-app.example.org” http://172.1.2.6:9999/ https://github.com/zalando/skipper
- 16. 16 Kube-ingress-aws-controller • Kubernetes Ingress Controller for AWS → ALB+TLS $ curl -k -H “Host: my-app.example.org” https://aws-5178-lb-z82sf5u4ae0v-1194901579.eu-central-1.elb.amazonaws.com/ https://github.com/zalando-incubator/kube-ingress-aws-controller
- 17. 17 External DNS • External DNS → DNS name $ curl https://my-app.example.org/ https://github.com/kubernetes-incubator/external-dns
- 18. 18 Skipper Intro
- 19. 19 Skipper: Predicate https://github.com/zalando/skipper Routing Table Route FilterPredicate Predicate Predicate Filter Filter Route Route Route Request
- 20. 20 Skipper: Filter Cookie pictogram created by iconoci from the Noun Project Filter Filter FilterResponse Filter Filter FilterRequest Filter Request_1Request ResponseResponse_1 /api /
- 21. 21 Skipper - syntax Ingress annotations: • zalando.org/skipper-predicate: pred1 && pred2 • zalando.org/skipper-filter: fltr1 -> fltr2
- 22. 22 Ship to production • Skipper highlevel patterns • Shadow traffic • Blue-green deployments
- 24. 24 Dev lifecycle - real world Dev TestFAIL Deploy
- 28. 28 Traffic Switching - Ingress configuration https://github.com/zalando/skipper
- 29. 29 Traffic Switching - interface $ kubectl plugin skipper traffic myapp v1 v2 10 # traffic <ingress> <svc-old> <svc-new> <perc> https://github.com/sszuecs/kubectl-plugins
- 30. 30 Test new features • Skipper highlevel patterns • Feature toggle • A/B tests
- 31. 31 Feature Toggle https://github.com/zalando/skipper Skipper alpha ?v=alpha caller
- 33. 33 A/B tests part 1 https://github.com/zalando/skipper Skipper A B 10% rest Request Response with Cookie
- 34. 34 A/B part 1 10% chance
- 35. 35 A/B part 1
- 36. 36 A/B tests part 2 https://github.com/zalando/skipper Skipper A B flavor=A flavor=B Request with Cookie Response
- 37. 37 A/B part 2 - matching cookie
- 38. 38 A/B part 2 - matching cookie
- 39. 54 Open Source LINKS Skipper HTTP Ingress Router https://github.com/zalando/skipper Skipper documentation https://zalando.github.io/skipper Kubectl plugin skipper https://github.com/szuecs/kubectl-plugins Kube AWS Ingress Controller https://github.com/zalando-incubator/kube-ingress-aws-controller External DNS https://github.com/kubernetes-incubator/external-dns Zalando Cluster Configuration https://github.com/zalando-incubator/kubernetes-on-aws
- 40. QUESTIONS? SANDOR SZÜCS TECH INFRASTRUCTURE SOFTWARE ENGINEER sandor.szuecs@zalando.de @sszuecs Illustrations by @01k
Editor's Notes
- #2: Welcome to DecompileD! Today, my talk is about - Kubernetes Ingress in production. To have some context I will show you some Zalando numbers.
- #3: There are about 2000 employees working for Zalando Tech. We have 7 tech hubs in Europe. My customers are all developer teams and we need to scale!
- #4: Let&apos;s have a brief look into zalando’s technology stack to have more technical context
- #5: We started as PHP magento shop. We rewrote it with Java and Postgres and deployed it into Linux containers. With a management shift we went to the AWS cloud and now evolve into a state of the art kubernetes infrastructure. We use Docker as deploy artifacts and Kubernetes to orchestrate them.
- #6: What is meant by large scale?
- #8: Let’s have a very brief look into Kubernetes objects relevant to the talk
- #9: A Deployment creates a set of Pods. &lt;wait&gt;
- #10: A Kubernetes service selects a set of PODs and acts as TCP loadbalancer to them &lt;wait&gt;
- #11: An ingress is an external access point to services &lt;wait&gt;
- #12: Because we have about 300 teams that want to deploy, we need automations that build loadbalancer infrastructure. We do this based on the Ingress definition. Let’s see what we want to build and how we do it.
- #13: There are 2 loadbalancer components involved: The application loadbalancer ALB, and skipper. You see the blue boxes.&lt;wait&gt; Request processing is going from top to bottom: First TLS is terminated on the ALB Skipper is target of all ALBs. Skipper runs on every worker node and does http routing. Skipper selects MyApp PODs via Kubernetes service MyApp boxes are your application PODs.
- #14: Technically, skipper bypasses Kubernetes service to reach PODs directly. Like this we can do proper loadbalancing and do retries on failing connections. &lt;wait&gt;
- #15: An ingress object is glueing the blue loadbalancer together with the green backends. You see two marked definitions: host is the host header for the frontend http routing And backend is used to find the application
- #16: If we created this ingress object, Skipper creates an HTTP route based on the provided configuration.&lt;wait&gt; From cluster nodes we can call a skipper endpoint with the specified Host header to reach our application. &lt;wait&gt;
- #17: Kube-ingress-aws-controller creates an ALB with attached certificates pointing to skipper. With this inplace, you can create an HTTPS request to an ALB.&lt;wait&gt; The ALB target shown is a route53 ALIAS record.&lt;wait&gt; With the correct host header set, a request will reach your application.&lt;wait&gt;
- #18: External DNS creates a public DNS record to the ALB.&lt;wait&gt; Now, we have everything we need to serve public traffic from the internet.&lt;wait&gt; Everything is automated and a deployer has only to provide an ingress definition.
- #19: To understand highlevel deployment patterns, I will give you a brief introduction to skipper.&lt;wait&gt; Skipper is a flexible cloud native http proxy router. It is made for frequently changing configurations.&lt;wait&gt; Additionally, skipper has 2 building blocks seen by users: Predicates and Filters
- #20: Skipper has a routing table proven to scale beyond 200.000 routes.&lt;wait&gt; A routing table consists of a number of routes.&lt;wait&gt; An http request will be mapped by Predicates to a specific Route.&lt;wait&gt; Each route has a set of filters.&lt;wait&gt;
- #21: HTTP requests and responses can be changed by Filters. &lt;wait&gt; For example we can change the path of the request from /api to / &lt;wait&gt;, which we might add in the response again.&lt;wait&gt; We can also set a Cookie in the response. &lt;wait&gt;
- #22: Predicates and Filters can both be set by Ingress annotations: &lt;wait&gt; skipper-predicate and skipper-filter &lt;wait&gt; You now have an understanding of required details for the next sections.
- #23: Besides the Kubernetes rolling update strategy, skipper supports &lt;wait&gt; Shadow traffic &lt;wait&gt; and blue-green deployments.&lt;wait&gt; Let’s see why..
- #24: A common development cycle looks like this.&lt;wait&gt; We develop and test and if these are successful.&lt;wait&gt; We deploy and go production.&lt;wait&gt; We do this all the day. &lt;wait&gt; If not we drink coffee and attend meetings.&lt;wait&gt;
- #25: In real world we see failures after new deployments, &lt;wait&gt; because the newer version might be slower than before.
- #26: One solution is to target your new application with current life traffic.&lt;wait&gt; Shadow traffic allows you to test with live traffic without notice of your users.&lt;wait&gt; Skipper can copy the request to a new target and drop the response from the new one.&lt;wait&gt; This we call shadow traffic&lt;wait&gt;
- #27: You can use the tee() filter to copy the full request to another URL target. This gives you flexibility however your new service is structured.
- #28: Another solution is to use blue-green deployments. Skipper can split traffic to different Kubernetes services. Like this you can rollout a version v2 and slowly ramp up traffic. How do you do it?
- #29: Again using ingress! You see the backend-weights annotation set to 90 and 10 for the 2 service backends for hostname “my-app.example.org”. Skipper will split the traffic as you defined in ingress. Here 90% of the traffic will target my-app-v1 and 10% v2.
- #30: As user interface you can use a kube control plugin to do traffic switching from v1 to v2. The last argument is the percentage of traffic you want to direct to the new service. The old one will get the rest of the traffic.
- #31: How do you downgrade a feature or test that a new feature is a success? Feature toggle and A/B tests can do that and skipper can help you to implement these.
- #32: A feature toggle can be easily downgraded on failure by your caller. If v equals alpha does not reply in time, call next time without this query. The caller can decide, if the feature is enabled or not.
- #33: To implement a feature toggle, you create an additional ingress. If a request matches the query “v equals alpha” and the host header, skipper will proxy to alpha service.
- #34: To check if implementation A is better than B, you can use A/B tests. &lt;wait&gt; A request without cookie matching our target has 10% chance to get a Cookie with “flavor equals A”. The rest will get “flavor equals B”
- #35: We see the traffic predicate matches the route by 10% chance. And skipper sets a Cookie with flavor A in the response. &lt;wait&gt;
- #36: Rest will get a cookie with flavor B &lt;wait&gt;
- #37: A request with cookie “flavor equals A” will be forwarded to service A. The same applies for B. Clients will stick to the chosen backend from part 1.
- #38: In case of a Cookie with flavor A, we call the backend a-app-svc &lt;wait&gt;
- #39: In case of of Cookie with flavor B, we call the backend b-app-svc &lt;wait&gt;
- #40: To run applications in production, you need to have visibility
- #41: How do you get all logs from one request across all backends? This what X-FlowID is for. Skipper sets an X-FlowID header, if not passed in the request. Applications only have to log this header in their handlers.
- #42: To find a log trace you can grep for the FlowID in this case: capital A
- #43: To answer the question if your backend application is slow, or returns errors, you want to have metrics from your loadbalancer
- #44: Skipper measures and exposes roundtrip metrics, errors, counters and histograms. We export metrics as json or Prometheus format.
- #45: To find which part of a service is slow, you should setup opentracing. This enables you to get waterfall charts to boil down which service in the chain is slow
- #46: Skipper can add automatically tracing headers to all incoming requests and reports to agents. This allows you to see skipper in traces shown before.
- #47: For resiliency, we have ratelimits and automatic retries. Additionally we also have circuitbreaker and you can also add throttling or packetloss, but I will not show this today.
- #48: Ratelimits can be used to protect your backends. You see incoming 1k requests per second and only 300 will be forwarded, rest will get a HTTP code 429
- #49: To allow 100 requests per second to the defined backend, we setup cluster Ratelimits as skipper Filter
- #50: Client side ratelimits can be used to protect your login page. For example allow 10 requests per hour rest will get a HTTP code 429
- #51: The shown cluster ratelimit filter with a third parameter allows 10 requests per hour per X-Forwarded-For header to the defined backend
- #52: Skipper can do retries. For example the first request goes to a POD which is ..
- #53: .. failing, so skipper will get a connect refused from the backend. Skipper will do ..
- #54: .. a retry to the other available POD. This is safe to do, because we only retry on errors if we did not send data.
- #55: That was it for today. We would like to hear from you in github issues or our skipper google group! We are also available in K8s Slack #sig-aws and #external-dns or ping me at twitter.
- #56: Questions? &lt;&lt;prev slide to Show the links&gt;&gt;