![11
Find external services using Envoy access logging
[2021-03-17T13:52:48.695Z] "GET /headers HTTP/1.1 -" 200 - "-" "-" 0 2128 63 62 "-" "curl/7.52.1"
"66e67f6d-4fa9-41e1-a103-1078b62221a5" "httpbin.org" "54.91.118.50:80" PassthroughCluster 10.44.0.7:34694
54.91.118.50:80 10.44.0.7:34692 - allow_any
log for HTTP:
curl http://httpbin.org/headers
[2021-03-17T13:56:04.611Z] "- - -" 0 - - - "-" 930 5818 153 - "-" "-" "-" "-" "54.91.118.50:443" PassthroughCluster
10.44.0.7:47868 54.91.118.50:443 10.44.0.7:47866 - -
log for HTTPS/TCP:
spec:
meshConfig:
accessLogFile: /dev/stdout
frontend](https://image.slidesharecdn.com/operationalizingmulticlusteristiolessonslearnedanddevelopingambientmesh-221207163907-8deef4b5/85/Operationalizing-Multi-Cluster-Istio_-Lessons-Learned-and-Developing-Ambient-Mesh-pptx-pdf-11-320.jpg)
![19
Verify secure traffic using Envoy access logging
format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%
%PROTOCOL% %DOWNSTREAM_TLS_VERSION%" %RESPONSE_CODE% …
$ kubectl logs reviews-v1-6549ddccc5-f28vj -c istio-proxy
[2021-03-17T00:52:08.689Z] "GET /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 295 8 7 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 11_2_3)
Add %DOWNSTREAM_TLS_VERSION% to your access log formatting:
Check the proxy logs:](https://image.slidesharecdn.com/operationalizingmulticlusteristiolessonslearnedanddevelopingambientmesh-221207163907-8deef4b5/85/Operationalizing-Multi-Cluster-Istio_-Lessons-Learned-and-Developing-Ambient-Mesh-pptx-pdf-19-320.jpg)
![22
AuthorizationPolicy
apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
name: "database-viewer"
namespace: default
spec:
selector:
matchLabels:
app: database
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/myapp-orders"]
to:
- operation:
methods: ["GET"]
paths: ["/data"]
orders
PUT
database
Can also target Ingress Gateway
Optional
Can also support IP blocks
Or DENY
https://istio.io/latest/docs/reference/config/security/authorization-policy/](https://image.slidesharecdn.com/operationalizingmulticlusteristiolessonslearnedanddevelopingambientmesh-221207163907-8deef4b5/85/Operationalizing-Multi-Cluster-Istio_-Lessons-Learned-and-Developing-Ambient-Mesh-pptx-pdf-22-320.jpg)
![23
Use access logs to create your rules
$ kubectl logs reviews-v2-76c4865449-6wblx -c istio-proxy
…
[2021-03-17T02:27:18.143Z] “POST /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 379 29 28 "-" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
"2dc83abd-71e1-40f5-9a6d-5f02dfceaa78" "reviews:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:59238 10.44.0.8:9080
10.44.0.9:37276 outbound_.9080_._.reviews.default.svc.cluster.local default
…
[2021-03-17T02:29:31.493Z] "GET /ratings/0 HTTP/1.1 -" 200 - "-" "-" 0 48 3 2 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"
"a51c2b7b-a8e3-417e-8566-4258b734c287" "ratings:9080" "10.44.0.7:9080" outbound|9080||ratings.default.svc.cluster.local
10.44.0.8:51176 10.56.48.22:9080 10.44.0.8:56738 - default
https://istio.io/latest/docs/tasks/observability/logs/access-log/
IP of pod sending the request
(kubectl get pods -o wide)
method
Outbound traffic going to ratings
Inbound traffic](https://image.slidesharecdn.com/operationalizingmulticlusteristiolessonslearnedanddevelopingambientmesh-221207163907-8deef4b5/85/Operationalizing-Multi-Cluster-Istio_-Lessons-Learned-and-Developing-Ambient-Mesh-pptx-pdf-23-320.jpg)
![24
Debugging Authorization Policies
$ istioctl x authz check mydatabasepod
ACTION AuthorizationPolicy RULES
ALLOW database-viewer.default 1
ALLOW _anonymous_match_nothing_ 1
$ curl —-request POST database:9080
RBAC: access denied
$ kubectl logs mydatabasepod -c istio-proxy
[2021-03-17T19:31:34.595Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-"
"curl/7.52.1" "8510e2fd-53f7-4a5a-95da-8796d90c51a2" “database:9080" "-" - - 10.44.1.8:9080 10.44.0.7:51260
outbound_.9080_._.database.default.svc.cluster.local -
Response to caller when denied:
Check what policies are applied on a pod:
Access log entry on the server sidecar:](https://image.slidesharecdn.com/operationalizingmulticlusteristiolessonslearnedanddevelopingambientmesh-221207163907-8deef4b5/85/Operationalizing-Multi-Cluster-Istio_-Lessons-Learned-and-Developing-Ambient-Mesh-pptx-pdf-24-320.jpg)
Operationalizing Multi Cluster Istio_ Lessons Learned and Developing Ambient Mesh.pptx.pdf
- 1. Operationalizing Multi Cluster Istio: Lessons Learned and Developing Ambient Mesh Kevin Dorosh Engineer @ Solo.io Istio Contributor
- 2. 01 02 03 04 05 06 2 Istio Intro Securing traffic Resilience Ambient Mesh Ambient Mesh Demo Brief Multicluster Considerations What this talk will cover
- 3. 3 Connectivity & communication challenges
- 4. 4 Solo.io solves API connectivity & communication challenges
- 5. 5 Communication without Istio frontend backend
- 6. 6 Istio • Service discovery / Load balancing • Secure service-to-service communication • Traffic control / shaping / shifting • Policy / Intention based access control • Traffic metric collection • Service resilience • API / programmable interface frontend backend Istio
- 8. 8 Secure outbound traffic Application Cloud Services API Endpoints Malicious website
- 9. 9 Find external services using the Kiali service graph
- 10. 10 Find external services using Prometheus https://istio.io/latest/blog/2019/monitoring-external-service-traffic/
- 11. 11 Find external services using Envoy access logging [2021-03-17T13:52:48.695Z] "GET /headers HTTP/1.1 -" 200 - "-" "-" 0 2128 63 62 "-" "curl/7.52.1" "66e67f6d-4fa9-41e1-a103-1078b62221a5" "httpbin.org" "54.91.118.50:80" PassthroughCluster 10.44.0.7:34694 54.91.118.50:80 10.44.0.7:34692 - allow_any log for HTTP: curl http://httpbin.org/headers [2021-03-17T13:56:04.611Z] "- - -" 0 - - - "-" 930 5818 153 - "-" "-" "-" "-" "54.91.118.50:443" PassthroughCluster 10.44.0.7:47868 54.91.118.50:443 10.44.0.7:47866 - - log for HTTPS/TCP: spec: meshConfig: accessLogFile: /dev/stdout frontend
- 12. 12 Create ServiceEntries for external services apiVersion: networking.istio.io/v1alpha3 kind: ServiceEntry metadata: name: httpbin-ext spec: hosts: - httpbin.org ports: - number: 80 name: http protocol: HTTP resolution: DNS location: MESH_EXTERNAL $ istioctl proxy-config endpoints ratings-v1-fd78f799f-gj8td | grep httpbin 34.199.75.4:80 HEALTHY OK outbound|80||httpbin.org 34.231.30.52:80 HEALTHY OK outbound|80||httpbin.org 54.166.163.67:80 HEALTHY OK outbound|80||httpbin.org 54.91.118.50:80 HEALTHY OK outbound|80||httpbin.org https://istio.io/latest/docs/reference/config/networking/service-entry/
- 13. 13 Block undefined external endpoints myapp envoy Change outboundTrafficPolicy to REGISTRY_ONLY NOTE: Traffic can be configured to bypass envoy using annotations: traffic.sidecar.istio.io/excludeOutboundIPRanges traffic.sidecar.istio.io/excludeOutboundPorts spec: meshConfig: outboundTrafficPolicy: mode: REGISTRY_ONLY https://istio.io/latest/docs/tasks/traffic-management/egress/egress-control/
- 14. 14 Direct external traffic thru egress gateways myapp envoy curl http://httpbin.org/headers Egress GW httpbin.org • Dedicated traffic exit points • Only Egress gateways can run on internet enabled nodes • Meet compliance requirements • Use Network Policies on workloads to prevent bypassing the egress gateway https://istio.io/latest/blog/2019/egress-traffic-control-in-istio-part-1/ https://istio.io/latest/docs/tasks/traffic-management/egress/egress-gateway/
- 15. 15 Encrypting traffic frontend backend frontend backend fake-backend
- 16. 16 Istio Identity $ istioctl dashboard envoy productpage-v1-5d9b4c9849-l8jjq.default http://localhost:15000
- 17. 17 Istio mutual TLS mode apiVersion: security.istio.io/v1beta1 kind: PeerAuthentication metadata: name: default namespace: istio-system # can be granular spec: mtls: mode: PERMISSIVE # or STRICT • Change PERMISSIVE to STRICT after onboarding all your workloads to the mesh • Can be disabled at the port level frontend backend Another App
- 18. 18 Verify secure traffic using Kiali
- 19. 19 Verify secure traffic using Envoy access logging format: "[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL% %DOWNSTREAM_TLS_VERSION%" %RESPONSE_CODE% … $ kubectl logs reviews-v1-6549ddccc5-f28vj -c istio-proxy [2021-03-17T00:52:08.689Z] "GET /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 295 8 7 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) Add %DOWNSTREAM_TLS_VERSION% to your access log formatting: Check the proxy logs:
- 20. 20 Check for envoy listeners $ istioctl pc listener reviews-v1-6549ddccc5-f28vj | grep inbound 0.0.0.0 15006 Trans: tls; Addr: *:9080 Cluster: inbound|9080|| PERMISSIVE: $ istioctl pc listener reviews-v1-6549ddccc5-f28vj | grep inbound 0.0.0.0 15006 Trans: tls; App: Istio HTTP Plain; Addr: *:9080 Cluster: inbound|9080|| 0.0.0.0 15006 Trans: raw_buffer; Addr: *:9080 Cluster: inbound|9080|| STRICT:
- 21. 21 Controlling workload access catalog orders GET GET PUT POST PUT ui database
- 22. 22 AuthorizationPolicy apiVersion: "security.istio.io/v1beta1" kind: "AuthorizationPolicy" metadata: name: "database-viewer" namespace: default spec: selector: matchLabels: app: database action: ALLOW rules: - from: - source: principals: ["cluster.local/ns/default/sa/myapp-orders"] to: - operation: methods: ["GET"] paths: ["/data"] orders PUT database Can also target Ingress Gateway Optional Can also support IP blocks Or DENY https://istio.io/latest/docs/reference/config/security/authorization-policy/
- 23. 23 Use access logs to create your rules $ kubectl logs reviews-v2-76c4865449-6wblx -c istio-proxy … [2021-03-17T02:27:18.143Z] “POST /reviews/0 HTTP/1.1 TLSv1.2" 200 - "-" "-" 0 379 29 28 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "2dc83abd-71e1-40f5-9a6d-5f02dfceaa78" "reviews:9080" "127.0.0.1:9080" inbound|9080|| 127.0.0.1:59238 10.44.0.8:9080 10.44.0.9:37276 outbound_.9080_._.reviews.default.svc.cluster.local default … [2021-03-17T02:29:31.493Z] "GET /ratings/0 HTTP/1.1 -" 200 - "-" "-" 0 48 3 2 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36" "a51c2b7b-a8e3-417e-8566-4258b734c287" "ratings:9080" "10.44.0.7:9080" outbound|9080||ratings.default.svc.cluster.local 10.44.0.8:51176 10.56.48.22:9080 10.44.0.8:56738 - default https://istio.io/latest/docs/tasks/observability/logs/access-log/ IP of pod sending the request (kubectl get pods -o wide) method Outbound traffic going to ratings Inbound traffic
- 24. 24 Debugging Authorization Policies $ istioctl x authz check mydatabasepod ACTION AuthorizationPolicy RULES ALLOW database-viewer.default 1 ALLOW _anonymous_match_nothing_ 1 $ curl —-request POST database:9080 RBAC: access denied $ kubectl logs mydatabasepod -c istio-proxy [2021-03-17T19:31:34.595Z] "GET / HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "-" "curl/7.52.1" "8510e2fd-53f7-4a5a-95da-8796d90c51a2" “database:9080" "-" - - 10.44.1.8:9080 10.44.0.7:51260 outbound_.9080_._.database.default.svc.cluster.local - Response to caller when denied: Check what policies are applied on a pod: Access log entry on the server sidecar:
- 26. Request Authentication apiVersion: "security.istio.io/v1beta1" kind: "RequestAuthentication" metadata: name: "jwt-example" spec: selector: matchLabels: app: orders jwtRules: - issuer: "example-provider" jwksUri: "https://example-provider.com/jwks.json" ui JWT orders https://istio.io/latest/docs/tasks/security/authorization/authz-jwt/
- 27. 27 Improve application resilience frontend backend
- 28. Timeouts, circuit breaking and rate limiting apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: reviews spec: hosts: - reviews http: - route: - destination: host: reviews subset: v2 timeout: 0.5s apiVersion: networking.istio.io/v1alpha3 kind: DestinationRule metadata: name: httpbin spec: host: httpbin trafficPolicy: connectionPool: tcp: maxConnections: 1 http: http1MaxPendingRequests: 1 maxRequestsPerConnection: 1 outlierDetection: consecutive5xxErrors: 1 interval: 1s baseEjectionTime: 3m maxEjectionPercent: 100 apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: filter-local-ratelimit-svc namespace: istio-system spec: workloadSelector: labels: app: productpage configPatches: - applyTo: HTTP_FILTER listener: filterChain: filter: name: "envoy.http_connection_manager" patch: … stat_prefix: http_local_rate_limiter token_bucket: max_tokens: 10 tokens_per_fill: 10 fill_interval: 60s … https://istio.io/latest/docs/tasks/traffic-management/request-timeouts/ https://istio.io/latest/docs/tasks/traffic-management/circuit-breaking/ https://istio.io/latest/docs/tasks/policy-enforcement/rate-limit/
- 30. Istio - The Industry’s Leading Service Mesh 2017 Istio Launched 2022 Ambient Mesh Launched Data Plane Enhancements 2019-20 7 New Community Releases 1000s Production Customers ~ 1000 Community Contributors 2022 CNCF 2019-2022
- 31. Introducing Istio Ambient Mesh 31 | Copyright © 2022 A new, open source contribution to the Istio project, that defines a new sidecar-less data plane. Solo.io and Google are the lead contributors to Istio Ambient Mesh. Cost Reduction Simplify Operations Improve Performance
- 32. What is Istio Ambient Mesh? P P P P P P P P P P P P P P P P P P Proxy Istio Sidecar Data Plane 1 Pod/Container = 1 Proxy Ambient Mesh Data Plane 1 Node = 1 Proxy Move from Sidecar Proxy per-pod architecture to a Proxy per-node architecture. “Making the Mesh Transparent to Applications” ● Reduced Compute Cost ● Improve Business Continuity ● Increase Business Flexibility ● Simplified Operations ● Reduced Maintenance ● Simplified Upgrades ● Easier to Add Applications ● Less Day-2-Day Complexity ● Adapt to Application Needs ● Offer SLAs for Applications ● Many Apps = 1 Platform Application Team ● Mesh is transparent to Apps ● Applications won’t break ● Flexible Performance Available ● Manage Security vs Performance Business Owner Platform Team
- 33. Istio enables Zero-Trust Security P P P P P P P P P P P P P P P P P P L4 Proxy P P P P P P P P P P P P P P P P P P Istio Security with Sidecar Proxy Istio Security with Ambient Mesh L4 Proxy L7 Proxy ● All traffic goes through Proxy ● Proxy manages mTLS, Identity ● Proxy manages L7 Application Filters | Policies ● All traffic goes through Proxy ● L4 Proxy manages mTLS, Identity ● L7 Proxy manages L7 Application Filters | Policies
- 34. Istio enables Service Mesh flexibility P P P P P P P P P P P P P P P P P P Proxy Istio Sidecar Data Plane 1 Pod/Container = 1 Proxy Ambient Mesh Data Plane 1 Node = 1 Proxy Istio Control Plane Cost Flexibility Operational Flexibility Performance Flexibility
- 37. HBONE - The protocol used to connect nodes HTTP Based Overlay Network Encapsulation protocol source: https://www.solo.io/blog/understanding-istio-ambient-ztunnel-and-secure-overlay/
- 38. Latest Ambient Developments Rust Ztunnel source: https://www.solo.io/blog/rust-ztunnel-istio-ambient-mesh/ https://www.semanticscholar.org/paper/A-work-stealing-scheduling-framework-supporting-Wang-Ji/2b956f8fcd744c4aaf8b0a ed417f4706cc046545
- 41. 41 | Copyright © 2022
- 42. 42 | Copyright © 2022
- 43. 43 | Copyright © 2022
- 44. 44 | Copyright © 2022
- 45. Thank You!