Log and control all service-to-service traffic in one place (Kelvin Wong)
0 likes143 views
The document discusses the challenges of migrating to microservices, focusing on issues like diagnosing network faults and managing fault-handling logic. It presents solutions such as API gateways and service meshes, highlighting their features and trade-offs. The apex system is introduced as a transitional architecture designed to simplify service-to-service traffic management and log aggregation during the migration process.
![NGINX’s ‘Router
Mesh’ model
“
81
[The Router Mesh
model] covers the
needs of a great many
microservices apps. 1
1. https://www.nginx.com/blog/microservices-reference-architecture-nginx-router-mesh-model/](https://image.slidesharecdn.com/apexpresentationforlondonmicroservicesmeetup2020-06-30-200705193510/85/Log-and-control-all-service-to-service-traffic-in-one-place-Kelvin-Wong-81-320.jpg)
Log and control all service-to-service traffic in one place (Kelvin Wong)
- 1. Migrating to microservices An open-source API proxy for logging and controlling traffic between microservices 1
- 2. Agenda 2 1) Challenges with microservices 2) Existing solutions 3) Apex design and architecture 4) Apex implementation and deployment
- 3. 1) Challenges with microservices 3
- 4. Challenges with microservices 4 1. Diagnosing network faults 2. Managing fault-handling logic
- 5. Microservices rely on the network 5
- 6. Monolith 6 With subsystems all running on same server…
- 7. Subsystem communication in monolith ● …calls between subsystems happen in-process ● …and are reliable and fast 7
- 8. Microservices 8 Now subsystems are ‘services’ running on different servers…
- 9. Subsystem communication between microservices 9 ● …calls between services are HTTP requests/responses ● …and are prone to latency and failure
- 10. Network latency ● Network hops measured in 10s or 100s of milliseconds 10
- 11. Network unreliability 11 ● Requests or responses can be dropped ● Even if service code is bug-free
- 13. Multi-service workflows are difficult to diagnose 13 ● E.g. placing order triggers multiple hops between services ● Fault can occur at any step
- 14. Multi-service workflows are difficult to diagnose 14 ● Each service outputs its logs in different place
- 15. Multi-service workflows are difficult to diagnose 15 ● Developer must trace request through multiple sets of logs ● Need all the logs in one place!
- 17. Sometimes, should just retry failed requests 17 ● Network faults can be random ● Retrying a request may be enough
- 18. Defining fault-handling logic requires care 18 ● Retry too soon / too often → overwhelm responding service
- 19. Different services can follow different rules 19 ● orders → shipping ○ Wait 3s ○ Retry 5 times ● By default ○ Wait 5s ○ Retry 2 times
- 20. All this logic has to be defined somewhere 20
- 21. Fault-handling logic in HTTP client libraries ● Client libraries imported into service code ● E.g. Ruby Gem, or Node Package 21
- 22. Other network concerns in HTTP client libraries ● Libraries also handle other network concerns ● E.g. rate-limiting, authentication, caching etc. 22
- 23. A new library for every language ● shipping’s owner writes a library for every language 23
- 24. Client libraries can become hard to manage ● Every new language requires a new library from every service it needs to call 24
- 25. Updating global rules becomes slower ● Changing traffic rules requires updating many libraries in many services ● E.g. “Rate-limit all services to 10,000 requests per minute” 25
- 26. Summary 26
- 27. Challenges with microservices 27 1. Diagnosing network faults ○ Esp. when workflows span multiple services, and logs are distributed 2. Managing fault-handling logic ○ Client libraries can be difficult to manage with many services
- 29. Solutions for challenges with microservices 29 1. API gateway 2. Service mesh
- 30. Solutions for challenges with microservices 30 1. API gateway 2. Service mesh Both are built upon proxy servers
- 31. Solutions for challenges with microservices 31 1. API gateway 2. Service mesh Both are built upon proxy servers
- 33. Microservices with API gateway ● Proxy server ● Intercepts all incoming requests into system 33
- 34. API gateway provides stable API for clients ● Provides stable API for clients ● Can change internal architecture, without updating clients 34
- 35. API gateway also provides one place to handle more networking concerns 35
- 36. Reminder: challenges with microservices 36 1. Diagnosing network faults 2. Managing fault-handling logic
- 37. Could we deploy an API gateway internally between services? 37
- 38. Theoretically yes, but existing API gateway solutions are not optimal ● Many features specific to handling client-server traffic ● Unnecessarily complex and poor fit for service-to-service traffic* 38* According to NGINX, maker of the popular open-source NGINX load balancer and web server
- 39. Still a useful pattern! ● Source of inspiration for Apex’s architecture 39
- 41. Service mesh has two main components 1) Sidecar proxies (data plane) 2) Configuration server (control plane) 41
- 42. 1st component Sidecar proxies (data plane) 42
- 43. Services and sidecar proxies ● Services talk through sidecar proxies ● Sidecar proxies manage retries, routing, caching etc. 43
- 44. Services and sidecar proxies ● Services code can focus on its core business logic ● Networking concerns outsourced to sidecar proxies 44
- 45. 2nd component Configuration server (control plane) 45
- 46. ● One configuration server with source of truth for config data ● Retries, rate limits, routing, access roles etc. 46 Service mesh and configuration server
- 47. ● Updates are made in one place, in the config server ● Sidecar proxies then update their cached copies 47 Service mesh and configuration server
- 48. Reminder: challenges with microservices 48 1. Diagnosing network faults 2. Managing fault-handling logic
- 49. ● Config server: one place to define and update fault-handling logic ● Sidecar proxies: execute fault-handling logic + log all requests and responses 49 Service mesh provides a strong solution
- 50. ● No single point of failure or traffic bottleneck → scalable and resilient to faults 50 Service mesh provides a strong solution
- 51. Service mesh trade-off - complexity ● Double # of components ● Istio, Linkerd 1 can be complex ● Few easy options if not containerised, or not on Kubernetes 51
- 53. API gateway vs service mesh 53 ● API gateways are simpler to deploy and operate… ● …but not optimized for service-to-service traffic
- 54. API gateway vs service mesh 54 ● Service meshes provide scalable and resilient solution… ● …but could require redeploying with containers and K8s ● …or be complex to deploy and operate
- 55. 3) Apex design and architecture 55
- 56. Apex is designed for teams migrating a monolith to microservices ● First few microservices (1 - 10 services) ● Deployed without containers and/or K8s, or deployed on PaaS solutions e.g. Heroku 56
- 57. Problem addressed by Apex ● Manage service-to-service traffic ● Aggregate logs and manage traffic rules in one place 57
- 58. Apex trade-offs (1) ● Simple to deploy and operate ● Preserve existing deployment patterns 58
- 59. Apex trade-offs (2) Willing to sacrifice: ● Rich built-in features ● High availability ● High scalability 59
- 60. Apex architecture - high-level overview 60
- 61. Microservices architecture with Apex 61 ● Apex mediates traffic between all services ● Akin to ‘internal API gateway’
- 63. Apex architecture with middleware 63
- 64. Apex as solution to microservices challenges 64
- 65. Reminder: microservices challenge #1 65 1. Diagnosing network faults ○ Esp. when workflows span multiple services, and logs are distributed 2. Managing fault-handling logic ○ Client libraries can be difficult to manage with many services
- 66. All requests and responses sent to apex-logs-db 66 ● No client libraries needed ● Similar functionality to log aggregators
- 67. Apex traces requests and responses belonging to same cycle 67
- 68. Apex can trace traffic across services in multi-service workflows 68
- 69. Logs can be queried on apex-admin-ui 69
- 70. Demo - querying logs 70 ● Send request from Postman ● Copy correlation_id from response ● Query apex-logs-db by correlation_id
- 71. Reminder: microservices challenge #2 71 1. Diagnosing network faults ○ Esp. when workflows span multiple services, and logs are distributed 2. Managing fault-handling logic ○ Client libraries can be difficult to manage with many services
- 72. apex-config-store stores credentials and retry logic in one place 72 ● Similar to config server in a service mesh ● ‘Service mesh lite’ architecture
- 73. apex-config-store stores credentials and retry logic in one place 73 ● Service credentials for authentication ● Service host names for routing
- 74. apex-config-store stores credentials and retry logic in one place 74 ● Global defaults: ○ default-default ● Service-specific overrides ○ orders-shipping ○ shipping-inventory
- 75. apex-proxy queries apex-config-store for every request 75 ● Authenticate service ● Route request to correct service ● Retrieve retry logic
- 76. apex-config-store provides one place to update config ● Updates can be made on apex-admin-ui 76
- 77. From Apex to service mesh 77
- 78. Apex vs service mesh 78 ● Apex trades off availability and scalability… ● …for ease of deployment and operation - especially without containers / K8s
- 79. Apex: a transitional architecture in adopting service mesh 79 “Internal API gateway” 1 “Service mesh lite” 2 “Router mesh” 3 1. https://konghq.com/blog/kong-service-mesh/ 2. https://developer-docs.citrix.com/projects/citrix-k8s-ingress-controller/en/latest/deploy/service-mesh-lite/ 3. https://www.nginx.com/blog/microservices-reference-architecture-nginx-router-mesh-model/
- 80. NGINX’s ‘Router Mesh’ model “ 80 As the intermediate model… The Router Mesh Model is relatively easy to implement, powerful, efficient, and fast. 1 1. https://www.nginx.com/blog/microservices-reference-architecture-nginx-router-mesh-model/
- 81. NGINX’s ‘Router Mesh’ model “ 81 [The Router Mesh model] covers the needs of a great many microservices apps. 1 1. https://www.nginx.com/blog/microservices-reference-architecture-nginx-router-mesh-model/
- 82. 4) Apex implementation and deployment 82
- 83. Apex architecture with technologies 83
- 84. apex-proxy: Node.js and Express.js ● Node.js: handle huge number of simultaneous connections with high throughput ● Express.js: easily extensible with middleware 84
- 85. apex-logs-db: TimescaleDB ● Time-series database: logs are time-series in nature 1 ● TimescaleDB: fast ingest rates (100,000+ rows per second, even with billions of rows) 85 1. https://blog.timescale.com/blog/what-the-heck-is-time-series-data-and-why-do-i-need-a-time-series-datab ase-dcf3b1b18563/
- 86. apex-config-store: Redis ● Redis: high read rate (72,000 requests per second) 86
- 87. Minimising latency ● Fast components ○ Node.js and Express.js ○ Redis ○ TimescaleDB ● Load-testing: ○ <1 second added latency per request-response cycle 87
- 88. Deploying with Docker Compose ● Docker & Docker Compose: easy to deploy ● AWS ECS 88
- 89. Deploying to VPC 89 ● Deploy to same VPC as services to minimise latency
- 90. Thanks! 90
- 91. Kelvin Wong Full-stack software engineer based in London kelvinjhwong.com kjhwong@gmail.com 91