Lunar Way and the Cloud Native "stack"
3 likes704 views
Kasper Nissen, a DevOps engineer at Lunar Way, discusses the company's cloud-native architecture, which utilizes Kubernetes and microservices to enhance scalability, autonomy, and continuous deployment. The presentation highlights the challenges faced in development, tooling, and maintaining visibility in a distributed system while leveraging CNCF projects for improved operations and monitoring. Nissen emphasizes the importance of flexibility, fault tolerance, and resource optimization in achieving a robust infrastructure for financial services, with plans to expand beyond Denmark into the Nordics.
Lunar Way and the Cloud Native "stack"
- 1. MAKE MONEY MATTER. Cloud Native Aarhus By Kasper Nissen (@phennex), DevOps Engineer @thelunarway
- 2. MAKE MONEY MATTER. Cloud Native Aarhus By Kasper Nissen (@phennex), DevOps Engineer @thelunarway
- 3. Lunar Way • The Partner model • Leverage the partner banks infrastructure • All money is in the partner bank • Currently only in Denmark, will move to the nordics in the near future
- 4. Kasper Nissen DevOps & Infrastructure Engineer @thelunarway Experience DevOps & Infrastructure Engineer @ LEGO (CITMABIS) (oursourced by IT Minds) for 5 months Senior/Software Engineer @ IT Minds (~4 years part time) Master thesis: KubeCloud - A Small-Scale Tangible Cloud Computing Environment. Interview with Software Engineering Daily: bit.ly/2paZ5lg Blogging about Cloud Native Tech @ www.kubecloud.io M. Eng. Computer Technology from Aarhus University - Department of Engineering. B. Eng. Information and Communication Technology from Aarhus University - School of Engineering
- 5. What do we have running? 19services 215containers in prod 13infrastructure services 3node rabbitmq cluster 1rails monolith 2100 GB postgresql DB’s 3kubernetes clusters 3AWS Accounts
- 6. Where are we running?
- 8. Service overview (with infrastructure) api nemid signup goals feeds topup bec-bank-nk bec-bank-ka credit support supportsystem prometheus elasticsearch pushgateway kibana alertmanager fluentd node-exporter rabbitmq social stream insight linkerd
- 9. Infrastructure RabbitMQ Cluster Kubernetes Cluster Elasticsearch Cluster AWS PostgreSQL database
- 10. Where are we heading? Horses vs Unicorns - Gene Kim
- 11. Cloud Native Utopia Microservice oriented Container packaged Dynamically scheduled
- 12. MICROSERVICES
- 13. Why ? • Development • Freedom and autonomy • Best tool for the job • Speed • Architecture • Fault tolerance • Flexibility • Coherence, decoupling, encapsulation • Deployment • Independence • Scalability • Speed • Resource utilisation 1. 2. 3. (our Microservice vision)
- 14. How are we building our services? Asynchronous first Decoupling in time and space allows for autonomy Shared dependencies Common packages, such as logging, monitoring, communication Each service has it’s own repository Containing source code, deployment spec, pipeline, etc. HTTP REQ event Repo
- 15. Killing our Rails monolith … slowly strangling Rails Rails Rails
- 16. Languages
- 17. Challenges - so far • Development • Tooling - e.g. Swagger for events? • Local env • Shared code • Deployment • Automation • Versioning • Architecture • Distributed monolith • Transparency • Complexity • Cross cutting concerns • Operation • Monitoring • Tracing events
- 18. CONTAINERS
- 20. Why? Isolation Services are isolated and contained in their environment Consistency in portability The container will run in the same way in local env as in prod Versioning Versioning a container is easy, rolling back and forth becomes easy Container App Runtime OS Container App Runtime OS Container App Runtime OS Container App Runtime OS Development Container App Runtime OS Production Container App V1 Runtime OS Container App V2 Runtime OS Container App Runtime OS image:v1 image:v2 image:v3
- 23. Why? Scheduling The scheduler will schedule your service on a node Resource optimization Scheduling allows for better packaging of services in hosts Resiliency If a node dies, the scheduler will reschedule on another node Scalability Scaling a dynamically cluster is easy, just add more nodes
- 24. SO, HOW DO WE WORK TOWARDS BECOMING A UNICORN?
- 25. Currently used CNCF projects
- 27. Why Kubernetes? Community 48k+ commits, 22+ GitHub stars, 1.1k contributors Crossplatform Multiple arcs, multiple cloud providers Resource optimization Packing nodes to utilize available resources Tooling A lot of great tools Scaling Great integration with auto-scaling, both on node- and container-level High availability Automatic failover, redundancy Pure Awesomeness! It’s just awesome! 1.5 release: Estimated 400 years of work hours
- 28. What does it do? Node Node Node Node Node Node Node Node Node Node big dataapp Bapp A database datacenter Cluster Manager
- 29. Where does it run? eu-west-1a eu-west-1b eu-west-1c Private Public
- 30. Services running in kubernetes Networking infrastructure elb internal elb api elb nginx-ingress linkerd route53 default-backend Logging es-proxy Dev Demo service 1 service # … service 1 service # … Monitoring prometheus Misc sanity-checks release-notifier pushgateway alertmanager grafana fluentd kibana node-exporter postgresql-exporter rabbitmq-exporter cloudwatch- exporter
- 31. What do we think of it? Freedom Squads can deploy and more or less implement how they see fit Autonomous services Squads can work independent of other squads Continuous Delivery Kubernetes allows us to deploy multiple times a day. It’s easy to rollback in case something went wrong Flexibility We run many different type of workloads in the cluster. Gives us mobility to become cloud agnostic Scalable infrastructure Scaling the infrastructure is easy, both on node and container level High availability Kubernetes takes care of container failures, AWS Auto Scaling groups takes care of node failures Easy maintenance We are using Kubernetes Operations to help us spin up our clusters, and maintain them.
- 33. Why fluentd? Simple and Easy Provides a simple interface for specifying input and output. Works great with Kubernetes and containers. Community Big community around fluentd, validates our choice. Small memory footprint Do not require a lot of resources in the cluster Proven reliability and performance It’s a fairly battle tested project
- 34. What does it do? <source> @type tail path /var/log/containers/*.log pos_file /var/log/containers/es-containers.log.pos tag kubernetes.application.* format json time_key event_time time_format %Y-%m-%dT%H:%M:%S.%3NZ </source> <filter kubernetes.application.**> @type kubernetes_metadata merge_json_log true preserve_json_log false </filter> <match kubernetes.application.**> type elasticsearch host es-proxy port 9200 include_tag_key true logstash_format true logstash_prefix application- reload_on_failure true </match> 1. 2. 3.
- 35. Logging setup AWS Elasticsearch Cluster fluentd aws-signing-proxy
- 36. What do we think of it? Very easy to use Set up is easy! Works great with Kubernetes Awesome plugin for adding Kubernetes metadata - making it easy to identify pods etc. Run as a daemonset Easy to run in every node of the cluster as a daemonset
- 39. What does it do? Multi-dimensional data model Time series identified by metric name and key/value pairs Flexible query language Comes with a builtin query language for al kinds of operations, sums, averages, increase, rate, etc. Easy and simple Easy to setup, and works great with Kubernetes service discovery Alerting and great integration with Grafana Prometheus has a builtin alerting system, and Grafana provides easy integration for making metrics visible in dashboards Pull-based approach Prometheus scrapes it’s targets at a regular interval
- 40. What metrics are we collecting? Kubernetes specific metrics Pods running, health of Kubernetes system components, etc. RabbitMQ Activity in queues, unacknowledged messages Nodes CPU, Memory Traffic Incoming traffic, upstream latency in cluster, etc. Containers CPU, Memory Application Specific metrics Relevant metrics, instrumented by the services owners
- 41. DEMO
- 42. What do we think of it? Provides great insights Provides valuable insights in the state of the cluster Makes is easy to developers to instrument their services We provide a simple package for instrumentation, making squads able to do their own monitoring. YOU BUILT IT, YOU RUN IT! Grafana integrations is sweet! Grafana and Prometheus works well together, making Grafana the interface for building dashboards and alerts Kubernetes <3 Prometheus The only thing a service owner has to do in the cluster to make Prometheus scrape their services is to add: annotations: prometheus.io/scrape: 'true'
- 43. gRPC Service to service communication with
- 44. What? gRPCserver Service #2Service #1 gRPCclient Protocol buffer proto req proto response
- 45. What do we use it for? It will be our default choice for synchronous calls Synchronous service to service communication will be aligned on gRPC Internal support system will use it to fetch data from our services Our internal support system needs information from different services on demand, the service will use gRPC to fetch the data service 4 service 3 service 1 service 2 Support
- 46. Why gRPC? Simple service definition using Protocol buffers Simpler request handling, no need for serialization and deserialization Binary protocol Less overhead in communication. Works across multiple languages and platforms gRPC has a widespread support for multiple languages, making it a perfect fit in our current polyglot architecture Works great with go and the rest of the ecosystem Docker, Kubernetes uses gRPC as long with Go. It’s a natural extension for service to service communication. Based on many years of Google experience!
- 48. Are we there yet? YES! and no.. We deploy multiple times a day Deployment is autonomous, squads can deploy to production as they please. We can easily scale to larger demands, if necessary Scaling our infrastructure is easy HOWEVER, doing microservices are complex! We still need to implement better tracing, using the CNCF project OpenTracing and Zipkin We need more insights and smarter routing in our service to service communication, we will be using linkerd. We can tolerate AZ failures to some extend Our services are spread across availability zones
- 49. DO YOU HAVE ANY QUESTIONS? Last thing
- 50. Thank you for listening! That was it for me! If you wanna know more, send me a message in the Cloud Native DK Slack Community. Remember to sign up at: https://cloudnative-dk.herokuapp.com/ Catch me on Twitter @phennex I will be speaking again: • CoDe:U - Continuous Delivery Users Århus (June 20th in INCUBA, Åbogade 15, Aarhus N) • Link: https://www.meetup.com/CoDe-U-AROS/events/239847862/ • GOTOCon Copenhagen - October 1st • Link: https://gotocph.com/2017/sessions/237