Microservices in AUTO1 by Alexander Egurtsov
0 likes313 views
This document describes Auto1's transition from a monolithic architecture to a microservices architecture. It discusses the original monolithic architecture's limitations around code duplication, shared databases, and difficulty with testing and deployments. It then outlines how Auto1 broke the monolith into over 150 microservices written in various languages. The rest of the document provides a "microservices cookbook" detailing best practices around service templates, communication patterns, CI/CD pipelines, monitoring, and other aspects of Auto1's microservices approach.
Microservices in AUTO1 by Alexander Egurtsov
- 1. Microservices in Auto1 by Alexander Egurtsov
- 2. Otsikko 2 About Auto1 Headquartered in Berlin, AUTO1 Group is Europe’s leading car trading platform. • Diverse team of over 3,500 employees • More than 55 nationalities • Over 1,000,000 private customers • Over 45,000 professional partners. The brand independent automotive company owns business units like AUTO1.com, wirkaufendeinauto.de and Autohero.
- 4. Otsikko 4 Original auto1 architecture • Code duplication • Shared database • Git Merge nightmare • Codebase complexity • No ownership • Regression testing is almost impossible • Long process from development to deploy
- 5. Otsikko 5 Breaking down monolith to microservices
- 6. Otsikko 6 Breaking down monolith to microservices • Code duplication Solved • Shared database Partially solved • Git Merge nightmare Solved • Codebase complexity Solved • No ownership Solved • Regression testing is almost impossible Solved • Long process from development to deploy Solved • Development time increased • System architecture complexity increased
- 7. Otsikko 7 Breaking down monolith to microservices
- 8. Otsikko 8 How does usual service look like
- 9. Otsikko 9 Microservice cookbook • Create new repo (@bat-bot setup repo <desired-repository-name> ) • Clone service template • Specify service properties (descriptor.yml) • Configure docker (docker-compose.yml) • Configure jenkins (Jenkinsfile + separate jenkins registry) • Configure terraform • Configure gatekeepers (in separate repo) • Deploy on qa, check health check
- 10. Otsikko 10 Microservice cookbook: template • File templates: - docker-compose.yml - .env.dist - Jenkinsfile - composer.json - /terraform - /jobs • Health check endpoint • Behat • Faker • PHPUnit • Service API libs • AWS integration libs
- 11. Otsikko 11 Microservice cookbook: behat + wiremock • User story tests • Integration scenarios • AWS mocking • BDD (not required) Behat (http://behat.org) Wiremock (http://wiremock.org) • API mocking • Internal/External services integration
- 12. Otsikko 12 Microservice cookbook: communication • API - generic bundle - configurable access through gatekeeper • AWS SNS & SQS - separate ECS task for each consumer - queues are created by terraform on each build • Changelog - each entity change saved in special table - any service can subscribe to any entity and any event type - subscriptions done by SNS (may be not sequential)
- 13. Otsikko 13 Microservice cookbook: api bundle # endpoint.yml getUser: method: 'GET' baseUrl: '%env(AUTHENTICATION_HOST)%' path: '/v1/auth/user/{userId}' requestClass: 'WKDADTOAuthenticationServiceUserRequestGetUserById' responseClass: 'WKDADTOAuthenticationServiceUserResponseUser' uploadFile: method: 'POST' baseUrl: '%env(INBOUND_PAYMENTS_HOST)%' path: '/v1/inbound-payments/file-data' requestFormat: 'stream' requestClass: 'WKDADTOInboundPaymentsServiceFileDataRequestPostFileUpload' responseClass: 'WKDADTOInboundPaymentsServiceFileDataResponseFileUpload'
- 14. Otsikko 14 Microservice cookbook: jenkins • CI job description stored inside service repo • CI pipeline executes on each PR creation • Pipeline contains - docker image creation - docker image to ECR push - code sniffer run - phpunit + behat tests run - terraform execution - swagger update
- 15. Otsikko 15 Microservice cookbook: terraform • Deploys image QA • Deploys same image on Prod • Creates all needed resources (queues, buckets, ...) • Starts console tasks (like consumers) • Starts service Demo: https://github.com/auto1-oss/aws-ecs-jenkins-terraform
- 16. Otsikko 16 Microservice cookbook: configuration • Hashicorp Vault • Jenkins secrets • Env variables • Configuration service
- 17. Otsikko 17 Microservice cookbook: security • Different users for different applications • Different gatekeepers for different platforms • Environment split support inbound-payments : routes: - path: /v1/inbound-payments/payment/{id} methods: PATCH filter: - type: wkda.api.filter.AccessTokenFilter - type: wkda.api.filter.ApiClientRoleFilter config: roles: admin
- 18. Otsikko 18 Microservice cookbook: monitoring • Logstash + kibana • Grafana • Elastalert
- 19. Otsikko 19 Microservice cookbook: monitoring name: Inbound payments errors type: any index: logstash-%Y.%m.%d num_events: 1 realert: minutes: 15 filter: - query: query_string : query: "(system.syslog.detail.exception.code:"500")" alert: - slack slack_channel_override : - "#a1-sales-alert"
- 20. Otsikko 20 Results • Over 150 microservices created (Java/PHP) • 655 repositories • 30 application databases • Over 150 microservices • Written on Java, PHP, Go, NodeJS • Over 200 Engineers • More than 20 teams (some of them are remote) • Each team owns 5-20 services • Average 10 releases per day
- 21. Otsikko 21 Thank you! Alexander Egurtsov: alexander.egurtsov@auto1.com Auto1 tech blog: http://auto1.tech Auto1 opensource: https://github.com/auto1-oss Questions?