Micro-serviços em Python usando Pyramid, Cornice e muito amor
0 likes342 views
The document outlines Briefy's transition to a microservices architecture, emphasizing an API-centric approach leveraging REST APIs for various platforms, and utilizing asynchronous communication through an event-based system. It details the tech stack including Python, Angular 4, and Kubernetes, alongside a strong focus on code quality, continuous integration, and reusability through shared libraries. The architecture supports complex workflows for visual content creation, while ensuring robust user permission management and event handling across various services.
Micro-serviços em Python usando Pyramid, Cornice e muito amor
- 1. From Legacy To a Microservices Architecture Python, ZCA, Pyramid, Angular 4 & Deis
- 2. About me Érico Andrei CTO at Briefy https://briefy.co From Mooca, based in Berlin Twitter @ericof erico@briefy.co
- 4. Briefy
- 6. The Team
- 8. Overview
- 9. A solution based on three pillars API CENTRIC To serve distinct use cases using the same codebase, Briefy built its solution around REST APIs. Web, Mobile and customer integration were the planned consumers. MICROSERVICES Single responsibility and distinct life cycles for parts of the solution pointed Briefy to use a micro services architecture. EVENTS BASED Most of the communication between the services could be done asynchronously, so an event-based — with messaging — solution was implemented.
- 10. Code quality and reuse as principles TESTING Codebase testing with Continuous Integration and Continuous deployment CODE QUALITY Style guide and documentation for the entire codebase DEVOPS Use existing toolset to minimise devops work. CODE REUSE Shared packages/libraries Build upon existing OSS codebase
- 11. Tech stack INFRA Storage: GDrive & S3 Messages: SQS Cluster: Kubernetes 1.7.8 DEIS 2.18 API Python 3.6.3 Zope Component Architecture Pyramid 1.9 / Cornice SQLAlchemy 1.2 Postgresql 9.6 DEV TOOLS Repo: GitHub CI: TravisCI Logging: Logstash Reporting: Sentry / NewRelic / Slack WEB FRONTEND Angular 4 Typescript Generic views Generic Listings Generic CTAs
- 12. Infrastructure
- 13. AWS THREE ENVIRONMENTS Live: eu-central-1 Staging: us-east-1 Development: eu-west-1 SERVICES Message routed using SQS S3 for static data & assets Kubernets Cluster on EC2 Dies Workflow
- 14. Kubernetes KUBERNETES OVER EC2 One master, n minions DEIS ON TOP OF KUBERNETES Workflow solution similar to Heroku PaaS EC2 WITH AUTO SCALING T2 and M4 instances.
- 15. Deis Workflow DOCKER Automatic build and save images SIMPLE git push deis master TWELVE-FACTOR And also support legacy if necessary
- 16. Code
- 17. Shared codebase BRIEFY.COMMON Events / Queue Base Models / Mixins Logging Workflow BRIEFY.WS REST API blueprints Validation Error handling BRIEFY.GDRIVE Common GDrive actions Create, Copy, Metadata, List, Download BRIEFY.S3 Common S3 actions Create, Copy, Metadata, List, Download
- 18. briefy.common QUEUE WORKER Queue worker to consume SQS messages and dispatch WORKFLOW Workflow engine to create solid and secure business processes EVENTS Event firing to internal component architecture and to SQS MODELS Base mixins and sqlalchemy models with versioning, local roles support and polymorphic types CACHE Cache object serialization on redis with dogpile.cache
- 19. briefy.ws FILTER Sort and filter listings automatically based on model fields RESOURCES Base classes to exposure models as API endpoints: REST, workflow transitions, versioning EVENTS Object lifecycle events firing SCHEMA Automatic colander schema generation for validation and deserialization SECURITY Resource factory to dynamically compute ACLs based on model workflow and local roles
- 20. Productivity tools CHECKSTYLE Linting rules and code style guide COOKIECUTTERS Templates to create new packages and services DOCKER IMAGES Python 3.6.3 Plone 5.0.8 Nginx / Varnish
- 21. Services
- 22. Core services MS.LAURE Automatic quality assurance and renaming of assets LEICA Workflow and scheduling solution. API-based CHOREOGRAPHER Events handling and routing. ROLLEIFLEX Authorisation and user management service. ALEXANDRIA Digital Assets management solution API-based
- 23. Support services CMS Content management backend for Briefy website. Using Plone CMS. THUMBOR Image server handling automatic cropping, resizing and image transforms BRIEFY.SLACK Service handling Slack notifications. BRIEFY.MAIL Service handling Mail notifications using Mandrill as the provider HOTELS CRM CRM solution to deal with Hotels acquisition for the City Packages initiative. LEADS Service handling leads and quotes coming from briefy.co website. SITEMAP Service that generates a sitemap.xml for CMS and static content. SEO helper API GATEWAY Gateway and router handling public accesses to Briefy API
- 24. Leica: from briefing to assets
- 25. The core of Briefy WORKFLOW & PERMISSIONS Every model have its own workflow HUNDREDS OF EVENTS Workflow transition, task execution, object creation and update. DATA MODELS Customer, Project, Order, Assignment, Creative and User profiles Being the scheduling and workflow solution for visual content creation, Leica is the core of Briefy operations.
- 26. Elements of Leica BRIEFY.FRONTEND Web application developed with Angular4, connects to Leica API. This app has 3 distinct users: Customer, Creative and Briefy team member, with distinct permissions. User Interface BRIEFY.LEICA Domain models, permissions, business logic and API endpoints. Codebase is well documented and 90% code coverage. API LEICA TASKS Scheduled tasks execute workflow transitions, generate reports and post KPIs. Share the same codebase as briefy.leica Tasks
- 27. Domain Models CUSTOMER Company requesting visual content creation. PROJECT A grouping of orders from the same customer. Usually attached to one contract (or agreement) ORDER A content creation request. When it is not confirmed, it is called Lead. Customer oriented. ASSIGNMENT A content creation execution schedule. Creative oriented. JOB POOL Groups available Orders to be offered to a group of pre-selected Creatives. CREATIVE A visual content creator - Photographer, videographer — profile. CUSTOMER USER Profile of a user attached to a Customer. BRIEFY USER Profile of a Briefy team member user.
- 28. Managing visual content creation at scale cost is complex Confidential Managing Visual Content Creation Is Complex
- 30. Every action, one event (1) ARIANA Assign a creative to an Order ORDER.WORKFLOW.ASSIGN 2017-07-27 15:46:10.03267 120 Times
- 31. Every action, one event (2) OPHELIE Validates a Creative PROFESSIONAL.WORKFLOW.VALIDATE 2017-07-31 15:55:11.436015 213 Times
- 32. Events trigger new actions ASSIGN. IS SCHEDULED Assignment is scheduled on Leica, generating an event EVENT IS ROUTED Actions are registered to this event, i.e.: Send email to Creative EMAIL SENT TO CREATIVE Using Mandrill, a message is sent to the Creative An event on Leica will be sent to Choreographer, there it will be routed to other micro services to execute new actions. This flow is asynchronous and all messages are sent using SQS queues. BRIEFY.LEICA BRIEFY.CHOREOGRAPHER BRIEFY.MAIL