Create a One Click Migration (OCM) process to Automate Repeatable Infrastructure Migration with Salt
0 likes325 views
The document outlines the development of a one-click migration process using SaltStack to automate the migration of a data warehouse from AWS to Evoswitch with the goal of improving efficiency and reducing human errors. It discusses the challenges faced, solutions implemented, and benefits achieved through the use of the Quantyca Migration Tool and systematic orchestration. Key improvements included faster task execution, resource management, and the creation of reusable migration workflows.
Create a One Click Migration (OCM) process to Automate Repeatable Infrastructure Migration with Salt
- 1. 11/18/2019
- 2. Create a One Click Migration (OCM) Process to Automate Repeatable Infrastructure migration With Salt
- 3. Giandomenico Avelluto Lead Site Reliability Engineer @Quantyca giandomenico.avelluto@quantyca.it https://www.linkedin.com/in/giandomenico-avelluto Who Am I ?
- 4. ➔ Introduction ➔ Project Overview ➔ Challenges ➔ Solutions ➔ Benefits Agenda
- 5. Introduction
- 6. A consulting company specialized in: ● Data Management utilizing advanced analytical tools ● Data Strategy making data a corporate asset What Quantyca is Goal And Context
- 7. ● Massive scale deployments ● Event-Driven ● Extensibility Why Quantyca Chooses SaltStack
- 10. ● A leading European travel company ● Portfolio of well-known brands Who LastMinute is
- 11. Overview Goal And Context Move Data Warehouse Infrastructure from AWS to EvoSwitch Data Center: ● 40 TB of data ● 5000 ETL Workflows ● 42 servers
- 13. Overview Architecture Components to migrate: ● ETL workflows ● Database ● Reporting tool
- 14. Challenges
- 15. ● Make Migration fast ● A repeatable process ● Trace the status of individual task execution Challenges Business Requirements
- 16. ● ETL workflows needs refactor ● Lots of operations needed. Client requested a commitment for additional resources Challenges Attention Points
- 18. Solutions
- 19. VS ● Lots of resources needed ● Possibility of human errors ● Strict coordination between teams necessary ● Migration can take long periods of time ● Fewer resources involved ● Coordination is ensured by the orchestration tool code ● Human error is avoided ● Better performance ● Code reusability
- 20. Solutions Automate Use SaltStack to: ● Automate overall process ● Take advantage of Salt State System and its idempotency model ● Perform parallel tasks, when possible ● Use Slack to trigger the migration and to track steps status
- 21. Solutions Integrate Integrate the “Quantyca Migration Tool (QMT)” in Salt to: ● Analyze workflow source code ● Apply changes automatically
- 22. Andrea Macchi Head of Quantyca Support Team “Flight Plan”“Flight Plan” A way to rule them all
- 23. Our “Flight” Plan Usefulness ● Definition of the individual migration activities ● Keep track of activities priorities and dependencies ● Identification of tasks that can be performed in parallel ● Simple transformation of a task in Salt objects
- 24. Flight Plan An example Env Compone nt Task Name Tasks Desc Pillar Target Dependen cies SLS Module Parallel Task #1 aws database dump Dump DB and transfer it aws: database: dump None aws. database. dump True Task #2 evo database restore Restore DB evo: database: restore #1 evo. database. restore True
- 25. Flight Plan Salt Fileserver Structure /srv/salt ├── aws │ └── database │ └── dump.sls ├── evos │ └── database │ └── restore.sls ● Follows the structure defined for our “Flight Plan” ● Steps names are the same as those defined on Flight Plan (Simple to locate on the filesystem)
- 26. Flight Plan Salt Pillars Structure ● It follows the structure defined on Flight Plan ● Steps names are the same as those defined on Flight Plan (Easy to locate on the filesystem) /srv/pillar ├── top.sls │ ├── aws │ └── database │ └── dump.sls ├── evos │ └── database │ └── restore.sls
- 28. Flight Plan Results ● We’ve generated more than 90 tasks ● We’ve enabled teams to manage multi- dependencies tasks in a very simple way ● We created a new asset
- 30. Salt States Overview A Salt State is a representation of the state in which a system should be in.
- 31. Salt Reactor Overview Reactor system gives Salt the ability to trigger actions in response to an event.
- 32. Salt State Workflow Overview We defined three different phases for our Salt States: ● Starting ● Execution ● Finished The Starting and Finished Phases utilizes Salt Macros to execute common actions without re- writing the Salt function in every SLS module
- 34. Salt State Overview Start Phase It fire on Salt bus a start event containing: ● Task ID ● Task Description ● Target { "mycompany/mystaterun/status/start":{ "_stamp":"2019-11-05T14:58:40.025229", "cmd":"_minion_event", "data":{ "desc":"Dump DB and transfer it", "taskId":"task #1", "target":"aws:database:dump" }, "id":"saltmaster", "tag":"mycompany/mystaterun/status/start" } }
- 35. Salt State Overview Start Phase When the event is fired on Salt Event Bus, a reaction on Salt Reactor System is triggered: ● It reads the event and extracts the needed data ● It sends that data on Slack
- 39. Salt State Overview Results We created a standard structure and workflow for our Salt States using: ● Macros for eliminating redundant code ● ChatOps to manage and trace task executions
- 41. Salt Orchestration Introduction It enables admins to coordinate the activities of multiple servers from a central place: ● Executed on the Master ● Master-level view and control over requisites
- 43. Timeline
- 44. Migration Strategy We decided to run the migration process three times (2 test, 1 real) before shutting down the old environment to: ● Fix possible bugs ● Make the process reliable
- 48. First “Dry-Run” Migration Review ● Some tasks failed and some failures generated an inconsistency state ● Some tasks were not performed simultaneously ● The Quantyca Migration Tool worked progressively but slowly Also, first data migration took a very long time ● Parallelism: ● Orchestration: ● Integration:
- 50. Second “Dry-Run” Migration Review ● Each activity was performed correctly ● Some tasks were not performed simultaneously ● Some bugs to be fixed on Quantyca Migration Tool ● Parallelism: ● Orchestration: ● Integration:
- 51. Attention point Three different levels of parallelism: ● System-level ● State-level ● Orchestration-level
- 53. Last Migration Run Review ● Each activity was performed correctly ● Every tasks were performed simultaneously ● The Quantyca Migration Tool analyzed code and performed code refactor correctly and rapidly ● Parallelism: ● Orchestration: ● Integration:
- 55. Benefits
- 56. ● Lower costs ● Visibility of operations ● Migration Process Reusability Customer Benefits
- 57. ● Flight Plan and Salt States can be reused for similar activities ● Documentation of activities is already available thanks to the Flight Plan ● Fewer resources commitment Our Benefits