Continuous DB migration based on carbon5 framework
3 likes4,339 views
This document discusses techniques for continuous database migration. It introduces Carbon5, an open source database migration framework. Carbon5 works by having each schema change stored as a separate SQL script file. When run, it checks which changes have not been applied and executes them. Carbon5 can be used in maven-driven or embedded modes. Maven-driven mode integrates with maven builds, while embedded mode performs checks on application startup. Best practices include feature-driven development of SQL files and keeping database configuration centralized. The document also discusses approaches for NoSQL databases, which may handle migrations differently.
Continuous DB migration based on carbon5 framework
- 1. Continuous DB migration Boris Trofimov @b0ris_1 www.btrofimoff.com
- 2. Agenda Dealing with changes Perfect solution or Continuous DB migration Carbon5 Introduction Maven-driven mode Embedded mode Pros/Cos and best practices
- 3. Dealing with Changes Applications evolve. God, bless bugs, refactoring and business changes. We love applications and configure CI, VCS in order to track source code/application version. But what’s about DB?
- 4. Dealing with Schema Change FACT: Some modifications might require changes in DB schema.
- 5. Dealing with Schema Change FACT: Some modifications might require changes in DB schema. Changes are simple and risky.
- 6. Dealing with Schema Change FACT: Some modifications might require changes in DB schema. Changes are simple and risky. Simple changes: – add one more table – add index
- 7. Dealing with Schema Change FACT: Some application modifications might require changes in DB schema. Changes are simple and risky. Simple changes: – add one more table – add index Risky changes: – rename/add column – change foreign key – migrate data from one table/column to another – denormalization on a fly
- 8. Dealing with Schema Change What’s about old apps upgrade and multiple schema changes? App v1 App v10
- 9. The Impact Risks to lose data Painful downtime Risks to break application (see #1 and #2) Resources, efforts and budget
- 10. Friday’s deployment in some teams
- 11. How people solve it Deny Friday’s deployments Manual deployment Hibernate-like schema update + SQLexec Self-developed tools based on version number approach
- 12. Perfect solution Dedicated reusable framework Every Feature request leads to separated SQL migration script. Framework tracks which changes were not applied and applies them. Configurable time to launch apply procedure Prevent Double changes Have Change Log Simple integration to existent project
- 13. Continuous DB Migration approach min + automatic migration = Continuous DB Migration
- 14. Carbon V
- 15. Why Carbon V DB-migration framework Open Source project Lightweight framework Simple usage
- 16. How it works Dedicated project folder for SQL migration scripts. This folder should be available when framework takes control. Each SQL migration script is pure SQL/DDL file with migration SQL code for specific feature Each Delta file should have name YYYYMMDDHHMMSS_<FEATURE>.sql When framework take control it checks and applies only new changes. C5 uses own JDBC connection. Way to re-usue existent DataSources with connection params
- 17. No magic, just… C5 creates own table “schema_version” inside your database C5 controls this table by itself (no needs to create or update). sql_file_name date duration 201405170900_user_auth.sql 2014-05-17 13:00:00 8 sec 201404130400_award_4565.sql 2014-04-14 14:00:00 6 sec
- 18. When to initiate migration procedure? Maven-driven approach Embedded mode
- 21. Configure Maven project Update POM definition Configure Explicit DB connection Can be configured depending on specific maven profiles (staging, production)
- 22. Add SQL migration files Use Project/src/main/db/migrations directory
- 23. Launch it performs migration $ mvn db-migration:migrate resets migration $ mvn db-migration:reset check if DB is up to date $ mvn db-migration:validate create new feature $ mvn db-migration:new -Dname=new_feature
- 24. Use cases Manual runs from developer machine Part of Continuous Deivery process – Add just one more Maven action inside specific CI configuration before deployment action
- 25. Embedded mode
- 26. Embedded mode Schema Check is constituent part of application. SQL changes are built-in into the application Any checks and possible DB migration is performed every time when application launches. Dedicated bean to carry out this responsibility
- 27. Create application bean import com.carbonfive.db.migration.DataSourceMigrationManager; import com.carbonfive.db.migration.DriverManagerMigrationManager; import com.carbonfive.db.migration.MigrationManager; import com.carbonfive.db.migration.ResourceMigrationResolver; public class DBMigratorImpl { DataSource dataSource; ... public void init(){ DataSourceMigrationManager migrationManager = new DataSourceMigrationManager(dataSource); migrationManager.setMigrationResolver(new ResourceMigrationResolver("classpath:/db/migrations")); migrationManager.migrate(); } }
- 28. Inject the bean into application context Seamless integrated with Spring DI It is possible to initialize ResourceMigrationResolver DataSourceMigrationManager directly though Spring DI without any line of code Two places where bean might be integrated inside Spring context Before Persistent beans After Persistent beans
- 29. Embedded vs maven driven E: Consistent and grace application upgrade E: No delays between deployment and schema upgrade M: Simple integration with CI M: External commands like validate & reset M: Quick manual migrations (hotfixes for instance)
- 30. Carbon5 Benefits Lightweight framework Brilliant extensibility (thanks to reasonable design) Support many DBMS in unofficial mode Simple integration Real continuous DB migration
- 33. Flyway framework Rich support for different build tools The same integration approach on Carbon V Pluggable architecture Java-based migrations (two options: through JDBC Connection & Spring JDBCTemplate) Extended service table
- 34. Best practices Feature driven development (each SQL feature change in own file) Make sql files safe in order to prevent it from double execution Keep database connection settings in single place (use shared DataSource) Do not make structure changes to the DB outside of your framework
- 37. NoSQL migration approach No dedicated explicit migration procedure (solved on business level in case of needs) No shutdown or downtime Rolling update – Every domain entity has version number – your v2 application should handle the v1 db format – Write code to convert entity to v2 on a fly (repository level) – Write modified entities to v2 on demand Some DBs like RavenDB are able to perform auto-migration
- 38. References https://code.google.com/p/c5-db-migration/ http://flywaydb.org/ Martin Fowler Evolutionary Database Design http://www.martinfowler.com/articles/evodb.html Refactoring Databases
- 39. Presentation in a nutshell Keep in mind this problem Do not invent wheels, use external frameworks/techniques And do not be afraid of Friday’s deployment then.
- 40. Thank you! Q&A