The Science of database CICD - UKOUG Breakthrough

Editor's Notes

• #3: What are we going to see today? First we will have a look at what are preconditions of database schema evolutions or database migrations? Second we will explore how database deployments or schema migrations are conducted And third we will have time for questions and then proceed with a hands-on lab.
• #34: Now If you have a version control system you will need to think about a directory structure of your repository. The main thing when it comes to structuring is storing production code separate from test code and test data. There are a couple of things to consider: Store the ddl of all of your objects per object type for easier navigation Store migration scripts separately Store tests and test data separate from production code Use the same naming for test packages and production packages to simplify navigation Use one test package per productive package that is tested Test data and tests should be stored separate from each other to ensure reusability of the test data in different test contexts Now that we have our directory structure we can have a look at our source code workflow.
• #39: Let’s have a look at the first precondition. The repository structure of our version control repository like git, and the workflow that we need to set up in our development process.
• #40: So The problem with long-running feature branches is that you can't test the commits in the right order. In both database and infrastructure development, it often depends on which patch or change is applied first before another is applied. If you have long running feature branches, you don't know what changes have been made to your application in the meantime. Therefore, testing is not very efficient or safe.
• #41: Well, Continuous Integration works best for databases and for infrastructure code when version control is trunk-based. Because many changes to database applications cannot be tested properly if feature branches are used! The same is true for infrastructure code. So if you are doing either infrastructure provisioning or database engineering, you should develop trunk-based. Let’s have a look why!
• #42: If we look at how trunk-based development and continuous integration work, the continuous integration pipeline is triggered with each commit on the mainline. CI is therefore executed when a new commit is made and starts: Migration scripts. Post-migration scripts such as recompiling, restoring synonyms, creating permissions. Tests are executed Metrics are calculated Static code analysis is performed The delivery process can then be started manually and only performs migrations and post-migration scripts.
• #44: Now there are two different approaches to implementing trunk-based development. Either you start completely trunk-based as shown here on the left, so that the whole application including the database is developed trunk-based. This requires a big amount of pair programming and putting changes of small size into production at a time. Or you work in Git with submodules, and split the application and the database into two repositories. The database repository is then included as a submodule in the application repository so that the CI/CD pipeline has access to both. The application part can be then developed for example with a gitflow workflow and the database trunk based. Database changes would then be deployed, when a new release of the application is deployed. This is easier to implement when a versioned API above the database exists that decouples the two parts.
• #47: If I tell people that they should to pair programming, I often hear that they don’t have time for pair programming, because they have lot’s of stuff to do. The thinking is that they are faster if they don’t do pair or mob programming. But the nature of things is, that you usually have dependencies in development. Those can be knowledge gaps that you have, where you need support from a colleague, or access dependencies. As an example the lead time is shown of the yellow task that needs both developers. It is a lot longer, than if those two would just sit together and do it in pair programming. With the side effect that if they would do pair programming, they would also share knowledge about the feature at the same time. So they would be faster and also smarter afterwards.
• #51: A database migration tool helps us migrate the database schema from one state to another. This means we start with an initial DDL script that builds database version 1. From here on we will always have migrations that transform our database into the next version. And our database migration tool tracks those migrations, checks if a migration script was already run against a database and if not, installs the migration. This way we can apply changes to our database like adding tables, adding foreign keys, removing data or refactoring the database with new migration scripts.
• #52: The tool that we are going to see today is flyway that follows a migration based approach. Database migration files are stored as a series of files with continuous enumeration, telling the tool in which order the scripts need to be run, to succeed. Usually there are migraiton scripts that run only once against a database instance and we have so called repeatable scripts, that run after every migration. Those scripts can set for example synonyms that might get lost during migrations or grants or create database views and procedures that don’t include persistence or state. Also recompilation script are quite common repeatable scripts, that check if all the database objects are still in a valid state and can be executed.
• #55: The third precondition is that we need to test our database.
• #58: If you have a java Application you can use jUnit and the jUnit extension DBUnit to write database integration tests or also database unit tests. The problem here is, that you need a running database instance to run the tests against, so they are more integration tests than they are unit tests, because you also test the interaction of the two systems with every test and cannot run them without having the database present.
• #63: We have primarily two kinds of teams, teams that provide services to others and teams that consume services from other teams. For application teams or software delivery teams it is especially important to have a service layer abstracting the infrastructure, to make them independent and allowing them to build and run their application independently of any other team. That’s the main goal of DevOps – You build it, you run it. So everything a team builds, they can also independently run in production. There’s no hand-off to an operations team that takes over production, the application teams does everything on its own. So, Especially above the infrastructure layer, a service layer is essential so that application teams can use them. These interfaces should be made available in different versions whenever possible.
• #66: Versioning looks something like this. Our service XY provides several versions of its API. This allows the consuming applications to change the version individually if they either want the benefit of new functionalities or if they want to invest time in the lifecycle of the application and switch to the latest service version. A very Important aspect is: Applications should be allowed to decide for themselves when they switch to newer versions of an API. Forcing applications to do this synchronously undermines the prioritization within an application team of what is important to them. It may be enough for a team to use functionalities from version 1, because they don’t require new functionalities of newer verisons.