Member-only story
Evolving a Database Schema
The elephant in the CI/CD pipeline
I have written in the past about trunk-based-development. I have read what others write about it too. The proverbial elephant in the room, speaking of trunks, is the database. A code change quite often goes hand-in-hand with a schema change, and it’s entirely reasonable to wonder how to manage this in a CI/CD context. We set ourselves the goal of having each push be production-ready. How does that work with a relational database?
I am not a DBA, nor a database expert by any means. I was, however, fortunate to spend many years in an engineering department that did not believe in working after hours. Not even for a deployment. The alternative to deploying after business hours is to deploy during business hours. While the software is in use. It might sound risky, but we reasoned it was less risky to deploy while we had all hands on deck and everyone was relatively fresh. Otherwise, if disaster struck, we’d all be asleep at first and then annoyed, exhausted, impatient, and running on (coffee) fumes.
The trick, then, is how to deploy without taking the software down “for maintenance and upgrades?” Ultimately, this was just another engineering challenge to overcome. Our initial solution was similar in many ways to a blue/green deployment. However, as Martin Fowler pointed out in the linked article, “Databases are tricky.” Well … we learnt a few tricks. I do mean “we.” What I present here is the finest group wisdom, triple distilled and cask aged.
Assumptions
I am making some assumptions in this article. Even if they don’t apply to your situation, much of the content may still be relevant. I’m being explicit about the assumptions so you can make an informed decision.
The biggest assumption is that you have multiple services (micro or otherwise) that are independently deployable and have independent schemas. It doesn’t matter whether they live on the same physical server or not, but Service A cannot read from Service B’s tables and vice versa. I am also assuming your services aren’t integrated at the data layer. They talk to each other through RPC, a message broker, or both. But not the database. Never the database.
