OpenStack Summit May 2014 Atlanta

This isn't a talk about OpenStack. This is a talk about how OpenStack does database migration continuous integration, and how other projects might be able to learn from what we've done.

OpenStack is an interesting project in that it runs in a variety of different scale scenarios. There are very small deployments of only a few machines, up to large public clouds of thousands of machines. The OpenStack project takes Continuous Integration (CI) very seriously, and runs a suite of unit and integration tests on all proposed patches.

One thing we weren't very good at testing was database upgrades though. We have unit tests for many of them, but we don't know how these upgrades affect real users in the field. As a response, a small team I manage have worked on adding database migration testing to our CI environment. This new test infrastructure is open source (https://github.com/stackforge/turbo-hipster), and extensible. It currently runs tests on smaller databases, as well as a largish real user database. It has already caught two significant performance problems in database migrations.

This talk will cover how the test infrastructure is built, what we've learned from the six months of running it that we'll have had by the time of the conference, as well as telling a story about some of the bugs we've found and how we fixed them.