Continuous Integration
People look at me like I’m crazy when I tell them that they might want to try an optimistic release cycle.
Most organization have a pessimistic release cycle. Projects are developed on isolated branches. Once completed the project code is usually integrated down onto the release branch and tested.
Different organization have different shades of pessimism. Some reserve their pessimism for high value sections of code, others use reserve it for complex projects, and others use reserve it for constantly changing applications.
I would like to propose an optimistic model for developing and deploying code on application which are constantly changing. Too often managers want to create a stable release candidate that they use to fix and regress bugs. By combining bug fixes and development on a single branch, it is impossible to tell the difference between bugs created during the development of new features, and the bugs created from bug-fixes. On this subject I say a bug is a bug, and far better to find and fix problems right after they occur. Constant integration and constant execution of repeatable tests finds bugs right away. Doing development on separate branches lengthens the time between creating a bug and fixing it. Lets face it code scheduled for release at some future date, doesn’t get a lot of quality oversight. Oh and did I mention that integration is a pain in the butt. I’ve seen code integration projects introduce more thorny bugs then the feature development!
Here is a description of a process which has worked well in the past. Web application code is released every week, with a full release followed by a patch release. User facing web applications get new code every week, but the service layers and older applications have less frequent updates.
The business unit has a single set of Java code which is compiled every time, via cruise-control) an update is made. The compilation also runs unit tests and API tests. The compilation produces jars. Each application, then manages their own dependencies, and during development they will suck out the jars they need. Tools like Ivy and Sage are helpful for managing dependencies.
All development is done on the head. Two days before deployment a release branch is created. Next a release candidate is created from this branch and deployed to a staging/integration environment. Next automated tests are run using Sybioware. Bugs are filed, fixed, and hopefully regressed. The release candidate is now the production release.
Once live, you may fix-forward by patching the branch and redeploying. Or you may roll back to the previous release. A patch release is a schedule fix-forward. Code changes to the release branch are also made to the head.
I would definitely consider this approach an optimistic one. No one can say for sure what is in the development branch at any given moment. The release branch is a snap-shot of development. So every week unwanted “features” may make it into the release candidate, but these are usually found before going live. In my experience this process produces only 2 rollbacks a year. I consider this optimistic process pretty good. In addition, it really saves time, by eliminating lengthy integration work.
Of course, there are exceptions to every rule and some long lived hairy development projects on static code are best done on a separate branch, but for frequently changing applications the integration phase only creates more bugs!
This process worked in with a fairly simple setup. The web application tier was stateless, and it are only served 16MM pages a day. It didn’t support multiple data centers active at the same time, so there was a limited need to synchronize message passing. I’m fairly certain that this release model could be extended to more complex environments, by doing the following:
- provide additional capability to commit, tag, and track code in set of files
- encourage constant project integration on a daily basis
- encourage repeatable full functional testing during development