Continuous Deployment changes your mind

Posted in agile, continuous delivery, continuous integration, deployment, management, software development, testing on August 14th, 2011 by Joerg – Be the first to comment

We are practicing Continuous Deployment since about a year. When looking back I realize the dramatic way this practice changed the way we work.

Just to draw the fine line between Continuous Delivery and Continuos Deployment I use the definition I once heard from Neal Ford. Continuous Delivery allows you to deploy every commit but you don’t do it every time. With Continuous Deployment you go further and really deploy every time.

I have recently been asked what the benefits of Continuous Deployment are compared to Continuous Delivery. I think the impact on your development practice is a huge thing.

In our company we practice agile development since about 2005. During that time I can remember many discussions about agile practices like:

  • Test Driven Development
  • Definition of Done
  • Potentially Shippable Product
  • Refactoring in baby steps
  • Not committing on a red build

Many of them do not easily reveal their benefits in daily business. So it becomes difficult to use them consequently.

The Definition of Done is a great example. It is easy to declare something finished or appropriately tested when there is usually a QA department that will find your mistakes during release tests. So the common recipe is write down your definition, post it on the wall and “be more disciplined”. Force yourself to write more tests, force yourself to declare things done only if they are really done and so on. This is not easy. In my experience it seldom works.

This changes dramatically when using continuous deployment. Between your commit and the production system is only an automated deployment pipeline, that usually takes something around 30 minutes. There is no QA department that does a extensive release test. Your mistakes will only be found by automated test that you wrote yourself. Well or they will be found by your Users.

Automated tests become your safety net. This again changes the way you think about TDD. It totally makes sense to have a near full coverage of your code. It also makes sense to have integration tests on API and GUI level to make sure your deployment pipeline catches most mistakes. It will not catch any mistake, but in my experience the same is true for most QA departments.

You will think differently about Potentially Shippable Products. Every commit has to be shippable and it will (not only potentially) be shipped.

This forces working in baby steps. It is just not possible to refactor a system by destroying it for several days.

If a build is red the deployment pipeline is halted. Nobody is able to apply more changes to the system unless this problem is fixed. Again something that used to be a discipline thing becomes a necessity. You need to help your teammates to fix the problem otherwise you can’t work yourself. The same is true if there is a red build in the evening. One of my teammates even talks about an addiction to fix the build before going home. It just gives you a good feeling to finish your day in a green state.

Continuous Delivery has several immediate business benefits I might tell you about another time. But most of them don’t need every commit to be deployed.

But I think the impact of deploying every commit on your development practice huge. Many agile practices make so much more sense. They fit naturally in this environment and you follow them easily. This will then result in benefits for your business on the long term.

Continuous Delivery Slides

Posted in agile, continuous delivery, continuous integration, deployment, java, management, software development, testing, version control on July 5th, 2011 by Joerg – Be the first to comment

The last Java User Group event in Berlin was hosted at Hypoport. It was about Continuous Delivery. We had about 100 guests.

There where two talks. One by Axel Fontaine who introduced Continuous Delivery especially in a Java environment. You can find his talk on Parleys or on his Blog.

I had the pleasure to introduce our experiences with Continuous Delivery. We are currently developing a new product. Since we started about a year ago we consequently used Continuous Delivery. We learned quite some lessons on our way which I presented that evening. My Slides are available here. (As this was a german event the slides are in german)

Thanks to everyone who attended the event. It was a really nice evening. Special thanks to everybody who helped organizing it.

Pre-tested “Commits” using Git

Posted in continuous delivery, continuous integration, deployment, tools, version control on May 29th, 2011 by Joerg – 20 Comments

The ability to use pre-tested commits is a feature of certain Continuous Integration servers like Teamcity. The whole concept is not easily transfered to a Git infrastructure. There are several approaches. Teamcity 6.5 now newly supports an approach called “Personal builds on branches” (see here). In our project we needed the feature a long time ago and created a Git workflow that fulfills the intention of pre-tested commits and does not need any special features of the CI server.

But first of all what is the problem that is solved by testing a commit before it is committed? I think Geek and Poke gets it perfectly. You want to be able to get the latest checkin from VCS an be sure that it builds. Classic CI is in the wrong sequence. You commit first and let the CI server test afterwards. Of course you recognize a problem fast but by then your fellow coders will already get your mess. The responsible coder will run the test on his machine first but what do you have a CI server for?

The solution is to let the CI server test your changes before they are available to other developers. The classic approach used by Teamcity is pretty complex. They use an IDE Plugin or a so called Command Line Runner. It works like this:

  1. Before you commit you will start the remote run using one of the tools. The tool will create a patch of your changes and send them to Teamcity.
  2. Teamcity will now checkout a fresh version from the VCS and apply the patch to it.
  3. Now your CI run will compile and test everything including your changes.
  4. If successful the plugin will proceed to really check in your changes into Version Control.

Aside from it’s complexity this solution did not work well with distributed VCS like Git or Mercurial.

With Version 6.5 there is another option “Personal builds on branches”. You commit your changes into special branches that follow a certain name pattern. If Teamcity recognizes this commit it will start a personal build and push to the master branch if the build was successful. A very similar approach has been described for Jenkins.

We are using Git as our VCS since more than a year but the new solution by Teamcity has only been added very recently. This situation forced us to create our own solution and to our surprise this was much easier than expected. The main reason was the distributed nature of Git. Some of the points are:

  • A commit is something that only effects your local repository as long as you don’t push it. This allows to test after a commit and before push. So you don’t need a special mechanism to create patches and send them somewhere.
  • The central repository is not the only one. Beside your local copy you can create as many repositories as you like. So you can push wherever you like to in order to e.g. run tests.
  • The central repository that you pull from doesn’t need to be the same that you push to.
  • Commits have a globally unique identifier. You can try to push a commit twice. Git will recognize it and don’t apply it the second time.

You can probably already guess what I am up to, so lets show a picture of the overall setup.

There is one central Git-repository that only contains pre-tested changes. I call this “Green Repository” because it should only contain changes that lead to green builds. Every developer pulls from this repository but nobody is allowed to push to it. Instead everybody has a personal repository (think fork if you were on GitHub). The CI Server watches those personal repositories. After a commit it starts the compile and test. If that was successful it pushes the changes to the Green Repository. Now everybody can pull your changes and can be sure they have been tested.

You might wonder about some details of this setup so I try to answer some questions here:

  • The remote server that contains the Green Repository and all personal remote repositories could be a simple git ssh setup. We are using a local installation of the Gitorious software instead. It is very easy to clone a central repository and to set access rights this way. There is also a commercial solution by the creators of GitHub called GitHub:FI.
  • The remote-run on the CI Server needs to include 2 steps. Running the build (e.g. maven) and pushing the changes after successful build. We are using build dependencies in Teamcity in order to do these two steps. The first step is a simple maven goal and the second step is a simple command line call “git push”.
  • In order to make the push to your personal remote repository easier you should create an alias. “git remote add remote-run [your remote repo url]” then it is easy to push your changes using “git push remote-run”.
  • This solution works best with small teams that do not change often. The setup procedure for a new team member requires good knowledge of the solution. The setup itself can be done in less than 15 Minutes.

The question is whether we will continue using our own solution or switch to the new feature of Teamcity. So far I can’t see any advantage of the Teamcity feature. With our solution we are even more flexible in regard of branch design. So I think our layout will live on for a while.

If your CI-Server has no support for pre-tested commits you might give our setup a try. I am eager to hear your experiences. I am also happy to answer any specific questions about the setup in comments.