Pre-tested “Commits” using Git

The ability to use pre-tested commits is a feature of certain Continuous Integration servers like Teamcity. The whole concept is not easily transfered to a Git infrastructure. There are several approaches. Teamcity 6.5 now newly supports an approach called “Personal builds on branches” (see here). In our project we needed the feature a long time ago and created a Git workflow that fulfills the intention of pre-tested commits and does not need any special features of the CI server.

But first of all what is the problem that is solved by testing a commit before it is committed? I think Geek and Poke gets it perfectly. You want to be able to get the latest checkin from VCS an be sure that it builds. Classic CI is in the wrong sequence. You commit first and let the CI server test afterwards. Of course you recognize a problem fast but by then your fellow coders will already get your mess. The responsible coder will run the test on his machine first but what do you have a CI server for?

The solution is to let the CI server test your changes before they are available to other developers. The classic approach used by Teamcity is pretty complex. They use an IDE Plugin or a so called Command Line Runner. It works like this:

  1. Before you commit you will start the remote run using one of the tools. The tool will create a patch of your changes and send them to Teamcity.
  2. Teamcity will now checkout a fresh version from the VCS and apply the patch to it.
  3. Now your CI run will compile and test everything including your changes.
  4. If successful the plugin will proceed to really check in your changes into Version Control.

Aside from it’s complexity this solution did not work well with distributed VCS like Git or Mercurial.

With Version 6.5 there is another option “Personal builds on branches”. You commit your changes into special branches that follow a certain name pattern. If Teamcity recognizes this commit it will start a personal build and push to the master branch if the build was successful. A very similar approach has been described for Jenkins.

We are using Git as our VCS since more than a year but the new solution by Teamcity has only been added very recently. This situation forced us to create our own solution and to our surprise this was much easier than expected. The main reason was the distributed nature of Git. Some of the points are:

  • A commit is something that only effects your local repository as long as you don’t push it. This allows to test after a commit and before push. So you don’t need a special mechanism to create patches and send them somewhere.
  • The central repository is not the only one. Beside your local copy you can create as many repositories as you like. So you can push wherever you like to in order to e.g. run tests.
  • The central repository that you pull from doesn’t need to be the same that you push to.
  • Commits have a globally unique identifier. You can try to push a commit twice. Git will recognize it and don’t apply it the second time.

You can probably already guess what I am up to, so lets show a picture of the overall setup.

There is one central Git-repository that only contains pre-tested changes. I call this “Green Repository” because it should only contain changes that lead to green builds. Every developer pulls from this repository but nobody is allowed to push to it. Instead everybody has a personal repository (think fork if you were on GitHub). The CI Server watches those personal repositories. After a commit it starts the compile and test. If that was successful it pushes the changes to the Green Repository. Now everybody can pull your changes and can be sure they have been tested.

You might wonder about some details of this setup so I try to answer some questions here:

  • The remote server that contains the Green Repository and all personal remote repositories could be a simple git ssh setup. We are using a local installation of the Gitorious software instead. It is very easy to clone a central repository and to set access rights this way. There is also a commercial solution by the creators of GitHub called GitHub:FI.
  • The remote-run on the CI Server needs to include 2 steps. Running the build (e.g. maven) and pushing the changes after successful build. We are using build dependencies in Teamcity in order to do these two steps. The first step is a simple maven goal and the second step is a simple command line call “git push”.
  • In order to make the push to your personal remote repository easier you should create an alias. “git remote add remote-run [your remote repo url]” then it is easy to push your changes using “git push remote-run”.
  • This solution works best with small teams that do not change often. The setup procedure for a new team member requires good knowledge of the solution. The setup itself can be done in less than 15 Minutes.

The question is whether we will continue using our own solution or switch to the new feature of Teamcity. So far I can’t see any advantage of the Teamcity feature. With our solution we are even more flexible in regard of branch design. So I think our layout will live on for a while.

If your CI-Server has no support for pre-tested commits you might give our setup a try. I am eager to hear your experiences. I am also happy to answer any specific questions about the setup in comments.

Be Sociable, Share!
  1. ansgar k. says:

    TeamCity now supports multiple build steps in a single build configuration. I’m wondering if you might be able to implement the test run + push in a single config and spare the build dependency.

    • Joerg says:

      Good point. I haven’t had a chance to try this yet but this multiple build step feature could turn out very useful for our overall setup not only for the pre-tested commit.

  2. Yiannis says:

    Great post Joerg! I’m in the process of designing a solution for a similar setup using TeamCity and Mercurial and was wondering how you resolve the potential creation of multiple heads when one tries to push to the green repository. Is this something automated or do you have to manually merge the changes?

    • Joerg says:

      I am not an expert for mercurial but I guess it will not be that much different.
      In git you can’t normally push to a remote repository when this would result in a merge. Your commit would simply be rejected. This requires you to pull again to your local repository and re-run the remote-run.
      A remote merge would be dangerous anyway as this merge would then contain a codebase that has not been tested.

  3. tim says:

    Hey Joerg, nice post – seems like you are following the Integration-Manager Workflow described here http://progit.org/book/ch5-1.html#integrationmanager_workflow

    I would like to go with this workflow too, however it seems very painful to setup new build configurations for every developer. Do you know any way around that? or somehow use one build configuration?

    • Joerg says:

      Hi Tim, yes it is more or less an automated version of the integration-manager workflow.

      You are right you have to create a configuration per developer. This is a pretty straightforward in Teamcity by just copying some old configuration. After creation every developer is responsible for maintaining his config.
      But if you have a large team or a high turnaround this configuration would probably not be a good fit.

      • sam holder says:

        Rather than copy a previous config I found that you can create a build config template where the github username is a build parameter, so you only need to create a new config from the template and fill in the github user name and bam! you have your new config.

        • Joerg says:

          Thats true. Templates have not been available when we originally created this workflow. Now the “Remote-Runs” are based on templates too.
          Regarding your other question. We are not using branches on our green repository. Therefore we did not have that problem. I guess you have to delete the branch manually.

  4. tim says:

    another question – how do the personal remote repos stay in sync with the green repo? isn’t it possible they could get out of sync, and so the push to the personal remote repos is different than the push to the green repo?

    • Joerg says:

      You always pull from green and push to your remote repository. So the sync happens via your local repository.
      This is an advantage of the distributed VCS and I am still amazed how stable this works. Kudos to Linus and the other creators of git.

  5. Lior Tal says:

    Nice article.

    An incentive for using TeamCity for this would be — less wasted space (multiple repos, each per developer), IDE integration (via the plugin), Performance (?)

    Have you compared the performance of TC versus this system? i wonder if it is even noticeable.

    • Joerg says:

      We are using Teamcity as CI Server to support this process. We are just not using their mechanism to start remote runs. This is for two reasons. First we used this process way before this feature was implemented in Teamcity and it still works great. Second our process supports multiple branches which is sometimes necessary.

      So I have not tested the alternative yet and can’t say anything about performance differences. The IDE Integration of Teamcity works perfectly in our process. I don’t really care about the “wasted space” of multiple repositories. With todays HD capacities this is barely noticeable.

  6. Conor Mullen says:

    Hi Jeorg

    Great article. Im just wondering do you find it a pain handling any merge conflicts when pushing from a developers remote repository to the green repository?

    • Joerg says:

      Hi Conor,

      No, because you can’t handle merge conflicts at this point at all. If it is not a fast forward git will refuse to push from developer remote to green. In that situation you have to pull from green to your local again, handle merge conflicts there, and then push again to your developer remote.

      Last but not least there is a simple rule for preventing merge conflicts. Commit often, pull often and push often. On average I do a commit, pull, push cycle every hour during development. If your teammates do the same there will be no problem.

      • Conor Mullen says:

        Ah I see thanks Joerg.

        While that would work my concern would be time to produce the feedback. With the above the compilation and test phases would execute followed by the push to the golden repository. The downside to this is that if your repository is not up to date, you compile, test and then push but the push fails, you will have just lost the time it took to compile,test. I was thinking possibly doing a pull before the compilation process so the CI’s clone of the developers remote repository can at least try and be up to date before undergoing build and test.

        • Joerg says:

          Hi Conor,

          The better and simpler way is to pull from green repository to your local repository before you push to your remote. It is better, because you can solve merge conflicts first and loose no time. It is simpler, because you can script it on your machine. This is what we usually do.

  7. NightOwl888 says:

    Joerg,

    Great post. I am curious to know where you put your long-running (integration and end-to-end) tests in this setup?

    If you put them in the “personal builds” stage, the personal builds could take an unreasonable amount of time depending on the tests involved and the developer would have to wait to see the result.

    However, if you put them in a daily build (which I presume would be done after the commit to the “green repository”), then the repository would technically not be safe to pull a deployment from (at least not from the HEAD) as all of the tests may not have been done.

    • Joerg says:

      Long running tests/integration tests run after the code has been commited to the green repo.
      This happens after each successful commit (not only daily). So technically you can pull something that is not completely valid but the long runtime of a personal build would be worse.
      To compensate this a little bit our Push-Script warns you if you try to push while the integration tests are broken.

  8. sam holder says:

    Hi Joerg, thanks for the great post, we have implemented something similar based on the information found here. However we have come across an issue with team city that we are not sure how to solve and I wondered if you have had the same problem and how you might have solved it.

    When a dev wants to delete a branch, the delete is only pushed to the personal fork. nothing seems to be triggered in TeamCity to get it to flow the delete from the fork to the green repository. have you experienced this? If so have you managed to overcome it?

    Thanks!

    Sam

  9. M says:

    Use Gerrit.

  1. There are no trackbacks for this post yet.

Leave a Reply