tools

Pre-tested “Commits” using Git

Posted in continuous delivery, continuous integration, deployment, tools, version control on May 29th, 2011 by Joerg – 19 Comments

The ability to use pre-tested commits is a feature of certain Continuous Integration servers like Teamcity. The whole concept is not easily transfered to a Git infrastructure. There are several approaches. Teamcity 6.5 now newly supports an approach called “Personal builds on branches” (see here). In our project we needed the feature a long time ago and created a Git workflow that fulfills the intention of pre-tested commits and does not need any special features of the CI server.

But first of all what is the problem that is solved by testing a commit before it is committed? I think Geek and Poke gets it perfectly. You want to be able to get the latest checkin from VCS an be sure that it builds. Classic CI is in the wrong sequence. You commit first and let the CI server test afterwards. Of course you recognize a problem fast but by then your fellow coders will already get your mess. The responsible coder will run the test on his machine first but what do you have a CI server for?

The solution is to let the CI server test your changes before they are available to other developers. The classic approach used by Teamcity is pretty complex. They use an IDE Plugin or a so called Command Line Runner. It works like this:

  1. Before you commit you will start the remote run using one of the tools. The tool will create a patch of your changes and send them to Teamcity.
  2. Teamcity will now checkout a fresh version from the VCS and apply the patch to it.
  3. Now your CI run will compile and test everything including your changes.
  4. If successful the plugin will proceed to really check in your changes into Version Control.

Aside from it’s complexity this solution did not work well with distributed VCS like Git or Mercurial.

With Version 6.5 there is another option “Personal builds on branches”. You commit your changes into special branches that follow a certain name pattern. If Teamcity recognizes this commit it will start a personal build and push to the master branch if the build was successful. A very similar approach has been described for Jenkins.

We are using Git as our VCS since more than a year but the new solution by Teamcity has only been added very recently. This situation forced us to create our own solution and to our surprise this was much easier than expected. The main reason was the distributed nature of Git. Some of the points are:

  • A commit is something that only effects your local repository as long as you don’t push it. This allows to test after a commit and before push. So you don’t need a special mechanism to create patches and send them somewhere.
  • The central repository is not the only one. Beside your local copy you can create as many repositories as you like. So you can push wherever you like to in order to e.g. run tests.
  • The central repository that you pull from doesn’t need to be the same that you push to.
  • Commits have a globally unique identifier. You can try to push a commit twice. Git will recognize it and don’t apply it the second time.

You can probably already guess what I am up to, so lets show a picture of the overall setup.

There is one central Git-repository that only contains pre-tested changes. I call this “Green Repository” because it should only contain changes that lead to green builds. Every developer pulls from this repository but nobody is allowed to push to it. Instead everybody has a personal repository (think fork if you were on GitHub). The CI Server watches those personal repositories. After a commit it starts the compile and test. If that was successful it pushes the changes to the Green Repository. Now everybody can pull your changes and can be sure they have been tested.

You might wonder about some details of this setup so I try to answer some questions here:

  • The remote server that contains the Green Repository and all personal remote repositories could be a simple git ssh setup. We are using a local installation of the Gitorious software instead. It is very easy to clone a central repository and to set access rights this way. There is also a commercial solution by the creators of GitHub called GitHub:FI.
  • The remote-run on the CI Server needs to include 2 steps. Running the build (e.g. maven) and pushing the changes after successful build. We are using build dependencies in Teamcity in order to do these two steps. The first step is a simple maven goal and the second step is a simple command line call “git push”.
  • In order to make the push to your personal remote repository easier you should create an alias. “git remote add remote-run [your remote repo url]” then it is easy to push your changes using “git push remote-run”.
  • This solution works best with small teams that do not change often. The setup procedure for a new team member requires good knowledge of the solution. The setup itself can be done in less than 15 Minutes.

The question is whether we will continue using our own solution or switch to the new feature of Teamcity. So far I can’t see any advantage of the Teamcity feature. With our solution we are even more flexible in regard of branch design. So I think our layout will live on for a while.

If your CI-Server has no support for pre-tested commits you might give our setup a try. I am eager to hear your experiences. I am also happy to answer any specific questions about the setup in comments.

Executing shell commands in Groovy

Posted in groovy, java, tools on September 30th, 2010 by Joerg – 15 Comments

Groovy is a good alternative to classic shell scripting for the Java experienced developer. I already wrote about this here. A common scenario is to call other shell commands from your scripts. Groovy makes this easy, but during a recent project I learned about some pitfalls. I will explain them here and also how to avoid them.

The simple way

Executing a simple command is just a matter of putting an .execute() after a string like that:

"mkdir foo".execute()

(Please ignore the fact for now, that you could do this using the Java standard functionality. This would be more effective in this example. Believe me, there are more complex requirements where you definitely need to call a shell command. I just keep the examples simple.)

Evaluating the exit value

What if you want to know whether this action was successful? In a shell script you usually check the exit value. Thats still straightforward:

def exitValue = "mkdir foo".execute().exitValue()
if(!exitValue)
  //do your error-handling

“execute()” just creates a java.lang.Process object. This simply returns an int with the exit value when calling exitValue() which you can use in your script.

Printing the output

Let’s say you want to see the result of the shell command during the runtime of your script. The easy approach would be:

println "ls".execute().text

If you want to combine text output and evaluation of the exit value it starts getting less elegant:

def process = "ls".execute()
process.text.eachLine {println it}
if(!process.exitValue())
...

This works, but not very well. It has two major issues:

  • The output will not happen until the shell command is finished. This is ok for a simple “ls” but will get ugly, when you have a long running command.
  • process.text contains only standard out by default, but not err out.

To solve this you can’t use the simple “.execute()” anymore. You need to fall back to java.lang.ProcessBuilder. This allows you to “redirectErrorStream()” which joins both streams together as it would look like when executing the command on a shell. Here is the example code:

  def process=new ProcessBuilder("ls").redirectErrorStream(true).start()
  process.inputStream.eachLine {println it}

The working directory

Another pitfall is the current working directory. In a simple shell script you would just “cd” into the new working directory and then continue with your script. This would not work in a groovy script. As groovy is using the underlying Java mechanisms the “cd” command has no effect to the following instances of Process. Instead you need to tell the ProcessBuilder where your working directory has to be. There are several ways to do this. The first would be to add a “directory(File)” call to the ProcessBuilder from the example above. This would look like that:

  def processBuilder=new ProcessBuilder("ls")
  processBuilder.redirectErrorStream(true)
  processBuilder.directory(new File("Your Working dir"))  // <--
  def process = processBuilder.start()
  ...

Fortunately there is a more convenient way when you use the execute method. You can use an overloaded version with two parameters. The first is a String array or a List containing environment variables or null if you don’t need them. The second is the working directory in form of a File object. Here is a code example:

   "your command".execute(null, new File("your working dir")) 

As a side note, if you need to set specific environment variables you also need to set them for each call.

Using wildcard characters

This pitfall is described in many Groovy books but this post would not be complete without it. Using Wildcards like in the following example would not work.

   "ls *.java".execute() 

Instead of listing all Java source files in the current directory as you might expect, it will try to list a file called “*.java”. It is very unlikely that this file is found. Wildcards like * are interpreted by the shell. The execute command is not using one. You need to call the shell first and let it interpret the wildcard-characters. On OSX and Linux this is simply done by adding a “sh -c” in front of the command. Something slightly different applies on Windows which I leave you to look up. The following command will do the expected.

   "sh -c ls *.java".execute() 

It doesn’t do any harm to add this prefix to all shell commands you like to start. You only need to take care of the operating system your script is running on. I am in the fortunate position to expect a UNIX-like shell on every computer in my project so I simply add “sh -c”.

Putting it all together

Because of all the issues mentioned above I ended up adding two methods to my Groovy scripts called executeOnShell. These methods solves the pitfalls mentioned and still result in a concise syntax when calling shell commands from the script.

def executeOnShell(String command) {
  return executeOnShell(command, new File(System.properties.'user.dir'))
}

private def executeOnShell(String command, File workingDir) {
  println command
  def process = new ProcessBuilder(addShellPrefix(command))
                                    .directory(workingDir)
                                    .redirectErrorStream(true) 
                                    .start()
  process.inputStream.eachLine {println it}
  process.waitFor();
  return process.exitValue()
}

private def addShellPrefix(String command) {
  commandArray = new String[3]
  commandArray[0] = "sh"
  commandArray[1] = "-c"
  commandArray[2] = command
  return commandArray
}

The method executeOnShell is overloaded, so that you can either call it with a working directory or without. It also returns the exitValue, so you can easily evaluate it. This is of course not complete, as it does not include the ability to set Environment variables and it works only on a Unix-like shell. It is sufficient for my current use cases.
Please feel free to use it and to add your own requirements. If you know of any other pitfalls when executing shell commands from Groovy please leave a comment.

Scripting magic with Groovy, Grape and Jetty

Posted in groovy, java, tools on December 31st, 2009 by Joerg – 4 Comments

Groovy 1.7 has just been release and it’s time to play again. One of the new features, Grape inspired me to the try the following. I created a script that acts as a simple webserver using embedded jetty without the need to install anything else than Groovy 1.7 on your computer. Just save the script as simpleWebServer, make it executable and start it like

./simpleWebServer -d someDirWithHTMLFiles -p 9000

Surf to http://localhost:9000 and you will see.

Here is the full script. I will explain the details further down.

#!/usr/bin/env groovy

@Grab('org.mortbay.jetty:jetty:6.1.22')
import org.mortbay.jetty.*;
import org.mortbay.jetty.handler.*

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

def server = new Server(opts.port.toInteger()?:8080)
def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                         resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

server.start()
server.join()

Here are the magic details about the script:

1. The Shebang

The first important thing about this script is the shebang.

#!/usr/bin/env groovy

If you are using a real operating system (sorry Windows users), this will let you start the script like a normal executable. I usually skip the .groovy appendix on script files, so it really feels like you are using a normal command line utility.

2. Grape

The next line uses the new feature of Groovy 1.7, called Grape. This is a dependency System that allows to load any dependency, that is available in the maven-repositories to be used inside a script.

@Grab('org.mortbay.jetty:jetty:6.1.22')

The syntax is pretty easy. It uses the maven notation: groupId:artifactId:version. There is an alternative version that uses separate attributes for each but I prefer the shortcut.
This line will lookup the dependency in the maven repository, download it and store it in ~/.groovy/grape. So the first start of the script might take a moment. The second time will be faster.
Grape allows to use the full java ecosystem in a simple groovy script without any additional install. Groovy 1.7 is enough. I think this makes scripting in groovy incredible powerful.

3. The CliBuilder
The script should really feel like a command line utility. Therefore it needs to deal with parameters. This is where the CliBuilder comes in.

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

Groovy includes Apache commons CLI. But where the Java version is already helpful in parsing parameters, the Groovy version gets really simple. The example above creates one CliBuilder, that provides a “DSL” for defining the parameters. Further down they are three parameters defined. This is all thats needed.
After the args are parsed we can just simply use them using Groovy properties syntax like opts.parameterName. Even a pretty usage statement can be printed, when –help is called.

4. Dynamic constructors
I used one of the embedded Jetty examples (FileServer.java) as the foundation of my script. The configuration of the handlers looked something like that in Java:

ResourceHandler resource_handler=new ResourceHandler();
resource_handler.setWelcomeFiles(new String[]{"index.html"});
resource_handler.setResourceBase(args.length==2?args[1]:".");

HandlerList handlers = new HandlerList();
handlers.setHandlers(new Handler[]{resource_handler,new DefaultHandler()});

Thats pretty long. Now the Groovy version looks like that:

def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                          resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

It uses dynamic constructors. Groovy allows to call a virtual constructor with a map of arguments, where the elements are actually bean properties. This allows a much more concise construction of an object, even if the original creator wasn’t so kind to provide a convenience constructor.

5. The Elvis Operator

You probably wondered what the following statement did in the last example:

opts.dir?:"."

Well, that was Elvis. In Java there is the ternary operator, which goes like this:

(condition)?(result if true):(result if false) 

In Grovvy there is something called the Groovy truth, which says if a statement is null, then it is false, else it is true. This makes the Elvis operator possible. Elvis says, if the statement is true (as in Groovy truth) then use the value of the statement, else use the value after the colon. This is a very concise way for realizing default values, which is often needed in scripts

With all the scripting features that were added in earlier versions and with Grape in 1.7 I think Groovy is finally a great alternative to usual scripting languages. When you are experienced in Java it’s probably much simpler to write a script in Groovy than in Perl, Ruby or even Bash. For me this is definitely true.

Well done, Groovy team!

IntelliJ IDEAs new directory-based project format

Posted in java, software development, tools on October 17th, 2009 by Joerg – 9 Comments

I am currently playing with the EAP-Version (Early Access Program) of my favorite IDE IntelliJ IDEA. Since the new version is not that far away it is time to learn the new features and I discovered one that surprised me. A small but pretty nice change.

(Update: Thanks to Strug I have realized that this feature is already present in Idea 8, you can use the action “Open in New (Directory based) format” to convert your project. Fortunately they renamed it to “Save as Directory-Based Format …” in Maia. )

IDEA has a new format to store the project files. It is called directory-based. Instead of using the three famous files .ipr .iml and .iws IDEA will now store all information into a directory, which is simply called .idea. It is located in the root folder of the project. This seems to be somewhat similar to eclipse’s .project folder. In a small example project I created the content of the new directory looks like this:

-rw-r--r--  1 joerg  joerg    163 17 Okt 09:53 ant.xml
-rw-r--r--  1 joerg  joerg   2107 17 Okt 09:53 compiler.xml
drwxr-xr-x  3 joerg  joerg    102 17 Okt 09:53 copyright
drwxr-xr-x  3 joerg  joerg    102 17 Okt 09:53 dictionaries
-rw-r--r--  1 joerg  joerg    277 17 Okt 09:53 encodings.xml
-rw-r--r--  1 joerg  joerg    170 17 Okt 09:53 fileColors.xml
-rw-r--r--  1 joerg  joerg   1595 17 Okt 09:53 misc.xml
-rw-r--r--  1 joerg  joerg    258 17 Okt 09:53 modules.xml
-rw-r--r--  1 joerg  joerg    207 17 Okt 09:53 templateLanguages.xml
-rw-r--r--  1 joerg  joerg    169 17 Okt 09:53 vcs.xml
-rw-r--r--  1 joerg  joerg  38245 17 Okt 10:21 workspace.xml

The files included in this directory depend on the settings you change in the IDE. So, if you would for instance set the SQL dialect for your project there will be another file called sqldialects.xml containing all settings about this.

There is one special file in there, which is workspace.xml. This file contains individual settings for the workspace, which are definitely not intended to be shared via version control. This is equivalent to the .iws file of the old format. IDEA will put this file automatically on the ignore list of your version control.

I see some advantages of the new format.

  • First of all it is easier to find specific settings. The filenames are meaningful and the files are small.
  • The new structure allows a very detailed control of which settings you want to share with your colleagues. If you don’t want to share e.g. your file coloring (another new feature of Maia) just put the file on the ignore list.
  • Directories of the projects saved in the new format will be recognized as projects in the open-project dialog. In the past this was one annoying additional click as you had to choose the .ipr file before.

The new directory based format is a small change but a very good one. It is often a sum of little detail-improvements that save a lot of trouble in daily work. So I am looking forward to the other details to be discovered.

By the way JetBrains released an open source version of IDEA two days ago. So if you want to try this just go to www.jetbrains.org and download the Community Edition.

XMind on Mac with Java 6

Posted in apple, java, tools on February 27th, 2009 by Joerg – 2 Comments

The Problem

XMind is a very nice mind mapping software. It is platform independent and it’s free. I started using it at work on a Windows PC. When I installed it on my Mac I had a bad surprise. When starting it, it stopped immediately with a message that the JVM terminated.

xminderror

Xmind is based on Eclipse RCP. Eclipse itself is using the Carbon framework on the Mac for it’s GUI. But Carbon is not supported on 64 Bit. But Java 1.6 on the Mac does only work on 64 Bit. So it will of course not work.

The Solution

You don’t need to uninstall Java 6 in order to run XMind. It turned out the solution was much simpler. You have to tell the XMind app what JVM to use. To do this use your favorite text editor and open the following file (I recommend VI :) ):

/Applications/XMind.app/Contents/Info.plist

Look for the following text and uncomment the -vm option:

<!-- to use a specific Java version (instead of the platform's default) uncomment 
         the following options:-->
<string>-vm</string>
<string>/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Commands/java</string>

Make sure it points to the right JVM. I had to change it to 1.5.0 as it did only point to Current, which was 1.6.

Save, start XMind again and have fun creating Your mindmaps.