groovy

Executing shell commands in Groovy

Posted in groovy, java, tools on September 30th, 2010 by Joerg – 23 Comments

Groovy is a good alternative to classic shell scripting for the Java experienced developer. I already wrote about this here. A common scenario is to call other shell commands from your scripts. Groovy makes this easy, but during a recent project I learned about some pitfalls. I will explain them here and also how to avoid them.

The simple way

Executing a simple command is just a matter of putting an .execute() after a string like that:

"mkdir foo".execute()

(Please ignore the fact for now, that you could do this using the Java standard functionality. This would be more effective in this example. Believe me, there are more complex requirements where you definitely need to call a shell command. I just keep the examples simple.)

Evaluating the exit value

What if you want to know whether this action was successful? In a shell script you usually check the exit value. Thats still straightforward:

def exitValue = "mkdir foo".execute().exitValue()
if(!exitValue)
  //do your error-handling

“execute()” just creates a java.lang.Process object. This simply returns an int with the exit value when calling exitValue() which you can use in your script.

Printing the output

Let’s say you want to see the result of the shell command during the runtime of your script. The easy approach would be:

println "ls".execute().text

If you want to combine text output and evaluation of the exit value it starts getting less elegant:

def process = "ls".execute()
process.text.eachLine {println it}
if(!process.exitValue())
...

This works, but not very well. It has two major issues:

  • The output will not happen until the shell command is finished. This is ok for a simple “ls” but will get ugly, when you have a long running command.
  • process.text contains only standard out by default, but not err out.

To solve this you can’t use the simple “.execute()” anymore. You need to fall back to java.lang.ProcessBuilder. This allows you to “redirectErrorStream()” which joins both streams together as it would look like when executing the command on a shell. Here is the example code:

  def process=new ProcessBuilder("ls").redirectErrorStream(true).start()
  process.inputStream.eachLine {println it}

The working directory

Another pitfall is the current working directory. In a simple shell script you would just “cd” into the new working directory and then continue with your script. This would not work in a groovy script. As groovy is using the underlying Java mechanisms the “cd” command has no effect to the following instances of Process. Instead you need to tell the ProcessBuilder where your working directory has to be. There are several ways to do this. The first would be to add a “directory(File)” call to the ProcessBuilder from the example above. This would look like that:

  def processBuilder=new ProcessBuilder("ls")
  processBuilder.redirectErrorStream(true)
  processBuilder.directory(new File("Your Working dir"))  // <--
  def process = processBuilder.start()
  ...

Fortunately there is a more convenient way when you use the execute method. You can use an overloaded version with two parameters. The first is a String array or a List containing environment variables or null if you don’t need them. The second is the working directory in form of a File object. Here is a code example:

   "your command".execute(null, new File("your working dir")) 

As a side note, if you need to set specific environment variables you also need to set them for each call.

Using wildcard characters

This pitfall is described in many Groovy books but this post would not be complete without it. Using Wildcards like in the following example would not work.

   "ls *.java".execute() 

Instead of listing all Java source files in the current directory as you might expect, it will try to list a file called “*.java”. It is very unlikely that this file is found. Wildcards like * are interpreted by the shell. The execute command is not using one. You need to call the shell first and let it interpret the wildcard-characters. On OSX and Linux this is simply done by adding a “sh -c” in front of the command. Something slightly different applies on Windows which I leave you to look up. The following command will do the expected.

   "sh -c ls *.java".execute() 

It doesn’t do any harm to add this prefix to all shell commands you like to start. You only need to take care of the operating system your script is running on. I am in the fortunate position to expect a UNIX-like shell on every computer in my project so I simply add “sh -c”.

Putting it all together

Because of all the issues mentioned above I ended up adding two methods to my Groovy scripts called executeOnShell. These methods solves the pitfalls mentioned and still result in a concise syntax when calling shell commands from the script.

def executeOnShell(String command) {
  return executeOnShell(command, new File(System.properties.'user.dir'))
}

private def executeOnShell(String command, File workingDir) {
  println command
  def process = new ProcessBuilder(addShellPrefix(command))
                                    .directory(workingDir)
                                    .redirectErrorStream(true) 
                                    .start()
  process.inputStream.eachLine {println it}
  process.waitFor();
  return process.exitValue()
}

private def addShellPrefix(String command) {
  commandArray = new String[3]
  commandArray[0] = "sh"
  commandArray[1] = "-c"
  commandArray[2] = command
  return commandArray
}

The method executeOnShell is overloaded, so that you can either call it with a working directory or without. It also returns the exitValue, so you can easily evaluate it. This is of course not complete, as it does not include the ability to set Environment variables and it works only on a Unix-like shell. It is sufficient for my current use cases.
Please feel free to use it and to add your own requirements. If you know of any other pitfalls when executing shell commands from Groovy please leave a comment.

Scripting magic with Groovy, Grape and Jetty

Posted in groovy, java, tools on December 31st, 2009 by Joerg – 4 Comments

Groovy 1.7 has just been release and it’s time to play again. One of the new features, Grape inspired me to the try the following. I created a script that acts as a simple webserver using embedded jetty without the need to install anything else than Groovy 1.7 on your computer. Just save the script as simpleWebServer, make it executable and start it like

./simpleWebServer -d someDirWithHTMLFiles -p 9000

Surf to http://localhost:9000 and you will see.

Here is the full script. I will explain the details further down.

#!/usr/bin/env groovy

@Grab('org.mortbay.jetty:jetty:6.1.22')
import org.mortbay.jetty.*;
import org.mortbay.jetty.handler.*

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

def server = new Server(opts.port.toInteger()?:8080)
def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                         resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

server.start()
server.join()

Here are the magic details about the script:

1. The Shebang

The first important thing about this script is the shebang.

#!/usr/bin/env groovy

If you are using a real operating system (sorry Windows users), this will let you start the script like a normal executable. I usually skip the .groovy appendix on script files, so it really feels like you are using a normal command line utility.

2. Grape

The next line uses the new feature of Groovy 1.7, called Grape. This is a dependency System that allows to load any dependency, that is available in the maven-repositories to be used inside a script.

@Grab('org.mortbay.jetty:jetty:6.1.22')

The syntax is pretty easy. It uses the maven notation: groupId:artifactId:version. There is an alternative version that uses separate attributes for each but I prefer the shortcut.
This line will lookup the dependency in the maven repository, download it and store it in ~/.groovy/grape. So the first start of the script might take a moment. The second time will be faster.
Grape allows to use the full java ecosystem in a simple groovy script without any additional install. Groovy 1.7 is enough. I think this makes scripting in groovy incredible powerful.

3. The CliBuilder
The script should really feel like a command line utility. Therefore it needs to deal with parameters. This is where the CliBuilder comes in.

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

Groovy includes Apache commons CLI. But where the Java version is already helpful in parsing parameters, the Groovy version gets really simple. The example above creates one CliBuilder, that provides a “DSL” for defining the parameters. Further down they are three parameters defined. This is all thats needed.
After the args are parsed we can just simply use them using Groovy properties syntax like opts.parameterName. Even a pretty usage statement can be printed, when –help is called.

4. Dynamic constructors
I used one of the embedded Jetty examples (FileServer.java) as the foundation of my script. The configuration of the handlers looked something like that in Java:

ResourceHandler resource_handler=new ResourceHandler();
resource_handler.setWelcomeFiles(new String[]{"index.html"});
resource_handler.setResourceBase(args.length==2?args[1]:".");

HandlerList handlers = new HandlerList();
handlers.setHandlers(new Handler[]{resource_handler,new DefaultHandler()});

Thats pretty long. Now the Groovy version looks like that:

def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                          resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

It uses dynamic constructors. Groovy allows to call a virtual constructor with a map of arguments, where the elements are actually bean properties. This allows a much more concise construction of an object, even if the original creator wasn’t so kind to provide a convenience constructor.

5. The Elvis Operator

You probably wondered what the following statement did in the last example:

opts.dir?:"."

Well, that was Elvis. In Java there is the ternary operator, which goes like this:

(condition)?(result if true):(result if false) 

In Grovvy there is something called the Groovy truth, which says if a statement is null, then it is false, else it is true. This makes the Elvis operator possible. Elvis says, if the statement is true (as in Groovy truth) then use the value of the statement, else use the value after the colon. This is a very concise way for realizing default values, which is often needed in scripts

With all the scripting features that were added in earlier versions and with Grape in 1.7 I think Groovy is finally a great alternative to usual scripting languages. When you are experienced in Java it’s probably much simpler to write a script in Groovy than in Perl, Ruby or even Bash. For me this is definitely true.

Well done, Groovy team!

XML to POJO via Groovy

Posted in groovy, java, software development on March 8th, 2009 by Joerg – 9 Comments

Transforming data in XML to several Java-Objects is a pretty common task. There are a lot of technologies that support this. One example would be JAXB. It is also not uncommon that the structure of the original XML and the Java objects does not match. The obvious choice for XML transformation in such a situation would be XSLT.
There is another approach that uses Groovy to transform XML directly into a POJO structure in one step. I will show you how this looks like.

Lets assume, we have a XML-file like this:

<data>
  <person>
    <firstname>Otto</firstname>
    <lastname>Mueller</lastname>
    <street>Lange Strasse</street>
    <city>Berlin</city>
  </person>
  <financial>
    <netincome>120000</netincome>
  </financial>
</data>

The target structure are two POJOs that look like this:
Target Objects

Both structures differ. There are also some small conversions to do. The XML includes yearly income, where the POJOs needs monthly income. The POJOs contains a field for zipcode, but the XML does not deliver it.

So, here is the Groovy-Code that does all the transformation:

public class XmlTransformer
{

  Person transform(File xml)  //1
  {
    def xmlData = new XmlSlurper().parse(xml)   //2

    Person person = new Person()

    person.firstName = xmlData.person.firstname   //3
    person.lastName = xmlData.person.lastname

    Address address = new Address()
    address.city = xmlData.person.city
    address.street = xmlData.person.street
    address.zipCode = ZipCodeFinder.find(xmlData.person.city.toString(),   //4
            xmlData.person.street.toString())
    person.address = address

    BigInteger yearlyIncome = new BigInteger(xmlData.financial.netincome.toString())
    person.monthlyIncome = yearlyIncome.divide(12)  //5

    return person
  }

}

What happens in detail (numbers on the list match the numbers in code comments):

  1. This is a simple Groovy class that contains one method called transform. This method gets the XML-file as input and returns a Person object. The Person class itself is defined in Java.
  2. XMLSlurper  is the Groovy XML-Parser that allows the easy access to all xml elements. For more details on it see here.
  3. This uses some Groovy magic. On the left side we assign the value to the property firstName of the person object. This is a shortcut for using person.setFirstName(). On the right side we see the slurper at work to get us the XML value.
  4. We can call any Java-class from within the transformation code. Here we call an example helper-class, that would return a zipcode for a given city and street.
  5. Finally we also do some calculations within our transformation to convert the yearly income to a monthly value.

Groovy can be called from Java in several ways. If you do not need to change the transformation often I recommend to just compile the Groovy code. In that case it will be called from Java as if it would be Java code:

XmlTransformer transformer = new XmlTransformer();
Person person = transformer.transform(inputFile);

This kind of transformation code can easily be embedded into a Java project. There are several advantages, when using the Groovy method:

  • Just one step for transformation including conversions.
  • Easy debugging, when using an IDE like IntelliJ IDEA.
  • Java-like syntax, no learning of XSLT required.
  • Easy code-structuring. The transformation can use all features of a dynamic object-oriented language.
  • Simple unit-testing. If you split the code into several methods, you can test each logical unit using established frameworks like JUnit or TestNG.

So what is your opinion? What are your current methods of transforming XML to POJOs?