Executing shell commands in Groovy

Groovy is a good alternative to classic shell scripting for the Java experienced developer. I already wrote about this here. A common scenario is to call other shell commands from your scripts. Groovy makes this easy, but during a recent project I learned about some pitfalls. I will explain them here and also how to avoid them.

The simple way

Executing a simple command is just a matter of putting an .execute() after a string like that:

"mkdir foo".execute()

(Please ignore the fact for now, that you could do this using the Java standard functionality. This would be more effective in this example. Believe me, there are more complex requirements where you definitely need to call a shell command. I just keep the examples simple.)

Evaluating the exit value

What if you want to know whether this action was successful? In a shell script you usually check the exit value. Thats still straightforward:

def exitValue = "mkdir foo".execute().exitValue()
if(!exitValue)
  //do your error-handling

“execute()” just creates a java.lang.Process object. This simply returns an int with the exit value when calling exitValue() which you can use in your script.

Printing the output

Let’s say you want to see the result of the shell command during the runtime of your script. The easy approach would be:

println "ls".execute().text

If you want to combine text output and evaluation of the exit value it starts getting less elegant:

def process = "ls".execute()
process.text.eachLine {println it}
if(!process.exitValue())
...

This works, but not very well. It has two major issues:

  • The output will not happen until the shell command is finished. This is ok for a simple “ls” but will get ugly, when you have a long running command.
  • process.text contains only standard out by default, but not err out.

To solve this you can’t use the simple “.execute()” anymore. You need to fall back to java.lang.ProcessBuilder. This allows you to “redirectErrorStream()” which joins both streams together as it would look like when executing the command on a shell. Here is the example code:

  def process=new ProcessBuilder("ls").redirectErrorStream(true).start()
  process.inputStream.eachLine {println it}

The working directory

Another pitfall is the current working directory. In a simple shell script you would just “cd” into the new working directory and then continue with your script. This would not work in a groovy script. As groovy is using the underlying Java mechanisms the “cd” command has no effect to the following instances of Process. Instead you need to tell the ProcessBuilder where your working directory has to be. There are several ways to do this. The first would be to add a “directory(File)” call to the ProcessBuilder from the example above. This would look like that:

  def processBuilder=new ProcessBuilder("ls")
  processBuilder.redirectErrorStream(true)
  processBuilder.directory(new File("Your Working dir"))  // <--
  def process = processBuilder.start()
  ...

Fortunately there is a more convenient way when you use the execute method. You can use an overloaded version with two parameters. The first is a String array or a List containing environment variables or null if you don’t need them. The second is the working directory in form of a File object. Here is a code example:

   "your command".execute(null, new File("your working dir")) 

As a side note, if you need to set specific environment variables you also need to set them for each call.

Using wildcard characters

This pitfall is described in many Groovy books but this post would not be complete without it. Using Wildcards like in the following example would not work.

   "ls *.java".execute() 

Instead of listing all Java source files in the current directory as you might expect, it will try to list a file called “*.java”. It is very unlikely that this file is found. Wildcards like * are interpreted by the shell. The execute command is not using one. You need to call the shell first and let it interpret the wildcard-characters. On OSX and Linux this is simply done by adding a “sh -c” in front of the command. Something slightly different applies on Windows which I leave you to look up. The following command will do the expected.

   "sh -c ls *.java".execute() 

It doesn’t do any harm to add this prefix to all shell commands you like to start. You only need to take care of the operating system your script is running on. I am in the fortunate position to expect a UNIX-like shell on every computer in my project so I simply add “sh -c”.

Putting it all together

Because of all the issues mentioned above I ended up adding two methods to my Groovy scripts called executeOnShell. These methods solves the pitfalls mentioned and still result in a concise syntax when calling shell commands from the script.

def executeOnShell(String command) {
  return executeOnShell(command, new File(System.properties.'user.dir'))
}

private def executeOnShell(String command, File workingDir) {
  println command
  def process = new ProcessBuilder(addShellPrefix(command))
                                    .directory(workingDir)
                                    .redirectErrorStream(true) 
                                    .start()
  process.inputStream.eachLine {println it}
  process.waitFor();
  return process.exitValue()
}

private def addShellPrefix(String command) {
  commandArray = new String[3]
  commandArray[0] = "sh"
  commandArray[1] = "-c"
  commandArray[2] = command
  return commandArray
}

The method executeOnShell is overloaded, so that you can either call it with a working directory or without. It also returns the exitValue, so you can easily evaluate it. This is of course not complete, as it does not include the ability to set Environment variables and it works only on a Unix-like shell. It is sufficient for my current use cases.
Please feel free to use it and to add your own requirements. If you know of any other pitfalls when executing shell commands from Groovy please leave a comment.

Be Sociable, Share!
  1. Johann says:

    I use Commons-Exec to run external programs at http://media.io but the default API of Commons Exec is still too low-level so I wrote anabstraction that has features like support for multiple commands, line queueing, ignoring of streams and that can also be used asynchronously in Executors.

    I’m not really 100 % satisfied with the API but I might hand it over to the Commons-Exec guys as a high-level class with a fluent API some day.

    If you’re interested, feel free to email me and I’ll send it to you to check out.

    • Joerg says:

      Hi Johann,

      sounds interesting. I guess your API is Java-based. Have you been using it from Groovy already? This gives another perspective regarding the API.

      An improved shell execution API available in Groovy scripts would certainly be good.

  2. Johann says:

    Joerg,

    yes, the code is in Java for performance and control reasons. Used with Groovy, it looks like this:

    ProcessBuilder builder = ProcessBuilder.run(“echo ${foo}”, “echo bar”, “echo blub”).in(“/tmp”).substitute([ foo: 'bar' ]).ignoreStdErr().lowPriority()

    It’s got one or two other tricks.

    • Joerg says:

      Nice,
      with Groovy 1.8, it’s extended command expressions and a static import it could even look like that:

      run “echo ${foo}” in “/tmp” substitute [foo:"bar] ignoreStdErr

      If the commons-exec people are not interested, may be the Groovy people are :)

      • Johann says:

        I think it would work better with Commons-Exec because Commons-Exec is currently missing a really good high-level support class. Also, it’s got a dependency on Commons-Exec anyway (and on SLF4J).

        Anyway, with the new syntax, it looks even better. You could of course use Categories and get

        use (ProcessBuilder) {
        [ "echo foo", "echo bla" ].run().ignoreStdErr().run()
        }

        I admit the double run method is a bit annoying. That’s why I said it’s not finished yet. ;-)

  3. tomi says:

    is there any way to quote parameter for command, say ls a file with a space in the filename, i will use ls ‘read it.txt’ in shell, but this doesn’t work when in groovy as following

    proc=”ls -l ‘/tmp/read me.txt’”.execute()

    it will be interpret as list 2 files with name ‘/tmp/read and me.txt’ as the proc.err.text output:
    ls: cannot access ‘/tmp/read: No such file or directory
    ls: cannot access me.txt’: No such file or directory

    • Joerg says:

      I am afraid there is no easy way. You will have to use the CommandArray Syntax I show in my last example, like this:

      def command = new String[3]
      command[0] = "sh"
      command[1] = "-c"
      command[2] = "ls 'read me.txt'"
      def process = new ProcessBuilder(command).start()
      println process.text
      
  4. Antonella says:

    Hello, nice blog.
    I’m a groovy newbie.
    How about linux pipes then? If I had to execute something like this:

    “cat somefile.txt | awk ‘($1>=3){print $2}’”

    what should I do?

    Thanks,
    Anthos

    • Joerg says:

      I have not tried pipes yet, but I think you will need to add ‘sh -c’ in front like in my last comment.

    • Ian says:

      for pipes you have to separate the processes
      proc1 = ‘ls’.execute()
      proc2 = ‘grep (something)’.execute()
      all = proc1 | proc2

      println all.text

  5. Dan says:

    Thanks for the post. Using processbuilder works much better than simply calling execute(). It is faster, and execute() was hanging for processes with very large outputs.

  6. Gitesh says:

    Hi..Thanks for such a useful post.
    Can u please give a code snippet to run command prompt and also pass commands and execute them in groovy script?

  7. Jeff says:

    Super helpful! I’d been stuck on a quirky multi-argument quoted command for a day now and your ProcessBuilder info did the trick.

    Thanks!

  8. Albert says:

    Hi Jörg,
    I’ve have done some tests in groovy with execute() and ProcessBuilder today.
    At the end of the day I can say that if you want to play with a shell then you HAVE TO use ProcessBuilder :)
    If you don’t, you will get a lot of problems with WHITE SPACES … at least on MAC !
    White spaces can be in the path or parameters (doesn’t matters if quoted or not) .

    btw: very nice post

  9. Tuomas says:

    Thanks about executing * wildcards, which I have forgotton how to use command array parameters.

  1. There are no trackbacks for this post yet.

Leave a Reply