java

Continuous Delivery Slides

Posted in agile, continuous delivery, continuous integration, deployment, java, management, software development, testing, version control on July 5th, 2011 by Joerg – Be the first to comment

The last Java User Group event in Berlin was hosted at Hypoport. It was about Continuous Delivery. We had about 100 guests.

There where two talks. One by Axel Fontaine who introduced Continuous Delivery especially in a Java environment. You can find his talk on Parleys or on his Blog.

I had the pleasure to introduce our experiences with Continuous Delivery. We are currently developing a new product. Since we started about a year ago we consequently used Continuous Delivery. We learned quite some lessons on our way which I presented that evening. My Slides are available here. (As this was a german event the slides are in german)

Thanks to everyone who attended the event. It was a really nice evening. Special thanks to everybody who helped organizing it.

Executing shell commands in Groovy

Posted in groovy, java, tools on September 30th, 2010 by Joerg – 15 Comments

Groovy is a good alternative to classic shell scripting for the Java experienced developer. I already wrote about this here. A common scenario is to call other shell commands from your scripts. Groovy makes this easy, but during a recent project I learned about some pitfalls. I will explain them here and also how to avoid them.

The simple way

Executing a simple command is just a matter of putting an .execute() after a string like that:

"mkdir foo".execute()

(Please ignore the fact for now, that you could do this using the Java standard functionality. This would be more effective in this example. Believe me, there are more complex requirements where you definitely need to call a shell command. I just keep the examples simple.)

Evaluating the exit value

What if you want to know whether this action was successful? In a shell script you usually check the exit value. Thats still straightforward:

def exitValue = "mkdir foo".execute().exitValue()
if(!exitValue)
  //do your error-handling

“execute()” just creates a java.lang.Process object. This simply returns an int with the exit value when calling exitValue() which you can use in your script.

Printing the output

Let’s say you want to see the result of the shell command during the runtime of your script. The easy approach would be:

println "ls".execute().text

If you want to combine text output and evaluation of the exit value it starts getting less elegant:

def process = "ls".execute()
process.text.eachLine {println it}
if(!process.exitValue())
...

This works, but not very well. It has two major issues:

  • The output will not happen until the shell command is finished. This is ok for a simple “ls” but will get ugly, when you have a long running command.
  • process.text contains only standard out by default, but not err out.

To solve this you can’t use the simple “.execute()” anymore. You need to fall back to java.lang.ProcessBuilder. This allows you to “redirectErrorStream()” which joins both streams together as it would look like when executing the command on a shell. Here is the example code:

  def process=new ProcessBuilder("ls").redirectErrorStream(true).start()
  process.inputStream.eachLine {println it}

The working directory

Another pitfall is the current working directory. In a simple shell script you would just “cd” into the new working directory and then continue with your script. This would not work in a groovy script. As groovy is using the underlying Java mechanisms the “cd” command has no effect to the following instances of Process. Instead you need to tell the ProcessBuilder where your working directory has to be. There are several ways to do this. The first would be to add a “directory(File)” call to the ProcessBuilder from the example above. This would look like that:

  def processBuilder=new ProcessBuilder("ls")
  processBuilder.redirectErrorStream(true)
  processBuilder.directory(new File("Your Working dir"))  // <--
  def process = processBuilder.start()
  ...

Fortunately there is a more convenient way when you use the execute method. You can use an overloaded version with two parameters. The first is a String array or a List containing environment variables or null if you don’t need them. The second is the working directory in form of a File object. Here is a code example:

   "your command".execute(null, new File("your working dir")) 

As a side note, if you need to set specific environment variables you also need to set them for each call.

Using wildcard characters

This pitfall is described in many Groovy books but this post would not be complete without it. Using Wildcards like in the following example would not work.

   "ls *.java".execute() 

Instead of listing all Java source files in the current directory as you might expect, it will try to list a file called “*.java”. It is very unlikely that this file is found. Wildcards like * are interpreted by the shell. The execute command is not using one. You need to call the shell first and let it interpret the wildcard-characters. On OSX and Linux this is simply done by adding a “sh -c” in front of the command. Something slightly different applies on Windows which I leave you to look up. The following command will do the expected.

   "sh -c ls *.java".execute() 

It doesn’t do any harm to add this prefix to all shell commands you like to start. You only need to take care of the operating system your script is running on. I am in the fortunate position to expect a UNIX-like shell on every computer in my project so I simply add “sh -c”.

Putting it all together

Because of all the issues mentioned above I ended up adding two methods to my Groovy scripts called executeOnShell. These methods solves the pitfalls mentioned and still result in a concise syntax when calling shell commands from the script.

def executeOnShell(String command) {
  return executeOnShell(command, new File(System.properties.'user.dir'))
}

private def executeOnShell(String command, File workingDir) {
  println command
  def process = new ProcessBuilder(addShellPrefix(command))
                                    .directory(workingDir)
                                    .redirectErrorStream(true) 
                                    .start()
  process.inputStream.eachLine {println it}
  process.waitFor();
  return process.exitValue()
}

private def addShellPrefix(String command) {
  commandArray = new String[3]
  commandArray[0] = "sh"
  commandArray[1] = "-c"
  commandArray[2] = command
  return commandArray
}

The method executeOnShell is overloaded, so that you can either call it with a working directory or without. It also returns the exitValue, so you can easily evaluate it. This is of course not complete, as it does not include the ability to set Environment variables and it works only on a Unix-like shell. It is sufficient for my current use cases.
Please feel free to use it and to add your own requirements. If you know of any other pitfalls when executing shell commands from Groovy please leave a comment.

Scripting magic with Groovy, Grape and Jetty

Posted in groovy, java, tools on December 31st, 2009 by Joerg – 4 Comments

Groovy 1.7 has just been release and it’s time to play again. One of the new features, Grape inspired me to the try the following. I created a script that acts as a simple webserver using embedded jetty without the need to install anything else than Groovy 1.7 on your computer. Just save the script as simpleWebServer, make it executable and start it like

./simpleWebServer -d someDirWithHTMLFiles -p 9000

Surf to http://localhost:9000 and you will see.

Here is the full script. I will explain the details further down.

#!/usr/bin/env groovy

@Grab('org.mortbay.jetty:jetty:6.1.22')
import org.mortbay.jetty.*;
import org.mortbay.jetty.handler.*

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

def server = new Server(opts.port.toInteger()?:8080)
def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                         resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

server.start()
server.join()

Here are the magic details about the script:

1. The Shebang

The first important thing about this script is the shebang.

#!/usr/bin/env groovy

If you are using a real operating system (sorry Windows users), this will let you start the script like a normal executable. I usually skip the .groovy appendix on script files, so it really feels like you are using a normal command line utility.

2. Grape

The next line uses the new feature of Groovy 1.7, called Grape. This is a dependency System that allows to load any dependency, that is available in the maven-repositories to be used inside a script.

@Grab('org.mortbay.jetty:jetty:6.1.22')

The syntax is pretty easy. It uses the maven notation: groupId:artifactId:version. There is an alternative version that uses separate attributes for each but I prefer the shortcut.
This line will lookup the dependency in the maven repository, download it and store it in ~/.groovy/grape. So the first start of the script might take a moment. The second time will be faster.
Grape allows to use the full java ecosystem in a simple groovy script without any additional install. Groovy 1.7 is enough. I think this makes scripting in groovy incredible powerful.

3. The CliBuilder
The script should really feel like a command line utility. Therefore it needs to deal with parameters. This is where the CliBuilder comes in.

def cli = new CliBuilder (usage:'simpleHtmlServer -p PORT -d DIRECTORY')
cli.with {
 h longOpt:'help', 'Usage information'
 p longOpt:'port',argName:'port', args:1, type:Number.class,'Default is 8080'
 d longOpt:'dir', argName:'directory', args:1, 'Default is .'
}

def opts = cli.parse(args)
if(!opts) return
if(opts.help) {
  cli.usage()
  return
}

Groovy includes Apache commons CLI. But where the Java version is already helpful in parsing parameters, the Groovy version gets really simple. The example above creates one CliBuilder, that provides a “DSL” for defining the parameters. Further down they are three parameters defined. This is all thats needed.
After the args are parsed we can just simply use them using Groovy properties syntax like opts.parameterName. Even a pretty usage statement can be printed, when –help is called.

4. Dynamic constructors
I used one of the embedded Jetty examples (FileServer.java) as the foundation of my script. The configuration of the handlers looked something like that in Java:

ResourceHandler resource_handler=new ResourceHandler();
resource_handler.setWelcomeFiles(new String[]{"index.html"});
resource_handler.setResourceBase(args.length==2?args[1]:".");

HandlerList handlers = new HandlerList();
handlers.setHandlers(new Handler[]{resource_handler,new DefaultHandler()});

Thats pretty long. Now the Groovy version looks like that:

def resourceHandler = new ResourceHandler(welcomeFiles:["index.html"],
                                          resourceBase:opts.dir?:".")
server.handler = new HandlerList(handlers:[resourceHandler, new DefaultHandler()])

It uses dynamic constructors. Groovy allows to call a virtual constructor with a map of arguments, where the elements are actually bean properties. This allows a much more concise construction of an object, even if the original creator wasn’t so kind to provide a convenience constructor.

5. The Elvis Operator

You probably wondered what the following statement did in the last example:

opts.dir?:"."

Well, that was Elvis. In Java there is the ternary operator, which goes like this:

(condition)?(result if true):(result if false) 

In Grovvy there is something called the Groovy truth, which says if a statement is null, then it is false, else it is true. This makes the Elvis operator possible. Elvis says, if the statement is true (as in Groovy truth) then use the value of the statement, else use the value after the colon. This is a very concise way for realizing default values, which is often needed in scripts

With all the scripting features that were added in earlier versions and with Grape in 1.7 I think Groovy is finally a great alternative to usual scripting languages. When you are experienced in Java it’s probably much simpler to write a script in Groovy than in Perl, Ruby or even Bash. For me this is definitely true.

Well done, Groovy team!

IntelliJ IDEAs new directory-based project format

Posted in java, software development, tools on October 17th, 2009 by Joerg – 9 Comments

I am currently playing with the EAP-Version (Early Access Program) of my favorite IDE IntelliJ IDEA. Since the new version is not that far away it is time to learn the new features and I discovered one that surprised me. A small but pretty nice change.

(Update: Thanks to Strug I have realized that this feature is already present in Idea 8, you can use the action “Open in New (Directory based) format” to convert your project. Fortunately they renamed it to “Save as Directory-Based Format …” in Maia. )

IDEA has a new format to store the project files. It is called directory-based. Instead of using the three famous files .ipr .iml and .iws IDEA will now store all information into a directory, which is simply called .idea. It is located in the root folder of the project. This seems to be somewhat similar to eclipse’s .project folder. In a small example project I created the content of the new directory looks like this:

-rw-r--r--  1 joerg  joerg    163 17 Okt 09:53 ant.xml
-rw-r--r--  1 joerg  joerg   2107 17 Okt 09:53 compiler.xml
drwxr-xr-x  3 joerg  joerg    102 17 Okt 09:53 copyright
drwxr-xr-x  3 joerg  joerg    102 17 Okt 09:53 dictionaries
-rw-r--r--  1 joerg  joerg    277 17 Okt 09:53 encodings.xml
-rw-r--r--  1 joerg  joerg    170 17 Okt 09:53 fileColors.xml
-rw-r--r--  1 joerg  joerg   1595 17 Okt 09:53 misc.xml
-rw-r--r--  1 joerg  joerg    258 17 Okt 09:53 modules.xml
-rw-r--r--  1 joerg  joerg    207 17 Okt 09:53 templateLanguages.xml
-rw-r--r--  1 joerg  joerg    169 17 Okt 09:53 vcs.xml
-rw-r--r--  1 joerg  joerg  38245 17 Okt 10:21 workspace.xml

The files included in this directory depend on the settings you change in the IDE. So, if you would for instance set the SQL dialect for your project there will be another file called sqldialects.xml containing all settings about this.

There is one special file in there, which is workspace.xml. This file contains individual settings for the workspace, which are definitely not intended to be shared via version control. This is equivalent to the .iws file of the old format. IDEA will put this file automatically on the ignore list of your version control.

I see some advantages of the new format.

  • First of all it is easier to find specific settings. The filenames are meaningful and the files are small.
  • The new structure allows a very detailed control of which settings you want to share with your colleagues. If you don’t want to share e.g. your file coloring (another new feature of Maia) just put the file on the ignore list.
  • Directories of the projects saved in the new format will be recognized as projects in the open-project dialog. In the past this was one annoying additional click as you had to choose the .ipr file before.

The new directory based format is a small change but a very good one. It is often a sum of little detail-improvements that save a lot of trouble in daily work. So I am looking forward to the other details to be discovered.

By the way JetBrains released an open source version of IDEA two days ago. So if you want to try this just go to www.jetbrains.org and download the Community Edition.

Scala Pattern Matching explained

Posted in java, scala, software development on April 26th, 2009 by Joerg – 1 Comment

I am currently reading Programming in Scala. For a programmer coming from an imperative world there many new concepts to learn. One of them is Pattern Matching. There are already lots of articles about it, but explaining something is still one of the best ways to learn. Here are the first 4 levels of understanding Pattern Matching.

Level 1 – Pattern Matching as simple switch/case replacement

Pattern Matching can simply be used to replace Java switch/case statements. This is called constant pattern and looks like this:

  def constantMatch(value:Any) = {
    value match {
      case "Hello" => "You said hello!"
      case 1 => "Oh, thats a 1!"
      case _ => "Don't know what you want!"
    }
  }

ConstantMatch is a function that takes a parameter of type Any. Any is the Scala pendant to java.lang.Object. This value is now matched against several constants. The first case is a String the second an Int and the last case is a wildcard for any other value.
There are a few differences to Java. First, there is no “fall through”. If a pattern has been matched its value will be returned. Thats the part after the => operator. This can be a full code block. The example above just shows the short version. The second difference is if no pattern matches Scala will throw a MatchError Exception. This is why the wildcard pattern is needed as the last line. This is more or less the same as the default clause of Java.

Level 2 – Introducing Variable patterns

Now level 1 was already more powerful than Java switch/case. But there are some more levels to go. The next step is introducing variable pattern. Here is an example:

 def variableMatch(value:Any) = {
    value match {
      case 1 =>"Oh, thats a 1!"
      case x =>"And thats a"+x+"!";
    }
  }

Instead of validating every possibly value the value itself is put into a variable. This variable can then be processed in the result block. You might recognize that there is no wildcard case(case _ => …). This is not necessary as the variable pattern will match any possible value. A wildcard would not even be possible. The compiler will complain that it is unreachable code.

Level 3 – Type Patterns

Type patterns allow to calculate the result based on the type of the value. Instead of using instanceof in Java or isInstanceOf in Scala this can just be done in a case statement. Let’s assume we have a class hierarchy used to calculate the financial value of different types of property.

abstract class Property {
  val value: BigDecimal
}

case class House(address: Address, override val value:BigDecimal) extends Property
case class Car(horsepower: Int, override val value:BigDecimal) extends Property
case class Horse(Age:Int, override val value:BigDecimal) extends Property
case class Address(street: String, city: String)

Ignore the case keyword for the moment. This is level 4. For all Java people, this code can be put in one file, although it is not recommended practice. I just do it here for simplicity. Property is the abstract base class. It defines a field value as all property needs to have a value. House, Car and Horse inherit from Property and add additional information. Address will be used in a later example.

For a special calculation the value will be applied based on the type of property. Houses will be applied with 80% of their value, Horses with 50% and the value of Cars will not be applied at all. Using Pattern Matching this looks like this:

  def applyPropertyValue(property:Property):BigDecimal = property match {
    case house:House => house.value * 0.8
    case horse:Horse => horse.value * 0.5
    case car:Car => 0
    case _ => property.value
  }

Without any instanceof or casting the value has been calculated based on the type of property. Here is a wildcard pattern used in the last line again. If there is another type of property the full value will be applied.

Level 4 – Constructor patterns and Case Classes
The last example already used case classes even if it would have worked with normal classes too. Case classes have some special characteristics.

  • There is a special factory method that allows the class to be instantiated without the new keyword. Car(150,20000) creates a Car object with 150 horsepower and a value of 20.000.
  • All parameters in the constructor are automatically defined as val. So they are fields and can be accessed from other objects.
  • The compiler generates meaningful implementations of equals, hashcode and toString.
  • And of course they can be used in constructor patterns.

In our example we need some very special cases to calculate the value. If a house is in Berlin 90% instead of 80% of the value can be applied. Cars with a horsepower of more than 150 can be applied with 50% of their value instead of 0. Our function now gets two more lines:

  def applyPropertyValue(property:Property):BigDecimal = property match {
    case House(Address(_,"Berlin"),value) => value * 0.9 //Deep match
    case Car(horsepower, value) if horsepower > 150 => value * 0.5 //Pattern Guard
    case house:House => house.value * 0.8
    case horse:Horse => horse.value * 0.5
    case car:Car => 0
    case _ => property.value
  }

The more special a case is the earlier it has to be validated. If the general case for Car would be matched first, the more specialized pattern would never be validated. In simple cases the Scala compiler complains about the code being unreachable.
The two new lines use the constructor pattern, that is only possible for case classes. But both also demonstrate other features of Pattern Matching. The House pattern shows a nested pattern. The important condition is in class Address, which is nested inside House. This kind of deep matches can save a lot of code. Imagine how many if statements, casts and instanceofs this needs in Java. The Car example shows another syntax for pattern matching. It is called Pattern Guards. An additional if after the pattern allows to validate conditions, before the pattern code will be excecuted.

Level 5 and above
This is still not all that can be learned about Pattern Matching. The story goes on with sequence patterns, sealed classes, partial functions and much more. But by now you should be able to understand most Pattern Matching statements that can be found in Scala. For more details I can only recommend reading Programming in Scala. It contains a lot of fun stuff like that.