10 things your static language can’t do

15 December 2014

Tags: groovy languages static dynamic java javascript scala C++

But maybe mine can…

For those of you who do not know me, and you are likely much more in that category than in the other, I’ve been working on the Groovy language full-time for more than 3 years now. I started as a user several years ago, in the context of DSLs, and eventually became a contributor before getting employed to work on the language. I love static typing, but not at the point of thinking that we should only reason about types. This is why I also love dynamic typing, and why I invested so much in Groovy.

Groovy is primarily a dynamic language. It is probably the most widely used alternative language on the JVM, with use cases ranging from DSLs (scripting Jenkins builds for example) to full blown applications on mobile devices (Android), going through full stack web applications with Grails. Despite being primarily a dynamic language, I spent a lot of time writing a static compiler for Groovy, making it a pretty unique language in the JVM world, but not limited to the JVM world:

Groovy is a language which supports dynamic typing and compile-time type checking. It makes it surprisingly powerful and versatile language, capable of adapting to a great variety of contexts.

When I tell people that I wrote the static compiler for Groovy, I often get a reaction which is "so you admit that dynamic languages are less powerful than static ones", and they see me as the one that made the language right. Oh no, I did not. In fact, I love the dynamic aspects of the language. It is always annoying that I, as a designer of a static compiler, have to defend dynamic languages, but it’s an interesting topic, especially those days where I read lots of articles doing dynamic-bashing.

So in this post, I’m going to illustrate 10 things that a static language (most likely) cannot do. It doesn’t mean that there are only 10 things that a static language cannot do compared to a dynamic one, but it is here to illustrate the fact that this idea that static languages are superior or more scalable just because they are type safe is IMHO stupid. Compare languages between them, but do not compare categories of languages. While static languages will be excellent in making type safety guarantees (errors at compile time), dynamic languages are often far superior in cutting down verbosity. That’s just an example which illustrates that comparing on the sole aspect of type safety is not enough.

In this post I will also illustrate how Groovy is special, because it is capable of handling things in a mixed mode, making it totally unique and incredibly powerful, bringing the best of the two worlds together. Last disclaimer, this post is mostly centered on the JVM world, because this is the one I know the best.

As expected I got lots of comments on various social media. Some are positive, some not, that’s ok, but again, I would like to remember that this is not static languages bashing, nor dynamic languages promotion. Maybe some of you will think "hey, but my static language can do it!" and yes, it is possible, because as the subtitle of this post says, mine can too. But when it does, often, there’s a drawback (like loosing type safety, decreased performance,… ) or you fall on dynamic behavior without even noticing it. When it is the case, I tried to be honest and tell about the possibilities. But I also voluntarily hide some static features of Groovy that make it a very interesting solution (flow typing for example). Last but not least, I am not saying that all dynamic features implement all those items. I am saying that by nature a dynamic language make it possible. An possible doesn’t mean required. So please read carefully before screaming!

Ready?

10. Object-oriented programming done right

This is a very academic point of view. Users of static languages tend to think that their language is object-oriented. Because C++ has a compiler, because Java has a compiler, means that statically typed languages have to be compiled. Object oriented programming does not require languages to be compiled. OOP is about message passing. Object-oriented programming does not imply type safety. It means that an object is something that receives messages. In the Java world, the message is a method with arguments, and the contract is enforced at compile time, but it is an implementation detail which reduces OO programming to a small subset of its capabilities.

The great Alan Kay himself explains it.

Groovy, as a dynamic language, supports commons OOP concepts (and also functional concepts) like class, interface, abstract classes or traits but also has a meta-object protocol. For those of you who did Smalltalk programming, it’s the same idea: the behavior of objects is not determined at compile time, it’s a runtime behavior determined by a meta-object protocol. In Groovy, it translates to the fact that to each class corresponds a meta-class which determines the behavior that an object will have when it receives a message (a method call).

This capability of doing things at runtime instead of compile time is the core of many features of dynamic languages and most of the points illustrated in this blog post derive from it.

I had some comments that the fact that dynamic languages can do OO right wasn’t really interesting. In fact, I insisted on keeping this because this is actually what makes most of the following items possible. So think of 10. as the basement for most of the following items.

9. Multimethods

Java supports overloaded methods. The question whether it is a good or a bad idea is beyond the scope of this post (and believe me it is a very interesting question both in terms of semantics and performance). The idea is that an object can have two methods of the same name accepting different parameters. Let’s illustrate this with Java code:

public static int foo(String o) { return 1; }
public static int foo(Date o) { return 2; }
public static int foo(Object o) { return 3; }

Then you call it like this:

public static void main(String[] args) {
    Object[] array = new Object[] { "a string", new Date(), 666 };
    for (Object o : array) {
        System.out.println(foo(o));
    }
}

What do you think it prints? Well, most beginners will probably answer something that looks natural when you know the contents of the array:

1
2
3

But the correct answer is:

3
3
3

Because the static type of o when the call to foo is made is Object. To say it more clearly, the declared type of o is Object so we are calling foo(Object). The reason for this is that the code is statically compiled so the compiler has to know at compile time which method is going to be called. A dynamic language like Groovy chooses the method at runtime (unless, of course, you use @CompileStatic to enforce static semantics), so the method which is going to be called corresponds to the best fitting arguments. So Groovy, unlike Java, will print the less surprising result:

1
2
3

It is theorically possible for a static language to do the same. But it comes at the price of performance. It would mean that the arguments have to be checked at runtime, and since static languages do not, as far as I know, implement an inlining cache, performance would be lower than those of a well designed dynamic language…

But to add something to a dynamic language, what if you remove the Object version of foo, and remove 666 from the array? As an exercise to the reader, would this Java code compile?

public static int foo(String o) { return 1; }
public static int foo(Date o) { return 2; }

public static void main(String[] args) {
    Object[] array = new Object[] { "a string", new Date() };
    for (Object o : array) {
        System.out.println(foo(o));
    }
}

If not, what do you have to do to make it pass? Yes, dynamic languages are superior here…

8. Duck typing

Duck typing has always been a selling point of dynamic languages. Basically imagine two classes:

class Duck {
   String getName() { 'Duck' }
}
class Cat {
   String getName() { 'Cat' }
}

Those two classes define the same getName method, but it is not defined explicitly in a contract (for example through an interface). There are many reasons why this can happen. For example, you didn’t write those classes, they are in a third party library and for some reason those methods were not intended to be part of the contract. Imagine that you have a list of objects containing either ducks, cats, or anything else definining a getName method. Then a dynamic language will let you call that method:

def list = [cat, dog, human, hal]
list.each { obj ->
   println obj.getName()
}

A static language like Java would force you to have a cast here. But since you don’t have an interface defining getName and implemented by all objects, you cannot cast to that type so you have to consider all types and delegate appropriately like in the following code:

if (obj instanceof Cat) {
   return ((Cat)obj).getName();
}
if (obj instanceof Duck) {
   return ((Duck)obj).getName();
}
if (obj instanceof Human) {
   return ((Human)obj).getName();
}
if (obj instanceof Computer) {
   return ((Computer)obj).getName();
}

The real solution in Java is to define either a common super class or an interface for all those, but again, sometimes you just cannot because you don’t have access to the code! Imagine that the Cat and Dog classes where designed like this for example:

public abstract class Something {} // should define getName, but does not for some obscure reason
public class Cat extends Something {
   public String getName() { return "Cat"; }
}
public class Dog extends Something {
   public String getName() { return "Dog"; }
}

Often the developer didn’t even realize that all objects share a common interface. That’s bad for you, and if you find this code you have no choice but the cascading instanceof solution. There are multiple issues with that code:

  • it is very repetitive, the only thing which changes is the type used in the test and the cast

  • it has to be extensive, that is to say that if your list happens to contain an object having a getName method but not in your list of cases to consider, the code is broken. This means that you have to think about changing that method if you add a new type in your list.

  • in the JVM world, as the number of cases to consider grows, the size of the method will increase to the point where the JIT (just-in-time compiler) decides it’s not worth inlining, potentially dramatically reducing performance.

Of course, one may say "but why the hell didn’t you use an interface". This is of course a good way to solve this in Java, but it is not always possible. Not for example if you don’t have access to the source code (think of the various classes being split in third party libraries). I often faced this problem in the past, and believe me it’s no fun (I look at you, Apache Lucene).

There are actually alternatives for static languages. In Java, you could use a reflective proxy: define an interface, then create a proxy implementing that interface which will delegate to the appropriate getName method. Of course it is overkill: for each object of your list you have a proxy instantiated… Another option, again in Java, is to make the call reflective. But in that case, the call becomes slow and in fact, what you are doing is a dynamic call like a dynamic language would do. A language like Groovy doesn’t have that problem because it implements smart techniques like call site caching and runtime bytecode generation which make it much faster than what a reflective call would do…

An elegant alternative used by other static languages is structural typing. This is for example what the Go language does. In this case, you define an interface, but the object does not have to explicitly implement the interface: the fact that the object defines a method corresponding to the method in the interface is enough to implement it. This is elegant but it changes the semantics of an interface as you define it in Java. Last but not least, this technique cannot be used on a platform like the JVM, because the virtual machine has no way to do it. Well, this is not totally true since now we have the invokedynamic bytecode instruction but guess what? You are relying on a dynamic feature of the VM… Can you hear it?

Some argued that this is very bad design. I must repeat that if you think so, you missed the point. The idea is to workaround poorly designed APIs (or APIs which were "optimized"). When I talked about Lucene it was for a very good reason. I faced the problem. Lucene is a highly optimized piece of code. It makes design decisions which are often based on performance: flattening as much as possible class hierarchies (the HotSpot JIT doesn’t like deep class hierarchies), make classes final, prefer abstract classes over interfaces, … So it is easy to find classes that you want to extend, but you can’t because they are final, or classes that implicitly implement a contract but do not define interfaces. This is a pain to work with, and the ability of a dynamic language to be able to call such methods without having to explicitly declare a contract is a real gain. Some static languages offer similar features through structural typing, but then you have to think about what it means (virtual table lookup?) and how it is implemented depending on the platform (on the JVM, relying on reflection is possible but you loose all type safety and have very bad performance). So everytime I used duck typing, it wasn’t on APIs that I had designed. It was on 3rd party APIs, that for some reason didn’t provide me with a way to call some methods.

7. Respond to non-existing methods

A dynamic language answers to messages (method calls) at runtime. This means that a well designed dynamic language should be able to let you answer any kind of method call, including… non existing methods! This feature is at the core of powerful facilitating frameworks like Grails. In Grails, you can define a domain class like this:

class Person {
   String firstName
   String lastName
   int age
}

The Person class does not define any method, nor does it have any explicit relation to a datastore, an ORM or SQL server. However, you can write code like this:

def adults = Person.findByAge { it>= 18 }

I will not dig into the details about how this is done, but the idea is to intercept the fact that the findByAge method does not exist, then parse the name of the method and build a query based on the method name and the rest of the arguments (here, a closure, an open block of code). Queries can be as complex as you wish, like findByLastNameAndAge or whatever you can think of. Of course Grails does some smart things here, like generating a new method at runtime, so that the next time this method is hit, it is not an unknown method anymore, and can be invoked faster! Only a dynamic language would let you do that. Say bye to infamous DAOs that you have to change everytime you have a new query, it is not necessary. One could say that they prefer safety at compile time rather than the ability to do this, but Grails also offers that possibility of checking that the syntax is correct at compile time, while still leveraging the dynamic runtime to make this work… It’s all about boilerplate removed, code verbosity and productivity…

The ability to react to arbitrary messages is actually at the core of many DSLs (domain specific languages) written in Groovy. They are at the core of builders for example, which will let you write code like:

catalog {
   book {
   	isbn 123
	name 'Awesome dynamic languages'
        price 11.5
        tags {
	   dynamic,
	   groovy,
	   awesome
	}
   }
}

Instead of the less readable Java 8 version (for the reader’s mental sanity, I will not write the Java 7 version):

builder.catalog( (el) -> {
  el.book ( (p) -> {
     p.setISBN("123");
     p.setName("Awesome dynamic languages");
     p.setPrice(11.5);
     p.setTags("dynamic","groovy","awesome");
  })
});

6. Mocking and monkey-patching

Mocking is at the core of many unit testing strategies. Most of static languages make use of an external library to do this. Why this can be true of dynamic languages too, this is often not strictly necessary. For example Groovy offers built-in stubbing/mocking capabilities, very easily thanks to its dynamic nature. Monkey patching rely on the very same behavior but is easier to explain so I will illustrate this concept here. Imagine that you use a closed-source library (I won’t judge you, I promise) or an open-source library for which you don’t want to/don’t have time to contribute to, but you have found a serious security issue in a method:

public class VulnerableService {
   public void vulnerableMethod() {
      FileUtils.recurseDeleteWithRootPrivileges("/");
   }
}

You know how to fix it, but you have to wait for the maintainer to upgrade the library. Unfortunately, you can’t wait because attackers are already leveraging the vulnerability on your production server (yeah, they like to). One option that a dynamic language can let you do is redefine the method at runtime. For example, in Groovy, you could write:

VulnerableService.metaClass.vulnerableMethod = {
   println "Well tried, but you have been logged to Santa's naughty guys list!"
}

Then a caller that would call the vulnerableMethod would call the monkey-patched version instead of the original one. Of course in a language like Groovy, this would only be true if the callee is dynamically compiled: if you use @CompileStatic to behave like a static compiler, you’re out of luck, because the method which will be invoked is selected at compile time, so you will be vulnerable even if you try to monkey patch… Groovy provides other extension mechanisms to work around this, but it’s not the topic here ;-)

5. Dynamic objects

Most dynamic languages let you create… dynamic objects. It is basically an object for which you attach methods and properties at runtime. Not that I am a big fan of it but there are some valid use cases (serialization, languages like Golo not supporting classes, prototype based construction, …). It can also be convenient if you want to rapidly prototype a class.

As an example, let’s see how you could create an arbitrary object to represent a person, without actually leveraging on a class, using the Groovy language:

def p = new Expando()
p.name = 'Cédric'
p.sayHello = { println "Hello $name" }

p.sayHello()

The code is totally dynamic here. It lets you create an arbitrary object, attach new methods to it, data, …, without relying on strong typing. Of course it is interesting when you see that the sayHello method is capable of referencing "pseudo-fields" which are themselves dynamic!

4. Scripting

Static languages can do scripting. But it is definitely not what I would call scripting. Having to write types is not natural in a script. I even worked in the past in a context where people who wrote scripts where not programmers. They didn’t even know what a type is, and they don’t care. The most popular scripting technologies like Bash do not have types, and it’s not a problem, so imagine the following. You arrive late at your office, your boss is very angry about that and shouts to you: "you have 5 minutes, not more, to give me the total number of followers of users who have submitted an accepted pull request on the Groovy repo recently". It’s a weird query, most probably your boss is going into social networking madness but you have no choice otherwise you’re fired.

In that case, most developers would think of:

  • using a Bash script combining curl, grep, regular expressions and hoping that man works

  • using a tool they know like Java, but since they have so little time, they will probably rely on a regular expression to parse the JSON feed until they realize they have to do a second HTTP query for each user

  • quiting their job

In Groovy, you would do:

import groovy.json.JsonSlurper

def json = new JsonSlurper().parse('https://api.github.com/repos/groovy/groovy-core/issues?state=closed'.toURL())
json.user.collectEntries { u ->
   // second query to fetch the nb of followers
   def followers = new JsonSlurper().parse(u.followers_url.toURL())
   [u.login,followers.size()]
}.values().sum()

What you can see here is that we use a facility, JSonSlurper which actually parses the JSON result. It is much more reliable that what you would have done with a quick hack like a regex, but not only:

  • all data is accessible in a path-like fashion (json.user.address.city.postalCode)

  • you don’t need a single type here

Even if you use a smart JSON parser with your static language, you would still have to write a collection of classes to unmarshall the JSON structure into beans/classes. For such a simple use case, you really don’t care. You just want things done, easily, quickly. You don’t need type safety. You don’t need it to be super clean and tolerant to future changes of the JSON format. Get. Things. Done. (and boss happy).

3. Runtime coercions

Another thing that dynamic languages are particularily good at is runtime coercions. In general static languages users know about one type of conversion, which is casting. Some are lucky enough to know about coercion (like the use of implicit in Scala), the others rely on the adapter pattern. In a dynamic language, runtime coercions are often easy to implement. A coercion differs from a cast in the sense that you want to convert an object of class A to an object of class B, but a B cannot be assigned to an A.

Groovy provides "natural" conversions for some widely used types: lists to objects, and maps to object, like in the example here:

Point p = [1,2] // coercion of a list literal into an object of class Point thanks to constructor injection
Point p = [x:1, y:2] // coercion of a map literal into an object of class Point thanks to setter injection

But if it happened to be that you cannot use maps or lists but really want to convert one type to another, you can just declare a converter:

class A {
   Object asType(Class<?> clazz) { new B(...) }
}

I can see you raising an eyebrow here, because I wrote the conversion code directly in class A, but remember it’s a dynamic language with a meta-object protocol, so nothing prevents you from writing this conversion code outside of the class A itself, through its metaclass, which would let you add conversion algorithms for classes which are beyond your control. It’s a win!

2. Dynamic binding

Dynamic binding is linked to DSL evaluation and scripting. Imagine the following script:

a+b

In this script, variables a and b are unbound. They are not known from a compiler, so if you tried to statically compile this with a compiler like Java (or C++, or Scala) it would definitely blow up. Not if you compile this with Groovy. Because it’s dynamic, it’s able to know that those variables will be eventually bound, when the script is executed. Groovy provides means to inject those variables when you need them. It is some kind of late binding, but it is the core of expression languages, and it is no surprise that products like ElasticSearch uses Groovy as the default scripting language: it allows it to be both compilable and late bound. But there is more, if you think you have an issue with not being able to resolve a and b at compile time and that you fear to write code which might fail at runtime…

1. Mixed mode compilation

The last thing that a dynamic language like Groovy is capable of doing is leveraging mixed mode compilation. Behind this curious term is a unique concept in programming languages: Groovy is able of mixing static code with dynamic code, but more, you can instruct the compiler how to do so. So if you design a DSL like in ElasticSearch where you know that some variables will be bound, that the number, names and types of those variables are fixed and known in advance, then you can instruct the compiler and switch to a statically compilable mode! This means that if the user uses an unknown variable, compilation will fail.

This technique is already used in Groovy itself, in the powerful Markup Template Engine. It is a template engine which is capable of generating markup-like contents with a very nice builder-like syntax, but all templates are statically compiled even if the code seems to be full of unresolved method calls or variables!

For those who are interested in this, I invite them to take an eye at my blog posts describing how you can do this.

Conclusion

In conclusion, I have highlighted some points where dynamic languages can do what static languages cannot. Users of the most widely used dynamic language, Javascript, probably have lots of ideas too. The point for me is not to tell which one is better than the other because I don’t care. In general, I am not much into the war behind those, because I really enjoy both. I do static typing most of time, but I really enjoy the dynamic nature of the language too because often I don’t want to be slowed down just to make a compiler happy. I, as a developer, should be happy. Making a compiler happy is secondary and often not necessary. Last but not least, you might have thought, reading this post, that your static language can do this or that. I won’t blame you here, because mine can too. The idea here is more to show that it is totally unnatural for a static language or it often comes with horrible drawbacks like verbosity, performance issues or simply difficult to implement.

Comments

Fixing Bluetooth debugging on Android Wear

20 October 2014

Tags: groovy android moto 360 wear

I have been working on making Groovy work on Android for several months now. In the last weeks, I even showed at SpringOne2GX an example of an application written in Groovy that worked on Android Wear. However, that code worked in an emulator. Recently, I got a real device, a Moto 360, so I wanted to see that application running on a real device. For some very obscure (understand buggy Android SDK) reason, it was far from being that easy…

I litteraly spent hours trying to figure out what was wrong, so I thought it would be interesting for those of you who face the same problem to have a blog post that explains how to deal with it.

The problem

If you follow the instructions on the Android documentation about how to enable bluetooth debugging, it’s in the end pretty simple. Basically, it’s about enabling USB debugging on your physical handheld, then enable bluetooth debugging on your wearable, and in the end enable bluetooth debugging in the Android Wear companion app.

The guide says:

In the Android Wear companion app, you should see the status change to:

Host: connected Target: connected

However, whatever I did, the Target: connected line never appeared for me. It was always Target: disconnected, so if I continued with the instructions:

adb forward tcp:4444 localabstract:/adb-hub; adb connect localhost:4444

Doing

adb devices

Only showed my wearable as offline:

List of devices attached

dcfbbafd	device
localhost:4444	offline
Why so evil?

I have searched for an answer in a lot of pages, including Stackoverflow where I tried very unlikely answers like in this page: execute both commands separately instead of doing them in the same line. I thought that maybe there was a timing issue and that the fact of separating both commands would give the toolkit a chance, but no, wasn’t that easy.

In the end, I was totally convinced that the problem was because I had previously associated my handheld with an emulator. I was convinced of it because even if I had now associated it with a real device (the Moto 360), in the Android Wear companion app, the device was recognized as an "emulator"… mmm… So I tried uninstalling the Android Wear app, clear its cache, but no matter what I did, after reinstalling, the settings were kept, and the Moto recognized as an emulator. So sad…

The solution

So you were looking at a solution, and here it is. Basically, the problem is that the Android companion app doesn’t store its settings under its cache. They are stored in the Google Play Services space, so here is the procedure that worked for me, and I sincerely hope it will do for you. On your handheld:

  1. open the applications settings, search for the Android Wear application, then force stop it

  2. clear its data and cache

  3. now search for Google Play Services (depending on your language settings, it can appear with a different name, on m y device it is "Services Google Play".

  4. click on Manage space. You will see that there’s a section for connected devices. I tried to clear data here, but it didn’t help, so you have to click on delete all data.

  5. reboot your phone

  6. reboot your Moto 360

I am unsure that the two last steps are really necessary, but I did it because I wanted to make sure that force stopping and clearing data did not introduce some weird behavior after that. When you reopen the Android Wear companion app, it should now be as if it was the first time you opened it and ask you to associate it with your watch. Do it, and now, you should be able to follow the normal procedure described in the Android documentation and…

yes baby

Success, now you can debug your application on a real device (which includes deploying it…).

Comments

Tip of the day: reversed git bisect

23 July 2014

Tags: groovy git bisect

I had an interesting use case for git bisect today and as my blog also consistutes a good archive for things I don’t want to loose, let’s take advantage of this to share the trick with you!

Normally, git bisect is used to find what commit introduced a regression in the codebase. For example, if you know that current HEAD is buggy but that at least, RELEASE_1_0 was good, then you can write:

git bisect start             (1)
git bisect bad               (2)
git bisect good RELEASE_1_0  (3)
1 start bisecting
2 tells that HEAD contains the regression
3 tells that RELEASE_1_0 is a tag corresponding to a version known not to have the bug

Git will checkout a revision that you can test, and you issue a list of git bisect good or git bisect bad commands until it determines what commit introduced the regression.

This is a very practical way to find a regression. In Groovy, I’ve used this more than once, it’s very useful.

Reversing the logic

But today, I wanted to reverse the logic. Actually, we had a bug report and we found out that the bug was already fixed, but we didn’t know in which version it was fixed. So actually, I didn’t want to find a regression, but a fix commit.

The idea to do this is to reverse the meaning of bad and good in bisect:

  • bad becomes "doesn’t produce a compile error"

  • good becomes "produces a compile error"

And since the range of revisions to test was pretty big (we know that the error was reported on Groovy 2.2.1, but master is 2.4.0), then I also took advantage of the git bisect run command, which automatically continues bisecting based on a command line return status code.

So basically, here’s what I wrote:

git bisect start                (1)
git bisect bad master           (2)
git bisect good GROOVY_2_2_1    (3)
git bisect run ./bisect.sh      (4)
1 start bisecting
2 master is known to have the fix, so we say bad is master
3 GROOVY_2_2_1 is known to have the bug, so we say good is GROOVY_2_2_1
4 start automatic bisecting thanks to the ./bisect.sh script

And what does bisect.sh consist of? Here you go:

bisect.sh
#!/bin/bash
export GROOVY_HOME=/tmp/testversion                                                                     (1)
./gradlew clean -x javadoc -x groovydoc -x javadocAll -x groovydocAll -PskipIndy=true installGroovy     (2)
/tmp/testversion/bin/groovy bisect.groovy || exit 0                                                     (3)
exit 1                                                                                                  (4)
1 tells where the local build version of Groovy will be installed
2 builds Groovy and installs it locally
3 executes the test script and if the test fails, return a success exit code
4 return a failure exit code

The trick is to reverse the exit codes in the script too: if the script compiles, then it means that the bug was fixed. Since we reverse the logic, we then need the script to return a bad exit code! In case the script fails, we will return a success (0) error code, because it means that the revision doesn’t have the fix. Easy, but needs some mental contorsion :-)

You will note that this script uses GROOVY_HOME and a local installation path. You can configure it using the $HOME/.gradle/gradle.properties file, and adding the following line in it:

gradle.properties
groovy_installPath=/tmp/testversion

Eventually, here is the groovy script which served as a test case (almost copied directly from the JIRA issue):

bisect.groovy
abstract  class Base<A> {
    abstract  void foo(A[] a)
}

class X {}

class Inheritor extends Base<X>{
    @Override
    void foo(X[] a) {}
//Groovyc: Can't have an abstract method in a non-abstract class.
//The class 'B' must be declared abstract
//or the method 'void foo([Ljava.lang.Object;)' must be implemented.
}

Inheritor

Note that each revision took around ~1 min 30s to test, even skipping the javadoc/groovydoc and indy versions of Groovy, so you can imagine what benefit you have here in using automatic bisecting.

In the end, after around 20 minutes of automatic processing, I received this nice message:

74d991f9f8c39d2730a054431bf28e6516e61735 is the first bad commit
commit 74d991f9f8c39d2730a054431bf28e6516e61735
Author: Cedric Champeau <cedric.champeau@gmail.com>
Date:   Sun Apr 27 18:06:24 2014 +0200

    GROOVY-6722: Compiler doesn't handle generic array covariant

:040000 040000 b61b92399ac86246c157f948b3232bf5ab0cf04f 5d0c56413a621c7693e7985aff0e4c84eb08889f M	src
bisect run success

What? "bad commit"? Yes, remember that the logic is reversed, so "bad" means actually "fixed it". So it says that the first commit which fixed the bug was actually 74d991f. And here we go, issue closed ;) One improvement I can see is to use a local clone of my repository instead of working directly in the same repository, so that I can continue working on my copy while bisecting is in progress.

Comments

The new Groovy website

18 July 2014

Tags: groovy templating gradle markup template engine

Last week, we revealed the beta of a brand new website for the Groovy language. This new website is open sourced and already received a few contributions. In order to make it even easier and as it a fully statically generated site that makes use of Groovy I wanted to give more technical details on the toolchain and how it is generated.

A static website

One of the first questions which arised was: why not use Grails/Spring Boot/Ratpack? In fact, the new Groovy website is fully statically generated. It offers multiple advantages:

  • hosting is much easier, as it only consists of static pages and assets

  • maintenance is simplified, no database to backup for example

  • everything is self contained, pages and data, into a single repository

  • no need for authentication

  • all content is public

Last but not least, we didn’t have any requirement for storing anything in a database, or that would require dynamic generation. Pull requests are enough so far. Eventually, we’re thinking about a blog, but even that can be statically generated even if you want to allow users to comment on articles (this blog is a perfect example). So in short, this decision was motivated by one mantra: the right tool for the right job.

The documentation, that you can find on this page, had already started migrating from the aging wiki to . It is generated independently of the website and integrated into it using iframes (we’re thinking about source integration though).

Structure of the project

Generator and site

The project is built using Gradle 2 and consists of 2 subprojects:

  • the generator project contains, as the name says, the static generator. It makes use of a template engine and provides the classes used in the model of the templates.

  • the site project contains the templates and data. If you’re looking into contributing contents, this is likely the place to look at.

Building and testing the site is easy:

git clone https://github.com/groovy/groovy-website.git          (1)
cd groovy-website
./gradlew generate                                              (2)
1 clones the repository
2 generates the website from templates

The output will be visible in <project directory>/site/build/site. There’s also a checkDeadlinks tasks that we will use once we get out of the beta phase to ensure that the generated pages do not contain any dead link.

Internally, we use our CI server to deploy changes to the master branch live. So any commit which is pushed onto the master branch is automatically published (in general, takes less than 2 minutes).

Adding contents

Even if the site is statically generated, we still have data. In this project, there’s an important file, named sitemap.groovy which handles a lot of the contents of the website. It is our "low cost" database and as you can see, it’s a DSL describing the contents of the website.

For example, you can see the menu section which looks like this:

menu {
    group('Groovy') {
        item 'Learn',                       'learn.html'
        item 'Documentation',               'documentation.html'
        item 'Download',                    'download.html'
        item 'Community',                   'community.html'
        item 'Ecosystem',                   'ecosystem.html'
    }

    group('About') {
        item 'Contributing',                'contribute.html'
        item 'Source code',                 'https://github.com/groovy/groovy-core'
        item 'Books',                       'learn.html#books'
        item 'Sponsors',                    'sponsors.html'
        item 'FAQ',                         'faq.html'
        item 'Search',                      'search.html'
    }

    // ...
}

It is a purely declarative description of the site menus. Actually, the "Groups" are used in the footer of the page, while the main Groovy group is used to generate the top navigation bar. Using a simple descriptive DSL is very interesting, because it decouples templates from the contents of the menu. We make sure that those templates do not contain any element which is hardcoded and reduce the risks of forgetting to update the footer, for example, if a section is added.

The same file is used to describe the list of downloads:

downloads {
    // ...
    distribution('Groovy 2.3') {
        description {
            yield 'Groovy 2.3 is our latest official '
            a(href: 'versioning.html', 'version')
            yield ' of Groovy.'
        }

        version('2.3.4') {
            stable true
            releaseNotes 'https://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10242&version=20432'
            windowsInstaller 'http://dist.codehaus.org/groovy/distributions/installers/windows/nsis/groovy-2.3.4-installer.exe'
        }
    }
    // ...
}

or the books which are listed on the learn page:

books {
    book('Groovy in Action, 2nd edition') {
        authors "Dierk König, Guillaume Laforge, Paul King, Cédric Champeau, Hamlet D'Arcy, Erik Pragt, and Jon Skeet"
        cover 'img/books/regina.png'
        url 'http://www.manning.com/koenig2/'
        description 'The undisputed definitive reference on the Groovy programming language, authored by core members of the development team.'
    }
    // ...
}

The same is done for all contents that need regular updates: user groups, events, projects of the ecosystem, … I think this DSL provides a very nice way to add contents to the website without caring about where it has to be done. You can really think of it as a small database, but making use of a Groovy DSL.

In addition, this file also declares the mapping between pages in the documentation section and the documentation page. Last but not least, it lists the individual pages that the website contain. Those pages make use of the markup template engine.

Eat your own dog food

In Groovy 2.3, we introduced a new markup template engine. We decided that the new web site was an excellent showcase of this template engine, and a real life use case. This template engine has several remarkable features, like static compilation of templates (even if the model is dynamic), layouts and of course a human readable builder like syntax:

html {
   head {
    title 'Groovy markup template engine in action!'
   }
   body {
    ul {
        features.each { f-> li(f.name) }
    }
   }
}

It has already been integrated into Spring Boot and Ratpack will use it in the next version (to be released on August 1st). A hint about its performance can be found here. If you are interested in details about how it works, you can find the documentation here and you can read my blog posts about it.

The website subproject is therefore organized accordingly. Inside the main source tree, you’ll find the following directories:

  • assets: contains static assets, like Javascript, CSS, images, …

  • html: contains pure HTML files which are easier to embed as is than using a markup syntax

  • includes: contains elements of code which are shared among multiple templates

  • layouts: contains template layouts, as defined in the documentation

  • pages: contains the main pages of the website

In general, consider pages as the entry point. A page generally makes use of one layout. As an example, let’s see how the Ecosystem page is generated. The source file consists of this:

ecosystem.groovy
layout 'layouts/main.groovy', true,                                             (1)
    pageTitle: 'The Groovy programming language - Ecosystem',                   (2)
    mainContent: contents {                                                     (3)
      div(id: 'content', class: 'page-1') {
        section(class: 'row') {
          div(class: 'row-fluid') {
            // ... snip side menu ...
            div(class: 'col-lg-8 col-lg-pull-0') {
              include template: 'includes/contribute-button.groovy'             (4)
              h1 {
                i(class: 'fa fa-leaf') {}
                yield ' Ecosystem'
              }
              p {
                yield '''
                    Beside the language and its API, Groovy gave birth   ...
                    on various themes such as web frameworks, desktop    ...
                    In this section, we will highlight a few of the most ...
                    which leverage Groovy at their core.
              '''
              }
              hr(class: 'divider')

              ecosys.eachWithIndex { e, index ->                                (5)
                def (name, item) = [e.key, e.value]
                article {
                  a(name: "${name}") {}
                  div(class:"content-heading clearfix media") {
                    div {
                      if (item.logo) {
                        img class: "pull-${(index % 2 == 0) ? 'left' : 'right'}",
                          src: item.logo, alt: name, hspace: '20px'
                      } else {
                        h2(name)
                      }
                      p(item.description)
                    }
                    a(href: item.url, target:'_blank', "Learn more...")
                  }
                }
                hr(class: 'divider')
              }
              // ...
            }
          }
        }
      }
    }
1 make use of the main layout
2 the layout requires a pageTitle variable
3 as well as a mainContent section corresponding to the main page contents
4 example of use of an include
5 iterates over the ecosys variable which contains the list of ecosystem projects as found in the sitemap

As you can see, this template format has the advantage of taking care of generating markup for you. You won’t hit your head again on the wall to find an unclosed tag. Everything is embedded, readable and concise.

Lessons learnt

Using the markup template engine for this project was interesting, because it was probably the first "real life" project to use it intensively. And as such, we discovered usability issues, but also bugs. Hopefully, none of those bugs or usability features were critical, and everything could be worked around, but expect some fixes in Groovy 2.3.5. It is also the reason why the project initially used Gradle 2: it comes with Groovy 2.3.2 which embeds the markup template engine, so it was possible to use it without organizing the project into separate modules like we have. In fact, the early versions of the site didn’t use subprojects. It’s only when we wanted to leverage improvements from Groovy 2.3.4 that we had to switch to that architecture.

A team work

In the end, I can’t finish this blog post without mentionning the team work it implied. In particular:

  • Damien Vitrac designed the website and produced HTML sketches. If you think the new site looks good, thank this guy!

  • Guillaume Laforge designed the site architecture, wrote contents, tweaked the CSS, that is to say produced almost all contents. He spent countless hours fixing responsiveness issues and digging into front-end dev.

  • I streamlined the process by setting up the Gradle project, designing the sitemap DSL, the integration of the markup template engine, CI integration, … that is to say pretty much all the "backend" stuff.

  • You, as a community, provided awesome pull requests within hours. Keep them coming, we love it!

Each of us have different skills. Guillaume is far better than I am in any kind of web design, styling issues, etc for example, so in the end, I think the combination works quite good and that the site as it is now is already pretty usable.

Let us know what you think, and don’t forget that you can contribute, it’s easy!

Comments


Older posts are available in the archive.