Tip of the day: reversed git bisect

23 juillet 2014

Tags: groovy git bisect

I had an interesting use case for git bisect today and as my blog also consistutes a good archive for things I don’t want to loose, let’s take advantage of this to share the trick with you!

Normally, git bisect is used to find what commit introduced a regression in the codebase. For example, if you know that current HEAD is buggy but that at least, RELEASE_1_0 was good, then you can write:

git bisect start             (1)
git bisect bad               (2)
git bisect good RELEASE_1_0  (3)
1 start bisecting
2 tells that HEAD contains the regression
3 tells that RELEASE_1_0 is a tag corresponding to a version known not to have the bug

Git will checkout a revision that you can test, and you issue a list of git bisect good or git bisect bad commands until it determines what commit introduced the regression.

This is a very practical way to find a regression. In Groovy, I’ve used this more than once, it’s very useful.

Reversing the logic

But today, I wanted to reverse the logic. Actually, we had a bug report and we found out that the bug was already fixed, but we didn’t know in which version it was fixed. So actually, I didn’t want to find a regression, but a fix commit.

The idea to do this is to reverse the meaning of bad and good in bisect:

  • bad becomes "doesn’t produce a compile error"

  • good becomes "produces a compile error"

And since the range of revisions to test was pretty big (we know that the error was reported on Groovy 2.2.1, but master is 2.4.0), then I also took advantage of the git bisect run command, which automatically continues bisecting based on a command line return status code.

So basically, here’s what I wrote:

git bisect start                (1)
git bisect bad master           (2)
git bisect good GROOVY_2_2_1    (3)
git bisect run ./bisect.sh      (4)
1 start bisecting
2 master is known to have the fix, so we say bad is master
3 GROOVY_2_2_1 is known to have the bug, so we say good is GROOVY_2_2_1
4 start automatic bisecting thanks to the ./bisect.sh script

And what does bisect.sh consist of? Here you go:

bisect.sh
#!/bin/bash
export GROOVY_HOME=/tmp/testversion                                                                     (1)
./gradlew clean -x javadoc -x groovydoc -x javadocAll -x groovydocAll -PskipIndy=true installGroovy     (2)
/tmp/testversion/bin/groovy bisect.groovy || exit 0                                                     (3)
exit 1                                                                                                  (4)
1 tells where the local build version of Groovy will be installed
2 builds Groovy and installs it locally
3 executes the test script and if the test fails, return a success exit code
4 return a failure exit code

The trick is to reverse the exit codes in the script too: if the script compiles, then it means that the bug was fixed. Since we reverse the logic, we then need the script to return a bad exit code! In case the script fails, we will return a success (0) error code, because it means that the revision doesn’t have the fix. Easy, but needs some mental contorsion :-)

You will note that this script uses GROOVY_HOME and a local installation path. You can configure it using the $HOME/.gradle/gradle.properties file, and adding the following line in it:

gradle.properties
groovy_installPath=/tmp/testversion

Eventually, here is the groovy script which served as a test case (almost copied directly from the JIRA issue):

bisect.groovy
abstract  class Base<A> {
    abstract  void foo(A[] a)
}

class X {}

class Inheritor extends Base<X>{
    @Override
    void foo(X[] a) {}
//Groovyc: Can't have an abstract method in a non-abstract class.
//The class 'B' must be declared abstract
//or the method 'void foo([Ljava.lang.Object;)' must be implemented.
}

Inheritor

Note that each revision took around ~1 min 30s to test, even skipping the javadoc/groovydoc and indy versions of Groovy, so you can imagine what benefit you have here in using automatic bisecting.

In the end, after around 20 minutes of automatic processing, I received this nice message:

74d991f9f8c39d2730a054431bf28e6516e61735 is the first bad commit
commit 74d991f9f8c39d2730a054431bf28e6516e61735
Author: Cedric Champeau <cedric.champeau@gmail.com>
Date:   Sun Apr 27 18:06:24 2014 +0200

    GROOVY-6722: Compiler doesn't handle generic array covariant

:040000 040000 b61b92399ac86246c157f948b3232bf5ab0cf04f 5d0c56413a621c7693e7985aff0e4c84eb08889f M	src
bisect run success

What? "bad commit"? Yes, remember that the logic is reversed, so "bad" means actually "fixed it". So it says that the first commit which fixed the bug was actually 74d991f. And here we go, issue closed ;) One improvement I can see is to use a local clone of my repository instead of working directly in the same repository, so that I can continue working on my copy while bisecting is in progress.

Comments

The new Groovy website

18 juillet 2014

Tags: groovy templating gradle markup template engine

Last week, we revealed the beta of a brand new website for the Groovy language. This new website is open sourced and already received a few contributions. In order to make it even easier and as it a fully statically generated site that makes use of Groovy I wanted to give more technical details on the toolchain and how it is generated.

A static website

One of the first questions which arised was: why not use Grails/Spring Boot/Ratpack? In fact, the new Groovy website is fully statically generated. It offers multiple advantages:

  • hosting is much easier, as it only consists of static pages and assets

  • maintenance is simplified, no database to backup for example

  • everything is self contained, pages and data, into a single repository

  • no need for authentication

  • all content is public

Last but not least, we didn’t have any requirement for storing anything in a database, or that would require dynamic generation. Pull requests are enough so far. Eventually, we’re thinking about a blog, but even that can be statically generated even if you want to allow users to comment on articles (this blog is a perfect example). So in short, this decision was motivated by one mantra: the right tool for the right job.

The documentation, that you can find on this page, had already started migrating from the aging wiki to . It is generated independently of the website and integrated into it using iframes (we’re thinking about source integration though).

Structure of the project

Generator and site

The project is built using Gradle 2 and consists of 2 subprojects:

  • the generator project contains, as the name says, the static generator. It makes use of a template engine and provides the classes used in the model of the templates.

  • the site project contains the templates and data. If you’re looking into contributing contents, this is likely the place to look at.

Building and testing the site is easy:

git clone https://github.com/groovy/groovy-website.git          (1)
cd groovy-website
./gradlew generate                                              (2)
1 clones the repository
2 generates the website from templates

The output will be visible in <project directory>/site/build/site. There’s also a checkDeadlinks tasks that we will use once we get out of the beta phase to ensure that the generated pages do not contain any dead link.

Internally, we use our CI server to deploy changes to the master branch live. So any commit which is pushed onto the master branch is automatically published (in general, takes less than 2 minutes).

Adding contents

Even if the site is statically generated, we still have data. In this project, there’s an important file, named sitemap.groovy which handles a lot of the contents of the website. It is our "low cost" database and as you can see, it’s a DSL describing the contents of the website.

For example, you can see the menu section which looks like this:

menu {
    group('Groovy') {
        item 'Learn',                       'learn.html'
        item 'Documentation',               'documentation.html'
        item 'Download',                    'download.html'
        item 'Community',                   'community.html'
        item 'Ecosystem',                   'ecosystem.html'
    }

    group('About') {
        item 'Contributing',                'contribute.html'
        item 'Source code',                 'https://github.com/groovy/groovy-core'
        item 'Books',                       'learn.html#books'
        item 'Sponsors',                    'sponsors.html'
        item 'FAQ',                         'faq.html'
        item 'Search',                      'search.html'
    }

    // ...
}

It is a purely declarative description of the site menus. Actually, the "Groups" are used in the footer of the page, while the main Groovy group is used to generate the top navigation bar. Using a simple descriptive DSL is very interesting, because it decouples templates from the contents of the menu. We make sure that those templates do not contain any element which is hardcoded and reduce the risks of forgetting to update the footer, for example, if a section is added.

The same file is used to describe the list of downloads:

downloads {
    // ...
    distribution('Groovy 2.3') {
        description {
            yield 'Groovy 2.3 is our latest official '
            a(href: 'versioning.html', 'version')
            yield ' of Groovy.'
        }

        version('2.3.4') {
            stable true
            releaseNotes 'https://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10242&version=20432'
            windowsInstaller 'http://dist.codehaus.org/groovy/distributions/installers/windows/nsis/groovy-2.3.4-installer.exe'
        }
    }
    // ...
}

or the books which are listed on the learn page:

books {
    book('Groovy in Action, 2nd edition') {
        authors "Dierk König, Guillaume Laforge, Paul King, Cédric Champeau, Hamlet D'Arcy, Erik Pragt, and Jon Skeet"
        cover 'img/books/regina.png'
        url 'http://www.manning.com/koenig2/'
        description 'The undisputed definitive reference on the Groovy programming language, authored by core members of the development team.'
    }
    // ...
}

The same is done for all contents that need regular updates: user groups, events, projects of the ecosystem, … I think this DSL provides a very nice way to add contents to the website without caring about where it has to be done. You can really think of it as a small database, but making use of a Groovy DSL.

In addition, this file also declares the mapping between pages in the documentation section and the documentation page. Last but not least, it lists the individual pages that the website contain. Those pages make use of the markup template engine.

Eat your own dog food

In Groovy 2.3, we introduced a new markup template engine. We decided that the new web site was an excellent showcase of this template engine, and a real life use case. This template engine has several remarkable features, like static compilation of templates (even if the model is dynamic), layouts and of course a human readable builder like syntax:

html {
   head {
    title 'Groovy markup template engine in action!'
   }
   body {
    ul {
        features.each { f-> li(f.name) }
    }
   }
}

It has already been integrated into Spring Boot and Ratpack will use it in the next version (to be released on August 1st). A hint about its performance can be found here. If you are interested in details about how it works, you can find the documentation here and you can read my blog posts about it.

The website subproject is therefore organized accordingly. Inside the main source tree, you’ll find the following directories:

  • assets: contains static assets, like Javascript, CSS, images, …

  • html: contains pure HTML files which are easier to embed as is than using a markup syntax

  • includes: contains elements of code which are shared among multiple templates

  • layouts: contains template layouts, as defined in the documentation

  • pages: contains the main pages of the website

In general, consider pages as the entry point. A page generally makes use of one layout. As an example, let’s see how the Ecosystem page is generated. The source file consists of this:

ecosystem.groovy
layout 'layouts/main.groovy', true,                                             (1)
    pageTitle: 'The Groovy programming language - Ecosystem',                   (2)
    mainContent: contents {                                                     (3)
      div(id: 'content', class: 'page-1') {
        section(class: 'row') {
          div(class: 'row-fluid') {
            // ... snip side menu ...
            div(class: 'col-lg-8 col-lg-pull-0') {
              include template: 'includes/contribute-button.groovy'             (4)
              h1 {
                i(class: 'fa fa-leaf') {}
                yield ' Ecosystem'
              }
              p {
                yield '''
                    Beside the language and its API, Groovy gave birth   ...
                    on various themes such as web frameworks, desktop    ...
                    In this section, we will highlight a few of the most ...
                    which leverage Groovy at their core.
              '''
              }
              hr(class: 'divider')

              ecosys.eachWithIndex { e, index ->                                (5)
                def (name, item) = [e.key, e.value]
                article {
                  a(name: "${name}") {}
                  div(class:"content-heading clearfix media") {
                    div {
                      if (item.logo) {
                        img class: "pull-${(index % 2 == 0) ? 'left' : 'right'}",
                          src: item.logo, alt: name, hspace: '20px'
                      } else {
                        h2(name)
                      }
                      p(item.description)
                    }
                    a(href: item.url, target:'_blank', "Learn more...")
                  }
                }
                hr(class: 'divider')
              }
              // ...
            }
          }
        }
      }
    }
1 make use of the main layout
2 the layout requires a pageTitle variable
3 as well as a mainContent section corresponding to the main page contents
4 example of use of an include
5 iterates over the ecosys variable which contains the list of ecosystem projects as found in the sitemap

As you can see, this template format has the advantage of taking care of generating markup for you. You won’t hit your head again on the wall to find an unclosed tag. Everything is embedded, readable and concise.

Lessons learnt

Using the markup template engine for this project was interesting, because it was probably the first "real life" project to use it intensively. And as such, we discovered usability issues, but also bugs. Hopefully, none of those bugs or usability features were critical, and everything could be worked around, but expect some fixes in Groovy 2.3.5. It is also the reason why the project initially used Gradle 2: it comes with Groovy 2.3.2 which embeds the markup template engine, so it was possible to use it without organizing the project into separate modules like we have. In fact, the early versions of the site didn’t use subprojects. It’s only when we wanted to leverage improvements from Groovy 2.3.4 that we had to switch to that architecture.

A team work

In the end, I can’t finish this blog post without mentionning the team work it implied. In particular:

  • Damien Vitrac designed the website and produced HTML sketches. If you think the new site looks good, thank this guy!

  • Guillaume Laforge designed the site architecture, wrote contents, tweaked the CSS, that is to say produced almost all contents. He spent countless hours fixing responsiveness issues and digging into front-end dev.

  • I streamlined the process by setting up the Gradle project, designing the sitemap DSL, the integration of the markup template engine, CI integration, … that is to say pretty much all the "backend" stuff.

  • You, as a community, provided awesome pull requests within hours. Keep them coming, we love it!

Each of us have different skills. Guillaume is far better than I am in any kind of web design, styling issues, etc for example, so in the end, I think the combination works quite good and that the site as it is now is already pretty usable.

Let us know what you think, and don’t forget that you can contribute, it’s easy!

Comments

Contributing for the Groovy language: setting up IntelliJ IDEA

24 juin 2014

Tags: groovy intellij gradle

Often people ask us how you can setup a development environment to contribute on the Groovy project. If you use IntelliJ IDEA, it’s actually very easy, so I decided to make a video for it. It’s my first screencast ever, I think I will try to do more in the future, so please excuse my hesitations and enjoy! You’ll notice that apparently Google Hangouts dislikes my face, but hopefully, the screensharing stuff worked ;-)

Here’s the video:

Of course, you can ask me if you have issues with the setup, but it should really be straightforward!

Comments

Groovy on Android, technical details

10 juin 2014

Tags: groovy android swift gr8conf

In my previous post, I have introduced how we could now use the Groovy language to develop Android applications. In this post, I will give you more details about how it works internally, giving you more hints about what makes it possible.

Compiling Groovy

The Groovy language is a JVM language which compiles to bytecode. Even if it has scripting capabilities, it’s always compiled. This means that in Groovy, you have two options: either a class is compiled to a .class file and used like any other class file on the JVM, using the groovyc compiler instead of javac, or a class can be compiled at runtime, for example if you rely on scripting. In the last case, the source script (or any Groovy source in general, not necessarily scripts) is available at runtime, and you rely on APIs that Groovy provide to compile those sources and execute them while your application is running. This is typical of scripting languages, but you must be aware that in any case, Groovy is not an interpreted language: everything is turned into bytecode.

A classic Android application

Since the Dalvik VM doesn’t use the same bytecode as the JVM, Android requires a bit of work in order to compile and execute Java programs. A special tool, called dex, is responsible for converting the JVM bytecode into Dalvik bytecode, and compile it into a classes.dex file. This is illustrated in the following schema:

Classic Android compilation process

In our case, we’re using the Gradle build tool, which is now the default for Android projects, so Gradle is responsible for this chain. Java sources are compiled into .class files, then those classes are processed by the dex tool, which converts bytecode for all classes and packages everything into a classes.dex file. Some additional steps can be done (such as calling ProGuard to reduce the size of the package), but in the end, an APK is produced, which is the deliverable application. When deployed on the device, there’s nothing else to do than loading the classes and executing the application.

Let’s see how the process differs in case of a Groovy application.

An application written in Groovy

In this case, we have .groovy files, corresponding to Groovy sources, but we may also have .java files. In the end, the process looks very similar:

Groovy Android compilation process

This schema exactly illustrates how the GR8Conf Agenda application is compiled. All is done at compile time, and nothing more at runtime. Groovy sources are compiled into JVM bytecode, which is in turn converted into Dalvik bytecode using dex. There’s absolutely no difference with Java, apart from the compiler, which is able to process both Groovy and Java sources!

One noticeable difference that some people have already reported to me is that since we embed the Groovy runtime, the number of method descriptors used in the classes.dex file is significantly higher than with an application written in pure Java. From my early tests, a Groovy application would consume around 8k methods, without optimizations. The classes.dex file has a limit of 65536 methods, so this is something you have to keep in mind. Anyway, I am not an Android specialist, but there seem to be workarounds available, like the multi-dex option.

In the end, Groovy is not different from any other library you would embed in your application, it’s "just" an additional jar. I also mentionned the fact that I recommended to use @CompileStatic on your classes. There’s a good reason for that. If you don’t, the classes that you will compile will use the dynamic runtime of Groovy. This is unlikely what you want on a mobile application, especially because on Android, it would use reflection, implying a major performance drop. On a normal JVM, Groovy would use call site caching techniques, like generation of classes at runtime or invokedynamic to improve performance. This is not possible on Android, so places where you use dynamic features of Groovy should be limited to small parts of the application, called not frequently. A good example would be the use of a builder for the UI. Builders are a very nice feature of Groovy, and for a UI, it would only be called once when the activity is loaded.

Using @CompileStatic, you will ensure that your classes are statically compiled, meaning that all method calls are resolved at compile time, leading to dramatically improved performance, very close, if not equal, to that of Java (depending on how you write your code, as usual, because idiomatic Groovy might not always be the fastest implementation).

In any case, you must recall that the first application that I wrote does not use a single class generated at runtime. Even if you remove all @CompileStatic annotations, it would use the dynamic runtime, but without creating classes at runtime.

Runtime scripting

One of the difficulties of adapting the Groovy language is that, as we said, Groovy is a highly dynamic language. One of its capabilities is executing scripts at runtime. So what if we wanted to type a Groovy script on the device, and execute it directly on it? Is it possible? In fact, yes, it is possible, given that you follow this process:

Groovy Android compilation process at runtime

You can have an application, written in Groovy or not, but in the end, once the application is running, you have Groovy source code that you want to execute. Then it needs to compile the classes, call dex directly on the device, package the result into a jar file on the local filesystem, then load it using a special classloader. So why this is possible, the process is very complex, not straightforward, but more importantly, it is dramatically slow.

This process is demonstrated in this application, which replicates the behavior of the well-known GroovyShell but directly on an Android device. To give you an idea, on my own device, a Samsung Galaxy Note 3, starting the application, written in Groovy, is blazing fast. Then if you hit the "execute" button, the first time, compilation of the little script will take around 3s.

Compiling Groovy at runtime

Subsequent compilations take much less time (around ~500ms), but the fact of having to dex files and write them to the filesystem is a performance killer. In anycase, this shows that it is possible, and even that you could rely on it for an application that would handle scripts. It would be possible, for example, to cache the jar files in order to avoid recompiling them…

Conclusion

In this post, I gave further details on how Groovy gets running onto an Android device. In a future post, I will give you more details about how you can get started with your own project. I will maybe give Lazybones, a project bootstraping tool, more love in the next days ;-)

Comments


Older posts are available in the archive.