The new Groovy website

18 juillet 2014

Tags: groovy templating gradle markup template engine

Last week, we revealed the beta of a brand new website for the Groovy language. This new website is open sourced and already received a few contributions. In order to make it even easier and as it a fully statically generated site that makes use of Groovy I wanted to give more technical details on the toolchain and how it is generated.

A static website

One of the first questions which arised was: why not use Grails/Spring Boot/Ratpack? In fact, the new Groovy website is fully statically generated. It offers multiple advantages:

  • hosting is much easier, as it only consists of static pages and assets

  • maintenance is simplified, no database to backup for example

  • everything is self contained, pages and data, into a single repository

  • no need for authentication

  • all content is public

Last but not least, we didn’t have any requirement for storing anything in a database, or that would require dynamic generation. Pull requests are enough so far. Eventually, we’re thinking about a blog, but even that can be statically generated even if you want to allow users to comment on articles (this blog is a perfect example). So in short, this decision was motivated by one mantra: the right tool for the right job.

The documentation, that you can find on this page, had already started migrating from the aging wiki to . It is generated independently of the website and integrated into it using iframes (we’re thinking about source integration though).

Structure of the project

Generator and site

The project is built using Gradle 2 and consists of 2 subprojects:

  • the generator project contains, as the name says, the static generator. It makes use of a template engine and provides the classes used in the model of the templates.

  • the site project contains the templates and data. If you’re looking into contributing contents, this is likely the place to look at.

Building and testing the site is easy:

git clone https://github.com/groovy/groovy-website.git          (1)
cd groovy-website
./gradlew generate                                              (2)
1 clones the repository
2 generates the website from templates

The output will be visible in <project directory>/site/build/site. There’s also a checkDeadlinks tasks that we will use once we get out of the beta phase to ensure that the generated pages do not contain any dead link.

Internally, we use our CI server to deploy changes to the master branch live. So any commit which is pushed onto the master branch is automatically published (in general, takes less than 2 minutes).

Adding contents

Even if the site is statically generated, we still have data. In this project, there’s an important file, named sitemap.groovy which handles a lot of the contents of the website. It is our "low cost" database and as you can see, it’s a DSL describing the contents of the website.

For example, you can see the menu section which looks like this:

menu {
    group('Groovy') {
        item 'Learn',                       'learn.html'
        item 'Documentation',               'documentation.html'
        item 'Download',                    'download.html'
        item 'Community',                   'community.html'
        item 'Ecosystem',                   'ecosystem.html'
    }

    group('About') {
        item 'Contributing',                'contribute.html'
        item 'Source code',                 'https://github.com/groovy/groovy-core'
        item 'Books',                       'learn.html#books'
        item 'Sponsors',                    'sponsors.html'
        item 'FAQ',                         'faq.html'
        item 'Search',                      'search.html'
    }

    // ...
}

It is a purely declarative description of the site menus. Actually, the "Groups" are used in the footer of the page, while the main Groovy group is used to generate the top navigation bar. Using a simple descriptive DSL is very interesting, because it decouples templates from the contents of the menu. We make sure that those templates do not contain any element which is hardcoded and reduce the risks of forgetting to update the footer, for example, if a section is added.

The same file is used to describe the list of downloads:

downloads {
    // ...
    distribution('Groovy 2.3') {
        description {
            yield 'Groovy 2.3 is our latest official '
            a(href: 'versioning.html', 'version')
            yield ' of Groovy.'
        }

        version('2.3.4') {
            stable true
            releaseNotes 'https://jira.codehaus.org/secure/ReleaseNote.jspa?projectId=10242&version=20432'
            windowsInstaller 'http://dist.codehaus.org/groovy/distributions/installers/windows/nsis/groovy-2.3.4-installer.exe'
        }
    }
    // ...
}

or the books which are listed on the learn page:

books {
    book('Groovy in Action, 2nd edition') {
        authors "Dierk König, Guillaume Laforge, Paul King, Cédric Champeau, Hamlet D'Arcy, Erik Pragt, and Jon Skeet"
        cover 'img/books/regina.png'
        url 'http://www.manning.com/koenig2/'
        description 'The undisputed definitive reference on the Groovy programming language, authored by core members of the development team.'
    }
    // ...
}

The same is done for all contents that need regular updates: user groups, events, projects of the ecosystem, … I think this DSL provides a very nice way to add contents to the website without caring about where it has to be done. You can really think of it as a small database, but making use of a Groovy DSL.

In addition, this file also declares the mapping between pages in the documentation section and the documentation page. Last but not least, it lists the individual pages that the website contain. Those pages make use of the markup template engine.

Eat your own dog food

In Groovy 2.3, we introduced a new markup template engine. We decided that the new web site was an excellent showcase of this template engine, and a real life use case. This template engine has several remarkable features, like static compilation of templates (even if the model is dynamic), layouts and of course a human readable builder like syntax:

html {
   head {
    title 'Groovy markup template engine in action!'
   }
   body {
    ul {
        features.each { f-> li(f.name) }
    }
   }
}

It has already been integrated into Spring Boot and Ratpack will use it in the next version (to be released on August 1st). A hint about its performance can be found here. If you are interested in details about how it works, you can find the documentation here and you can read my blog posts about it.

The website subproject is therefore organized accordingly. Inside the main source tree, you’ll find the following directories:

  • assets: contains static assets, like Javascript, CSS, images, …

  • html: contains pure HTML files which are easier to embed as is than using a markup syntax

  • includes: contains elements of code which are shared among multiple templates

  • layouts: contains template layouts, as defined in the documentation

  • pages: contains the main pages of the website

In general, consider pages as the entry point. A page generally makes use of one layout. As an example, let’s see how the Ecosystem page is generated. The source file consists of this:

ecosystem.groovy
layout 'layouts/main.groovy', true,                                             (1)
    pageTitle: 'The Groovy programming language - Ecosystem',                   (2)
    mainContent: contents {                                                     (3)
      div(id: 'content', class: 'page-1') {
        section(class: 'row') {
          div(class: 'row-fluid') {
            // ... snip side menu ...
            div(class: 'col-lg-8 col-lg-pull-0') {
              include template: 'includes/contribute-button.groovy'             (4)
              h1 {
                i(class: 'fa fa-leaf') {}
                yield ' Ecosystem'
              }
              p {
                yield '''
                    Beside the language and its API, Groovy gave birth   ...
                    on various themes such as web frameworks, desktop    ...
                    In this section, we will highlight a few of the most ...
                    which leverage Groovy at their core.
              '''
              }
              hr(class: 'divider')

              ecosys.eachWithIndex { e, index ->                                (5)
                def (name, item) = [e.key, e.value]
                article {
                  a(name: "${name}") {}
                  div(class:"content-heading clearfix media") {
                    div {
                      if (item.logo) {
                        img class: "pull-${(index % 2 == 0) ? 'left' : 'right'}",
                          src: item.logo, alt: name, hspace: '20px'
                      } else {
                        h2(name)
                      }
                      p(item.description)
                    }
                    a(href: item.url, target:'_blank', "Learn more...")
                  }
                }
                hr(class: 'divider')
              }
              // ...
            }
          }
        }
      }
    }
1 make use of the main layout
2 the layout requires a pageTitle variable
3 as well as a mainContent section corresponding to the main page contents
4 example of use of an include
5 iterates over the ecosys variable which contains the list of ecosystem projects as found in the sitemap

As you can see, this template format has the advantage of taking care of generating markup for you. You won’t hit your head again on the wall to find an unclosed tag. Everything is embedded, readable and concise.

Lessons learnt

Using the markup template engine for this project was interesting, because it was probably the first "real life" project to use it intensively. And as such, we discovered usability issues, but also bugs. Hopefully, none of those bugs or usability features were critical, and everything could be worked around, but expect some fixes in Groovy 2.3.5. It is also the reason why the project initially used Gradle 2: it comes with Groovy 2.3.2 which embeds the markup template engine, so it was possible to use it without organizing the project into separate modules like we have. In fact, the early versions of the site didn’t use subprojects. It’s only when we wanted to leverage improvements from Groovy 2.3.4 that we had to switch to that architecture.

A team work

In the end, I can’t finish this blog post without mentionning the team work it implied. In particular:

  • Damien Vitrac designed the website and produced HTML sketches. If you think the new site looks good, thank this guy!

  • Guillaume Laforge designed the site architecture, wrote contents, tweaked the CSS, that is to say produced almost all contents. He spent countless hours fixing responsiveness issues and digging into front-end dev.

  • I streamlined the process by setting up the Gradle project, designing the sitemap DSL, the integration of the markup template engine, CI integration, … that is to say pretty much all the "backend" stuff.

  • You, as a community, provided awesome pull requests within hours. Keep them coming, we love it!

Each of us have different skills. Guillaume is far better than I am in any kind of web design, styling issues, etc for example, so in the end, I think the combination works quite good and that the site as it is now is already pretty usable.

Let us know what you think, and don’t forget that you can contribute, it’s easy!

Comments

Contributing for the Groovy language: setting up IntelliJ IDEA

24 juin 2014

Tags: groovy intellij gradle

Often people ask us how you can setup a development environment to contribute on the Groovy project. If you use IntelliJ IDEA, it’s actually very easy, so I decided to make a video for it. It’s my first screencast ever, I think I will try to do more in the future, so please excuse my hesitations and enjoy! You’ll notice that apparently Google Hangouts dislikes my face, but hopefully, the screensharing stuff worked ;-)

Here’s the video:

Of course, you can ask me if you have issues with the setup, but it should really be straightforward!

Comments

Groovy on Android, technical details

10 juin 2014

Tags: groovy android swift gr8conf

In my previous post, I have introduced how we could now use the Groovy language to develop Android applications. In this post, I will give you more details about how it works internally, giving you more hints about what makes it possible.

Compiling Groovy

The Groovy language is a JVM language which compiles to bytecode. Even if it has scripting capabilities, it’s always compiled. This means that in Groovy, you have two options: either a class is compiled to a .class file and used like any other class file on the JVM, using the groovyc compiler instead of javac, or a class can be compiled at runtime, for example if you rely on scripting. In the last case, the source script (or any Groovy source in general, not necessarily scripts) is available at runtime, and you rely on APIs that Groovy provide to compile those sources and execute them while your application is running. This is typical of scripting languages, but you must be aware that in any case, Groovy is not an interpreted language: everything is turned into bytecode.

A classic Android application

Since the Dalvik VM doesn’t use the same bytecode as the JVM, Android requires a bit of work in order to compile and execute Java programs. A special tool, called dex, is responsible for converting the JVM bytecode into Dalvik bytecode, and compile it into a classes.dex file. This is illustrated in the following schema:

Classic Android compilation process

In our case, we’re using the Gradle build tool, which is now the default for Android projects, so Gradle is responsible for this chain. Java sources are compiled into .class files, then those classes are processed by the dex tool, which converts bytecode for all classes and packages everything into a classes.dex file. Some additional steps can be done (such as calling ProGuard to reduce the size of the package), but in the end, an APK is produced, which is the deliverable application. When deployed on the device, there’s nothing else to do than loading the classes and executing the application.

Let’s see how the process differs in case of a Groovy application.

An application written in Groovy

In this case, we have .groovy files, corresponding to Groovy sources, but we may also have .java files. In the end, the process looks very similar:

Groovy Android compilation process

This schema exactly illustrates how the GR8Conf Agenda application is compiled. All is done at compile time, and nothing more at runtime. Groovy sources are compiled into JVM bytecode, which is in turn converted into Dalvik bytecode using dex. There’s absolutely no difference with Java, apart from the compiler, which is able to process both Groovy and Java sources!

One noticeable difference that some people have already reported to me is that since we embed the Groovy runtime, the number of method descriptors used in the classes.dex file is significantly higher than with an application written in pure Java. From my early tests, a Groovy application would consume around 8k methods, without optimizations. The classes.dex file has a limit of 65536 methods, so this is something you have to keep in mind. Anyway, I am not an Android specialist, but there seem to be workarounds available, like the multi-dex option.

In the end, Groovy is not different from any other library you would embed in your application, it’s "just" an additional jar. I also mentionned the fact that I recommended to use @CompileStatic on your classes. There’s a good reason for that. If you don’t, the classes that you will compile will use the dynamic runtime of Groovy. This is unlikely what you want on a mobile application, especially because on Android, it would use reflection, implying a major performance drop. On a normal JVM, Groovy would use call site caching techniques, like generation of classes at runtime or invokedynamic to improve performance. This is not possible on Android, so places where you use dynamic features of Groovy should be limited to small parts of the application, called not frequently. A good example would be the use of a builder for the UI. Builders are a very nice feature of Groovy, and for a UI, it would only be called once when the activity is loaded.

Using @CompileStatic, you will ensure that your classes are statically compiled, meaning that all method calls are resolved at compile time, leading to dramatically improved performance, very close, if not equal, to that of Java (depending on how you write your code, as usual, because idiomatic Groovy might not always be the fastest implementation).

In any case, you must recall that the first application that I wrote does not use a single class generated at runtime. Even if you remove all @CompileStatic annotations, it would use the dynamic runtime, but without creating classes at runtime.

Runtime scripting

One of the difficulties of adapting the Groovy language is that, as we said, Groovy is a highly dynamic language. One of its capabilities is executing scripts at runtime. So what if we wanted to type a Groovy script on the device, and execute it directly on it? Is it possible? In fact, yes, it is possible, given that you follow this process:

Groovy Android compilation process at runtime

You can have an application, written in Groovy or not, but in the end, once the application is running, you have Groovy source code that you want to execute. Then it needs to compile the classes, call dex directly on the device, package the result into a jar file on the local filesystem, then load it using a special classloader. So why this is possible, the process is very complex, not straightforward, but more importantly, it is dramatically slow.

This process is demonstrated in this application, which replicates the behavior of the well-known GroovyShell but directly on an Android device. To give you an idea, on my own device, a Samsung Galaxy Note 3, starting the application, written in Groovy, is blazing fast. Then if you hit the "execute" button, the first time, compilation of the little script will take around 3s.

Compiling Groovy at runtime

Subsequent compilations take much less time (around ~500ms), but the fact of having to dex files and write them to the filesystem is a performance killer. In anycase, this shows that it is possible, and even that you could rely on it for an application that would handle scripts. It would be possible, for example, to cache the jar files in order to avoid recompiling them…

Conclusion

In this post, I gave further details on how Groovy gets running onto an Android device. In a future post, I will give you more details about how you can get started with your own project. I will maybe give Lazybones, a project bootstraping tool, more love in the next days ;-)

Comments

Groovy on Android

05 juin 2014

Tags: groovy android swift gr8conf

Yesterday ended GR8Conf Europe, a Groovy language conference in Copenhagen that was once again a successful event. I want to thank the crew again for this, and of course the Groovy language community which is so friendly and always helpful. This year turned out to be very special for me, because just two days after the announcement by Apple of the Swift language, I was talking about running Groovy on Android! As Guillaume Laforge noticed, there are a lot of similarities between the Swift language and Groovy language.

The timing is almost perfect, because Android users will want to have a language which is as modern as Swift is, but running on Android. And I see no better candidate than Groovy language here.

The presentation

Should you be interested in the slides, you can find them below. Since it was presented at GR8Conf Europe, it gives a bit of history of the changes needed to have the runtime working on Android too:

For those of you who are new to Groovy, you have to know that this language is derived from Java, meaning that the learning curve is almost 0, but it also removes a lot of its verbosity and adds a lot of features to it, such as closures (similar to lambdas in Java 8, but Java 8 is not available for Android developers), builders, runtime and compile-time metaprogramming.

As an example of how Groovylang can be used to reduce the verbosity of Java on Android, I will take a simple example: asynchronous tasks. Asynchronous tasks are required as soon as a task takes too much time to be executed on the UI thread. That is the case, by default, for any network based operations, in order to guarantee that the UI remains snappy even if network is slow or unavailable. The problem is that those asynchronous tasks are incredibly verbose. You have to write a lot of code, that I would tend to name "inner class hell", just for this. Let’s imagine that you need to parse a JSON feed, then update the UI accordingly. Then you would have to write something like this (no kidding):

public class FeedActivity {
    TextView mTextView;

    ...

    void updateFeed() {
    	new FeedTask().execute("http://path/to/feed");
    }

    class FeedTask extends AsyncTask<String, Void, String> {
        protected String doInBackground(String... params) {
            DefaultHttpClient httpclient = new DefaultHttpClient(new BasicHttpParams());
            HttpPost httppost = new HttpPost(params[0]);

            InputStream inputStream = null;
            String result = null;
            try {
                HttpResponse response = httpclient.execute(httppost);
                HttpEntity entity = response.getEntity();

                inputStream = entity.getContent();
                // json is UTF-8 by default
                BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream, "UTF-8"), 8);
                StringBuilder sb = new StringBuilder();

                String line = null;
                while ((line = reader.readLine()) != null) {
                    sb.append(line).append("\n");
                }
                result = sb.toString();
            } catch (Exception e) {
                // Oops
            } finally {
                try {
                    if (inputStream != null) {
                        inputStream.close();
                    }
                } catch (Exception squish) {
                }
            }
            StringBuilder speakers = null;
            try {
                JSONObject jObject = new JSONObject(result);
                JSONArray jArray = jObject.getJSONArray("speakers");
                speakers = new StringBuilder();
                for (int i = 0; i < jArray.length(); i++) {
                    speakers.append(jArray.getString(i));
                    speakers.append(" ");
                }
            } catch (JSONException e) {
                // do something?
            }
            return speakers.toString();
        }

        @Override
        protected void onPostExecute(String s) {
            mTextView.setText(s);
        }
    }
}

So now, let’s see the equivalent in Groovy (no kidding either):

public class FeedActivity {
    TextView mTextView

    ...

    void updateFeed() {
    	Fluent.async {
            def json = new JsonSlurper().parse([:], new URL('http://path/to/feed'), 'utf-8')
            json.speakers.join(' ')
        } then {
	    mTextView.setText(it)
        }
    }
}

I think you can start to see the advantage of using Groovy in your own Android projects. Of course, the Fluent class that I’m using here is a support class which I implemented in my first Android project (which is open sourced btw), but it’s really simple and gives an example of how Android users could benefit from Groovification of their APIs.

Feeling the pain

This is actually a key point of my talk: I hadn’t written any Android application before this talk, and I definitely wanted to be able to write an application in Groovy on Android. Why? Because it’s been some time already that I use Groovy everyday, and there’s no turning back. I wanted to feel the pain of the Java developers on Android, so that I can write better tools for them. And that’s actually where you, as a user, come in action: there are so many Groovy libraries out there whose sole objective is to ease the pain, make things that shouldn’t be complicated a breeze. This is exactly the point. My example with Fluent is one example of simplification of usages of Android APIs, but you have many more to invent, especially because you must have been as annoyed as I did by all those asynchronous tasks, XML files (think of builders!), callback hell, etc…

The beginning of a new era

In upcoming posts, I will try to demonstrate that we’re just at the beginning. Some people are already asking for a {swift} alternative for Android. It’s there guys, you have it, so spread the word and let’s make it happen! I am convinced that it is the beginning of a new era for Android and Groovy. Google already switched their main build system to Gradle, which is, by the way, using Groovy, so I think it’s time to move over and show your love!

All you need to get started is explained in the slide deck above, and you can find the source code of the sample android application on GitHub:

Update 2: build instructions with the Gradle plugin

You can now use a Gradle plugin to integrate Groovy with Android. The plugin can be found here: groovy-android-gradle-plugin. As of version 0.2, the plugin supports the Android plugin 0.10+.

Update: build instructions

If you want to try it by yourself, here’s how you can do it. First of all, official support for Android will be in Groovy 2.4. Before the first beta, you’ll have to build it from sources, and here is the quickest way:

git clone https://github.com/melix/groovy-core.git --branch master
cd groovy-core
./gradlew -PskipIndy=true install

Then you can clone the sample application:

cd ..
git clone https://github.com/melix/gr8confagenda.git

This contains a project that you can open using Android Studio.

If you want to use the Groovy language in your own Android project, a requirement is that it is using Gradle. If so, you can update your build.gradle file as is:

build.gradle
android {
   ...
   packagingOptions {
        // workaround for http://stackoverflow.com/questions/20673625/android-gradle-plugin-0-7-0-duplicate-files-during-packaging-of-apk
        exclude 'META-INF/LICENSE.txt'
        exclude 'META-INF/groovy-release-info.properties'
    }
}

repositories {
    mavenLocal()
    jcenter()
}

dependencies {
    compile 'org.codehaus.groovy:groovy:2.4.0-SNAPSHOT:grooid'
    // the following dependency is necessary if you want JSON support
    compile ('org.codehaus.groovy:groovy-json:2.4.0-SNAPSHOT') {
        transitive = false
    }
}

// add support for Groovy to existing configurations
android.applicationVariants.all {
    task "groovy${name}Compile"(type: GroovyCompile) {
        source = javaCompile.source + fileTree('src/main/java').include('**/*.groovy')
        destinationDir = javaCompile.destinationDir
        classpath = javaCompile.classpath
        groovyClasspath = classpath
        sourceCompatibility = '1.6'
        targetCompatibility = '1.6'
        doFirst {
            def runtimeJars = plugins.findPlugin(com.android.build.gradle.AppPlugin).runtimeJars
            classpath = files(runtimeJars) + classpath
        }
    }
    javaCompile.dependsOn("groovy${name}Compile")
    javaCompile.enabled = false
}

And that’s all! Now, one option for you is to write support libraries and make them available to the community. Enjoy!

Comments


Older posts are available in the archive.