Gradle and Kotlin, a personal perspective

22 May 2016

Tags: jvm groovy gradle kotlin

Gradle embraces Kotlin, what about Groovy?

First of all, it’s been a long time since I last blogged, and I’d like to remind that everything written here are opinions of my own and not the views of my employer, which happens to be Gradle Inc as write those lines.

A few days ago, Gradle and Jetbrains announced a partnership to make Kotlin a first class language for Gradle builds, both for build scripts and plugins. Most likely, you know Gradle has been using Groovy since its inception. Lots of people think that Gradle is written in Groovy, which is actually wrong. Most of Gradle is written in Java. The builds scripts are written in Groovy, lots of plugins are written in Groovy, our test cases are written in Groovy (using the best testing framework out there, Spock), but Gradle itself is written in Java.

From my perspective, this is situation has been very disturbing and continues to be so. I have very good friends in the Groovy community, and this move has been seen by some of them as a betrayal. As an Apache Groovy committer, and someone who spent almost 4 years full time implementing new features of the language, most importantly its static compiler, seeing Kotlin promoted as the language of choice for Gradle’s future, it’s a little strange. One could legitimely say, WTF? I’ve been aware of this work for several months now, and my colleagues Rodrigo B. de Oliveira and Chris Beams have done an amazing job in a very short period of time. From a long time Groovy user and Groovy developer point of view, it’s hard not to make this move an emotional thing. However, business is not about emotions. In particular, what are we trying to acheive with Gradle? We’re trying to help developers build their applications. We’re trying to make this elegant, reproducible, scalable and fast. We’re language agnostic. We can build Java, Groovy, Scala, Kotlin, C++, Python, …​ Gradle has never been the tool to build Groovy applications: it’s been a tool to build software. It’s a tool about automation. And I’ve been complaining enough about communities that build their own tool for their very specific language to understand that this is super important: Gradle is (or aims at) the best tool for building any kind of software. In short, we must think in terms of what is best for our users, and sometimes, this means changing technogies. A product should not be bound to a technology, but a company should even less be bound to it. And given the response that we had after the announcement, supporting Kotlin seem to drive a lot of excitement around Gradle, and that’s a very good thing. So, let’s take that out, and think what it means for Groovy.

Groovy support is not abandoned

First of all, I already said it several times, but better continue to spread the message: support for Groovy in Gradle is not deprecated nor removed. You can still write your scripts in Groovy, you can write your plugins in Groovy, and you will still be able to do it. But Gradle will likely encourage users to migrate to Kotlin. To be clear, Kotlin support is incubating, and there’s a lot to do to make it as usable as the Groovy version. Second, there are tens of thousands of builds written using Groovy, hundreds of plugins written in Groovy, so it’s not tomorrow that Kotlin is going to replace Groovy. However, we care about the future, so we need to think about what it means in the long term. Should we be excited about supporting Kotlin? Yes we should, because Kotlin is an amazing language. Should we continue to be excited about Groovy? Of course we should, because it’s also an amazing language. But it’s old and as such brings a lot of legacy with it. As someone who implemented the static compiler for Groovy, I know it very well. There are things that are hard to change, because a large part of the Groovy community is very fond of its dynamic behavior.

So let’s focus on the two major aspects that led to embracing Kotlin in Gradle. The fist one, and principal, is IDE support. Let’s face it: even before I joined Gradle, when I was giving talks about it, people were complaining about IDE support. Compared to a tool like Maven, supporting Gradle build scripts is complicated. Supporting XML is easy (to some extent). Supporting a dynamic DSL is not. Some say it’s Groovy’s fault, and I want to correct this statement right now: it’s not Groovy’s fault. While Groovy let’s you design dynamic DSLs, the design of the DSL can be changed to make it easier for tools to "discover" things. But when Gradle was designed, there wasn’t any statically compiled Groovy. The idiomatic way to write DSLs in Groovy, at that time, was to heavily rely on runtime metaprogramming. While loving metaprogramming, I’ve always prefered compile time metaprogramming over runtime metaprogramming. For multiple reasons:

  • because in most cases, what you want to do at runtime can be done in a unique, setup phase. For example, create your metaclasses, enrich existing types, configure property missing, method missing, …​ If it’s setup, it’s better done at compile time, because you can report errors, and because it gives higher performance. This led the way I designed the static compiler, and more features of Groovy after that (traits, type checking extensions, …​) : describe what you want to do at compile time.

  • because it makes the life of tools easier. While IntelliJ or Eclipse support DSL descriptors that help them provide completion, those are hard to implement, and often inaccurate. They can only approximate what is going to happen at runtime. And in the end, you’re doing the same job twice: you’re writing a runtime for your DSL, which is dynamic, then you need to write a DSL descriptor for the IDE to understand it. Wouldn’t it be better if all was done in a unique place? Something that both the compiler and the IDE can understand?

So while we know we can describe dynamic Groovy DSLs so that they are understood by IDEs, it’s effectively a lot of work. And if you want to support multiple IDEs, it’s even more work. But in the case of Gradle, it’s even worse: each plugin can provide it’s own "micro DSL". While there’s an "idiomatic" way to configure Gradle builds, it’s no single rule. One can implement it’s own Groovy DSL within the Gradle build. And no luck the IDE would ever understand it. Another pain point is that Gradle adds complexity to complexity in terms of DSL capabilities. For example, when you have a build script that has:

dependencies {
   compile libraries.groovy
}

greeter {
   message = 'hello'
}

sign {
   signature = top
}

often people do not realize that: - dependencies is found on a Project instance - libraries is a user declared variable, that can be found in a plugin, another build script, a project properties file, …​ (how does the IDE find about it?) - greeter is a convention object, defined by a plugin, to configure the default values of its task - sign is a task, which has a signature property, and top references an extension property from the project

So while this build script is simple to read, it’s hard to understand how it effectively works, because objects can be found at different places, can be provided by different providers (plugins, properties, extensions), but everything is accessed using a single notation. This is bad, because it makes it almost impossible for an IDE to understand what is going on.

The question is, is it Groovy’s fault? My answer is not totally. The fault is mostly on the DSL design, and Groovy made it too easy to do so. But again, that was designed at a time when dynamic Groovy was the rule. I gave a talk, recently, about building modern DSLs with Groovy, where I discourage such practices, and encourage the use of static DSLs instead.

That leads me to the second main reason of embracing Kotlin in Gradle: performance. When we talk about performance, lots of folks tend to think that Groovy is slow. This is not the case. Groovy is pretty fast. However, depending on the design of the DSL, you can easily fall into traps that can lead to catastrophic performance. Before I go further with it, I’m reading way to often that Gradle is slow because it’s written in Groovy and that Groovy is dynamic so it’s slow. F* no, those who tell you that just didn’t profile a build. As I said, Gradle is mostly written in Java. And I’ve spent the last 3 months optimizing the performance of Gradle, and I can tell you that of the dramatic performance improvements that one can see in Gradle 2.13 and 2.14, almost none was obtained by rewriting Groovy to Java, or rewriting Groovy code. None! Most of the hotspots were pure Java code. Period. However, as soon as you use plugins, which are today mostly written in dynamic Groovy, or that your build scripts imply a lot of nested closures, things start to become complicated for "Groovy". Let me explain that clearly. I think at some point, someone made a terrible design decision in Groovy. I don’t know who it was, but the idea was to rely on exceptions to control the flow of resolution of properties. This means that when a property is missing, typically in a closure, an exception is thrown. When a method is not found, an exception is thrown. When a property is not found, an exception is thrown. That seemed to be a good idea, because in the end, you want to provide the user with an error, but in practice, this is catastrophic, because Groovy can capture those exceptions. Typically, in a delegation chain (nested closures), a containing closure or class can actually have this property defined, or implement property missing/method missing. Now, re-think a second about the "simple" example of Gradle build above: how do you now where to look up for message, top, signature, …​? Now you know: f* exceptions are thrown, stack traces are filled, and eventually captured because some composite dynamic object finally wants to answer the message…​ In practice, for some builds I have profiled, it was tens of thousands of exceptions being thrown and stack traces filled for nothing. And that has a terrible impact on performance. So even if we have implemented strategies in Gradle to try to avoid throwing those exceptions (which are responsible for part of the performance improvements in 2.14), this is very hard to do it, and we’re still throwing way too many of them. A static language doesn’t have this problem, because every single reference in source is resolved at compile time. So, if you’re writing a plugin in Groovy, for the sake of performance, please add @CompileStatic.

So there goes Kotlin. Kotlin has excellent static builders support, that make it practical both for IDE support, which will dramatically improve user experience in terms of understanding what do write, what is an error, having documentation, refactorings, …​ and is a very pleasant language to work with. Honestly, I don’t have anything bad to say about the language (apart from the fun keyword that I don’t like). To some degree, it’s not very surprising: Kotlin has heavily inspired by Groovy and another popular JVM language: Scala. And again, being the one behind the static compiler of Groovy, I can’t blame them for doing what I like about static languages. Their builder support is awesome, and very elegant. And it’s supported out of the box by IntelliJ of course, but also Eclipse.

A static DSL for Groovy?

Ok, so one might think at this point that I’m mad. I wrote a "competing" language, and I’m happy to see Kotlin being promoted in Gradle. I wrote the static compiler, that is capable of doing everything Kotlin can do (minus reified generics, plus superior scripting support, type checking extensions, …​), so wtf? Ok, so let’s be very clear: I have absolutely no doubt that Groovy can do everything that we’ve done with the Kotlin support in Gradle. It can be statically compiled, provide an elegant DSL that is statically compiled, and it can be understood by the IDE. I had no doubt before the Kotlin work started, I have even less doubts now. And I can say I have no doubts because I tried it: I implemented experimental support for statically compiled Gradle scripts, written in Groovy. Here’s an example:

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'groovy'
apply plugin: GreetingPlugin

repositories {
    mavenCentral()
}

dependencies {
    compile 'commons-lang:commons-lang:2.5'
    compile "commons-httpclient:commons-httpclient:3.0"
    compile "commons-codec:commons-codec:1.2"
    compile "org.slf4j:jcl-over-slf4j:1.7.10"
    compile "org.codehaus.groovy:groovy:2.4.4"
    testCompile 'junit:junit:4.12'
    runtime 'com.googlecode:reflectasm:1.01'
}

tasks.configure('test', Test) {
    jvmArgs '-XX:MaxPermSize=512m', '-XX:+HeapDumpOnOutOfMemoryError'
}

dependencies {
    compile 'org.codehaus:groovy:groovy-all:2.4.4'
}

extension(GreetingPluginExtension) {
    message = 'Hi'
    greeter = findProperty('greeter')?:'static Gradle!'
}

tasks.create('dependencyReport', DependencyReportTask) {
    outputs.upToDateWhen { false }
    outputFile = new File( project.buildDir, "dependencies.txt")
}

class GreetingPlugin implements Plugin<Project> {
    void apply(Project project) {
        project.extensions.create("greeting", GreetingPluginExtension)
        project.task('hello') << {
            println "${project.extension(GreetingPluginExtension).message} from ${project.extension(GreetingPluginExtension).greeter}"
        }
    }
}

class GreetingPluginExtension {
    String message
    String greeter
}

This is an example Gradle build that is compiled statically. It has none of the problems I described about the Groovy implementation in Gradle above. It uses all the techniques that static Groovy provides: extension methods, powerful scripting with implicit imports, type checking extensions, …​ All this works. And interestingly, the work that is done to enable support for Kotlin also benefits to statically compiled Groovy, and Java! Let’s not forget about the latter, which is years behind in terms of "modern" languages support. So if this works, why do we need Kotlin? To be honest, I asked it to myself many times. It was very difficult to me, because I knew Groovy could do it. Again, I had no doubt about the language capabilities, no doubt about the performance impact of doing this. However, I missed two critical points:

  1. IDE support. Even if support of Groovy in IntelliJ is by far the most advanced of all other IDEs, it still lacks behind when static compilation is on. But more importantly, it doesn’t know that my script is statically compiled, nor does it now about my custom extension methods. I tried to implement a GDSL descriptor to make it aware of them, and it somehow worked: I do have code completion, but errors are not marked as errors, and the IDE still doesn’t understand that it should only suggest to me what is relevant in the context. With Kotlin scripts which are natively static, there’s no such issue. The IDE understands everything natively, in IntelliJ and Eclipse. So, I have no doubt that Jetbrains can implement support for this, just like I had no doubt I could implement a static Groovy DSL, but who is going to write this? Me? Gradle? I don’t have the time to do it. And it’s not Gradle’s job to write IDE plugins. And what about Eclipse? One big issue that the Groovy community has, today, is that nobody is supporting Eclipse since Pivotal dropped sponsorship of Groovy. After more than one year, nobody took over the development of Groovy Eclipse. Nobody. While Groovy itself saw lots of new contributors, while we saw a lot of bugfixes, new contributors and that the download numbers where never as high as they are today, IDE support is critical. And nobody took over the development of it. I saw some people referring to what Jetbrains is doing as "blackmailing". Seriously? Jetbrains? Think of what they’ve done for Groovy. Groovy would never has been as popular as it is without them. They provided us with the best Groovy IDE possible. They are constantly supporting new features of the language, adding support for AST transformations, traits, …​ They even added the ability, in IDEA 14, to use Groovy (and not Kotlin, guys!) as the language to evaluate expressions in the debugger. And they would try to kill Groovy? Kill part of their business? Come on guys! So yes, they invested a lot in Kotlin and want to promote it, but how could it be otherwise? And it’s not like if the language sucked: it’s awesome!

  2. Does it make sense? Now that we made the decision to support Kotlin, that we proved it would provide the level of user friendliness we want and that it is statically compiled by default, does it make sense to put resources to support static Groovy in addition? I don’t have an answer to this. I thought yes, but now I’m not sure. Kotlin does the job. And honestly, they have great engineers working on the language. Even if it lacks behind in terms of scripting and compilation times compared to Groovy, I have no doubt they will fix it. How arrogant would we be if we thought other languages could not do what we’ve done with Groovy?

The future of Groovy

The last point I want to address is what it means for the future of Groovy, and what it means for my future in Groovy. First of all, I always thought that the future of Groovy was in the hands of its community. It’s not Gradle that has Groovy’s future in its hands. It’s you. The move to the Apache Software Foundation was also done for this very same reason: community first. If you want to continue to use Groovy, to improve it, to support it, all you have to do is f* do it! And I will continue! I love this language, I know too well how far it can go in terms of DSL support, AST transformations, now in 2.5 we have macros, that’s just a crazily powerful language that’s super fun to use. Should we fear competition? No, we shouldn’t. Competition is good. It should be inspiring. And if Gradle moving to Kotlin means the death of Groovy, maybe the problem is elsewhere. And even if lots of people get introduced to Groovy through Gradle, it’s not the only entry point. Grails is another. Jenkins (through Flow) is another. And many, many more. There was a tweet a few days ago which showed the 100 most popular dependencies in GitHub projects. Groovy was one of them. No Kotlin. No Scala. Groovy. It’s everywhere, and it’s going to be there for a long time.

Part of the fears of the community is, after the Pivotal demise, if Groovy is a dying language. It’s not. It has never been so widely used. The move to Apache Software Foundation drove a lot of attention and brought us many more contributors. But the community has to realize what the problems with Groovy are, and it has to face them: the introduction of the static compiler was too late. IDE support is important. Java 9 support is going to be super important. If you love your language, contribute. Help it. Help yourselves. The future of Groovy must be in your hands. I can’t recall how many times I told this, since I joined VMware, a few years ago, to develop Groovy. In every talk I give, I’m always telling how important it is that you contribute. Jetbrains is not going to write Groovy Eclipse for you.

And I would like to finish with one word: if people move from Groovy to Kotlin, is it really a problem? Isn’t any technology inspired by another? Aren’t we, developers, always rebuilding the same things, but improving them, learning lessons from the past? Is Kotlin a better Groovy? I don’t have the answer yet. Maybe it is. Maybe not. Today Groovy remains greatly superior in terms of scripting, DSL support, but it comes with a price that Gradle doesn’t want to pay. And let’s not forget the original community of Groovy: a dynamic language for the JVM. There are still lots of people who like this aspect of the language (and I do too, typically when I write Groovy scripts in place of bash scripts, I don’t care about types). It’s compile time metaprogramming features also make it incredibly powerful. Modern Groovy definitely doesn’t deserve its "bad press". Would you compare Java 8 with Java 1? No. So don’t compare Groovy 2.4 with Groovy 1 either. Reputation should change, and you can help there too.

This leads me to what I should do. And there, I’m a bit lost, to be honest. I work for a company that embraced Groovy, that is now embracing Kotlin. I love my job, I love working with Gradle, I love Groovy, and I quite enjoy Kotlin. I’m a passionate developer. I just want to continue having fun. But if you think that as such, I’m not a good representative of the Groovy community anymore, maybe I should step off from the Groovy project. I would hate that, but I’ve kind of been hurt by the bad comments we (Gradle) received from some members of the Groovy community. I don’t want to fall into a language war, I don’t care about this. I care about users. What I love to do is helping people, period.

I would like to finish this post with a thought about what I’m going to do, as a Gradle developer, for you, Groovy users. In particular, I am convinced that the success of Gradle is largely due to its Groovy DSL, despite its problems. The fact that it’s simple, easy to read, is super important. I joined the Groovy project because I was using Groovy as a DSL platform in a natural language processing context. Groovy is super powerful for this. And I learnt a lot in terms of DSL design. In particular, I will try to make sure that it doesn’t become a Kotlin API. What I mean by that is that I think we should elevate from a Groovy DSL to a Gradle language. And this language is meant at describing builds. And our users are not Kotlin developers. Most of them are not Groovy developers either. They are, as I described earlier, from different horizons. And I would hate if a user would have to understand concepts like generics or type inference to write a build script. That would be horribly wrong. A build author should understand how to model an application, not what is a type, what is an extension method, or generic return type inference. It’s different for plugin authors, but for a build author, it’s super important. So I will try to make sure that Kotlin scripting support improves, even if it means that it would go even closer to what Groovy supports. I would do this not because I want Groovy to die, I don’t (and it wouldn’t help my royalties for Groovy in Action 2 ;)), but it would help users or Gradle. That’s what I care most about, just like I care about what Groovy users want when I work on the Groovy project.

As for talking about Gradle, Groovy and its future, I’ll be a GR8Conf next week, I’d be happy to answer you in person there too!

Keep on Groovying!