GitHub Build Status Apache License 2

This plugin adds support for including Git repositories as source dependencies in Gradle builds, making multi-repository development much easier to deal with!

This plugin has been successfully tested on Gradle 6.2, 6.3, 6.4, 6.5, 6.6, 6.7.1, 6.8.2, 6.9, 7.1.1, 7.2, 7.3-rc-3. Earlier releases are not supported. The following Gradle versions are known to be broken with this plugin: 6.0.x, 6.1.x.

Rationale

Multi-repository development, while providing architectural advantages (reduced scope of libraries, faster development cycles, …​), are often painful for developers, especially when the number of modules being involved grows.

For example, to develop a single feature, a developer may have to work on more than one repository at a time: a core module, living in repository core, and a library module depending on core, living in a repository library.

The problem is that the development of a particular feature may require changes to both core and library. In that case, developers, in the Java ecosystem, typically rely on publishing snapshots in their local Maven repository: make a change in core, publish a snapshot, then make changes to library, publish a snapshot, and repeat the loop.

There are several problems with this approach:

  • it requires switching between projects, publishing intermediate artifacts to the local filesystem

  • makes it difficult to collaborate with others as they also need to checkout several projects

  • it requires integrating mavenLocal() as a repository for your local builds, which is considered a bad practice (because it introduces non-reproducibility and makes the builds brittle)

  • for CI, it requires publishing to a snapshot repository for downstream builds to pick up the changes, meaning that you will often have to merge work-in-progress just to be able to test changes

  • it simply doesn’t work if the library that you want to work with is not a first level dependency

To improve the situation, Gradle users can use composite builds to avoid publishing to a local repository. This makes the development cycle much faster already, by avoiding the publishToMavenLocal dance. However, there are some limitations with composite builds:

  • the included builds must be available locally, in a directory

  • it makes it hard, or even impossible, to setup on CI servers, unless you create an "integrating" project which hardcodes checkouts

  • it forces to change the configuration of the build to use composites

In addition, there’s a process problem with developing in a multi-repository environment: if the feature requires changes to multiple modules, in order to be able to integrate the changes, in particular on CI, you have to publish either snapshots or pre-releases. The problem is that this is not necessarily acceptable: for example you might want to develop a feature in a branch of each repository, and only merge once the full feature is ready.

Gradle also provides experimental support for source dependencies, but there are addressing a different problem. In particular, source dependencies are a replacement for regular dependencies: they require to change the dependency notation in builds with "source dependencies", including the branches. What we want to do, instead, is to keep our build files untouched, and substitute binary dependencies with sources.

This plugin provides a solution to this problem by allowing to include (in the sense of Gradle included builds) Git repositories, and specifying what branches/tags should be used.

If you’re looking for a synthetic view of pros and cons of each solution, please refer to this section in the docs.

Configuration

This plugin is a settings plugin which must be applied to your settings.gradle(.kts) file:

Applying the plugin
plugins {
    id 'me.champeau.includegit' version '0.1.1'
}
import me.champeau.gradle.igp.gitRepositories

plugins {
    id("me.champeau.includegit") version "0.1.1"
}

Declaring included Git repositories

The plugin defines a gitRepositories extension which is used to declare included Git repositories.

For example, say that you want to include jdoctor which is hosted at https://github.com/melix/jdoctor/, then you should write:

Including a Git repository
gitRepositories {
    include('jdoctor') {
        uri = 'git@github.com:melix/jdoctor.git'
        // optional, set what branch to use
        branch = 'feature1'
        // you can also use a tag
        tag = 'v1.0'
    }
}
gitRepositories {
    include("jdoctor") {
        uri.set("git@github.com:melix/jdoctor.git")
        // optional, set what branch to use
        branch.set("feature1")
        // you can also use a tag
        tag.set("v1.0")
    }
}

By default, the plugin will clone the included Git repositories in the checkouts directory of the project. If the repository is already cloned, the plugin will automatically perform an update every 24 hours. Alternatively, you can force it to update by adding -Drefresh.git.repositories to your Gradle command line.

Using local copies instead of cloning

Declaring included Git repositories will automatically make the plugin clone the remote repositories. This makes it very convenient to use on CI, since you will now be able to have branches which use other working branches from other repositories. However, it is likely that you already have local clones that you are already modifying and that you’d like to use for development. In this case, you can set the local.git.XXXX Gradle property, where XXXX is the included repository name, in your gradle.properties file, to point to your local copy. It is recommended to use the gradle.properties file located in your user home directory in this case:

local.git.jdoctor=/home/me/development/jdoctor
Note
if the property is found, the branch or tag configuration will be ignored.

Automatic local copies

Alternatively, you may have one or more directory with your checked out projects. In this case, the plugin provides a convenience which is going to automatically map directories to Git repository names. For this, you need to set the auto.include.git.dirs to the list of directories to scan. For example, say that you have:

/home/me
      └── development
          ├── gradle
          │ ├── foo-gradle-plugin
          │ └── gradle-core
          └── micronaut
              ├── micronaut-core
              └── micronaut-data

Then you can set this in your gradle.properties file:

auto.include.git.dirs=/home/me/development/gradle,/home/me/development/micronaut

The plugin will automatically scan the gradle and micronaut directories, and map the foo-gradle-plugin, gradle-core, micronaut-core and micronaut-data directories to potential included Git repositories. If a build is including a repository named micronaut-core, then it will automatically pick it from the micronaut-core directory.

This mechanism makes it extremely convenient to work with complex codebases with multiple Git repositories.

Authentication

The plugin supports 3 different authentication mechanisms:

  • basic authentication (username + password)

  • ssh with public key

  • ssh with password

Authentication can be configured per repository:

Configuring authentication per repository
gitRepositories {
    include('myrepo') {
        // ...
        authentication {
            basic {
                username = '...'
                password = '...'
            }
            // or
            sshWithPublicKey()
            // or
            sshWithPublicKey {
                privateKey = file("/path/to/private/key")
            }
            // or
            sshWithPassword {
                password = '...'
            }
        }
    }
}
gitRepositories {
    include("myrepo") {
        // ...
        authentication {
            basic {
                username.set("...")
                password.set("...")
            }
            // or
            sshWithPublicKey()
            // or
            sshWithPublicKey {
                privateKey.set(file("/path/to/private/key"))
            }
            // or
            sshWithPassword {
                password.set("...")
            }
        }
    }
}

It is also possible to configure a default authentication mechanism, which will be used when authentication isn’t configured specifically on a repository:

Configuring the default authentication mechanism
gitRepositories {
    defaultAuthentication {
        sshWithPublicKey()
    }
}
gitRepositories {
    defaultAuthentication {
        sshWithPublicKey()
    }
}

Comparison of solutions

This table summarizes some of the pros and cons of each solution, so that you can make a sound decision.

Snapshots Included builds Source dependencies This plugin

Works for transitive dependencies

No

Yes

No

Yes

Transparent to build scripts

No

Yes

No

Yes

Works consistently on CI and local

No

No

No

Yes

Handles cloning/checkout

No

No

Yes

Yes

Avoids publishing to artifact repository

No

No

No

Yes

Supports multiple branches

No

No

Yes

Yes

Works cross build tools

Yes

No

No

No

Supports same build tool, different versions

Yes

Depends on builds

Depends on builds

Depends on builds

Continous upstream testing

No

Manual

Depends on dependencies

Yes

Here’s a description of the different columns. This comparison is made for the multi-repository setup. It doesn’t mean that it would be the same, say, for a Gradle composite build living in a single repository:

  • Works for transitive dependencies: a build defines "direct" dependencies, which are typically used directly in source code, but often what you need to test is a transitive dependency. This column indicates if the solution makes it possible to substitute a transitive dependency with sources, transparently

  • Transparent to build scripts: some solutions, typically SNAPSHOTS, require changes to build scripts because you need to introduce mavenLocal, put a particular version, or introduce a first level dependency so that the changes are visible. Other solutions like this plugin only require applying the plugin, but leave your dependency declarations untouched.

  • Works consistently on CI and local: does the technical solution works consistently locally and on CI? Snapshots are the typical example of things which are hard to reason about because the local Maven repo may contain different dependencies than the remote snapshot repository. It also requires sync’ing and refreshing dependencies. Other solutions like composite builds work well for local development, but break as soon as you push on CI because the local repositories wouldn’t be available.

  • Handles cloning/checkout: does the solution handle checking out (or cloning in Git terminology) the dependency for you? Will it make the dependency visible as sources in your IDE?

  • Avoids publishing to artifact repository: Snapshots typically require publishing artifacts to a binary repository, or local file system, for other builds to "see" the changes. Some solutions like included builds do not, since they handle the dependency using sources instead.

  • Supports multiple branches: Snapshots work well, except when you need to integrate changes from different branches: either you have to publish different artifacts with different coordinates or versions to be able to test them in downstream projects, or you have to merge changes and push a snapshot. On the contrary, source dependencies handle branches gracefully because they don’t require any publication to a binary repository.

  • Works cross build tools: Snapshots can be consumed from different build tools, typically both Maven and Gradle. Source dependencies, included builds and this plugin require all participating builds to use Gradle and therefore are not suitable if you have a mix of build tools.

  • Supports same build tool, different versions: Snapshots are binary dependencies so the build tool which was used doesn’t matter. Included builds and source dependencies will use the version of the build tool which includes the other builds as the "driver". If there are incompatibilities between versions of the main build and the included ones, builds may fail.

  • Continous upstream testing: Does the solution make it possible to continuously test upstream dependencies? Typically, without changing your build scripts, it would be nice if you could test that the project is compatible with the latest master branch of a dependency. This plugin makes it quite simple to implement, while included builds require some manual setup. Snapshots won’t help.

Known limitations

The plugin won’t work for plugin substitutions (e.g includeBuild in the pluginManagement section).