Included Git repositories plugin for Gradle

This plugin adds support for including Git repositories as source dependencies in Gradle builds, making multi-repository development much easier to deal with!

This plugin has been successfully tested on Gradle 7.1.1, 7.2, 7.3, 8.11. Earlier releases are not supported. The following Gradle versions are known to be broken with this plugin: 6.0.x, 6.1.x.

Rationale

Multi-repository development, while providing architectural advantages (reduced scope of libraries, faster development cycles, …), are often painful for developers, especially when the number of modules being involved grows.

For example, to develop a single feature, a developer may have to work on more than one repository at a time: a core module, living in repository core, and a library module depending on core, living in a repository library.

The problem is that the development of a particular feature may require changes to both core and library. In that case, developers, in the Java ecosystem, typically rely on publishing snapshots in their local Maven repository: make a change in core, publish a snapshot, then make changes to library, publish a snapshot, and repeat the loop.

There are several problems with this approach:

it requires switching between projects, publishing intermediate artifacts to the local filesystem
makes it difficult to collaborate with others as they also need to checkout several projects
it requires integrating mavenLocal() as a repository for your local builds, which is considered a bad practice (because it introduces non-reproducibility and makes the builds brittle)
for CI, it requires publishing to a snapshot repository for downstream builds to pick up the changes, meaning that you will often have to merge work-in-progress just to be able to test changes
it simply doesn’t work if the library that you want to work with is not a first level dependency

To improve the situation, Gradle users can use composite builds to avoid publishing to a local repository. This makes the development cycle much faster already, by avoiding the publishToMavenLocal dance. However, there are some limitations with composite builds:

the included builds must be available locally, in a directory
it makes it hard, or even impossible, to setup on CI servers, unless you create an "integrating" project which hardcodes checkouts
it forces to change the configuration of the build to use composites

In addition, there’s a process problem with developing in a multi-repository environment: if the feature requires changes to multiple modules, in order to be able to integrate the changes, in particular on CI, you have to publish either snapshots or pre-releases. The problem is that this is not necessarily acceptable: for example you might want to develop a feature in a branch of each repository, and only merge once the full feature is ready.

Gradle also provides experimental support for source dependencies, but there are addressing a different problem. In particular, source dependencies are a replacement for regular dependencies: they require to change the dependency notation in builds with "source dependencies", including the branches. What we want to do, instead, is to keep our build files untouched, and substitute binary dependencies with sources.

This plugin provides a solution to this problem by allowing to include (in the sense of Gradle included builds) Git repositories, and specifying what branches/tags should be used.

If you’re looking for a synthetic view of pros and cons of each solution, please refer to this section in the docs.

Configuration

Warning

This plugin is a settings plugin which must be applied to your settings.gradle(.kts) file, not to a project build.gradle(.kts) file.

Applying the plugin

plugins {
    id 'me.champeau.includegit' version '0.2.0'
}

import me.champeau.gradle.igp.gitRepositories

plugins {
    id("me.champeau.includegit") version "0.2.0"
}

Declaring included Git repositories

The plugin defines a gitRepositories extension which is used to declare included Git repositories.

For example, say that you want to include jdoctor which is hosted at https://github.com/melix/jdoctor/, then you should write:

Including a Git repository

gitRepositories {
    include('jdoctor') {
        uri = 'git@github.com:melix/jdoctor.git'
        // optional, set what branch to use
        branch = 'feature1'
        // you can also use a tag
        tag = 'v1.0'
    }
}

gitRepositories {
    include("jdoctor") {
        uri.set("git@github.com:melix/jdoctor.git")
        // optional, set what branch to use
        branch.set("feature1")
        // you can also use a tag
        tag.set("v1.0")
    }
}

By default, the plugin will clone the included Git repositories in the checkouts directory of the project. If the repository is already cloned, the plugin will automatically perform an update every 24 hours. Alternatively, you can force it to update by adding -Drefresh.git.repositories to your Gradle command line.

Using local copies instead of cloning

Declaring included Git repositories will automatically make the plugin clone the remote repositories. This makes it very convenient to use on CI, since you will now be able to have branches which use other working branches from other repositories. However, it is likely that you already have local clones that you are already modifying and that you’d like to use for development. In this case, you can set the local.git.XXXX Gradle property, where XXXX is the included repository name, in your gradle.properties file, to point to your local copy. It is recommended to use the gradle.properties file located in your user home directory in this case:

local.git.jdoctor=/home/me/development/jdoctor

Note	if the property is found, the `branch` or `tag` configuration will be ignored.

Automatic local copies

Alternatively, you may have one or more directory with your checked out projects. In this case, the plugin provides a convenience which is going to automatically map directories to Git repository names. For this, you need to set the auto.include.git.dirs to the list of directories to scan. For example, say that you have:

/home/me
      └── development
          ├── gradle
          │ ├── foo-gradle-plugin
          │ └── gradle-core
          └── micronaut
              ├── micronaut-core
              └── micronaut-data

Then you can set this in your gradle.properties file:

auto.include.git.dirs=/home/me/development/gradle,/home/me/development/micronaut

The plugin will automatically scan the gradle and micronaut directories, and map the foo-gradle-plugin, gradle-core, micronaut-core and micronaut-data directories to potential included Git repositories. If a build is including a repository named micronaut-core, then it will automatically pick it from the micronaut-core directory.

This mechanism makes it extremely convenient to work with complex codebases with multiple Git repositories.

Configuring the included build

By default, the root directory of the cloned repository will be automatically included. You can tweak the configuration of the included build by calling the includeBuild method of the gitRepositories extension:

Configuring the included build

gitRepositories {
    include('jdoctor') {
        uri = 'git@github.com:melix/jdoctor.git'
        includeBuild {
            name = 'other-name'
        }
    }
}

gitRepositories {
    include("jdoctor") {
        uri.set("git@github.com:melix/jdoctor.git")
        includeBuild {
            name = "other-name"
        }
    }
}

Including sub-directories instead of the root

In some cases, the root directory of the cloned project may not be the directory you want to include, or you may want to include several sub-directories as separate included builds. For this purpose, you can use the includeBuild statement which works exactly like Gradle’s includeBuild, except that the root directory is the checked out directory:

Including a sub-directory of a Git repository

gitRepositories {
    include('jdoctor') {
        uri = 'git@github.com:melix/jdoctor.git'
        // This will include the "build-logic" directory of the repository
        // instead of the whole project
        includeBuild 'build-logic'
    }
}

gitRepositories {
    include("jdoctor") {
        uri.set("git@github.com:melix/jdoctor.git")
        // This will include the "build-logic" directory of the repository
        // instead of the whole project
        includeBuild("build-logic")
    }
}

Note	You may use several `includeBuild` statements from a single repository.

Authentication

The plugin supports 3 different authentication mechanisms:

basic authentication (username + password)
ssh with public key
ssh with password

Authentication can be configured per repository:

Configuring authentication per repository

gitRepositories {
    include('myrepo') {
        // ...
        authentication {
            basic {
                username = '...'
                password = '...'
            }
            // or
            sshWithPublicKey()
            // or
            sshWithPublicKey {
                privateKey = file("/path/to/private/key")
            }
            // or
            sshWithPassword {
                password = '...'
            }
        }
    }
}

gitRepositories {
    include("myrepo") {
        // ...
        authentication {
            basic {
                username.set("...")
                password.set("...")
            }
            // or
            sshWithPublicKey()
            // or
            sshWithPublicKey {
                privateKey.set(file("/path/to/private/key"))
            }
            // or
            sshWithPassword {
                password.set("...")
            }
        }
    }
}

It is also possible to configure a default authentication mechanism, which will be used when authentication isn’t configured specifically on a repository:

Configuring the default authentication mechanism

gitRepositories {
    defaultAuthentication {
        sshWithPublicKey()
    }
}

gitRepositories {
    defaultAuthentication {
        sshWithPublicKey()
    }
}

Configuring checkout directories

The plugin supports 2 different ways to configure the checkout directory:

either by configuring the root directory where all repositories are going to be checked out (by default, checkouts)

Configuring the root checkout directory

gitRepositories {
    checkoutsDirectory.set(file('.'))
}

gitRepositories {
    checkoutsDirectory.set(file("."))
}

or by configuring a checkout directory per included repository

Configuring the root checkout directory

gitRepositories {
    include('myrepo') {
        // ...
        checkoutDirectory = file('lib')
    }
}

gitRepositories {
    include("myrepo") {
        // ...
        checkoutDirectory.set(file("lib"))
    }
}

Performing actions before the build is included

It is possible to perform actions right after a project has been cloned and before it is included. This can be useful, for example, if the project needs to be analyzed in order to properly configure the included builds.

The action will always be called, even if sources are already available locally. It is not recommended to mutate sources as part of this callback, since this will likely break updating the sources later.

Executing code before the build is included

gitRepositories {
    include('my-project') {
    // ...
        codeReady { event ->
            println("Project cloned in ${event.checkoutDirectory}")
        }
    }
}

gitRepositories {
    include("my-project") {
        // ...
        codeReady {
            println("Project cloned in ${checkoutDirectory}")
        }
    }
}

Comparison of solutions

This table summarizes some of the pros and cons of each solution, so that you can make a sound decision.

	Snapshots	Included builds	Source dependencies	This plugin
Works for transitive dependencies	No	Yes	No	Yes
Transparent to build scripts	No	Yes	No	Yes
Works consistently on CI and local	No	No	No	Yes
Handles cloning/checkout	No	No	Yes	Yes
Avoids publishing to artifact repository	No	No	No	Yes
Supports multiple branches	No	No	Yes	Yes
Works cross build tools	Yes	No	No	No
Supports same build tool, different versions	Yes	Depends on builds	Depends on builds	Depends on builds
Continous upstream testing	No	Manual	Depends on dependencies	Yes

Snapshots

Included builds

Source dependencies

This plugin

Works for transitive dependencies

Yes

Transparent to build scripts

Yes

Works consistently on CI and local

Yes

Handles cloning/checkout

Yes

Avoids publishing to artifact repository

Yes

Supports multiple branches

Yes

Works cross build tools

Yes

Supports same build tool, different versions

Yes

Depends on builds

Continous upstream testing

Manual

Depends on dependencies

Yes

Here’s a description of the different columns. This comparison is made for the multi-repository setup. It doesn’t mean that it would be the same, say, for a Gradle composite build living in a single repository:

Works for transitive dependencies: a build defines "direct" dependencies, which are typically used directly in source code, but often what you need to test is a transitive dependency. This column indicates if the solution makes it possible to substitute a transitive dependency with sources, transparently
Transparent to build scripts: some solutions, typically SNAPSHOTS, require changes to build scripts because you need to introduce mavenLocal, put a particular version, or introduce a first level dependency so that the changes are visible. Other solutions like this plugin only require applying the plugin, but leave your dependency declarations untouched.
Works consistently on CI and local: does the technical solution works consistently locally and on CI? Snapshots are the typical example of things which are hard to reason about because the local Maven repo may contain different dependencies than the remote snapshot repository. It also requires sync’ing and refreshing dependencies. Other solutions like composite builds work well for local development, but break as soon as you push on CI because the local repositories wouldn’t be available.
Handles cloning/checkout: does the solution handle checking out (or cloning in Git terminology) the dependency for you? Will it make the dependency visible as sources in your IDE?
Avoids publishing to artifact repository: Snapshots typically require publishing artifacts to a binary repository, or local file system, for other builds to "see" the changes. Some solutions like included builds do not, since they handle the dependency using sources instead.
Supports multiple branches: Snapshots work well, except when you need to integrate changes from different branches: either you have to publish different artifacts with different coordinates or versions to be able to test them in downstream projects, or you have to merge changes and push a snapshot. On the contrary, source dependencies handle branches gracefully because they don’t require any publication to a binary repository.
Works cross build tools: Snapshots can be consumed from different build tools, typically both Maven and Gradle. Source dependencies, included builds and this plugin require all participating builds to use Gradle and therefore are not suitable if you have a mix of build tools.
Supports same build tool, different versions: Snapshots are binary dependencies so the build tool which was used doesn’t matter. Included builds and source dependencies will use the version of the build tool which includes the other builds as the "driver". If there are incompatibilities between versions of the main build and the included ones, builds may fail.
Continous upstream testing: Does the solution make it possible to continuously test upstream dependencies? Typically, without changing your build scripts, it would be nice if you could test that the project is compatible with the latest master branch of a dependency. This plugin makes it quite simple to implement, while included builds require some manual setup. Snapshots won’t help.

Known limitations

The plugin won’t work for plugin substitutions (e.g includeBuild in the pluginManagement section).