Dependency Management with Gradle

Part 2 - Handling Conflicts and Customizing Resolution

Cédric Champeau (@CedricChampeau) & Louis Jacomet (@ljacomet) - Gradle Inc.

Who are we?

who s who

Dependency management team

Declaring and Analyzing Dependencies in Gradle

Including a quick recap of

Dependency Management with Gradle Part 1 – Fundamentals

Gradle Build Tool

Build and automation tool with rich dependency management functionality

  • JVM based and implemented in Java

  • 100% Free Open Source - Apache Standard License 2.0

  • Agnostic build tool (Java, Groovy, Kotlin, Scala, Android, C, C++, Swift, …​)

plugins {
    `java-library`
}
repositories {
    mavenCentral()
}
dependencies {
    implementation("org.slf4j:slf4j-api:1.7.2")
}

Build Scans and Gradle Enterprise

scan general

Modules live in Repositories

  • Identified by group, name and version (e.g. org.slf4j:slf4j-api:1.7.2)

  • Consists of artifacts (e.g. .jar) and metadata (.module, .pom or ivy.xml)

  • Repositories are visited sequentially

Inspecting dependencies

By default transitive dependencies are brought in

dependencies {
    implementation("org.apache.httpcomponents:httpclient:4.5.9")
}

Show dependency graph:

gradle dependencies --configuration compileClasspath

compileClasspath - Compile classpath for source set 'main'.
\--- org.apache.httpcomponents:httpclient:4.5.9
     +--- org.apache.httpcomponents:httpcore:4.4.11
     +--- commons-logging:commons-logging:1.2
     \--- commons-codec:commons-codec:1.11

Focusing on a dependency

Show path to a dependency:

gradle dependencyInsight
             --configuration compileClasspath
             --dependency codec

Result:

...
commons-codec:commons-codec:1.11
\--- org.apache.httpcomponents:httpclient:4.5.9
     \--- compileClasspath
...

Inspect dependencies in build scan

scans.gradle.com (--scan) or Gradle Enterprise provide a dependency inspector

scan httpclient

Dependency constraints

  • Introduce additional constraints on versions

  • Does not override versions of transitive dependencies

dependencies {
    constraints {
        implementation("org.slf4j:slf4j-api:1.7.26")
        implementation("org.apache:commons-lang3:3.3.0")
    }
    dependencies {
        implementation("org.slf4j:slf4j-api") // no version
    }
}

Rich dependency versions

  • Work also with dependency constraints

  • Does not override versions of transitive dependencies

dependencies {
   implementation("info.picocli:picocli") {
       version {
          strictly("[3.9, 4.0[")
          prefer("3.9.5")
       }
       because("Provides command-line argument parsing")
   }
}

Platforms to share versions

platform/build.gradle.kts
plugins {
   `java-platform`
}

dependencies {
    constraints {
       api("org.slf4j:slf4j-api:1.7.26")
       runtime("org.slf4j:slf4j-simple:1.7.26")
    }
}
cli/build.gradle.kts
dependencies {
   api(platform(project(":platform")))

   implementation("org.slf4j:slf4j-api") // <-- no version here
}

Semantics and dependencies

Gradle has a strong modelling of dependencies:

  • Semantic difference between compilation and runtime

  • Semantic difference between building a library and building against a library

  • Ability for a module to produce more than one variant

Understanding conflict resolution

Different kinds of conflicts

  • Version conflicts

    • 2 or more dependencies want a different version of the same thing

  • Implementation conflicts

    • there are more than one implementation of the same thing

Definition of version conflict

There’s a version conflict whenever two components:

  • depend on the same module D

  • depend on different versions of module D

For example:

  • A → D:1.0

  • B → D:1.1

Maven vs Gradle

asciidoctor diagram conflict1

  • Maven selects version 20.0

  • Gradle selects version 25.1-android

Why?

Maven’s Nearest first

  • "Nearest" dependency version wins

  • root → guava:20.0 which is closer than root → guice → guava:25.1-android

  • So 20.0 wins

Problems with nearest first

Nearest is order dependent!

asciidoctor diagram maven1

Selects guava:25.1-android

Nearest is order dependent!

asciidoctor diagram maven2

Selects guava:20

Nearest first is basically unpredictable for any reasonably sized dependency graph

Typical Maven workarounds

  • Adding exclusions + first level dependency

    • leaks dependencies

  • Adding a dependencyManagement block

    • but non transitive!

How Gradle works

  • Performs full conflict resolution

  • All opinions matter

  • Highest wins: we always choose 25.1-android whatever the order

Gradle dependency constraints

dependencies {
   constraints {
      implementation("com.google.guava:guava:guava:25.1-android")
   }
}

are honored transitively

Ivy conflict resolution

  • Ivy uses pluggable conflict resolvers

  • can be different per dependency

  • hard to reason about

Handling conflict resolution in Gradle

Failing on version conflict

configurations {
   compileClasspath {
       resolutionStrategy.failOnVersionConflict()
   }
}

  • Forces you to think about conflicts

  • Requires an explicit way to select a version

Dependency resolution rules

  • Offer one way to solve conflicts

  • Limited to the configuration being resolved

configurations.all {
    resolutionStrategy.eachDependency {
        if (requested.group == "org.apache.commons" &&
            requested.name == "commons-lang3") {
            useVersion("3.3.1")
            because("tested with this version")
        }
    }
}

Demo

When metadata is wrong

  • Sometimes metadata of a module is simply wrong

    • "hard" dependency when it should be optional

    • dependency on broken version

    • missing dependencies

    • …​

Component metadata rules

  • Allow fixing metadata

  • Applied to all configurations (unconditionally true)

e.g: DeltaSpike container control dependency in Camel CDI should be optional

fun DependencyHandler.fixBadMetadata() = components {
    withModule("org.apache.camel:camel-cdi", FixApacheCamel::class.java)
}

class FixApacheCamel: ComponentMetadataRule {
    override fun execute(context: ComponentMetadataContext) = context.details.run {
        if (id.version.startsWith("2.15")) {
            allVariants {
                withDependencies {
                    removeAll { it.group.contains("deltaspike") }
                }
            }
        }
    }
}

Handling mutually exclusive modules

  • A capability describes a feature that is mutually exclusive

  • All components provide their GAV as a capability by default

The famous logging problem

  • Slf4j and is bridging / replacement modules

  • log4j vs. slf4j-log4j12 vs. log4j-over-slf4j

  • And similarly for java.util.logging

  • And again for commons-logging

Relocated libraries

  • asm:asm now known as org.ow2.asm:asm

  • junit:junit-dep and junit:junit

  • cglib:cglib-nodep and cglib:cglib

Detecting conflicts

fun DependencyHandler.detectLoggerConflicts() = components {
    // both commons-logging and jcl-over-slf4j implement the commons-logging api
    withModule("org.slf4j:jcl-over-slf4j", CommonLoggingCapabilities::class.java)
    withModule("org.apache.commons:commons-logging", CommonLoggingCapabilities::class.java)

    // both slf4j-simple and slf4j-log4j12 implement the slf4j api
    withModule("org.slf4j:slf4j-simple", Slf4jImplementationCapabilities::class.java)
    withModule("org.slf4j:slf4j-log4j12", Slf4jImplementationCapabilities::class.java)

    // ...

Detecting conflicts

class CommonLoggingCapabilities : ComponentMetadataRule {
    override fun execute(context: ComponentMetadataContext) = context.run {
        details.allVariants {
            withCapabilities {
                addCapability("org.apache.commons", "commons-logging-impl", "1.0")
            }
        }
    }

}

class Slf4jImplementationCapabilities : ComponentMetadataRule {
    override fun execute(context: ComponentMetadataContext) = context.run {
        details.allVariants {
            withCapabilities {
                addCapability("org.slf4j", "slf4j-impl", "1.0")
            }
        }
    }
}

Selecting a module over another one

  • Uses dependency substitution

  • This effectively replaces one module with another

fun Configuration.preferSlf4JSimple() = resolutionStrategy.dependencySubstitution {
    substitute(module("org.slf4j:slf4j-log4j12"))
            .because("prefer slf4j-simple over other implementations")
            .with(moduleInGraph("org.slf4j:slf4j-simple"))

    substitute(module("log4j:log4j"))
            .because("use the slf4j bridge for log4j")
            .with(module("org.slf4j:log4j-over-slf4j:1.7.16"))
}

fun Configuration.useSlf4jEverywhere() = resolutionStrategy.eachDependency {
    if (requested.name == "commons-logging") {
        useTarget("org.slf4j:jcl-over-slf4j:0")
    }
}

Excluding dependencies

What is it?

  • An option for consumers to prune transitive sections of the graph

  • Maven and Gradle have different handling of exclusions

Maven exclusion

  • With Maven if a dependency is excluded on a given edge, it is excluded for the whole graph

asciidoctor diagram process3

C is excluded in Maven

Gradle exclude

  • With Gradle all paths need to exclude for a dependency to be excluded

asciidoctor diagram process4

C is excluded

Configuration level exclude

  • A dependency is many times and deep into the dependency graph and you don’t need it

  • Exclude it on all paths

configurations.all {
   exclude group:'commons-logging', module:'commons-logging'
}

However:

  • Leaks to downstream consumers

  • Even worse: added to all first level dependencies at publication

When to use excludes: last resort

  • Semantics should be: "I don’t use this library that this module says it needs"

  • Should be limited to cases where the module actually needs it but your execution paths don’t

    implementation("org.apache.spark:spark-core_2.12:2.4.3") {
        withoutFeatureWhichRequires("org.apache.ivy", "ivy")
    }

// ...

fun ModuleDependency.withoutFeatureWhichRequires(group: String, module: String) {
    exclude(mapOf("group" to group, "module" to module))
    because("we don't use the features of ${name} which require ${group}:${module}")
}

A word about forcing

  • Forcing dependencies is not recommended

  • Behavior can be non reproducible (ordering issues in case of competing forces)

  • Use it only if you don’t have consumers

  • But prefer resolution rules

Disabling transitive resolution

  • Sometimes you don’t care about transitivity

  • e.g: download a set of files

configurations {
   myFiles {
      transitive = false
   }
}

// ...

tasks {
   downloadFiles {
      from configurations["myFiles"]
   }
}

Transitive version conflict and consequences

Understanding the consequences of decisions

  • 2 categories of consumers

    • libraries: can be consumed, including transitively

    • applications: at the end of the chain

Understanding the consequences of decisions

  • Wrong decisions on libraries affect the whole chain

    • e.g: excluding a dependency

    • e.g: forcing a version

    • e.g: adding a constraint

    • e.g: using ranges

Improving graph consistency

The challenge?

  • Dependencies do not work in isolation

    • Most projects will use more than a couple

    • Having a consistent set of dependencies is a different problem than having the right dependency at the right version

  • Organizations may want to

    • Share information on recommended dependencies, including versions and rejections

    • Control available dependencies and their origin

Multiple options available

  • Repository content filtering

  • Bill Of Materials (BOM) / Recommendation platforms

  • Alignment of library modules

  • and more

Repository content filtering

Modules live in repositories

  • Identified by group, name and version (e.g. org.slf4j:slf4j-api:1.7.2)

  • Consists of artifacts (e.g. .jar) and metadata (.module, .pom or ivy.xml)

  • Repositories are visited sequentially

Content filtering

  • Gives control to where dependencies can be sourced from

  • Improves the reproducibility aspect

  • Has security implications as well

repositories {
    maven {
        url = 'https://repo.company.com/'
        content {
            includeGroupByRegex​ "my\\.company.*"
        }
    }
    jcenter() {
         content {
             excludeGroup "org.slf4j"
             excludeGroupByRegex​ "my\\.company.*"
         }
    }
}

Bill Of Materials support in Gradle

What is a Maven BOM?

  • Defines versions for dependencies (and more)

  • Used for obtaining a consistent set of versions

    • For modules composing a library, that work better together at the same version

    • For heterogeneous modules that form an ecosystem

  • Allows to omit versions in the build file that imports it

Example

Importing a BOM in Gradle

dependencies {
    implementation(
        platform(
            'org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE'
        )
    )
}

Particularities in Gradle

  • BOM dependency declarations are imported as constraints

  • BOMs are like regular dependencies, they can be visible by consumers

    • When declared in api for example

  • Gradle always considers all versions in a dependency graph

This creates an important difference with application of BOMs in Maven:

In Gradle, the transitive dependency graph can still resolve a dependency with a different version than specified in the BOM.

Enforced BOMs

dependencies {
    implementation(
        enforcedPlatform(
            'org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE'
        )
    )
}

Will result in constraints that force versions instead of participating in version conflict resolution.

If you have consumers, you should not use this.

Replacement of Spring dependency management plugin

  • ✅ for most use cases

  • Unsupported scenarios:

    • If you want to override properties defined in the BOM to have a different set of constraints

    • Local dependency declaration causing a downgrade

Recommendation platforms in Gradle

  • By using the java-platform plugin

  • Full feature set requires Gradle Module Metadata publication

  • Will translate to a Maven BOM for the most part

    • ⚠️ In Maven order matters, so order in your Gradle build file matters if you need the compatibility

Recommendation platform example

plugins {
  `java-platform`
}

javaPlatform {
    allowDependencies()
}

dependencies {
  constraints {
      runtime('org.postgresql:postgresql:42.2.6')
  }
  api(platform('com.fasterxml.jackson:jackson-bom:2.9.9'))
  api(platform('org.springframework.boot:spring-boot-dependencies:2.1.5.RELEASE'))
}

Aligning a group of dependencies

What is the problem?

  • A group of dependencies are designed to work together.

    • Think jackson or even the Spring framework

  • Through transitive dependency updates, modules end up having different versions

So the goal would be to have the tools to make sure that any upgrade of any module would cause the whole set to upgrade its version.

Demo

Virtual platforms for published libraries

  1. Name a virtual platform module

  2. Enhance library modules metadata to declare they belongsTo the platform

  3. The platform collects the modules that belongs to it

    • When jackson-dataformat-yaml says it belongs to the platform, all platform versions now point to it as well

  4. Alignment!

Usability details

  • A virtual platform can only align to a version it sees in the graph

  • A virtual platform will attempt alignment on all modules

Attempted alignment example

  • com.fasterxml.jackson.core:jackson-databind:2.8.8.1 ✅

  • com.fasterxml.jackson.core:jackson-core:2.8.8.1 ❌

  • The platform will align modules existing in 2.8.8.1 to that version,

  • Other modules will have the highest lower version that exists for them

    • Saw 2.8.8.1, 2.8.8, …​ then others most likely in 2.8.8

    • Saw 2.8.8.1, 2.7.8, …​ then others mostl likely in 2.7.8

Can I leverage an existing BOM?

  • Yes, but

    • You still need the belongsTo

    • Alignment will be according to the BOM definitions

Using a Gradle platform for alignment

  • Leverages the ability to declare the cycle in Gradle

  • A platform depends on all the modules

  • Each modules depends on the platform

  • Requires Gradle Module Metadata to be fully functional

Alignment platform

plugins {
  `java-platform`
}

dependencies {
    constraints {
        api(project(':common'))
        api(project(':server'))
        api(project(':client'))
    }
}

For the projects:

dependencies {
    api(project(':platform'))
}

Conclusion

Tools for a complex world

  • Real world projects will face dependency conflicts

  • Gradle provides tools to model conflicts

    • not all conflicts are equal (version, implementation, …​)

    • better modeling → better decisions

  • Sometimes a hammer is an acceptable solution

We’re adding tooling to cover more cases!

Coming next

Focus on publishing and the importance of Gradle Module Metadata

  • Publishing components

  • Maven and Ivy compatibility

  • Published capabilities

  • Published alignment

  • Feature variants

  • Test fixtures