Feature Spotlight: Incremental Builds

In this month’s installment of the feature spotlight, I would like to discuss how to speed up your build by giving Gradle enough information to perform incremental builds.

Task Inputs, Outputs, and Dependencies

Built-in tasks, like JavaCompile, declare a set of inputs (Java source files) and a set of outputs (class files). Gradle uses this information to determine if a task is up-to-date and needs to perform any work. If none of the inputs or outputs have changed, Gradle can skip that task. We call this kind of build an “incremental build”.

To take advantage of the incremental build support, you need to provide Gradle with information about your task’s inputs and outputs. It is possible to configure a task to only have outputs. Before executing the task, Gradle checks the outputs and will skip execution of the task if the outputs have not changed. In real builds, a task usually has inputs — source files, resources, and properties. Gradle checks that neither the inputs nor outputs have changed before executing a task.

Often a task’s outputs will serve as the inputs to another task. It is important to get the ordering between these tasks correct, or the tasks will run in the wrong order or not at all. Gradle does not rely on the order that tasks are defined in the build script. New tasks are unordered, therefore execution order can change from build to build. You can explicitly tell Gradle about the order between two tasks by using a dependency (consumer.dependsOn producer).

 

Declaring Explicit Task Dependencies

Let us take a look at an example project that contains a common pattern. For this project, we need to create a zip file that contains the output from a generator task. The manner in which the generator task creates files is not interesting–it produces files that contain an incrementing number.

build.gradle

apply plugin: 'base'

task generator() {
    doLast {
        def generatedFileDir = file(“$buildDir/generated”)
        generatedFileDir.mkdirs()
        for (int i=0; i<10; i++) {
            new File(generatedFileDir, “${i}.txt”).text = i
        }
    }
}
task zip(type: Zip) {
    dependsOn generator
    from “$buildDir/generated”
}

The build _works_, but the build script has some issues. The output directory for the generator task is repeated in the zip task, and dependencies of the zip task are explicitly set with `dependsOn`. Gradle appears to execute the generator task each time, but not the zip task. This is a good time to point out that Gradle’s up-to-date checking is different from other tools, such as Make. Gradle compares the checksum of the inputs and outputs instead of only the timestamp of the files. Even though the generator task runs each time and overwrites all of its output files, the content does not change and the zip task does not need to run again. The checksum of the zip task’s inputs have not changed. Skipping up-to-date tasks lets Gradle avoid unnecessary work and speeds up the development feedback loop.

Declaring Task Inputs and Outputs

Now, let us understand why the generator task seems to run every time. If we take a look at Gradle’s –info level logging, we will see the reason:

Executing task ‘:generator’ (up-to-date check took 0.0 secs) due to:
Task has not declared any outputs.
We can see that Gradle does not know that the task produces any output. By default, if a task does not have any outputs, it must be considered out of date.  Outputs are declared with TaskOutputs. Task outputs can be files or directories.

build.gradle

task generator() {
    def generatedFileDir = file("$buildDir/generated")
    outputs.dir generatedFileDir
    doLast {
        generatedFileDir.mkdirs()
        for (int i=0; i<10; i++) {
            new File(generatedFileDir, "${i}.txt").text = i
        }
    }
}

If we run the build two more times, we will see that the generator task says it is up-to-date after the first run. We can confirm this if we look at the `--info` output again: > Skipping task ‘:generator’ as it is up-to-date (took 0.007 secs). But we have introduced a new problem. If we increase the number of files generated (say, from 10 to 20), the generator task does not re-run. We could work around this by doing a clean build each time we need to change that parameter, but this workaround is error-prone.

We can tell Gradle what can impact the generator task and requires rebuilds. We can use TaskInputs to declare certain properties as inputs to the task as well as input files. If any of these inputs change, Gradle will know to execute the task.

build.gradle

task generator() {
    def fileCount = 10
    inputs.property "fileCount", fileCount
    def generatedFileDir = file("$buildDir/generated")
    outputs.dir generatedFileDir
    doLast {
        generatedFileDir.mkdirs()
        for (int i=0; i<fileCount; i++) {
            new File(generatedFileDir, "${i}.txt").text = i
        }
    }
}

We can check this by examining the `--info` output after we change the value of fileCount: > Executing task ‘:generator’ (up-to-date check took 0.007 secs) due to: > Value of input property ‘fileCount’ has changed for task ‘:generator’

Inferring Task Dependencies

So far, we have only worked on the generator task, but we have not reduced any of the repetition we have in the build script. We have an explicit task dependency and a duplicated output directory path. Let us try removing the task dependencies by relying on how CopySpec.from() evaluates arguments with Project.files(). Gradle can automatically add task dependencies for us. This also adds the output of the generator task as inputs to the zip task.

build.gradle

task zip(type: Zip) {
    from generator
}

Inferred task dependencies can be easier to maintain than explicit task dependencies when there is a strong producer-consumer relationship between tasks. When you only need some of the output from another task, explicit task dependencies will usually be cleaner. There is nothing wrong with using both explicit task dependencies and inferred dependencies, if that is easier to understand.

Simplifying with a Custom Task

We call tasks like generator “ad-hoc” tasks. They do not have well-defined properties nor predefined actions to perform. It is okay to use ad-hoc tasks to perform simple actions, but a better practice is to move ad-hoc tasks intocustom task classes. Custom tasks let you remove a lot of boilerplate and standardize common actions within your build.

Gradle makes it really easy to add new task types. You can start playing around with custom task types directly in your build file. When using annotations like @OutputDirectory, Gradle will create output directories before your task executes, so you do not have to worry about making the directories yourself. Other annotations, like@Input and @InputFiles, have the same effect as manually configuring a task’s TaskInputs.

Try creating a custom task class named Generate that produces the same output as the generator task above. Your build file should look like the following:

build.gradle

task generator(type: Generate) {
    fileCount = 20
}

task zip(type: Zip) {
    from generator
}

Here is our solution:

build.gradle

class Generate extends DefaultTask {
    @Input
    int fileCount = 10
    @OutputDirectory
    File generatedFileDir = project.file("${project.buildDir}/generated")
    @TaskAction
    public void perform() {
        for (int i=0; i<fileCount; i++) {
            new File(generatedFileDir, "${i}.txt").text = i
        }
    }
}

Notice that we no longer need to create the output directory manually. The annotation on `generatedFileDir`takes care of this for us. The annotation on `fileCount `tells Gradle that this property should be considered an input in the same way we used `inputs.property` before. Finally, the annotation on `perform()` defines the action for `Generate `tasks.

Final Notes about Incremental Builds

When developing your own build scripts, plugins and custom tasks, declaring task inputs and outputs is an important technique to keep in your toolbox. All of the core Gradle tasks use this to great effect.  If you would like to learn about other ways to make your tasks incremental at a lower level, take a look at the incubating incremental task support.  Incremental tasks provide a fine-grained way of building only what has changed when a task needs to execute.