4-5: Caching Mechanism

Reusing existing compile caches has a significant benefit for accelerating compilation speed. This section will discuss the caching and incremental build mechanisms based on Gradle's two main lifecycles. Specifically, the first-layer cache includes:

1. Caching during the Execution phase operates at the Task level, which consists of the Up-to-Date Check and the Incremental Build.
2. Caching during the Configuration phase, namely the Configuration Cache, a new feature introduced from Gradle 6.6.

These caches are stored within the current Project directory by default (.gradle/ and build/).

A second-layer cache, known as the Build Cache, resides under the user directory at .gradle/caches.

The above three caches form a caching system closely related to the Gradle Plugin development, which is the focal point of our discussion. There are other caches you might be familiar with:

1. The Android Gradle Plugin (AGP) had previously implemented its Task cache, but it was removed from AGP 4.1 onward.
2. The Gradle Daemon can cache some build containers and libraries, transforming cold starts into hot starts. However, this isn't a primary concern for Plugin developers.
3. Gradle's dependency cache is outside the scope of this book, which predominantly discusses Plugins development within the Android ecosystem.

We will not delve deeper into them in this section. Figure 4.5.1 is a diagram offering an overview of Gradle caches. The parts marked with an asterisk will be our main focus.

Gradle searches for caches from left to right, as indicated by the arrows in the top-right area of the diagram. Note that the Build Cache is designed to include local storage as well as remote storage (shared with all users connected to the service). It aims to support both Configuration and Execution phase caching. However, as of Gradle version 7.4.2, this mechanism only supports the second-layer cache during the Execution phase. Configuration phase caching is not yet supported.

We will further elaborate on adhering to caching design guidelines, which will be instrumental in enhancing the Slack notification Plugin.

The code used in this section is within the slack project:

• The Plugin project resides at slack/plugins/slack-cache-rules-compliance.
• Tinker with the Plugin configuration at slack/plugin-config/src/main/kotlin/slack-cache-rules-compliance.gradle.kts.
• The Android Application test project, a bearer of the Plugin, resides in slack/app. Remember to activate the appropriate Plugin in app.gradle.kts during testing.

4-5-1: Configuration Cache

The time consumed during the configuration phase mainly revolving around parsing the build.gradle(.kts) script, running the Plugin's apply() method, and calculating the Task Graph. Generally, enabling the Configuration Cache can reduce configuration waiting time by several seconds to minutes, given that no changes are made to the entire build script. As of version 7.4.2, this feature remains in preview and is not yet fully developed.

To experience this optimization, include --configuration-cache when executing the Gradle command, such as ./gradlew help --configuration-cache. Since Configuration Cache isn't fully stable in Gradle 7.4, to always enable it, use the following properties:

org.gradle.unsafe.configuration-cache=true

During the first clean full build, you'll notice a prompt indicating Task Graph calculation:

Calculating task graph as no configuration cache is available for tasks: 
:test-app:assembleDebug

Once successful, a prompt will appear at the end of the build:

Configuration cache entry stored.

If there's no change to the build script in the next build, the previous cache will be reused:

Reusing configuration cache.
...
51 actionable tasks: 2 executed, 49 up-to-date
Configuration cache entry reused.

Caches are stored in the .gradle/configuration-cache directory at the project root. Cached content must be serializable/deserializable. Currently, successful activation of Configuration Cache requires all scripts and Plugins in the project to adhere to development guidelines, ensuring the safe reuse of the Task Graph. For medium to large scale projects, a step-by-step adaptation plan is essential to address all compatibility issues.

Plugin User Adaptation Guide

As a Gradle Plugin user, if you encounter unsupported problems in commonly used Plugins, you can search to see if similar issues have already arisen. For instance, the problem that emerged when the Kotlin Gradle Plugin 1.4.32 worked in conjunction with Gradle 7.0 (Figure 4.5.2).

The solution can be found in the JetBrains YouTrack issue: upgrading the Kotlin Plugin version to 1.5.0 or higher ensures compatibility.

Notably, key Plugins like AGP, Kotlin Gradle Plugin (KGP) have supported Configuration Cache in their latest versions. You can gather more information from the respective documentation.

For other third-party open-source Plugins, it's recommended to check their latest versions and issue lists to ensure no missed adaptations.

If a Plugin or some custom Tasks don't support the Configuration Cache and aren't persistent in your App, you can temporarily bypass them:

• If only used when packaging the Release version and not needed during regular Debug development, consider disabling the corresponding Task of the Plugin. Refer to Section 8-3 for details.
• If one or more Tasks are not required for local development and are mainly used in CI or other special scenarios, consider using the Task#notCompatibleWithConfigurationCache() API to mark these Tasks as incompatible. This allows for the regular application of the Configuration Cache mechanism if the current build's Task Graph does not include such Tasks. It also doesn't impact full builds in other environments, avoiding exceptions due to incompatibility.

Guide for Plugin Developers: Adapting to the Configuration Cache

As a Plugin developer, one must pay close attention to the intricate details surrounding the adaptation to Configuration Cache. As of Gradle 7.4.2, there are seven best practices recommended by the documentation. We will discuss these in terms of Configuration and Execution phases.

During the Configuration Phase:

1. Discontinue the use of listeners such as BuildListener and TaskExecutionListener. Instead, embrace the new set of APIs based on BuildService, namely OperationCompletionListener and BuildEventsListenerRegistry. Also, cease using beforeTask() and afterTask(). Refer to Section 4-1 for more details.

2. Unless necessary, avoid reading system or environment variables during the configuration phase. Alterations to these variables might render the previous cache obsolete. Although restrictions have been relaxed since version 7.4, it's still recommended to adopt the Provider variant of variable fetching. This delays the consumption of system/environment variables until Task runtime using ProviderFactory#systemProperty() and ProviderFactory#environmentVariable(). The ProviderFactory can be accessed via the Project context as project.providers. Note that some older versions of the Gradle adaptation guide might recommend the forUseAtConfigurationTime() API for fetching environment variables at the configuration phase, but it has been deprecated.

3. During the configuration phase, avoid using JDK or language-level file reading APIs, like file("some.conf").readText(). This issue can be addressed using ProviderFactory#fileContents().

   providers.fileContents(layout.projectDirectory
     .file("some.conf")).asText

During the Execution Phase:

1. Refrain from accessing instances of other Tasks within a Task.
2. Avoid referencing instances from the following three major categories, which can be classified as context types holding some state:
   1. Live JVM state types, e.g., ClassLoader, Thread, OutputStream, Socket, etc.
   2. Gradle model types like Gradle, Settings, Project, SourceSet, Configuration, etc. A common solution is to directly declare the relevant attributes you wish to reference as Task inputs. For instance, if you need project.version, simply declare an abstract val version: Property<String> and pass the attribute into it. Also, you can leverage Managed Classes/Properties, as discussed in Section 4-3, to allow Gradle to automatically inject certain supported utility classes or context instances.
   3. Dependency management types such as Configuration, SourceDirectorySet, and dependency resolution results like ArtifactResolutionQuery, ResolvedArtifact, ArtifactResult, etc. Consider using the lazy retrieval APIs of the Provider type, like ArtifactCollection.getResolvedArtifacts() which returns a Provider<Set<ResolvedArtifactResult>>. Alternatively, you can directly pass the final data needed from the dependency resolution result, using supported data types like String as the carrier.
3. Don't access methods or properties related to Project (directly or indirectly) within a Task.

A commonly encountered issue can be illustrated using the Slack notification Plugin as an example. When executing the demo command ./gradlew clean assembleAndNotifyDebug --configuration-cache, an issue arises that needs rectification:

\* What went wrong:
Configuration cache problems found in this build.
 
2 problems were found storing the configuration cache, 
1 of which seems unique.
\- Task `:app:assembleAndNotifyDebug` of type \
   `me.xx2bab.buildinaction.slackcache\
   .SlackNotificationTask`: invocation of 'Task.project' at\
   execution time is unsupported.

Inspecting the detailed stack trace in the Configuration Cache HTML Report reveals the root of the issue: the reference of Project in the Task. (Figure 4.5.3)

Given the lengthy stack trace, let's focus on the portion relevant to our code:

org.gradle.api.InvalidUserCodeException: 
Invocation of 'Task.project' by Task ':app:assembleAndNotifyDebug' 
at execution time is unsupported.
  ...
  at api.DefaultTask.getProject(DefaultTask.java:59)
  at ...SlackNotificationTask.notifyBuildCompletion(...Task.kt:87)
  ...

Clearly, the error originates from line 87 of SlackNotificationTask.kt. To address this, simply replace project.logger with this.logger, transitioning the call context from Project to Task:

//project.logger.lifecycle("...")
logger.lifecycle("...")

After this correction, the project's Configuration Cache should operate seamlessly. For more intricate examples like avoiding Project references, check the example in Section 3-6. The Configuration Cache feature is still under development and some scenarios might not yet be supported. The support from the Android Gradle Plugin (AGP) or other third-party Plugins is progressively evolving. For comprehensive support details, including replacements for Project-related APIs, consider the following resources:

1. Stay updated with the latest releases of Configuration Cache (https://docs.gradle.org/current/userguide/configuration_cache.html#config_cache:requirements)
2. For detailed development rules and use cases, it's recommended to review AGP's journey with Configuration Cache (addressing over 400 issues): Configuration caching deep dive - Android Developers (https://medium.com/androiddevelopers/configuration-caching-deep-dive-bcb304698070)
3. The Gradle team maintains the android-cache-fix-gradle-plugin (https://github.com/gradle/android-cache-fix-gradle-plugin) repository. For unique problems related to AGP build cache and Configuration Cache, this repository might offer the exact solution your project requires.

Successfully implementing the Configuration Cache in a project brings various benefits, for instance,

• Tasks within the same project may execute in parallel (at the Task level, not the Worker level, without needing --parallel).
• Dependency resolution results can be cached as well.

These benefits stem from the decoupling of state type instances and not consuming certain environment or dependency-related parameters during the configuration phase.

Lastly, Gradle also incubates the Configure On Demand feature, which is distinct from Configuration Cache:

• Configure On Demand determines if a Project participates in the current build's Configuration phase, avoiding unnecessary project configurations.
• Configuration Cache, when a project's configuration is confirmed to be evaluated, offers cache results from previous runs to prevent the re-evaluation of unchanged configuration scripts.

Configure On Demand relies on Decoupled Projects state. For more discussion, refer to Section 4-4.

4-5-2: Incremental Build and UP-TO-DATE Check

Generally speaking, we refer to Execution phase caching as either Incremental Builds or UP-TO-DATE Checks. For those familiar with modern Gradle, the UP-TO-DATE Check is a feature available right out of the box. While the official documentation lumps both concepts together, referencing them as "Incremental Build" (a.k.a. "UP-TO-DATE Check"), we'll break them apart here for a clearer understanding of their individual functions.

Regarding the UP-TO-DATE Check, there's no need for additional activation, though some adaptation might be necessary for developers. As discussed in Section 2-6, the UP-TO-DATE Task status essentially mirrors this feature's effect. Its primary concern revolves around determining if inputs and outputs have changed. When no changes are detected, the cache is reused, skipping the Task execution.

On the other hand, Incremental Build is built on top of the UP-TO-DATE Check. When there's a partial change in the input and the previous output cache still exists, the Task can choose to execute incrementally. This means it only runs the core Task logic for the changed parts, significantly reducing execution time. Keep in mind that not all Tasks are compatible with the Incremental Build feature. Their feasibility largely depends on the Task's input-output format and cardinality. Further restrictions are discussed in the adaptation guide below.

By default, this caching mechanism points to the current Project's build/ directory which involves input-output related content. Consequently, running the clean Task typically deletes this directory, making the cache void.

One characteristic of Execution phase caching is its autonomy. Unlike the Configuration Cache, which requires global support for optimization (otherwise, it would throw errors), the main cache unit during execution is the Task. These Tasks are relatively independent. As long as most time-consuming Tasks are cached and the cache is reusable, overall performance during the Execution phase is ensured. This characteristic is applicable to Build Cache as well.

Adaptation Guide for Plugin Users

For the UP-TO-DATE Check:

1. Carefully observe during local development to see if any Tasks never reach the UP-TO-DATE status. If a Task seems cacheable (e.g., doesn't involve network services and only deals with local file operations), you should raise an issue with the Plugin maintainers.
2. Refrain from input parameters like timestamps that lead to the Task being re-executed every time.

For Incremental Builds, it's challenging for Plugin users to externally determine if a Task is incremental. This check is typically left up to developers.

Adaptation Guide for Plugin Developers

For the UP-TO-DATE Check:

1. Refer to Section 2-6 for input-output specifications, preferring Gradle's file-related APIs over native JDK File operations. Remember that while Task inputs are optional, Task outputs are mandatory.
2. Section 3-2 discussed scenarios where caching becomes invalidated in a following non-clean build.

For Incremental Build:

With well-configured inputs and outputs, our focus shifts to incremental modifications. Let's illustrate with a scenario: writing a Task to generate MD5 and SHA256 checksums for all local source files. Task inputs can be retrieved from the Android Gradle Plugin's (AGP) Variant-related API, while outputs are declared as a DirectoryProperty.

val allSources: Provider<List<Directory>> = appVariant.sources.java.all
val sourceVerificationTask = project.tasks
  .register<SourceToVerificationCodesTask>(
  "${appVariant.name}SourceToVerificationCodes"
) {
  sources.from(project.files(allSources))
  outputDir.set(project.layout
    .buildDirectory.dir("src_verification"))
}

The crux of supporting Incremental Build lies in the @get:Incremental annotation and the Task Action parameter, compute(inputChanges: InputChanges).

@CacheableTask
abstract class SourceToVerificationCodesTask : DefaultTask() {

  @get:Incremental  // [I]
  @get:InputFiles
  @get:PathSensitive(PathSensitivity.RELATIVE)
  abstract val sources: ConfigurableFileCollection

  @get:OutputDirectory
  abstract val outputDir: DirectoryProperty

  @TaskAction
  fun compute(inputChanges: InputChanges) { // [II]
    val outDir = outputDir.get().asFile

    if (inputChanges.isIncremental) {
      // [III]
      inputChanges.getFileChanges(sources).forEach { change ->
        if (change.fileType == FileType.DIRECTORY) return@forEach
        when (change.changeType) {
          ChangeType.ADDED, ChangeType.MODIFIED -> {
            processSourceFile(change.file, outDir)
          }

          ChangeType.REMOVED -> {
            removeTargetFile(change.file, outDir)
          }
        }
      }
    } else {
      sources.forEach {
        it.walk().forEach { inputFile ->
          processSourceFile(inputFile, outDir)
        }
      }
    }
  }

  private fun processSourceFile(inputFile: File, outDir: File) {
    if (inputFile.isFile
      && (inputFile.extension == "java" 
      || inputFile.extension == "kt")
    ) {
      val fullFileName = inputFile.fullFileName
      val sha256MD5 = File(outDir, fullFileName)
      sha256MD5.createNewFile()
      val content = inputFile.readText()
      sha256MD5.writeText(
        """
        |SHA256: ${content.sha256()}
        |MD5: ${content.md5()}
        """.trimMargin()
      )
    }
  }

  private fun removeTargetFile(inputFile: File, outDir: File) {
    if (inputFile.extension == "java" 
      || inputFile.extension == "kt") {
      val fullFileName = inputFile.fullFileName
      val sha256MD5 = File(outDir, fullFileName)
      sha256MD5.delete()
    }
  }

  ...
}

The @get:Incremental annotation on an input property indicates its support for incremental checks (at [I]). Following that, Gradle passes in an InputChanges tool during the Task Action (at [II]). We can use this to determine whether the build is incremental (or full) and in the former case, retrieve a list of changed files via inputChanges.getFileChanges(sources) (at [III]).

These changes can be additions, modifications, or deletions. Depending on the change type and whether it's a file or directory, we can appropriately respond. For instance, when a file is added or modified, we rerun the Task's core logic. However, if a file is removed, we simply delete the corresponding output file.

This example leads us to some limitations of Incremental Builds:

1. Only three Gradle file input types are supported: RegularFileProperty, DirectoryProperty, and ConfigurableFileCollection. Others like ListProperty<RegularFile> and ListProperty<Directory> are not supported so far. This is because, during the API design, the Gradle team believed that ConfigurableFileCollection was sufficient for incremental scenarios, since FileType can be either FILE or DIRECTORY, FileCollection accommodates both scenarios. Thus, when interacting with AGP (which extensively uses Provider based file containers), one can wrap unsupported file types using project.files(...), as shown in the sources.from(project.files(allSources)) line.
2. Typically, one should avoid reading the content of a file marked with @Incremental and then using that content as an output filename. This is because when a file is removed (triggering ChangeType.REMOVED), you can't read the deleted file and thus can't modify the corresponding output.
3. A Task should only have one Task Action that accepts InputChanges as an input.
4. A "one-to-one" or "one-to-many" arrangement of input-output is ideal for incremental processing. Aggregative many-to-one scenarios require clarity on whether the target output can be easily modified.

Considering the last limitation, even within the AGP, the ratio of incremental to non-incremental Tasks is roughly 1:7, based on a count among IncrementalTask, NewIncrementalTask, and NonIncrementalTask. Same for AGP Synergy Plugin development, opportunities to apply Incremental Build are limited. However, always be vigilant when designing Plugins to check if they meet the criteria for this support.

4-5-3: The Secondary Cache System: Build Cache

The Build Cache is a feature that's not enabled by default in Gradle and currently only supports caching during the Execution phase. The term "secondary" signifies:

1. When the primary cache within a project is cleared (e.g., ./gradlew clean) or invalidated (e.g., switching Git branches), Gradle will check either locally or remotely (based on user configuration) for a cache that matches the current Task fingerprint. If found, it will apply this cache to the current Task, skipping its execution, and the Task status will be marked as FROM-CACHE. The default local cache resides in the .gradle/caches directory in the user's root directory, this directory also contains caches for compiled scripts and dependencies. The remote cache location can be configured based on server specifications.
2. The Build Cache can store the results of multiple Tasks as long as they don't exceed the pre-set maximum capacity. This means a Task can reuse the cache from not only its last run but from all its prior runs.

To enable caching during the configuration phase, use the --build-cache flag for temporary activation or add the following parameter to gradle.properties for persistent activation:

org.gradle.caching=true

Adaptation Guide for Plugin Users

As mentioned, the activation of Build Cache does not require the support of all Tasks. Plugin users can determine whether to apply the optimization based on their project's build circumstances.

A good practice is to periodically export build result reports to local development machines using the --scan flag. Alternatively, use third-party Plugins to monitor the cache reuse status of Tasks, comparing them to identify optimizable Tasks. On remote CI servers, we generally disable any caching optimizations when performing a full compilation. However, if your team has additional "fast release" channels that permit cache acceleration for debuggable package deliveries, enabling the Build Cache is certainly feasible.

For discussions comparing multiple Build Scan reports, see Section 8-1. Additionally, in functional tests, you can use GradleRunner to acquire Task statuses, checking if results from subsequent runs satisfy the FROM-CACHE status. For more on this, see Section 4-6.

Adaptation Guide for Plugin Developers

It's crucial to understand that not all Tasks require this mechanism. Typical examples including:

1. Tasks without Task Action, such as lifecycle Tasks, assemble, which are essentially empty hook implementations.
2. Tasks without outputs, such as uploading files to a remote server only.
3. Simple local file operations without complex logic, such as the Copy or Jar Tasks provided by Gradle. In these cases, reading the cache from the .gradle directory and writing to the current build directory has similar costs to direct copying, making caching unnecessary.
4. For complex Tasks, the cost of computing the Cache Key and futher comparison are higher than usual Tasks, however the cache is seldom reused. For instance, the Build Cache of Dex building related tasks can be disbaled in the CI that runs for whole APK/AAB builds.

For Tasks ensuring the activation of caching, annotate the Task's Class with @CacheableTask to inform Gradle that the Task is now managed by Build Cache.

@CacheableTask
abstract class SlackNotificationPlugin : Plugin<Project> {...}

Tasks with Build Cache activated must have clear input and output definitions, akin to the UP-TO-DATE Check. A difference is that @CacheableTask requires input properties to be explicitly marked with their respective @PathSensitive(...) annotations. Typically, avoid using PathSensitivity.ABSOLUTE to minimize Task reruns due to unchanged content but altered paths, preserving shared remote caches.

Cache lookup depends on the Cache Key derived from Task properties and type. Factors affecting the Cache Key include:

1. The Task type and its associated classpath.
2. The names of the Task's output properties.
3. Property names and values marked by Gradle Task custom annotations (both inputs and outputs), like the previously mentioned @Inputs, @Internal, @Nested, etc.
4. Input property names and values that dynamically added through the TaskInputs class.
5. The classpaths of Gradle itself, buildSrc, and Plugins.
6. Operations in build scripts, like build.gradle(.kts) and settings.gradle(.kts), which might impact Task execution. For instance, adding a beforeTask(...) callback (use of such hook insertions is discouraged).

Lastly, we present a Build Cache runtime API: cacheIf(...). In essence, it serves the same purpose as @CacheableTask, but it allows dynamic configuration based on a logic evaluation during the configuration phase. This can be correlated with the TaskInputs/TaskOutputs runtime API covered in Section 2-6.

project.tasks.register<SourceToVerificationCodesTask>(
  "${appVariant.name}SourceToVerificationCodes"
) {
  ...
  outputs.cacheIf { true }
}

4-5-4: In Summary

From our discussion, we can concisely summarize the characteristics of the two cache types:

1. The Execution phase caching rules are clear-cut, straightforward to modify, and allow selective Task exemption from the cache. The table below offers a direct comparison between the two execution phase caches.

Feature	Incremental Build/ UP-TO-DATE Check	Build Cache
Supported Version	Gradle 1.6	Gradle 3.5
Task Status When Cache Hits	UP-TO-DATE	FROM-CACHE
Local Cache Path	Project's build directory	Default in Project root's .gradle directory
Remote Cache Path	N/A	Configured based on the server (includes support from Gradle Enterprise)
Cache Capacity	Single result	Multiple, with configurable volume size

2. Configuration Cache rules are more complex since they are unstable with error messaging is not entirely intuitive. It requires more time to adapt. Currently, in order to make this caching possible, the complete global configuration must follow the aforementioned guidelines. Gradle is expected to activate the Configuration Cache by default in version 9.0, giving third-party developers an estimated 1 to 2 years to prepare. It's hoped that future iterations will streamline and potentially include Build Cache support for it, and Gradle releases will introduce more developer-friendly migration tools and APIs, simplifying the overall development process.