3-5: Artifact API Internal

As we tread along the intricate path of the AGP configuration, this next leg of our journey unravels the mystery of how AGP Artifacts are configured and threaded into the subsequent processing steps. Peeking behind this curtain not only offers a treasure trove of design insights but also gears us up for crafting the Polyfill toolkit in our upcoming section.

3-5-1: Public Artifact API Exploration

First of all, behind the Variant#artifacts API is the com.android.build.api.artifact.Artifacts interface, and its implementation is com.android.build.api.artifact.impl.ArtifactsImpl.

To get a clear snapshot of how Artifacts come to life and evolve, let's spotlight two pivotal methods:

fun <FILE_TYPE : FileSystemLocation> get(
  type: SingleArtifact<FILE_TYPE>): Provider<FILE_TYPE>

fun <TaskT : Task> use(
  taskProvider: TaskProvider<T>): TaskBasedOperationImpl<T>

As for get(...) method that deals with the SingleArtifact type, the implementation is pretty direct:

override fun <FILE_TYPE : FileSystemLocation> get(
  type: SingleArtifact<FILE_TYPE>
): Provider<FILE_TYPE> = getArtifactContainer(type).get()

The process is to obtain an ArtifactContainer of the respective type and invoke its get() method to secure the ultimate Provider outcome. A deeper dive unveils yet another succinct call:

internal fun <FILE_TYPE : FileSystemLocation> getArtifactContainer(
  type: Single<FILE_TYPE>
): SingleArtifactContainer<FILE_TYPE> {
  return storageProvider.getStorage(type.kind)
    .getArtifact(objects, type)
}

Here, we're pulling out a TypedStorageProvider aligning with the Single<FILE_TYPE> of the SingleArtifact. This magic is performed by StorageProviderImpl, which is the storageProvider property as you can see.

private val singleStorage = mutableMapOf<Artifact.Single<*>,
  SingleArtifactContainer<T>>()

@Synchronized
internal fun getArtifact(
  objects: ObjectFactory,
  artifactType: Artifact.Single<T>
): SingleArtifactContainer<T> {
  return singleStorage.getOrPut(artifactType) {
    SingleArtifactContainer {
      SinglePropertyAdapter(propertyAllocator(objects))
    }
  }
}

The TypedStorageProvider keeps a neat map, dubbed singleStorage, to catalog the SingleArtifactContainer. The content we've unfolded thus far predominantly relates to containers. Rather than diving too deep into the nitty-gritty, why not sketch a classes map? It’s a great way to gain a bird's-eye view of the interconnected framework we're grappling with. (Figure 3.5.1)

Beyond the diagram, there are some tooling aids from code, like SinglePropertyAdapter and propertyAllocator that come into play. Their primary role is to adjust different file types and fix those tedious API differences between Gradle's Property and ListProperty. Remember to dive into Section 4-2, where Property is unwrapped in detail.

interface PropertyAdapter<T> {
  fun set(with: Provider<T>)
  fun get(): Provider<T>
  fun disallowChanges()
  fun from(source: PropertyAdapter<T>) {
    set(source.get())
  }
}

class SinglePropertyAdapter<T: FileSystemLocation>(val prop: Property<T>)
  : PropertyAdapter<T> {
  override fun disallowChanges() {
    prop.disallowChanges()
  }
  override fun get(): Provider<T> {
    return prop
  }
  override fun set(with: Provider<T>) {
    prop.set(with)
  }
}

The from(...) method is the center of attention in this block of code. Think of it as a bridge that combines the Propertys of two Adapters or, in plain word, does prop1.set(prop2). This function isn't a stranger in our journey, it pops up in several places. With that in mind, our next stop is the SingleArtifactContainer, where we'll inspect its dynamics alongside its predecessor, ArtifactContainer.

internal class SingleArtifactContainer<T: FileSystemLocation>(
  val allocator: () -> SinglePropertyAdapter<T>
) {
  private val agpProducer = allocator()

  init {
    current.from(agpProducer)
    final.from(current)
  }

  fun setInitialProvider(taskProvider: TaskProvider<*>,
    with: Provider<T>) {
    if (needInitialProducer().compareAndSet(true, false)) {
      agpProducer.set(with)
      currentTaskProviders.add(taskProvider)
    }
  }

}

internal abstract class ArtifactContainer<T, U>(
  private val allocator: () -> U) where U: PropertyAdapter<T> {
  internal var current = allocator()
  val final = allocator()
  fun get(): Provider<T> = final.get()
}

Let's decode what's going on. A peek at the get() method reveals its reliance on the final attribute. If you disregard actions like modifying the Artifact (i.e., methods like transform(...), append(...)), the essence of final echoes current, which in turn mirrors agpProducer, a function passing from the setInitialProvider(...) method's parameters.

However, with a maze of countless calls to the setInitialProvider(...) method in AGP, diving into it is like searching for a needle in a haystack. Let's pivot our approach, tracing our path through the enum type of Artifacts, specifically homing in on how SingleArtifact.MERGED_MANIFEST from Section 3-3 is referenced. (Figure 3.5.2)

As with the last section, we will only cover the relevant classes of the Application module in this section. It is not difficult to discover that the ProcessApplicationManifest.kt reference points have the following code:

class CreationAction(
  creationConfig: ApkCreationConfig
) : VariantTaskCreationAction<ProcessApplicationManifest,
  ApkCreationConfig>(creationConfig) {
  ...

  override fun handleProvider(
    taskProvider: TaskProvider<ProcessApplicationManifest>
  ) {
    super.handleProvider(taskProvider)
    val artifacts = creationConfig.artifacts
    artifacts.setInitialProvider(
      taskProvider,
      ProcessApplicationManifest::mergedManifest
    )
      .on(SingleArtifact.MERGED_MANIFEST)  [I]

    ...
  }

  override fun configure(
    task: ProcessApplicationManifest
  ) { ... }

  ...
}

This logic is contained within the CreationAction class, which performs one of the steps of the AGP creation Task described at the end of the previous section. We've found the source of the SingleArtifact.MERGED_MANIFEST Artifact, which "should" be the ProcessApplicationManifest::mergedManifest at [I], with the following declaration:

abstract class ProcessApplicationManifest
  : ManifestProcessorTask() {
  @get:OutputFile
  abstract val mergedManifest: RegularFileProperty
  ...
}

Why is "should" used here? Note the target object of this invocation, artifacts.setInitialProvider which is ArtifactsImpl, not ArtifactContainer. So, we must trace this code again:

class ArtifactsImpl(...): Artifacts {
  ...
  internal fun <FILE_TYPE, TASK> setInitialProvider(
    taskProvider: TaskProvider<TASK>,
    property: (TASK) -> FileSystemLocationProperty<FILE_TYPE>
  ): SingleInitialProviderRequestImpl<TASK, FILE_TYPE>
      where FILE_TYPE : FileSystemLocation,
          TASK: Task
      = SingleInitialProviderRequestImpl(this, taskProvider, property)
}

internal class SingleInitialProviderRequestImpl<
  TASK: Task, FILE_TYPE: FileSystemLocation>(
  private val artifactsImpl: ArtifactsImpl,
  private val taskProvider: TaskProvider<TASK>,
  private val from: (TASK) -> FileSystemLocationProperty<FILE_TYPE>
) {
  var fileName: String? = null
  private var buildOutputLocation: String? = null
  private var buildOutputLocationResolver: (
    (TASK) -> Provider<Directory>)? = null

  fun atLocation(loc: String?): SingleInitialProviderRequestImpl<...> {
    buildOutputLocation = loc
    return this
  }

  fun atLocation(loc: (TASK) -> Provider<Directory>)
      : SingleInitialProviderRequestImpl<...> {
    buildOutputLocationResolver = loc
    return this
  }

  fun withName(name: String): SingleInitialProviderRequestImpl<...> {
    fileName = name
    return this
  }

  fun on(type: Single<FILE_TYPE>) {
    val artifactContainer = artifactsImpl.getArtifactContainer(type)
    taskProvider.configure {
      ...
      val outputAbsolutePath = when {
        buildOutputLocation != null -> {
          File(buildOutputLocation, fileName.orEmpty())
        }
        buildOutputLocationResolver != null -> {
          val resolver = buildOutputLocationResolver!!
          File(resolver.invoke(it).get().asFile, fileName.orEmpty())
        }
        else -> {
          artifactsImpl.getOutputPath(type,
            if (artifactContainer.hasCustomProviders()) {
              taskProvider.name
            } else "",
            fileName.orEmpty())
        }
      }
      // since the taskProvider will execute, resolve its output path.
      from(it).set(outputAbsolutePath)
    }
    artifactContainer.setInitialProvider(
      taskProvider, taskProvider.flatMap { from(it) })  [I]
  }
}

Diving straight in from SingleInitialProviderRequestImpl#on(...), we can see it's juggling a couple of Tasks: it interprets, assembles, and tailors the file related to the output Artifact. Additionally, it invokes ArtifactContainer#setInitialProvider(...) (at [I]) to cache both the TaskProvider and the associated Artifact. If you recall our previous discussion, this method ties together the process of associating Task output with the caching mechanism.

Here's a clear picture of the upper half of the Artifact design: As the AGP Task springs to life, certain output attributes are mapped to the Artifact's InitialProvider (as the original source) via CreationAction#handleProvider. These are then safely cached in the internal ArtifactContainer. Now, when you fetch the Artifact using Artifacts#get(...)/getAll(...), it leans on AGP's internal Task utilizing the TaskProvider's implicit dependency handling. This, in essence, draws a clear boundary between the external Artifact API and AGP's inner workings.

Moving to the lower half of the design, let's turn our attention to the use() method. On the interface level, it returns a TaskBasedOperation. But if you peek under the hood, it's implemented as TaskBasedOperationImpl<TaskT>:

interface Artifacts {
  ...
  fun <TaskT: Task> use(t: TaskProvider<TaskT>): TaskBasedOperation<TaskT>
}

class ArtifactsImpl(
  project: Project,
  private val identifier: String
): Artifacts {
...
  fun <...> use(t: TaskProvider<TaskT>): TaskBasedOperationImpl<TaskT>
}

Remember our discussion in Section 3-3 about TaskBasedOperation? It's equipped with four handy wiredWith*() methods that correspond to four *OperationRequest interfaces. These are neatly housed in the OperationRequests Kotlin file. And, as for the implementations of these interfaces? They reside in the OperationRequestsImpl Kotlin file as *OperationRequestsImpl.

The object we get from use(...) is one of these *OperationRequestsImpl. Our next move? Invoke one of the trinity: toCreate(...), toAppend(...), toTransform(...). These actions enable us to interact with the Artifact – either swap the original Artifact, add on an extra Artifact, or tweak the current one. Just so you know, these methods are top-tier and find their home in the OperationRequestsImpl file. Let's zoom in on toTransform(...) to see the magic inside:

private fun <TaskT: Task,
  FileTypeT: FileSystemLocation, ArtifactTypeT> toTransform(
  artifacts: ArtifactsImpl,
  taskProvider: TaskProvider<TaskT>,
  from: (TaskT) -> FileSystemLocationProperty<FileTypeT>,
  into: (TaskT) -> FileSystemLocationProperty<FileTypeT>,
  type: ArtifactTypeT)
    where ArtifactTypeT : Single<FileTypeT>,
        ArtifactTypeT : Artifact.Transformable {
  val artifactContainer = artifacts.getArtifactContainer(type)
  val currentProvider = artifactContainer.transform(
    taskProvider, taskProvider.flatMap { into(it) })  // [I]
  val fileName = if (type.kind is ArtifactKind.FILE
    && type.getFileSystemLocationName().isNullOrEmpty()) {
    DEFAULT_FILE_NAME_OF_REGULAR_FILE_ARTIFACTS
  } else ""
  taskProvider.configure {  // [II]
    from(it).set(currentProvider)
    // Now that the task will execute, let's resolve its output path.
    into(it).set(
      artifacts.getOutputPath(type,
        taskProvider.name,
        fileName
      )
    )
  }
}

Before diving deep, let's unpack the calls inside [I] a bit:

internal abstract class ArtifactContainer<T, U>(
  private val allocator: () -> U) where U: PropertyAdapter<T> {

  open fun transform(taskProvider: TaskProvider<*>,
    with: Provider<T>): Provider<T> {
    hasCustomTransformers.set(true)
    val oldCurrent = current
    current = allocator()
    current.set(with)
    currentTaskProviders.clear()
    currentTaskProviders.add(taskProvider)
    final.from(current)
    return oldCurrent.get()
  }
}

Initially, a fresh container object is assigned to current. The output of the Task executing transform(...) (a Provider that matches the current Artifact's type) is mapped to this new current. This is done via taskProvider.flatMap { into(it) }. Post this, the relevant TaskProvider is recorded, and the contents of the final container are matched with the new current. Wrapping up, the Provider of the prior current node, which is oldCurrent.get(), is retrieved as a return value.

Looping back to toTransform(...), after fetching the Provider from the oldCurrent container at [II], we earmark it as the Task's input for toTransform(...). Then, we allocate a newly minted file, courtesy of ArtifactsImpl.getOutputPath(...), to its output. Sounds familiar? It mirrors what we spotted in SingleInitialProviderRequestImpl#on(...).

Though the logic feels straightforward, it's the key for realizing the Artifact design's lower half. Time for some visual aids: Let's dissect the roles of several internal attributes through the next illustrative diagram. (Figure 3.5.3)

Dive into this: it's like a backward-linked list, with multiple nodes hanging out in there. The external consumer comes in at the final stage. The nodes in between? They're PropertyAdapter container crafted by the allocator function. They're initialized as empty, and destined to be filled later on. For our current journey, let's think of them as just Property. To populate them, we use the versatile Property.set(...), it can handle different file types and even other Property objects (check out Section 4-2 for more on that). Here's a quick rundown:

1. agpProducer: This is the starting node, and what goes in here is typically some AGP Task's output attribute.
2. current: Our present focus. It starts off pointing to the Provider of agpProducer. Once toTransform(...) takes the stage, it shifts the cursor to the current Task's output attribute. But here's a twist: unlike our first and last nodes, every time toTransform(...) gets a go, we're looking at a brand new container object, loaded with the corresponding Task output attribute and then added to our linked list.
3. final: Our endgame. Starts off by pointing at the Provider of current. So, in essence, it's mirroring the head node or AGP Task output. Every time toTransform(...) calls, it updates to keep track of the latest current container's Provider.

The design of having fixed head and tail nodes is a classic in linked list data structures. With final, you can blissfully ignore the evolving nature of current, how often toTransform(...) pops in. For the end user, they just have to call get(...)/getAll(...) to snag the most recent outcome. Meanwhile, the current node is your connector, linking the past Task's output to the next one, ensuring the entire chain stays intact. You can also dive deeper into toAppend(...) and toCreate(...) using the same lens.

Now, let's take a breather and recap. We've pretty much demystified the second half of use(...) + toTransform(...). Reflecting on the entire sequence, here are the big takeaways:

1. From the first half, it's clear that we're working with a versatile Artifact caching system, tailored for a range of Artifact and file types. At its core, it's built around SingleArtifactContainer (we're only zooming in on SingleArtifact).
2. Often, the Artifact's source data provider emerges from the process created by the AGP Task (we'll uncover more scenarios in the upcoming section).
3. Thanks to the mechanism of Provider, the multiple cache containers weave a backward chain. With the strategic placement of the final node, rest assured, get(...)/getAll(...) always fetches the freshest data.

3-5-2: Internal Artifact API Exploration

In the previous analysis of the source data provider, we shared a snippet of the ProcessApplicationManifest.kt code, but there is actually an additional noteworthy piece of code in the handleProvider(...):

class CreationAction(creationConfig: ApkCreationConfig)
  : VariantTaskCreationAction<ProcessApplicationManifest, ApkCreationConfig>(creationConfig) {
  ...

  override fun handleProvider(
    taskProvider: TaskProvider<ProcessApplicationManifest>
  ) {
    super.handleProvider(taskProvider)
    val artifacts = creationConfig.artifacts
    artifacts.setInitialProvider(
      taskProvider,
      ProcessApplicationManifest::mergedManifest
    )
      .on(SingleArtifact.MERGED_MANIFEST)  [I]

    artifacts.setInitialProvider(
      taskProvider,
      ManifestProcessorTask::mergeBlameFile
    )
      .withName("manifest-merger-blame-"
        + creationConfig.baseName + "-report.txt")
      .on(InternalArtifactType.MANIFEST_MERGE_BLAME_FILE)  [II]
    artifacts.setInitialProvider(
      taskProvider,
      ProcessApplicationManifest::reportFile
    )
      .atLocation(
        FileUtils.join(
          creationConfig.services.projectInfo.getOutputsDir(),
          "logs"
        )
          .absolutePath
      )
      .withName("manifest-merger-"
        + creationConfig.baseName + "-report.txt")
      .on(InternalArtifactType.MANIFEST_MERGE_REPORT)  [III]
  }
  ...
}

In the snippets [II] and [III] you might've noticed, instead of using SingleArtifact.XXX from [I], the code references InternalArtifactType. Let's jog our memory back to Section 3-3, where we pointed out that both SingleArtifact and MultipleArtifact expose a combined total of 11 public Artifacts.

But wait, there's more! The AGP has a plenty of Artifacts that it doesn't openly advertise. Dive into InternalArtifactType, and you'll see it extends Artifact.Single<T>. If you recall, back in Section 3-3, we stumbled upon several internal classes tied to Artifact, such as Category, ContainsMany, and Transformable. To shine a light on Artifact.Single<T>, check out its four implementations in Figure 3.5.4.

Zooming into InternalArtifactType<T>, it encompasses the lion's share of intermediate and terminal products throughout AGP— a massive 212 in total. Here's the catch: the method to access these internal Artifacts isn't in the public-facing Artifacts, but neatly tucked away in ArtifactsImpl:

class ArtifactsImpl(
  project: Project,
  private val identifier: String
): Artifacts {

  // The usual public API
  override fun <FILE_TYPE : FileSystemLocation> get(
    type: SingleArtifact<FILE_TYPE>
  ): Provider<FILE_TYPE> = getArtifactContainer(type).get()

  // The AGP's closely guarded secret API
  fun <FILE_TYPE : FileSystemLocation> get(
    type: Single<FILE_TYPE>
  ): Provider<FILE_TYPE> = getArtifactContainer(type).get()

  ...
}

Beyond the Artifacts we've just talked about, the AGP holds more "gems" for us to unearth. Take ProcessApplicationManifest.kt for instance. When the Application module goes about merging AndroidManifest.xml, it eagerly looks for its input:

abstract class ProcessApplicationManifest : ManifestProcessorTask() {

  private var manifests: ArtifactCollection? = null
  private var featureManifests: ArtifactCollection? = null

  @InputFiles
  @PathSensitive(PathSensitivity.RELATIVE)
  fun getManifests(): FileCollection {
    return manifests!!.artifactFiles
  }

  @InputFiles
  @Optional
  @PathSensitive(PathSensitivity.RELATIVE)
  fun getFeatureManifests(): FileCollection? {
    return if (featureManifests == null) {
      null
    } else featureManifests!!.artifactFiles
  }

    class CreationAction(
    creationConfig: ApkCreationConfig
  ) : VariantTaskCreationAction<ProcessApplicationManifest,
    ApkCreationConfig>(creationConfig) {

    ...
    override fun configure(
      task: ProcessApplicationManifest
    ) {
      ...
      // This includes the dependent libraries.
      task.manifests = creationConfig  [I]
        .variantDependencies
        .getArtifactCollection(
          ConsumedConfigType.RUNTIME_CLASSPATH,
          ArtifactScope.ALL,
          AndroidArtifacts.ArtifactType.MANIFEST
        )
      ...
      if (variantType.isBaseModule) {
        task.featureManifests = creationConfig
          .variantDependencies
          .getArtifactCollection(
            ConsumedConfigType.REVERSE_METADATA_VALUES,
            ArtifactScope.PROJECT,
            AndroidArtifacts.ArtifactType.REVERSE_METADATA_FEATURE_MANIFEST
          )
      } else if (variantType.isDynamicFeature) {
        ...
      }
    }

  }

}

Let's take a deeper dive into this: with the code at [I], fetching the AndroidManifest.xml of the majority of submodules is a piece of cake— well, except for the Feature module. Now, VariantDependencies is like your go-to guide, holding the list of all dependencies for your current Variant. Digging further with getArtifactFileCollection(..) lets you peek into the metadata from these Artifacts. The method's signature is:

fun getArtifactCollection(
  configType: ConsumedConfigType,
  scope: ArtifactScope,
  artifactType: AndroidArtifacts.ArtifactType,
  attributes: AndroidAttributes? = null
)

Breaking it down:

• ConsumedConfigType: Represents the type of the dependency. Common types include: COMPILE_CLASSPATH, RUNTIME_CLASSPATH.
• ArtifactScope: Specifies the range for artifact retrieval. Available scopes are: ALL, EXTERNAL and PROJECT?
• ArtifactType: Indicates the specific kind of artifact. Some examples are: CLASSES, CLASSES_JAR, SOURCES_JAR, JAVA_RES, to MANIFEST. There are over 80 artifact types available.

When it comes to AGP's internal Artifacts, our exploration so far has only scratched the surface with two main points: getArtifactCollection(...) and InternalArtifactType. But to sum it up:

1. By analyzing the different stages of Artifacts, it's pretty evident that AGP is gearing up to reveal more Artifacts in the future. Just imagine - there are around 300 internal Artifacts that haven never been exposed.
2. While AGP 8.0 isn't locking the doors on internal code access yet, our current insights into these two internal Artifacts serve as a handy reference, especially if you're crafting third-party Plugins. In order to make the AGP team aware of internal Artifacts use cases from the community and ultimately result in the exposure of more Artifacts APIs, we might need to submit more use cases in the issue tracker.

3-5-3: In Summary

By the end of our journey, we've unearthed that the bedrock of Artifact API v2 lies in the caching system, linked data structures, and of course, the charm of Gradle's Provider API. And here's the kicker, the design blueprints of the Artifact API v2 aren't exclusive, they're a shared treasure both inside and out. This gives us a solid theoretical foundation when the creative itch strikes to build our third-party Artifact API toolkits.