1-4: Building An Android App by Hand

Consider the following scenario: if there were no Gradle, AGP, or even the Ant, how would the most basic Android application be compiled? Before diving into the nitty-gritty of Gradle and AGP, it's vital to have a firm grasp on the major build procedures of an Android app using build tools from the Android SDK only.

The source code for this section is located in the manual-build project.

1-4-1: Preparation

This section ignores the intricacies of Kotlin usage, multi-modules, external dependencies, and obfuscation. Instead, we concentrate on a single template with fundamental resources, Java code and manually generate the final APK. The build.sh file is used for the entire build configuration and execution. A simplified building process for an APK is shown in Figure 1.4.1.

Here are some conventions:

• The build is based on bash scripts, permitting only the use of Android SDK and JDK tools, with scripts tested on Ubuntu and macOS.
• The build is based on Android SDK version 30 (including build tools, etc.), utilizes AAPT2, D8/R8, and other modern compilation tools. Remember to add the build-tools folder under Android SDK 30 to the PATH of the system, so that the executable commands in the script can be found.

Once the tool version is determined, we can use Android Studio to create a new Android application project and remove AndroidX package and theme dependencies. This means not only removing dependencies from build.gradle(.kts), but also ensuring that the project code and resources do not use any external libraries.

Let's create a build.sh script and initialize the first piece of code to create a build folder.

## 1. Let's get started
  ### Create build dir
  buildDir="./out"
  rm -rf ${buildDir}
  mkdir ${buildDir}
  
  ### Setup global params
  sdk=YOUR_ANDROID_SDK_PATH
  androidJar=${sdk}/platforms/android-30/android.jar

When building with AGP and Gradle, there is a comprehensive build directory that distinguishes intermediates and final outputs. Here, for simplicity, we flatten all generated files into one folder, ./out. For the cleaning step, we simply cleared the entire build folder since incremental compilation is evidently not our objective (i.e., only the modified portions of the input are processed, while the unmodified portions are reused from the preceding result's cache).

After creating the build folder, we defined two variables sdk and androidJar, which will be used in a variety of places later.

• sdk is the path to the Android SDK, please make sure that the local environment has configured the Android SDK and export ${android-sdk-path}/build-tools/${sdk-version}/ to the environment variable.
• androidJar is one of the major portions of the SDK, containing not only the Java interface consistent with JDK but also a variety of interfaces and resources provided by the Android system to higher-level applications. Whether you want to reference Button or TextView, the program has to look it up here.

Note that androidJar is a dependency solely for compilation; the Android operation system supplies the runtime version.

1-4-2: Compile and Link Resources

AAPT2's processing is divided into Compile and Link, which resemble the compilation process of certain low-level programming languages (i.e. C language):

During the Compile stage, the original resource files undergo processing to generate .flat files. These .flat files consist of binary data encompassing numerous protobuffer messages. The relevant parameters employed in this stage are explained as follows:

• When the --dir parameter is utilized, it facilitates the scanning of an entire folder, enabling the compilation of all files present within. The resultant output is a compressed zip package (._ap), which contains all the compiled files in the form of .flat packages. Since it is not feasible to directly incorporate the zip package into the subsequent linking process, an additional step of unpacking becomes necessary.
• Conversely, if the --dir parameter is not employed, it becomes imperative to manually traverse through each resource file and pass them individually as arguments to the compile command. Consequently, the compilation result will also manifest as individual .flat files.

As for Link stage, it involves the integration of all the .flat files to generate the res.apk package. This package includes the /res resource folder, the resources.arsc file, the AndroidManifest.xml, as well as the R.java file. It should be noted that this package is merely a prototype of the APK and does not yet include the .dex files, .so files as well as the final signatures. The relevant parameters pertaining to this stage are explained as follows:

• The inclusion of the android.jar file is necessary at the current stage. This is because resource files may make references to system-related resources.
• The R.java file will not be generated by default. However, if desired, the --java parameter can be specified, along with the designated file generation path (the package name does not need to be explicitly specified as it will be generated automatically).

## 2. Compile and Link resources
  echo "Compile resources"
  appResDir="./src/main/res"
  manifest="./src/main/AndroidManifest.xml"
  compileTargetArchive=${buildDir}/compiledRes
  compileTargetArchiveUnzip=${buildDir}/compiledResDir
  linkTarget=${buildDir}/res.apk
  r=${buildDir}/r
  
  ### Compile
  echo "AAPT2 compiling"
  aapt2 compile -o ${compileTargetArchive} --dir ${appResDir}
  unzip -q ${compileTargetArchive} -d ${compileTargetArchiveUnzip}
  echo -e "aapt2 intermediates \r\n  - compiled res zip archive : \
    ${compileTargetArchive} \r\n  - unzip res from above : \
    ${compileTargetArchiveUnzip} "

  
  ### Link
  echo "AAPT2 Linking"
  linkInputs=$(find ${compileTargetArchiveUnzip} -type f | tr '\r\n' ' ')
  aapt2 link -o ${linkTarget} -I ${androidJar} \
    --manifest ${manifest} --java ${r} ${linkInputs}
  echo -e "aapt2 generated \r\n  - R.java : ${r} \r\n \
    - res package : ${linkTarget}"

To maintain simplicity, the process described above does not account for resource merging. For advanced AGP resource handling, please refer to Chapter 5.

1-4-3: Compile Java Source Code

The process of compiling Java code should be a concept well-understood by most, as it involves two distinct stages:

1. The transformation of .java files into .class files (which may include .jar files).
2. The conversion of .class files into .dex files.

## 3. Compile Java code
  echo "Compile classes"
  classesOutput=${buildDir}/classes
  mainClassesInput="./src/main/java/$package/manualbuild/*.java"
  rDotJava=${r}/me/xx2bab/buildinaction/manualbuild/R.java
  mkdir ${classesOutput}
  echo "Created a classes output directory at ${classesOutput}"
  
  ## .java -> .classes
  echo "javac (.java -> .classes)"
  javac -d ${classesOutput} ${mainClassesInput} ${rDotJava} \
    -classpath ${androidJar}
  echo "javac generated ${classesOutput}"
  
  ## .classes -> .dex
  echo "D8 (.classes -> .dex)"
  dexOutput=${buildDir}/dex
  mkdir ${dexOutput}
  d8 ${classesOutput}/me/xx2bab/buildinaction/manualbuild/*.class \
    --lib ${androidJar} --output ${dexOutput}
  echo "D8 generated ${dexOutput}"

The initial step of changing .java files into .class files can be efficiently accomplished using javac, the standard Java compiler included with the Java Development Kit (JDK). This compiler translates Java source code into bytecode, which is then stored in .class files or packed into .jar files, a format commonly used for aggregating numerous Java classes into a single distributable archive.

I would like to highlight a couple of important aspects that pertain to the process of Java code compilation:

• The source code must incorporate the R.java file, which is generated during a previous resource compilation stage. This file plays a pivotal role as it serves as an intermediary between the application's code and its resources, mapping all resource IDs to their corresponding Java objects.
• Similar to the resource compilation phase, it is necessary to include the android.jar file as a fundamental dependency in compile classpath. During the actual runtime stage, the Android system itself will provide this environment.

Compared to the AGP build procedure, we have not included the following:

• The Annotation Processor (AP): If AP is required, we need to traverse the annotations in the code using the -processor parameter before initiating the actual Java compilation. The Kotlin compiler has a similar mechanism, more details can be found in Chapter 6.
• Using other languages: When writing Android applications in languages other than Java, such as Groovy or Kotlin, the build process will utilize corresponding compilation tools to substitute javac. These tools will carry out similar tasks to javac, but tailored to the syntax and constructs of the respective languages. Though they are beyond the scope of this discussion, it's important to be aware of their existence and use.

The second step involves converting .class files into .dex (Dalvik Executable) files. This step is essential because the Android runtime, known as the Dalvik Virtual Machine, utilizes a different instruction set than standard Java Virtual Machine. Consequently, JVM (or Java) bytecode must be transformed into a format that the Dalvik VM can interpret, which is where .dex files come into play.

To complete the aforementioned task, we used the D8 compiler. The Android SDK currently provides two builders for *.dex: D8 and R8. R8 has further features like obfuscation, but doing so necessitates self-compilation for R8 tool since it isn't included by default in the SDK folder, making it less convenient for our needs. So, we settled on D8 as the most straightforward and affordable tool.

To better grasp the roles and distinctions of these tools, let's examine their historical development. In the early Android build toolchain, the DX tools were used to compile .dex files. DX translates Java .class or .jar files to .dex format similarly to D8, but the conversion result is not adequately optimized and more memory was consumed during compilation. Later, Google introduced a novel toolchain known as Jack-and-Jill. This tool set aimed to bypass the intermediary .class file stage and directly generate the final .dex file from .java files. Jack served as a replacement for the DX compiler with several advantages:

• It ran as an independent service, eliminating the need to start a JVM to load and run Jack for each instance. This was similar to the Gradle Daemon from the compiler tool perspective and allowed control over the number of concurrent tasks to prevent overloading.
• It had the ability to pre-dex and cache pre-processed files for libraries, enabling incremental compilation to reduce compilation time (though with certain limitations).
• Jack also has a built-in obfuscation function, which can be compatible with ProGuard rules.

Jill was an auxiliary tool that converted early .jar packages into Jayce, the internal file format of Jack. However, the Jack-and-Jill toolchain faced several challenges:

• Omitting the .class file compilation process made it incompatible with many existing toolchains, leading to significant migration costs. Tools such as javac were bypassed, complicating the Annotation Processor support.
• Incorporating support for the Kotlin ecosystem resulted in additional compatibility issues, it will inevitably increase the effort required to support Kotlin, and it is impossible to obtain future optimization from the corresponding development tools, such as javac and kotlinc.
• There were also substantial external pressures related to changes in the associated toolchains for third-party tools migration.

By 2017, the Android team had released partial support for Java 8 on Android and announced that it would support both Jack and DX (using Desugaring tools combined with the AGP Transform API). It made it clear that they would abandon Jack and return temporarily to DX.

Later that year, the first version of D8 released:

• It inherits the complete chain of .java -> .class -> .dex, trying to keep the original toolchain intact.
• It compiles faster and outputs smaller DEX files than DX.
• It is designed to work in combination with R8, which is a tool for code shrinking and optimization.
• It offers better and more efficient desugaring of Java 8 language. The desugaring process is post-positioned between the end of the AGP Transform and the conversion to .dex, ensuring that all bytecodes can be correctly executed desugaring.

In subsequent updates, the Android team announced the deprecation of DX and the AGP no longer supported it starting from version 7.0.0. For D8 and R8, they share the same codebase and use the same basic toolchain. While D8 is equipped to carry out simple optimization operations, R8 offers a comprehensive suite of optimizations, encompassing obfuscation among others.

Back to the script itself, the D8 command requires the input of the .class files and android.jar output from the prior step. Upon successful execution, the command yields .dex files as output. These output files can be located in the ./app/out/dex directory.

1-4-4: Package APK

The final stage is to package the entire APK file.

## 4. Build APK
  echo "Package and Sign the APK"
  tools=${sdk}/tools/lib
  originApk=${buildDir}/manual-build-unaligned-unsigned.apk
  alignedApk=${buildDir}/manual-build-aligned-unsigned.apk
  zipAlignedSignedApk=${buildDir}/manual-build-aligned-signed.apk
  
  ## Package APK zip file
  echo "Packaging"
  java -cp $(echo ${tools}/*.jar | tr ' ' ':') \
    com.android.sdklib.build.ApkBuilderMain ${originApk} \
    -u -v -z ${linkTarget} -f ${dexOutput}/classes.dex
  echo "[PG][ManualBuild] Built apk by ApkBuilderMain at ${originApk}"
  
  ## Zipalign the original APK
  echo "Zip aligning"
  zaTool=$(dirname $(which aapt2))/zipalign
  ${zaTool} 4 ${originApk} ${alignedApk}
  echo "[PG][ManualBuild] APK aligned ${alignedApk}"
  
  ## Sign the aligned APK
  echo "Signing"
  apksigner sign --ks ./debug.keystore --ks-key-alias androiddebugkey \
    --ks-pass pass:android --key-pass pass:android \
    --out ${zipAlignedSignedApk} ${alignedApk}
  echo "APK is signed"
  echo "Build completed, check the final apk at ${zipAlignedSignedApk}"

At the outset of the script, we establish a collection of tool variables as usual, and then we initiate the packaging process:

• Initially, we leverage the ApkBuilderMain class from the sdklib to generate an unsigned, raw APK. Its primary role involves adding classes.dex to the APK package produced by AAPT2, and outputting a rudimentary APK file. ApkBuilderMain is the only deprecated utility within the script. Currently, the APK packaging in AGP no longer depends on this tool. Here, for the sake of simplicity, we have chosen it. You could alternatively utilize Zip related commands to accomplish the same task, albeit with manual verification of the relevant parameters and execution of the merge operation.
• Subsequently, we utilize the ZipAlign tool to perform a 4-byte alignment optimization on the APK, thereby enhancing the system's efficiency in accessing APK resources.
• Ultimately, we use ApkSigner to sign the APK. There are notable differences for JarSigner and ApkSigner:
  • JarSigner is essentially designed for Jar packages. Despite its usage in the early Android build tools, the subsequent APK signature process has been entirely replaced by ApkSigner since SDK 24.
  • Some of the signature standards for Jars are not necessary for APKs, which may introduce superfluous checks and warnings. For instance, not adding -tsa or -tsacert when signing will throw a "is not timestamped" warning.
  • By default, JarSigner in JDK 7 and above defaults the signature algorithm to SHA256. However, this algorithm is not supported by Android 4.2 and below, necessitating the switch to different algorithms during the signing process.
  • Android's signature standards are constantly updated, and the v2 and later standards can only be signed with ApkSigner.

For different Android output formats, the following processing standards are applied:

• If the APK uses JarSigner, it needs to be signed first, followed by optimization with ZipAlign.
• If the APK uses ApkSigner, it needs to be optimized with ZipAlign first and then signed.
• The signature generation of Android Archive Library (AAR) files is optional, however required in some Maven repository publications.
• Android Application Bundle (AAB) currently uses JarSigner for signing.

Upon completion of the signing process, the final APK (named manual-build-aligned-signed.apk) can officially be installed and executed on the Android device (Figure 1.4.2).

1-4-5: In Summary

In this section, we have delved into the intricacies of Android APK packaging tools, the resource building process, Java code compilation, and APK assembly. By initiating an investigation with the low-level tools, we've sought to comprehend the basic APK build procedure, including the input and output files at each stage.

Additionally, whether one is a seasoned developer or a beginner to Android, it is advisable to familiarize oneself with the most recent build tools. On the one hand, older versions of these tools lack optimizations compatible with modern Android operating systems; on the other hand, newer tools can offer increased customization options, better fulfilling the requirements of Android custom builds. In the following chapters, we will go into more detail regarding custom builds.

To distill the essence of the building process, we would like to introduce two keywords: Transformation and Aggregation.

• Transformation: This encapsulates the process of converting a source file into a target artifact, which can be either compile-type or non-compile-type.
  • Compile-type transformations, such as those carried out by javac, kotlinc, aapt2 compile, etc., transform human-readable source code files into lower-level programs that are better suited for parsing or execution by computers.
  • Non-compile-type transformations, such as obfuscation, essentially simplify and replace text without altering the file type or downgrading from a higher to a lower level.
• Aggregation: This denotes the process of collecting and integrating a specific type of source file or artifact. The tool that performs this operation is termed an Aggregator in this book. For instance, before compiling .dex, all .class files need to be aggregated as input, and .arsc requires the aggregation of all resource files as input. In Sections 3-6, 6-3, and 7-5, we will examine more implementations and applications of aggregators.