Code Coverage not reported for instrumented tests (Android library module) on AGP 7.0

[raquezha]

Hi I'm trying this on my test multi-module project before trying this on the other project I'm working on. I've followed the steps and run ./gradlew rootCoveragePlugin but got this report:

--
com.raquezha.heograpiya.splash is 0% but when I run it in Android Studio local coverage it recognize the tests:

--

here's the file structure
as you can see Android Studio recognizes the local code coverage of splash module and says its 100% lines covered

here's my top-level build.gradle

// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {
    apply from: rootProject.file('dependencies.gradle')

    repositories {
        google()
        mavenCentral()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:7.0.0-alpha15'
        classpath 'org.jetbrains.kotlin:kotlin-gradle-plugin:1.5.0'
        classpath 'com.google.gms:google-services:4.3.5'
        classpath 'nl.neotech.plugin:android-root-coverage-plugin:1.4.0'

        // NOTE: Do not place your application dependencies here; they belong
        // in the individual module build.gradle files
    }
}

plugins {
    id "nl.neotech.plugin.rootcoverage" version "1.4.0"
}

apply plugin: 'nl.neotech.plugin.rootcoverage'

jacoco {
    toolVersion = "0.8.7"
}

tasks.withType(Test) {
    useJUnitPlatform()
    testLogging {
        exceptionFormat "full"
        events "started", "skipped", "passed", "failed"
        showStandardStreams true
    }
}

subprojects {
    afterEvaluate { project ->
        if (project.hasProperty('android')) {
            android {
                compileSdk 30
                buildToolsVersion "30.0.3"

                defaultConfig {
                    minSdk 26
                    targetSdk 30
                    versionCode 1
                    versionName "1.0"

                    testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
                }

                buildTypes {
                    release {
                        minifyEnabled false
                        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
                    }

                    debug {
                        testCoverageEnabled true
                    }
                }

                compileOptions {
                    sourceCompatibility JavaVersion.VERSION_1_8
                    targetCompatibility JavaVersion.VERSION_1_8
                }

                kotlinOptions {
                    jvmTarget = '1.8'
                }

                buildFeatures {
                    viewBinding true
                }

                sourceSets {
                    main {
                        java.srcDirs = ['src/main/kotlin']
                    }
                    androidTest {
                        java.srcDirs = ['src/androidTest/kotlin']
                    }
                    test {
                        java.srcDirs = ['src/test/kotlin']
                    }
                }
            }
        }
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

here's the splash module build.gradle:

plugins {
    id 'com.android.library'
    id 'kotlin-android'
}

dependencies {
    implementation _libs.lottie

    implementation project(':shared')

    testImplementation project(path: ':shared', configuration: 'testDependencies')
    androidTestImplementation project(path: ':shared', configuration: 'androidTestDependencies')
}

tasks.withType(Test) {
    useJUnitPlatform()
    testLogging {
        exceptionFormat "full"
        events "started", "skipped", "passed", "failed"
        showStandardStreams true
    }
}

I'm using:

1. Android Studio Arctic Fox | 2020.3.1 Canary 15
2. Gradle JDK: AdoptOpenj9-11
3. gradle-wrapper: distributionUrl=https://services.gradle.org/distributions/gradle-7.0-bin.zip

I'm using canary because this is only a test project I want to test new stuff

[Rolf-Smit]

I'm not exactly sure what causes this, but I suspect it has something to do with using:

subprojects {
    afterEvaluate { project ->
        // Modifying srcDirs here
    }
}

It could be that the plugin gets configured before you configure the new srcDirs, meaning that the plugin will not see these? You could try to move the plugin apply block to after this block. But this is just me guessing.

Can you share this project with me so I can do some testing?

[raquezha]

Hi thanks for helping! you can check the project here: https://github.com/raquezha/heograpiya

[raquezha]

I've tried applying the apply block after subProjects{ } block but it did not work still 0%

[Rolf-Smit]

Hi @raquezha took me a while to take a look at this, but I found a few things:

Looking at the report that is generated, it is missing all execution data, which is needed to calculate coverage. You can check this for your self by going to the sessions part of the report:

This can have multiple causes, but in this case this happens because the Android Gradle Plugin updated where these files are located since version 7.0. This means the Plugin needs a small update.

However I also spotted this, which will also prevent coverage (even if I update the plugin):

In the root project build.gradle file you have this line:

getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"))

This basically overrides any of the default set execution data location's with the location you provided. Your location only looks at the buildDir of the root project, but normally execution data is generated in all the different module buildDir's, so this will definitely not work if you ask me.

TLDR: This plugin needs an update due to changes in the Android Gradle Plugin 7.0. I'm going to take care of this. (see update this is no longer something I can fix)

[raquezha]

Thank you! looking forward for the fix, If you need someone to test, I'll gladly help!

[Rolf-Smit]

Update:
I've done some work on making the plugin work with AGP 7.0. The integration test for this plugin now succeeds (which it didn't before because some coverage was missing).

However, on your specific project, there is still coverage missing. So I'm investigating what exactly causes that.

[Rolf-Smit]

Update 2:

Kotlin 1.5 is the issue?

Did some more testing and I found that even though there is a small change in AGP 7.0 (they moved the location of the .exec for unit tests to a different location). After fixing that there is still no unit-test coverage (instrumented unit-tests work fine).

I found that the generated .exec file for unit-tests is basically empty, which is weird.

When I downgrade Kotlin to version 1.4.x everything works as expected, the ` In other words, it seems this is not specifically a AGP 7.0 issue, but more likely a JaCoCo or Kotlin issue. On the other hand AGP 7.0 itself depends on Kotlin 1.4, so maybe that is what's causing the issue?

I also tried upgrading to Gradle 7.1, JaCoCo 8.8 (snapshot), Kotlin 1.5.10. But that didn't work.

For now I only know that downgrading to Kotlin 1.4, seems to work (as a workaround).

Coverage in Android Studio

Now the reason you are seeing coverage in Android Studio is because you are probably running the Intellij code coverage runner, and not the Jacoco coverage runner. If you switch to Jacoco you will notice that Android Studio will not report any coverage.

I will try to search further but as of now I don't have any idea's left. Maybe I should file a bug? But where? JaCoCo? Kotlin?

[kherink]

hi, is there any progress on the kotlin 1.5 coverage issue? if help is needed let me know.

[Rolf-Smit]

@kherink I could use some help here in finding out which tool actually causes the missing coverage. I'm currently working on a bare-minimal example that reproduces this (without my Plugin being involved), it's work in progress but at least I'm able to reproduce it*.

*Although it is already reproducible by running code coverage via Android Studio (IntelliJ) using JaCoCo, thus skipping this plugin, the Gradle JaCoCo plugin and Android Gradle plugin.

[kherink]

@Rolf-Smit I will have a look and let you know what I find.

[CoenQian]

@kherink I could use some help here in finding out which tool actually causes the missing coverage. I'm currently working on a bare-minimal example that reproduces this (without my Plugin being involved), it's work in progress but at least I'm able to reproduce it*.

*Although it is already reproducible by running code coverage via Android Studio (IntelliJ) using JaCoCo, thus skipping this plugin, the Gradle JaCoCo plugin and Android Gradle plugin.

@Rolf-Smit hi, you can reproduce it by this example.
Wrong coverage after update Kotlin to 1.5.10

[kherink]

@Rolf-Smit i'll look at it this weekend

[AshMosahebTAB]

Hi I don't use this plugin.. But I came across this issue when finding the same problem with code coverage with my own custom script after upgrading to AGP 7+ and Kotlin 1.5+.
After extensive messing around with my script and other configurations, I found that this line seemed to be the culprit...

testCoverageEnabled true
in here:

android {
    buildTypes {
        debug {
            testCoverageEnabled true
        }
    }
}

Coverage now seems to now work for me (at least my test reports now show hits), by omitting this line. Not sure what has changed in AGP, and if this effects anything else, but I thought i'd share here anyway, just it case it helps someone, or helps progress the investigation further.

[raquezha]

This library needs testCoverageEnabled true in order to work. But thanks anyway, this can be useful for someone who encounter problem same as yours.

[Rolf-Smit]

@AshMosahebTAB

Thanks for taking the time to add this comment here! I also noticed this but figured it was a bug, since you would expect that when this property is true it would actually instrument the classes, omitting it would make me think it would not instrument the classes.

Anyhow, this library indeed uses the testCoverageEnabled property to see which modules to include in the combined coverage report. If removing this property works I can rework the library to no longer use that property, possibly introducing a different method to tell the plugin which modules to use for the combined report.

[trietbui85]

Hi @Rolf-Smit Does the testCoverageEnabled mean to run coverage for JUnit test (/test - both pure Java/Kotlin & Android unit test), or Instrumented test (/androidTest), or both?

[AshMosahebTAB]

It's worth noting, that this bug has been raised here https://issuetracker.google.com/issues/195860510

[Rolf-Smit]

@AshMosahebTAB thanks for sharing that here. I'm at this point 100% sure that this is not something I can fix, but this is something that needs fixing on the Android (Google) side of things.

[Rolf-Smit]

@anticafe

Hi @Rolf-Smit Does the testCoverageEnabled mean to run coverage for JUnit test (/test - both pure Java/Kotlin & Android unit test), or Instrumented test (/androidTest), or both?

This library does not introduce testCoverageEnabled (it is a property that is part of the Android Gradle Plugin), this library just looks at that property to determinate whether or not to create coverage report tasks for a certain module. This property does not dictate whether or not certain tests run.

In the README you can find information on how to configure this library in such a way that it either executes instrumented tests (test that run on an Android device) or pure Java unit tests (tests that run on your machine): https://github.com/NeoTech-Software/Android-Root-Coverage-Plugin#3-configuration

[Rolf-Smit]

Update:

It seems Google finally stepped in and fixed the issue in AGP version 7.2.0: https://issuetracker.google.com/issues/195860510#comment12

Unfortunately the fix will not be released this month, they expect to release a beta version of AGP 7.2.0 that includes this fix around December 13th.

[Rolf-Smit]

A SNAPSHOT release is now available that supports Android Gradle Plugin version 7.2.0-alpha06.

Since this SNAPSHOT is only available in Maven Central (Gradle Plugin Portal does not support SNAPSHOT releases), you cannot use this release using the normal id block, instead you will need to add it using an older method:

apply plugin: 'nl.neotech.plugin.rootcoverage'

buildscript {
    repositories {
        // ...
        mavenCentral()
    }
   dependencies {
      classpath 'nl.neotech.plugin:android-root-coverage-plugin:1.5.0-SNAPSHOT'
   }
}

Once there is a stable 7.2. release I will release a stable 1.5 release (or when Google decides to fix this on top of 7.0.4 or 7.1-beta I will create a special release for those).

[jernej7]

@Rolf-Smit
I tried to switch from 1.4.0 to 1.5.0-SNAPSHOT using the same code you provided above, but Gradle reports an error Could not find nl.neotech.plugin:android-root-coverage-plugin:1.5.0-SNAPSHOT., while it has no problems with 1.4.0 using the same method.

Searched in the following locations:
  - https://repo.maven.apache.org/maven2/nl/neotech/plugin/android-root-coverage-plugin/1.5.0-SNAPSHOT/maven-metadata.xml
  - https://repo.maven.apache.org/maven2/nl/neotech/plugin/android-root-coverage-plugin/1.5.0-SNAPSHOT/android-root-coverage-plugin-1.5.0-SNAPSHOT.pom

At these locations only versions up to 1.4.0 are available (at least publicly visible).
I am not familiar with snapshot versions. Do I have to add a specific maven (snapshot) repository for the 1.5.0-SNAPSHOT to work?

[Rolf-Smit]

@jernej7

You indeed need to use a special repository, the snapshot repository from Maven Central:

maven {
    url 'https://s01.oss.sonatype.org/content/repositories/snapshots/'
}

You must add this reposiotry to your plugin repository section. Also, since this is not the official Gradle Plugin Portal repository, you might also need to map the plugin ID to an actual maven module (only needed for SNAPSHOTS, or when not using the official Gradle Plugin Portal repository):

settings.gradle (top of the file):

pluginManagement {
    repositories {
        mavenCentral()
        maven {
            url 'https://s01.oss.sonatype.org/content/repositories/snapshots/'
        }
        gradlePluginPortal()
    }
}

[Rolf-Smit]

Edit:

I might have been stupid and never published a special Gradle Plugin Marker artifact to maven central, without a Marker artifact Gradle cannot map the plugin ID to a maven module.

For SNAPSHOTS this is now fixed (I just published a new one), so you no longer need this to use SNAPSHOTS:

    resolutionStrategy {
        eachPlugin {
            if (requested.id.id == 'nl.neotech.plugin.rootcoverage') {
                useModule("nl.neotech.plugin:android-root-coverage-plugin:${requested.version}")
            }
        }
    }