Java: Include Gradle dependencies in CodeQL database

[Marcono1234]

Because CodeQL only includes files which are compiled during build in the database, a query might not yield all desired results in case dataflow or taintflow occurs in a third party dependency and CodeQL itself does not model that dependency.

Therefore I tried to see whether it is possible to modify a Gradle build to also compile the sources of dependencies. The result is the following Gradle Kotlin DSL buildscript modification:

tasks.compileJava {
    doFirst {
        // Cannot make CodeQL include dependency sources when there is a module-info.java file, see below
        if (sourceSets.main.get().allJava.any {it.name == "module-info.java"}) {
            logger.log(LogLevel.WARN, "Source contains module-info.java; cannot include dependency sources for CodeQL")
            return@doFirst
        }

        logger.info("Preparing dependency sources")

        /*
         * Get source artifacts, see
         * - https://stackoverflow.com/a/39981143
         * - https://github.com/gradle/gradle/blob/5ec3f672ed600a86280be490395d70b7bc634862/subprojects/kotlin-dsl-tooling-builders/src/main/kotlin/org/gradle/kotlin/dsl/tooling/builders/KotlinLibSources.kt#L104-L113
         */
        val componentIdToArtifact =
            configurations
                .compileClasspath
                .get().incoming
                //.artifactView { lenient(true) }
                .artifacts
                .associateBy { it.id.componentIdentifier }
        val artifactsToRemove = mutableSetOf<File>()
        val sourceFileArtifacts = mutableSetOf<File>()
        dependencies.createArtifactResolutionQuery()
            .forComponents(componentIdToArtifact.keys)
            .withArtifacts(JvmLibrary::class.java, SourcesArtifact::class.java)
            .execute()
            .resolvedComponents
            .flatMap { it.getArtifacts(SourcesArtifact::class.java) }
            .filterIsInstance<ResolvedArtifactResult>()
            .forEach {
                artifactsToRemove.add(componentIdToArtifact[it.id.componentIdentifier]!!.file)

                // Manually unzip sources JAR and remove module-info.java files, otherwise CodeQL does not include
                // the files as source files in the database, possibly related to https://github.com/github/codeql/issues/5556
                // Once that is solved can replace this with just: sourceFileArtifacts.add(it.file)
                // Can then also remove module-info.java check at beginning

                val codeQlSourcesDir = project.buildDir.toPath().resolve("codeql-dependency-sources")
                val dependencySourceDir = codeQlSourcesDir.resolve(it.file.name)

                if (!Files.isDirectory(codeQlSourcesDir)) {
                    Files.createDirectory(codeQlSourcesDir)
                } else if (Files.isDirectory(dependencySourceDir)) {
                    logger.info("Sources for ${it.id.displayName} already exist; skipping extraction")
                    return@forEach
                }

                copy {
                    from(zipTree(it.file))
                    into(dependencySourceDir)
                    exclude("**/module-info.java")
                }
                sourceFileArtifacts.add(dependencySourceDir.toFile())
            }

        logger.info("Finished preparing dependency sources")

        // Remove files from classpath for which source artifacts exist
        classpath = classpath.filter {it !in artifactsToRemove}
        // Adjust sourcepath to include source artifacts
        options.sourcepath = files(sourceSets.main.get().allJava.map(File::getParent)).plus(files(sourceFileArtifacts))
        options.sourcepath!!.forEach {
            logger.info("Source: $it")
        }
    }
}

This is probably a rather hacky and unreliable solution (suggestions are welcome), but for a small demo project using Gson, this seemed to work fine. It works by requesting the source artifacts of the dependencies, including them with javac's --source-path, and removing the corresponding JAR artifact from the classpath. I chose --source-path here because it only specifies where source files can be found, without requiring all of them to be compiled (in case they are not used). For example if the main Java sources only use Gson's FieldNamingPolicy, then only the classes FieldNamingPolicy and FieldNamingStrategy (the superinterface) are compiled.

There are however some limitations:

• The dependency needs to have a source artifact.
• The sources of the source artifact need to be compilable with the same configuration as the current project. (Maybe there are some corner cases where this is not possible.)
• The build script section above modifies the default compileJava task; for a real project this is most likely not desired and instead a separate task should be used.
• The --source-path approach might not actually be that efficient compared to treating the third-party sources as regular main sources and might still end up compiling large parts of the complete sources of the dependencies.
• There is a limitation (or bug?, maybe related to Java: CompilationUnit elements in Module are .class files instead of .java files #5556) with CodeQL where it does not extract source files specified by --source-path if a module-info.java file is present. The script above currently tries to work around that, but it might not be able to handle all cases.

Hopefully this is nonetheless useful for someone :)

[smowton]

Bear in mind it isn't necessary for the build of both projects to happen in a single Gradle run. You could simply make a small shell script that builds GSON using Maven and then your project using Gradle, or pass the -c option to codeql database create twice for the same effect. In either case you should pass database create the --source-root option specifying a directory that contains the source files for both projects (this defaults to the current working directory).

[Marcono1234]

Thanks for the reminder! That would definitely be a good workaround for cases where the above build script does not work.