Create a Cyclone DX SBOM equivalent to the current build build-info json metadata #2753

andrew-m-leonard · 2021-10-11T09:44:39Z

As part of the strategy to supply Cyclone DX SBOM info for Adoptium binaries, this issue covers the initial task of creating a Cyclone DX SBOM equivalent of the current binary metadata build-info.
This metadata can be seen here as an example: https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17%2B35/OpenJDK17-jdk_x64_linux_hotspot_17_35.tar.gz.json
To view these files it is best to download and view using a suitable JSON viewer. (you can download, and open in a browser).
This JSON contains numerous pieces of "build-info",eg:

"version" : Version info for the binary, eg.what openjdk tag, build, ..
scmRef: What adoptium openjdk mirror git tag it was built from
buildRef : What level (git commit SHA) of temurin-build scripts were used to build it
full_version_output: the built binary "java -version" string
makejdk_any_platform_args: the temurin build make arguments used
configure_arguments: the openjdk "Configure" arguments used
make_command_args: the openjdk make arguments used
openjdk_built_config: the Temurin build scripts configuration generated
openjdk_source: the openjdk repo source that was built (git SHA)
build_env_docker_image_digest: if built within a docker container, this is the exact SHA digest of the image it was built from
dependency_version_alsa|freetype|freemarker: various dependencies built with, URLs to the version used

The intention of this task is to take this current build-info and generate a CycloneDX SBOM for it.

andrew-m-leonard · 2021-10-11T09:47:53Z

@SehrishHussain fyi

SehrishHussain · 2021-10-11T13:26:34Z

@andrew-m-leonard So, I searched in CycloneDX Tool Center as well as the github repository for CycloneDX (github CycloneDX) yet I couldn't find the right tool to convert the above Json file into CycloneDX SBoM.
What am I missing here? Your help is highly appreciated.

andrew-m-leonard · 2021-10-11T14:28:36Z

@SehrishHussain this link provides some useful "usecases": https://cyclonedx.org/use-cases/
We're going to have to be a bit inventive, and apply one of these use cases to our openjdk binaries. Probably a "library" type Component.
@spoole167 Steve, do you have any CycloneDX SBOM experts that can give guidance on how to get started please? I think CycloneDX could do with the "Hello World" of SBOM's!

smlambert · 2021-10-11T14:31:47Z

related: #2681

andrew-m-leonard · 2021-10-11T14:33:56Z

@SehrishHussain Looking at the use case examples, take the Inventory example, for us we could generate:

{
  "bomFormat": "CycloneDX",
  "specVersion": "1.3",
  "serialNumber": "urn:uuid:x-y-z",
  "version": 1,
  "components": [
    {
      "type": "library",
      "name": "temurin-binary",
      "version": "11.0.12"
    }
  ]
}

SehrishHussain · 2021-10-12T05:52:56Z

@andrew-m-leonard my question was/is still the same i.e. how/where do we generate these CycloneDX SBoM? I have been searching over the internet and asked Patrick Dwyer he is the core team of CycloneDX, we have scheduled a call this week. So far, we have to use multiple tools to generate SBoM.
Meanwhile I want to work on another task in parallel. What do you suggest please.

SehrishHussain · 2021-10-12T11:58:13Z

So far this is what I have learnt in generating SBOMs.

Steps to generate CycloneDX SBOMs:

From CycloneDX SBOM Tool Center forked the CycloneDX CLI tool to local repo
Downloaded the relevant Binaries.
However, the latest release of binaries for CycloneDX CLI doesn't execute on my machine for some reason. I'm in contact with Patrick Dwyer regarding the issue.
Run the relevant commands to generate required use case of SBOM.

@andrew-m-leonard @sxa you want to add to the process I mentioned above?

SehrishHussain · 2021-10-13T17:50:42Z

@andrew-m-leonard @smlambert Industry best practices suggest to automate SBOM but for starters I have created one manually. Please, take a look.
Where can we access the build-info for Adoptium binaries, since to replicate CycloneDX SBOM use cases additional information is required.

`
{
"bomFormat": "CycloneDX",
"specVersion": "1.3",
"serialNumber": "urn:uuid:3e671687-395b-41f5-a30f-a58921a69b79",
"version": 1,
"metadata": {
"vendor": "Eclipse Adoptium",
"os": "linux",
"arch": "x64",
"variant": "hotspot",
"version": {
"minor": 0,
"patch": null,
"msi_product_version": "17.0.0.35",
"security": 0,
"pre": null,
"adopt_build_number": null,
"major": 17,
"version": "17+35",
"semver": "17.0.0+35",
"build": 35,
"opt": null
},
"scumRef": "jdk-17+35_adopt",
"buildRef": "https://github.com/adopt…rin-build/commit/c3a40ff",
"version_data": "jdk17",
"binary_type": "jdk",
"hashes": [
{
"alg": "SHA-256",
"content": "6f1335d9a7855159f982dac557420397be9aa85f3f7bc84e111d25871c02c0c7"
}

    ],
    "full_version_output": "openjdk version \"17\" 2021-09-14\nOpenJDK Runtime Environment Temurin-17+35 (build 17+35)\nOpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing)\n",
    "make_command_args": "make product-images legacy-jre-image   test-image\n",
    "openjdk_source": "https://github.com/adoptium/jdk17/commit/a5afad28437",
    "build_env_docker_image_digest": "[adoptopenjdk/centos6_build_image@sha256:e9fa19de1a830399a91044a277a6cca7bbd915322187825bfd4cfa752917adab]\n",
    "dependency_version_alsa":  "https://ftp.osuosl.org/pub/blfs/conglomeration/alsa-lib/alsa-lib-1.1.6.tar.bz2\n"


}

}
`

andrew-m-leonard · 2021-10-13T18:10:28Z

@SehrishHussain great work Sehrish, that looks a great first schema.
So agree automation through one of the available plugins would be the desired way, but as you can see what we have is rather bespoke at the moment.
We need to break down the current build-info, into constituent components, some info is "version/input" info, some is "how it was made" info (Assembly), and some is "3rd party dependencies".
The BUILT_CONFIG info from the build-info for example needs better organization and maybe bits of it categorizing:
https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17%2B35/OpenJDK17-jdk_x64_linux_hotspot_17_35.tar.gz.json

SehrishHussain · 2021-10-14T12:46:11Z

@andrew-m-leonard thank you, I'm learning to understand it better. I will organize the BUILT_CONFIG info. Meanwhile would you assign this task to me. I assume that's how we do it for Outreachy applicants.

SehrishHussain · 2021-10-14T18:05:39Z

@andrew-m-leonard I have categorized each key-value pairs for "BUILT_CONFIGURATION".

"BUILD_CONFIGURATION": {
"ARCHITECTURE": "x64",
"TARGET_OS": "linux",
"VARIANT": "hotspot",
"JAVA_TO_BUILD": "jdk17",
"TEST_LIST": [
"sanity.openjdk",
"sanity.system",
"extended.system",
"sanity.pref",
"sanity.functional",
"extended.functional",
"extended.openjdk",
"extended.pref",
"special.functional"
],
"DYNAMIC_LIST": [
"extended.openjdk"
],
"NUM_MACHINES": 3,
"SCM_REF": "jdk-17+35_adopt",
"BUILD_ARGS": "",
"NODE_LABEL": "build&&linux&&x64",
"ADDITIONAL_TEST_LABEL": "",
"KEEP_TEST_REPORTDIR": true,
"ACTIVE_NODE_TIMEOUT": 0,
"CODEBUILD": false,
"DOCKER_IMAGE": "adoptopenjdk/centos6_build_image",
"DOCKER_NODE": "",
"DOCKER_REGISTRY": "",
"DOCKER_CREDENTIAL": "",
"PLATFORM_CONFIG_LOCATION": "adoptium/temurin-build/master/build-farm/platform-specific-configurations",
"CONFIGURE_ARGS": "--enable-dtrace",
"OVERRIDE_FILE_NAME_VERSION": "",
"USE_ADOPT_SHELL_SCRIPTS": true,
"RELEASE": true,
"PUBLISH_NAME": "jdk-17+35",
"ADOPT_BUILD_NUMBER": "",
"ENABLE_TESTS": true,
"ENABLE_INSTALLERS": true,
"ENABLE_SIGNER": true,
"CLEAN_WORKSPACE": true,
"CLEAN_WORKSPACE_AFTER": false,
"CLEAN_WORKSPACE_BUILD_OUTPUT_ONLY_AFTER": true
}

JSON file:
{ "bomFormat": "CycloneDX", "specVersion": "1.3", "serialNumber": "urn:uuid:3e671687-395b-41f5-a30f-a58921a69b79", "version": 1, "metadata": { "vendor": "Eclipse Adoptium", "os": "linux", "arch": "x64", "variant": "hotspot", "version": { "minor": 0, "patch": null, "msi_product_version": "17.0.0.35", "security": 0, "pre": null, "adopt_build_number": null, "major": 17, "version": "17+35", "semver": "17.0.0+35", "build": 35, "opt": null }, "scumRef": "jdk-17+35_adopt", "buildRef": "https://github.com/adopt…rin-build/commit/c3a40ff", "version_data": "jdk17", "binary_type": "jdk", "hashes": [ { "alg": "SHA-256", "content": "6f1335d9a7855159f982dac557420397be9aa85f3f7bc84e111d25871c02c0c7" } ], "full_version_output": "openjdk version \"17\" 2021-09-14\nOpenJDK Runtime Environment Temurin-17+35 (build 17+35)\nOpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing)\n", "make_command_args": "make product-images legacy-jre-image test-image\n", "openjdk_source": "https://github.com/adoptium/jdk17/commit/a5afad28437", "build_env_docker_image_digest": "[adoptopenjdk/centos6_build_image@sha256:e9fa19de1a830399a91044a277a6cca7bbd915322187825bfd4cfa752917adab]\n", "dependency_version_alsa": "https://ftp.osuosl.org/pub/blfs/conglomeration/alsa-lib/alsa-lib-1.1.6.tar.bz2\n" }, "BUILD_CONFIGURATION": { "ARCHITECTURE": "x64", "TARGET_OS": "linux", "VARIANT": "hotspot", "JAVA_TO_BUILD": "jdk17", "TEST_LIST": [ "sanity.openjdk", "sanity.system", "extended.system", "sanity.pref", "sanity.functional", "extended.functional", "extended.openjdk", "extended.pref", "special.functional" ], "DYNAMIC_LIST": [ "extended.openjdk" ], "NUM_MACHINES": 3, "SCM_REF": "jdk-17+35_adopt", "BUILD_ARGS": "", "NODE_LABEL": "build&&linux&&x64", "ADDITIONAL_TEST_LABEL": "", "KEEP_TEST_REPORTDIR": true, "ACTIVE_NODE_TIMEOUT": 0, "CODEBUILD": false, "DOCKER_IMAGE": "adoptopenjdk/centos6_build_image", "DOCKER_NODE": "", "DOCKER_REGISTRY": "", "DOCKER_CREDENTIAL": "", "PLATFORM_CONFIG_LOCATION": "adoptium/temurin-build/master/build-farm/platform-specific-configurations", "CONFIGURE_ARGS": "--enable-dtrace", "OVERRIDE_FILE_NAME_VERSION": "", "USE_ADOPT_SHELL_SCRIPTS": true, "RELEASE": true, "PUBLISH_NAME": "jdk-17+35", "ADOPT_BUILD_NUMBER": "", "ENABLE_TESTS": true, "ENABLE_INSTALLERS": true, "ENABLE_SIGNER": true, "CLEAN_WORKSPACE": true, "CLEAN_WORKSPACE_AFTER": false, "CLEAN_WORKSPACE_BUILD_OUTPUT_ONLY_AFTER": true } }

SehrishHussain · 2021-11-01T18:19:48Z

@andrew-m-leonard I've created first automated SBOM. It's very basic but its automated. Used CycloneDX CLI tool and above Json file. I need to build Temurin locally, for some unkown reason my machine isn't building. Anyhow, I thought to share it here. I will get a new machine with Ubuntu and hopefully will build jdk binaries locally as it is the first step for automating SBOM. Its a JSON file so can't attach it here. Here is the content of file.
{ "bomFormat": "CycloneDX", "specVersion": "1.3", "serialNumber": "urn:uuid:a29e03ec-6cdb-49d3-899e-3263a6d04486", "metadata": { "timestamp": "2021-11-01T17:46:46Z", "tools": [ { "vendor": "CycloneDX", "name": "CycloneDX CLI", "version": "0.19.0.0" } ] }, "components": [] }

andrew-m-leonard · 2021-11-01T18:50:34Z

That's great @SehrishHussain , let me know if you need help getting your local build working.
It would perhaps be nice to do a quick 2min demo?

SehrishHussain · 2021-11-02T06:31:46Z

@andrew-m-leonard yes please, that would be very helpful. Where can we do this demo?

andrew-m-leonard · 2021-12-06T10:16:46Z

@SehrishHussain the above CycloneDX JSON looks good. What we need to do next is get that produced as an output of the build. That probably means something along the lines of the following:

Add CycloneDX tooling to build infrastructure...so it is available at least on certain nodes to build the SBOM. What tools need to be available?
Design how you will invoke and gather the current metadata into the Cyclone DX SBOM...?
Info on the current metadata is here: https://github.com/adoptium/temurin-build/blob/master/README.md#metadata
Code is here for current metadata construction: https://github.com/adoptium/ci-jenkins-pipelines/blob/9556d1f74dfe9ad5cb2dcf14df33468a171533f0/pipelines/build/common/openjdk_build_pipeline.groovy#L715
Actual writing of the Metadata JSON is here: https://github.com/adoptium/ci-jenkins-pipelines/blob/9556d1f74dfe9ad5cb2dcf14df33468a171533f0/pipelines/build/common/openjdk_build_pipeline.groovy#L921
Metadata groovy class : https://github.com/adoptium/ci-jenkins-pipelines/blob/b9b8f47ae6f6828734ede7a4869eee8bae8c6505/pipelines/library/src/common/MetaData.groovy

Feel free to discuss your ideas and designs here..
thanks

andrew-m-leonard · 2021-12-15T14:13:10Z

CycloneDX core-java tool provides Bom json generators which we can use to call from the Jenkins pipline:
https://cyclonedx.github.io/cyclonedx-core-java/org/cyclonedx/generators/json/BomJsonGenerator13.html

The pipeline write metadata code here: https://github.com/adoptium/ci-jenkins-pipelines/blob/9556d1f74dfe9ad5cb2dcf14df33468a171533f0/pipelines/build/common/openjdk_build_pipeline.groovy#L921
can call a Java app using context.sh(script: "java AdoptiumGenSBOM ...")
The provisor to this that we ensure a JDK is available...

The Java app AdoptiumGenSBOM, could be have a command line interface like:

java AdoptiumBom --create bom.json --name Temurin_x64Linux --version jdk-18+26
java AdoptiumBom --update bom.json --component freetype --version 1.18 --hash 24abcdef4545

andrew-m-leonard · 2021-12-17T10:47:42Z

@SehrishHussain the PR for the CycloneDX jenkins build integration is now merged: #2805

To allow you to extend and test changes/additions to TemurinGenSBOM.java, you can run it as follows:
Prereqs locally:

Install a suitable Java, eg.from here: https://adoptium.net/releases.html
Ensure "ant" installed

Building/running..

Clone temurin-build: git clone [email protected]:adoptium/temurin-build.git
cd temurin-build/cyclonedx-lib
ant -f build.xml clean
ant -f build.xml build
ant -f build.xml run

build.xml "run" target just invokes TemurinGenSBOM with no args, you can change that to test whatever you need..

Next step is to "design" how we transform/gather the temurin build-info to call this app, so look at what "build-info" we currently have, and how that would be represented in the SBOM? The design a command line to achieve this, example:

java TemurinGenSBOM --create bom.json --name Temurin_x64Linux --version jdk-18+26 --type application
java TemurinGenSBOM --add_dependency bom.json --name freetype --version 1.18 --hash 24abcdef4545
...

andrew-m-leonard · 2022-04-14T10:13:32Z

This is now complete, and can be enabled by specifying a buildArg of --create-sbom

andrew-m-leonard added the enhancement Issues that enhance the code or documentation of the repo in any way label Oct 11, 2021

github-actions bot added the docker Issues related to our docker files and docker scripts label Oct 11, 2021

andrew-m-leonard mentioned this issue Oct 11, 2021

Prototype "Reproducible Build" using current available Adopt jenkins job & infra framework and "build info" #2594

Closed

andrew-m-leonard assigned SehrishHussain Oct 14, 2021

andrew-m-leonard mentioned this issue Nov 4, 2021

Create a CycloneDX SBOM for the full reproducible build_info #2785

Closed

andrew-m-leonard added the reproducible-build label Nov 4, 2021

smlambert added this to Secure Software Development Activities Mar 1, 2022

andrew-m-leonard closed this as completed Apr 14, 2022

andrew-m-leonard moved this to Done in Secure Software Development Activities Apr 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Cyclone DX SBOM equivalent to the current build build-info json metadata #2753

Create a Cyclone DX SBOM equivalent to the current build build-info json metadata #2753

andrew-m-leonard commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

SehrishHussain commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

smlambert commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

SehrishHussain commented Oct 12, 2021 •

edited

Loading

SehrishHussain commented Oct 12, 2021 •

edited

Loading

SehrishHussain commented Oct 13, 2021

andrew-m-leonard commented Oct 13, 2021

SehrishHussain commented Oct 14, 2021 •

edited

Loading

SehrishHussain commented Oct 14, 2021 •

edited

Loading

SehrishHussain commented Nov 1, 2021

andrew-m-leonard commented Nov 1, 2021

SehrishHussain commented Nov 2, 2021 •

edited

Loading

andrew-m-leonard commented Dec 6, 2021

andrew-m-leonard commented Dec 15, 2021

andrew-m-leonard commented Dec 17, 2021 •

edited

Loading

andrew-m-leonard commented Apr 14, 2022

Create a Cyclone DX SBOM equivalent to the current build build-info json metadata #2753

Create a Cyclone DX SBOM equivalent to the current build build-info json metadata #2753

Comments

andrew-m-leonard commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

SehrishHussain commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

smlambert commented Oct 11, 2021

andrew-m-leonard commented Oct 11, 2021

SehrishHussain commented Oct 12, 2021 • edited Loading

SehrishHussain commented Oct 12, 2021 • edited Loading

SehrishHussain commented Oct 13, 2021

andrew-m-leonard commented Oct 13, 2021

SehrishHussain commented Oct 14, 2021 • edited Loading

SehrishHussain commented Oct 14, 2021 • edited Loading

SehrishHussain commented Nov 1, 2021

andrew-m-leonard commented Nov 1, 2021

SehrishHussain commented Nov 2, 2021 • edited Loading

andrew-m-leonard commented Dec 6, 2021

andrew-m-leonard commented Dec 15, 2021

andrew-m-leonard commented Dec 17, 2021 • edited Loading

andrew-m-leonard commented Apr 14, 2022

SehrishHussain commented Oct 12, 2021 •

edited

Loading

SehrishHussain commented Oct 12, 2021 •

edited

Loading

SehrishHussain commented Oct 14, 2021 •

edited

Loading

SehrishHussain commented Oct 14, 2021 •

edited

Loading

SehrishHussain commented Nov 2, 2021 •

edited

Loading

andrew-m-leonard commented Dec 17, 2021 •

edited

Loading