feat: add generated SPDX file on bottling #16594

SMillerDev · 2024-02-05T22:54:25Z

Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same change?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your changes? Here's an example.
Have you successfully run brew style with your changes locally?
Have you successfully run brew typecheck with your changes locally?
Have you successfully run brew tests with your changes locally?

Attached is an example SBOM.
spdx.sbom.json

This should allow us to have some more tracking of what goes into our bottles, but also allow others to have some insight into it.

MikeMcQuaid

Thanks for the PR!

feels a bit weird this living on Tab when it's got little in common there
do we have requests for these files anywhere you can link to?

Library/Homebrew/dev-cmd/bottle.rb

Library/Homebrew/tab.rb

MikeMcQuaid · 2024-02-06T14:46:44Z

Library/Homebrew/tab.rb

@@ -407,4 +430,151 @@ def to_s
    end
    s.join(" ")
  end
+
+  sig { returns(Hash) }
+  def to_spdx_sbom


Is this an officially defined format somewhere? If so: we should be using some sort of validator here ideally both at brew bottle and brew tests times.

I believe the SPDX spec is defined here: https://spdx.github.io/spdx-spec/v2.3/

In terms of a Ruby validator, I'm not sure. I can look into this 🙂

full ack. get the results validated. the current implementation produces invalid data.
see #16594 (comment)

SMillerDev · 2024-02-06T15:15:56Z

feels a bit weird this living on Tab when it's got little in common there

Yeah, I was considering splitting it out of tab, but it is sort of the same thing so wanted to get it public first.

do we have requests for these files anywhere you can link to?

No, but I can see the tooling being available in the larger ecosystem be useful. And I chatted with some people about this and they seemed interested.
@gdams for example.

MikeMcQuaid · 2024-02-12T14:05:43Z

Yeah, I was considering splitting it out of tab, but it is sort of the same thing so wanted to get it public first.

Cool, all good, as long as done before merged 👍🏻

No, but I can see the tooling being available in the larger ecosystem be useful. And I chatted with some people about this and they seemed interested.
@gdams for example.

I think this is the sort of thing I'd like to see some more requests for before we consider integration here.

gdams · 2024-02-12T16:53:38Z

Yeah this is certainly something that I see to be useful in homebrew. With the significant pressure companies/projects are being put under to provide SBOMs it would be useful for projects to be able to easily determine the exact set of deps in homebrew formulas.

github-actions · 2024-03-05T00:06:16Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

Library/Homebrew/tab.rb

jkowalleck · 2024-03-23T21:36:54Z

Attached is an example SBOM.
spdx.sbom.json

The resulting data structure seams invalid to the SPDX 2.3 JSON schema.
Tested with https://www.jsonschemavalidator.net/ and https://www.liquid-technologies.com/online-json-schema-validator

Library/Homebrew/sbom.rb

SMillerDev · 2024-03-24T10:00:51Z

New generated SBOM:
sbom.spdx.json

That should pass validation and have some more information than the previous one.

MikeMcQuaid

Looking better!

Blockers needed before merge:

tests, ideally with 100% coverage of sbom.rb
uses JSONSchemer.schema (like github_packages.rb) to do schema validation on write

SMillerDev · 2024-03-24T21:07:25Z

uses JSONSchemer.schema (like github_packages.rb) to do schema validation on write

Didn't know we had that, it sounds awesome. Do we use that for the API already? Otherwise I'm adding that to my list.

MikeMcQuaid · 2024-03-25T08:37:10Z

Didn't know we had that, it sounds awesome.

It is good for catching problems for sure.

Do we use that for the API already?

No(t yet). We'd need to create and publish a schema, too. Might want to sync up with @apainintheneck and save this for API v3 rather than create a v2 schema that won't be around in a year.

apainintheneck · 2024-03-26T06:59:56Z

I sent @SMillerDev some info but honestly API v3 still seems a ways off since the way we handle dependencies for formulae is still undecided and cask v3 is currently blocked by potential scope creep. Either we hold off on validating API v2 for now or we add validation knowing it might get removed in a few months.

SMillerDev · 2024-03-30T18:52:37Z

Does anyone know where stdlib information comes from in the install tab etc? Or is that only for Linux and that's why I can't find it?

SMillerDev · 2024-04-21T11:23:10Z

Any suggestions to fix this?

  Error: bottling failed
  Error: cannot load such file -- json_schemer

MikeMcQuaid · 2024-04-21T18:53:24Z

Library/Homebrew/sbom.rb

+# typed: true
+# frozen_string_literal: true
+
+require "json_schemer"


@SMillerDev This will only be available for dev-cmd so may need to get moved into a method call instead. I suspect that's needed to fix CI failures.

I'm not sure what you mean by a method call, do you mean to make a command to generate an SBOM?

Move the require inside a def something

Bo98 · 2024-04-25T15:20:16Z

We cannot vendor it for all as it unfortunately depends on a native extension (via simpleidn -> unf -> unf_ext).

We already have rexml in multiple groups so it isn't really a problem having it in potentially many groups.

SMillerDev · 2024-04-27T18:20:25Z

I think/hope this is ready for a final review

MikeMcQuaid

Great work so far @SMillerDev, think this is pretty close.

Library/Homebrew/dev-cmd/bottle.rb

Library/Homebrew/sbom.rb

Library/Homebrew/test/sbom_spec.rb

Library/Homebrew/sbom.rb

SMillerDev · 2024-05-03T12:35:28Z

Not sure how to resolve this

Library/Homebrew/sbom.rb:79:5: C: Homebrew/InstallBundlerGems: Only use Homebrew.install_bundler_gems! in dev-cmd.
    Homebrew.install_bundler_gems!(groups: ["bottle"])
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

MikeMcQuaid · 2024-05-03T13:04:40Z

Not sure how to resolve this

Need to move that call to the relevant dev-cmd instead e.g. bottle.

Library/Homebrew/dev-cmd/bottle.rb

SMillerDev · 2024-05-06T18:19:39Z

Okay, after my latest changes it definitely hits those paths, but I guess codecov isn't based on the online run?

MikeMcQuaid · 2024-05-07T07:45:25Z

Okay, after my latest changes it definitely hits those paths, but I guess codecov isn't based on the online run?

It should be: https://github.com/Homebrew/brew/actions/runs/8973658512/job/24644407661?pr=16594#step:13:52

We don't run online tests on macOS in case that's it?

CC @Bo98 for ideas.

SMillerDev · 2024-05-07T18:35:45Z

Checked the patches and simplified the test a little bit since I think it ran as generic (and would never take those patches). I think we're good to merge now.

MikeMcQuaid

Thanks again @SMillerDev!

MikeMcQuaid reviewed Feb 6, 2024

View reviewed changes

github-actions bot added the stale No recent activity label Mar 5, 2024

woodruffw reviewed Mar 5, 2024

View reviewed changes

Library/Homebrew/tab.rb Outdated Show resolved Hide resolved

jkowalleck reviewed Mar 5, 2024

View reviewed changes

Library/Homebrew/tab.rb Outdated Show resolved Hide resolved

github-actions bot removed the stale No recent activity label Mar 6, 2024

SMillerDev force-pushed the feat/sbom/install_spdx branch from 0e3db6e to 8c63729 Compare March 23, 2024 19:36

jkowalleck reviewed Mar 23, 2024

View reviewed changes

Library/Homebrew/sbom.rb Show resolved Hide resolved

SMillerDev force-pushed the feat/sbom/install_spdx branch 2 times, most recently from 2166dde to af7e6fe Compare March 24, 2024 09:59

SMillerDev force-pushed the feat/sbom/install_spdx branch from af7e6fe to 8a52c56 Compare March 24, 2024 10:44

MikeMcQuaid reviewed Mar 24, 2024

View reviewed changes

SMillerDev force-pushed the feat/sbom/install_spdx branch from 8a52c56 to 18ae7dc Compare March 30, 2024 18:40

SMillerDev force-pushed the feat/sbom/install_spdx branch 3 times, most recently from 4cf1dd2 to fd8dad2 Compare March 30, 2024 19:04

SMillerDev force-pushed the feat/sbom/install_spdx branch 2 times, most recently from f55bd13 to 919d77e Compare April 18, 2024 16:36

SMillerDev marked this pull request as ready for review April 21, 2024 11:23

MikeMcQuaid reviewed Apr 21, 2024

View reviewed changes

SMillerDev force-pushed the feat/sbom/install_spdx branch from 72365d6 to 2ad8e6b Compare April 25, 2024 15:13

SMillerDev force-pushed the feat/sbom/install_spdx branch from 2ad8e6b to 7b040c1 Compare April 25, 2024 15:26

MikeMcQuaid reviewed Apr 29, 2024

View reviewed changes

SMillerDev commented May 3, 2024

View reviewed changes

Library/Homebrew/sbom.rb Outdated Show resolved Hide resolved

SMillerDev commented May 3, 2024

View reviewed changes

Library/Homebrew/sbom.rb Outdated Show resolved Hide resolved

SMillerDev commented May 3, 2024

View reviewed changes

Library/Homebrew/sbom.rb Outdated Show resolved Hide resolved

SMillerDev commented May 3, 2024

View reviewed changes

Library/Homebrew/sbom.rb Outdated Show resolved Hide resolved

SMillerDev force-pushed the feat/sbom/install_spdx branch from 6ad273d to 44c4ca3 Compare May 3, 2024 12:27

MikeMcQuaid reviewed May 3, 2024

View reviewed changes

Library/Homebrew/dev-cmd/bottle.rb Outdated Show resolved Hide resolved

SMillerDev force-pushed the feat/sbom/install_spdx branch 6 times, most recently from 8b64aeb to 5b5eda0 Compare May 6, 2024 18:09

SMillerDev force-pushed the feat/sbom/install_spdx branch from 5b5eda0 to e9bb075 Compare May 7, 2024 18:28

feat: add generated SPDX file on bottling

a43b746

SMillerDev force-pushed the feat/sbom/install_spdx branch from e9bb075 to a43b746 Compare May 7, 2024 18:33

MikeMcQuaid approved these changes May 7, 2024

View reviewed changes

MikeMcQuaid enabled auto-merge May 7, 2024 18:40

MikeMcQuaid merged commit 0d1bebc into Homebrew:master May 7, 2024
24 checks passed

github-actions bot added the outdated PR was locked due to age label Jun 7, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jun 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add generated SPDX file on bottling #16594

feat: add generated SPDX file on bottling #16594

SMillerDev commented Feb 5, 2024

MikeMcQuaid left a comment

MikeMcQuaid Feb 6, 2024

woodruffw Mar 5, 2024

jkowalleck Mar 23, 2024 •

edited

Loading

SMillerDev commented Feb 6, 2024

MikeMcQuaid commented Feb 12, 2024

gdams commented Feb 12, 2024

github-actions bot commented Mar 5, 2024

jkowalleck commented Mar 23, 2024 •

edited

Loading

SMillerDev commented Mar 24, 2024

MikeMcQuaid left a comment

SMillerDev commented Mar 24, 2024

MikeMcQuaid commented Mar 25, 2024

apainintheneck commented Mar 26, 2024

SMillerDev commented Mar 30, 2024

SMillerDev commented Apr 21, 2024

MikeMcQuaid Apr 21, 2024

SMillerDev Apr 22, 2024

MikeMcQuaid Apr 22, 2024

Bo98 commented Apr 25, 2024 •

edited

Loading

SMillerDev commented Apr 27, 2024

MikeMcQuaid left a comment

SMillerDev commented May 3, 2024

MikeMcQuaid commented May 3, 2024

SMillerDev commented May 6, 2024

MikeMcQuaid commented May 7, 2024

SMillerDev commented May 7, 2024

MikeMcQuaid left a comment

feat: add generated SPDX file on bottling #16594

feat: add generated SPDX file on bottling #16594

Conversation

SMillerDev commented Feb 5, 2024

MikeMcQuaid left a comment

Choose a reason for hiding this comment

MikeMcQuaid Feb 6, 2024

Choose a reason for hiding this comment

woodruffw Mar 5, 2024

Choose a reason for hiding this comment

jkowalleck Mar 23, 2024 • edited Loading

Choose a reason for hiding this comment

SMillerDev commented Feb 6, 2024

MikeMcQuaid commented Feb 12, 2024

gdams commented Feb 12, 2024

github-actions bot commented Mar 5, 2024

jkowalleck commented Mar 23, 2024 • edited Loading

SMillerDev commented Mar 24, 2024

MikeMcQuaid left a comment

Choose a reason for hiding this comment

SMillerDev commented Mar 24, 2024

MikeMcQuaid commented Mar 25, 2024

apainintheneck commented Mar 26, 2024

SMillerDev commented Mar 30, 2024

SMillerDev commented Apr 21, 2024

MikeMcQuaid Apr 21, 2024

Choose a reason for hiding this comment

SMillerDev Apr 22, 2024

Choose a reason for hiding this comment

MikeMcQuaid Apr 22, 2024

Choose a reason for hiding this comment

Bo98 commented Apr 25, 2024 • edited Loading

SMillerDev commented Apr 27, 2024

MikeMcQuaid left a comment

Choose a reason for hiding this comment

SMillerDev commented May 3, 2024

MikeMcQuaid commented May 3, 2024

SMillerDev commented May 6, 2024

MikeMcQuaid commented May 7, 2024

SMillerDev commented May 7, 2024

MikeMcQuaid left a comment

Choose a reason for hiding this comment

jkowalleck Mar 23, 2024 •

edited

Loading

jkowalleck commented Mar 23, 2024 •

edited

Loading

Bo98 commented Apr 25, 2024 •

edited

Loading