Bug: Output of harden-runner insight should use commit hash instead

[jauderho]

I just testing and implementing harden-runner after starting with the scorecard action. Repo is here: https://github.com/jauderho/psfiles

So if the push is for actions to use commit hashes instead of version tags, the output page of the insights should utilize the commit hash instead.

For example, https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1731266664

Recommendation is for

- uses: step-security/harden-runner@v1
  with:
    allowed-endpoints: 
      api.github.com:443
      github.com:443

Instead, it really should be (and changing as necessary as the action gets updated)

- uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    allowed-endpoints: 
      api.github.com:443
      github.com:443

Similarly, the README.md for this repo should indicate/recommend the use of hashes instead.

[varunsh-coder]

Thanks @jauderho for using harden-runner!

Good feedback. I will address it ASAP.

I observed you are setting the allowed-endpoints, but the egress-policy is set to audit. As a result the traffic is not restricted. The default value for egress-policy is block. So you can either remove egress-policy and just keep the allowed-endpoints or set egress-policy to block.

I will address this in the insights page as well and set egress-policy to block explicitly.

I will be happy to fix your workflows for you. Please let me know.

[jauderho]

@varunsh-coder , I was leaving it to audit for a bit to see if any additional egress points get caught. I will change these to block in a few days.

[jauderho]

@varunsh-coder while you are at it, if you can make the copy block on the insights page be the entire workflow (or at least make than an option), it will make it easier to do a complete copy and replace.

[varunsh-coder]

@varunsh-coder while you are at it, if you can make the copy block on the insights page be the entire workflow (or at least make than an option), it will make it easier to do a complete copy and replace.

@jauderho Great idea! I will make it same experience as the home page https://app.stepsecurity.io, where there is an editor and one can just copy and paste it without having to fix indentation etc.

Do let me know if you have more ideas to improve user experience or any other feedback. Thanks a lot!

[jauderho]

I do have one nit but it's mostly me being OCD.

If I check the Add step-security/harden-runner action on https://app.stepsecurity.io/, I'd like to see an additionally newline inserted before the next step. It's a little easier to distinguish between the steps that way.

Also, do the insight links expire after a while? For example, https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1731266664

[varunsh-coder]

I do have one nit but it's mostly me being OCD.

If I check the Add step-security/harden-runner action on https://app.stepsecurity.io/, I'd like to see an additionally newline inserted before the next step. It's a little easier to distinguish between the steps that way.

Good feedback! I will address it. Please keep the feedback coming!

Also, do the insight links expire after a while? For example, https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1731266664

No, they do not expire. Did you want them to expire?

[jauderho]

Might be nice to have them expire in a few days (and an option to retain if needed). I’m just wondering about the use case where a private repo might accidentally leak endpoints if this is integrated in the workflow.

[jauderho]

Also, the way things are set up means that we have to take two passes, once to audit and once to block.

I’m trying to think of a safe way to do this in one pass. I suppose one could have an action that modifies things after the fact but that would mean that the action needs write access and self modifying code could have unexpected outcomes.

Although, I think your tool appears to be able to have some knowledge of various actions so maybe that can be included as part of the suggested block of yaml. For instance, the CodeQL action is almost always going to be talking to the same set of hosts for everyone so it would make sense to just return github.com etc. as part of the allow list.

Does that make sense?

[varunsh-coder]

Might be nice to have them expire in a few days (and an option to retain if needed). I’m just wondering about the use case where a private repo might accidentally leak endpoints if this is integrated in the workflow.

Hi @jauderho harden-runner does not show insights for private repos as of now (I will add this to the readme). I think it would not be right for me to show the insights to public for a private repo. Other services like codecov show the test coverage to public for public repos, but require authentication and check repo access before showing results of private repos. I was thinking of doing the same. thoughts?

[varunsh-coder]

Also, the way things are set up means that we have to take two passes, once to audit and once to block.

I’m trying to think of a safe way to do this in one pass. I suppose one could have an action that modifies things after the fact but that would mean that the action needs write access and self modifying code could have unexpected outcomes.

Yes, it would be ideal if one did not have to change the workflow while going from audit to block mode. In fact some users have asked for a warn-only mode for similar reason. In warn-only mode, one would select the allowed-endpoints in the insights website, and they will get saved in the backend (not in the workflow file). If a new endpoint is called, it will notify in annotations (as a warning). Then the developer can click on the insights link and add that to the allowed list if it should be allowed, or investigate why that outbound call was made. So, one option is for the block mode to also store allowed-endpoints in the backend. But some users have asked for it to be in the workflow yaml file, since it makes it clear what is allowed and what is not.

Although, I think your tool appears to be able to have some knowledge of various actions so maybe that can be included as part of the suggested block of yaml. For instance, the CodeQL action is almost always going to be talking to the same set of hosts for everyone so it would make sense to just return github.com etc. as part of the allow list.

Does that make sense?

That knowledge base is to give users some information about why a certain domain is called. But restricting based on that will not work for run commands.

[jauderho]

Might be nice to have them expire in a few days (and an option to retain if needed). I’m just wondering about the use case where a private repo might accidentally leak endpoints if this is integrated in the workflow.

Hi @jauderho harden-runner does not show insights for private repos as of now (I will add this to the readme). I think it would not be right for me to show the insights to public for a private repo. Other services like codecov show the test coverage to public for public repos, but require authentication and check repo access before showing results of private repos. I was thinking of doing the same. thoughts?

Yeah, not allowing for private repos without an explicit opt-in would be good. For example, I now have a GHA template repo which I plan to use to copy to new repos. Not allowing private repos will prevent inadvertent exposure.

[varunsh-coder]

@jauderho since you have a repo related to docker images, and you run security tools on those images, I wanted to ask you about this feature I am thinking about - automatic SBOM (software bill of materials)/ provenance generation.

As you know harden-runner monitors the runner and so it knows what dependencies are being downloaded, from where, and what artifact is generated. So, I would like to automatically generate SBOM, and then upload it either along with the docker container, or in the release tab (if there is a release).

This way developers will not need to think about SBOM generation, it will just happen as part of the build.

What do you think about this? Thanks!

[jauderho]

It's on my todo list to figure out how to implement cosign and SBOMs for containers. I just figured it out for Go binaries (see bl3auto repo) but have not had cycles to work on this yet. Hopefully in the next few weeks...

[varunsh-coder]

@jauderho I have fixed the commit hash issue that you reported. The readme and the insights page have been updated.

For all your other suggestions, I have created separate issues to track them. I have improved the copy-paste experience, but it will take me more time to have the full workflow show up there. I will get it done soon though.

Thanks again for the feedback! I have added a Slack channel link in the readme, in case you want to join. Would love to have more discussions.

I will go ahead and close this issue. Thanks!

[jauderho]

I see the fix. However, to match exactly as the other page, there should be a name: field before the uses:, so something like.

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      95s5acprodeus1file6.blob.core.windows.net:443
      api.github.com:443
      artifactcache.actions.githubusercontent.com:443
      auth.docker.io:443
      codeload.github.com:443
      dl-cdn.alpinelinux.org:443
      ghcr.io:443
      github.com:443
      production.cloudflare.docker.com:443
      proxy.golang.org:443
      registry-1.docker.io:443

[varunsh-coder]

@jauderho sorry, missed that. re-opening issue. will add it soon.

[jauderho]

One observation: My https://github.com/jauderho/dockerfiles repo was previously set to default read/write for Actions but with all of these perm changes, I now have the repo set to default read.

While this is better, I've noticed that this changes the output of the insights URL from say https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1741506002 to https://app.stepsecurity.io/github/***/psfiles/actions/runs/1741506002

Looks like the username is masked now. Not a showstopper but just to point out that the URL will not work. You might want to have a more verbose error message for folks that hit the https://app.stepsecurity.io/github/***/<REST_OF_URL endpoint to manually add their username back in.

[varunsh-coder]

https://github.com/jauderho/dockerfiles

One observation: My https://github.com/jauderho/dockerfiles repo was previously set to default read/write for Actions but with all of these perm changes, I now have the repo set to default read.

While this is better, I've noticed that this changes the output of the insights URL from say https://app.stepsecurity.io/github/jauderho/psfiles/actions/runs/1741506002 to https://app.stepsecurity.io/github/***/psfiles/actions/runs/1741506002

Looks like the username is masked now. Not a showstopper but just to point out that the URL will not work. You might want to have a more verbose error message for folks that hit the https://app.stepsecurity.io/github/***/<REST_OF_URL endpoint to manually add their username back in.

Thanks for reporting this! This is definitely going to cause problems for some users as they will not know what is wrong with the URL. I will create another issue for it.

[jauderho]

Noticed that https://github.com/step-security/harden-runner/blob/main/action.yml appears to be calling node12.

My understanding is that node 12 support was EOL (normal not security) a while ago. See https://endoflife.date/nodejs

Should this be bumped to say node 16?

[varunsh-coder]

Noticed that https://github.com/step-security/harden-runner/blob/main/action.yml appears to be calling node12.

My understanding is that node 12 support was EOL (normal not security) a while ago. See https://endoflife.date/nodejs

Should this be bumped to say node 16?

Thanks a lot for feedback! I will create an issue to address.

[jauderho]

If api.snapcraft.io is blocked for https://github.com/jauderho/dockerfiles/actions/runs/1742841293, should the job not fail so that someone can look into it?

Or should there be a block-hard-fail option type?

Also, the corresponding output https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1742841293 does not call out that api.snapcraft.io was blocked for some reason.

[varunsh-coder]

If api.snapcraft.io is blocked for https://github.com/jauderho/dockerfiles/actions/runs/1742841293, should the job not fail so that someone can look into it?

Or should there be a block-hard-fail option type?

Yes, I think I need user feedback on this. api.snapcraft.io is a special case. It is not used from the workflow, but from snapd. I had reported it to GitHub, but looks like it might not be fixed.

Let me create an issue to investigate block-and-fail mode, where job fails if there is call to endpoints not in the allowed list. Do you think block-and-fail should be the same as block or should it be a different option? I have already gotten feedback to have a warn-only mode, where allowed-endpoints are not stored in the workflow, but on the back-end, and there is just a warning if a new endpoint is called...So, I don't want to have too many options, else it gets hard for developers to decide.

[jauderho]

Maybe it should be hard-fail and soft-fail or is that not sufficiently explanatory?

So you end up with just:

• audit
• hard-fail
• soft-fail

It's unclear to me how much value is added by having a warn-only mode but maybe I'm failing to see the use case.

[jauderho]

So I've run into a potential issue around how the allowed-endpoints are defined. If you look at my workflow for ansible (https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/ansible.yml), it has a build matrix of building for both ubuntu and alpine.

Now the dependencies and hence the allowed-endpoints are different for ubuntu vs. alpine as some python wheels have to be manually built for alpine dependencies. In order to make this run successfully without having to break out the matrix into different jobs, I would have to end up using whatever the more permissive list would be.

Not a showstopper but might catch some folks down the line.

Example: https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1742888941

[varunsh-coder]

So I've run into a potential issue around how the allowed-endpoints are defined. If you look at my workflow for ansible (https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/ansible.yml), it has a build matrix of building for both ubuntu and alpine.

Now the dependencies and hence the allowed-endpoints are different for ubuntu vs. alpine as some python wheels have to be manually built for alpine dependencies. In order to make this run successfully without having to break out the matrix into different jobs, I would have to end up using whatever the more permissive list would be.

Not a showstopper but might catch some folks down the line.

Example: https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1742888941

Interesting case. Yes, I think the union of all endpoints for the matrix would need to be used in the workflow file. When the experience of showing the entire workflow in the editor in the insights page gets done, getting a list of all the endpoints across jobs in a matrix should get easier.

[varunsh-coder]

Maybe it should be hard-fail and soft-fail or is that not sufficiently explanatory?

So you end up with just:

• audit
• hard-fail
• soft-fail

It's unclear to me how much value is added by having a warn-only mode but maybe I'm failing to see the use case.

The main difference is in terms of where the allowed-endpoints are stored. If stored in the workflow file, and they change often (which might not be the case for many workflows), then one would have to update the workflow often. In such a case one may want to store the allowed-endpoints in the backend using the insights page.

[jauderho]

Where is this backend that you are referring to? I think that's where I'm missing understanding of.

[varunsh-coder]

Where is this backend that you are referring to? I think that's where I'm missing understanding of.

Backend is the API and data store that stores the correlated outbound traffic.
e.g. when you visit https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1742888941, it fetches the data for that workflow run and shows it from the backend API.

Now, the idea is that instead of adding the allowed-endpoints in the workflow YAML file, you can simply check the endpoints that should be allowed, and save the policy using the insights page using a save policy button. Then the policy will get saved along with the insights in the backend. Next time the same workflow runs, harden-runner will fetch the policy from the backend and then hard-fail/soft-fail based on that policy. This way, if there is a new endpoint, one does not need to change the workflow YAML file (so no need to do pull request and update the YAML file). Just go to the insights page, and approve the new endpoint.

The downside is that the allowed-endpoints are not visible in the YAML file.

[varunsh-coder]

@jauderho I have fixed the UI bug. Can you please verify? The editor should show harden-runner in same way as the landing page.

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      95s5acprodeus1file6.blob.core.windows.net:443
      api.github.com:443
      archive.ubuntu.com:80

[jauderho]

@jauderho since you have a repo related to docker images, and you run security tools on those images, I wanted to ask you about this feature I am thinking about - automatic SBOM (software bill of materials)/ provenance generation.

As you know harden-runner monitors the runner and so it knows what dependencies are being downloaded, from where, and what artifact is generated. So, I would like to automatically generate SBOM, and then upload it either along with the docker container, or in the release tab (if there is a release).

This way developers will not need to think about SBOM generation, it will just happen as part of the build.

What do you think about this? Thanks!

You may want to track signing with Docker here : docker/roadmap#269

[jauderho]

@jauderho I have fixed the UI bug. Can you please verify? The editor should show harden-runner in same way as the landing page.

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      95s5acprodeus1file6.blob.core.windows.net:443
      api.github.com:443
      archive.ubuntu.com:80

This appears to work. Just tested on one of my actions.

[jauderho]

BUG: https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1747044469 does not appear to return any output.

This is tied to a pretty large job with 150+ subtasks (51 simultaneous). Workflow file is here: https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1747044469

It is unclear to me if the large number of jobs is causing an issue somewhere.

[varunsh-coder]

https://app.stepsecurity.io/github/jauderho/dockerfiles/actions/runs/1747044469

I had created a bug to investigate this earlier - #68.

[varunsh-coder]

@jauderho I have fixed the UI bug. Can you please verify? The editor should show harden-runner in same way as the landing page.

- name: Harden Runner
  uses: step-security/harden-runner@14dc64f30986eaa2ad2dddcec073f5aab18e5a24 # v1
  with:
    egress-policy: block
    allowed-endpoints: 
      95s5acprodeus1file6.blob.core.windows.net:443
      api.github.com:443
      archive.ubuntu.com:80

This appears to work. Just tested on one of my actions.

Thanks! I will go ahead and close this bug. I will enable discussions on this repo.

[jauderho]

@jauderho since you have a repo related to docker images, and you run security tools on those images, I wanted to ask you about this feature I am thinking about - automatic SBOM (software bill of materials)/ provenance generation.

As you know harden-runner monitors the runner and so it knows what dependencies are being downloaded, from where, and what artifact is generated. So, I would like to automatically generate SBOM, and then upload it either along with the docker container, or in the release tab (if there is a release).

This way developers will not need to think about SBOM generation, it will just happen as part of the build.

What do you think about this? Thanks!

So.... I think I have keyless cosign and SBOM generation working for a container image. There are a bunch of moving parts so this may yet change.

Workflow is here: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/age.yml

Output is here: https://github.com/jauderho/dockerfiles/actions/runs/1755633128

EDIT: I have not yet figured out how to use cosign to sign the scan output but it does seem possible. See https://github.com/pvnovarese/oss-2021-sbom-complete-workflow-demo/blob/main/.github/workflows/build-and-publish.yaml

[varunsh-coder]

@jauderho since you have a repo related to docker images, and you run security tools on those images, I wanted to ask you about this feature I am thinking about - automatic SBOM (software bill of materials)/ provenance generation.
As you know harden-runner monitors the runner and so it knows what dependencies are being downloaded, from where, and what artifact is generated. So, I would like to automatically generate SBOM, and then upload it either along with the docker container, or in the release tab (if there is a release).
This way developers will not need to think about SBOM generation, it will just happen as part of the build.
What do you think about this? Thanks!

So.... I think I have keyless cosign and SBOM generation working for a container image. There are a bunch of moving parts so this may yet change.

Workflow is here: https://github.com/jauderho/dockerfiles/blob/main/.github/workflows/age.yml

Output is here: https://github.com/jauderho/dockerfiles/actions/runs/1755633128

EDIT: I have not yet figured out how to use cosign to sign the scan output but it does seem possible. See https://github.com/pvnovarese/oss-2021-sbom-complete-workflow-demo/blob/main/.github/workflows/build-and-publish.yaml

Thanks @jauderho! I have created a discussion topic for SBOM and automatic signing.

[jauderho]

@varunsh-coder

A somewhat related question for you. One of the OpenSSF Scorecard recommendations is to switch from using Docker image tag to digests. Given that you have an Action to covert from versions to commit hashes for GHA, do you know of or have any plans to create a similar page to https://app.stepsecurity.io/ that will allow for a cut/paste conversion for Docker tags?

[varunsh-coder]

@varunsh-coder

A somewhat related question for you. One of the OpenSSF Scorecard recommendations is to switch from using Docker image tag to digests. Given that you have an Action to covert from versions to commit hashes for GHA, do you know of or have any plans to create a similar page to https://app.stepsecurity.io/ that will allow for a cut/paste conversion for Docker tags?

@jauderho I was thinking of the exact same thing. I will add it. Thanks for the idea!

[jauderho]

One item that's unclear to me is we should include the version number while using the digest for point to an image. See jauderho/dockerfiles#146

[varunsh-coder]

One item that's unclear to me is we should include the version number while using the digest for point to an image. See jauderho/dockerfiles#146

Yes, I think we should.