Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add expect-error input to force PR check green on expected failure #1177

Merged
merged 22 commits into from
Aug 16, 2022
Merged

Add expect-error input to force PR check green on expected failure #1177

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
b7d4497
Add expect-error to expected failure job check
angelapwen Aug 12, 2022
4fb1cdc
Remove "failure expected" in PR check name
angelapwen Aug 12, 2022
c756bae
Update `expect-errors` input description
angelapwen Aug 12, 2022
59d39e3
Add double quotes to input description
angelapwen Aug 12, 2022
f5a4ea9
Throw error if caller isn't codeql-action
angelapwen Aug 15, 2022
14763b6
Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…
angelapwen Aug 15, 2022
e31c33d
Fix string equality comparison
angelapwen Aug 15, 2022
46f3540
Move error throwing location
angelapwen Aug 15, 2022
3606edb
Add print debug messages
angelapwen Aug 16, 2022
334b121
Fix expected repo URL
angelapwen Aug 16, 2022
65fcacd
Set failure if input is set incorrectly
angelapwen Aug 16, 2022
3d8828a
Update `expect-errors` input description
angelapwen Aug 16, 2022
6be3892
Address comments, throw failure if expected and didn't fail
angelapwen Aug 16, 2022
0736537
Merge remote-tracking branch 'origin/main' into angelapwen/expect-err…
angelapwen Aug 16, 2022
1e1da28
Set failure in other spot for expect-error but not thrown
angelapwen Aug 16, 2022
66bfa45
Test removing runWrapper's failure
angelapwen Aug 16, 2022
70d3299
Check error input is not default value
angelapwen Aug 16, 2022
41f86d9
Test remove runWrapper setting failure
angelapwen Aug 16, 2022
5aec66f
Test run where error expected but not received
angelapwen Aug 16, 2022
98d4d17
Fail with ram:1 in failure workflow
angelapwen Aug 16, 2022
f8f36a7
Comment fixups
angelapwen Aug 16, 2022
b67600f
Remove extraneous log lines
angelapwen Aug 16, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/debug-artifacts-failure.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
strategy:
matrix:
os: [ubuntu-latest, macos-latest]
name: Failure Expected - Upload debug artifacts
name: Upload debug artifacts after failure in analyze
continue-on-error: true
timeout-minutes: 45
runs-on: ${{ matrix.os }}
Expand All @@ -45,6 +45,7 @@ jobs:
- uses: ./../action/analyze
id: analysis
with:
expect-error: true
ram: 1
download-and-check-artifacts:
name: Download and check debug artifacts after failure in analyze
Expand Down
4 changes: 4 additions & 0 deletions analyze/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,10 @@ inputs:
default: ${{ github.token }}
matrix:
default: ${{ toJson(matrix) }}
expect-error:
description: For PR checks. If true, the Action will not set Actions status to failure.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May need to make this more descriptive so that our users don't set this to true when they don't mean to? I don't think I see a hidden option for input parameters.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about this?

Suggested change
description: For PR checks. If true, the Action will not set Actions status to failure.
description: [Internal] If true, the Action will not set Actions status to failure.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, this caused some failures — I wonder if the bracket symbols aren't able to be parsed correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding double quotes around it resolved it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider throwing an error inside the code if this is set but the repo is not github/codeql-action or a fork. (You should be able to get that information from the Actions event context.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added this code — still doing a final test of:

  • that if it's set in another repo, the error throws
  • that if it's set in this repo, the test passes
  • that if it's set in a fork of this repo, the test passes
  • that if it's set in this repo but no error is found, an error indicating this is thrown

required: false
default: "false"
outputs:
db-locations:
description: A map from language to absolute path for each database created by CodeQL.
Expand Down
8 changes: 6 additions & 2 deletions lib/analyze-action.js
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion lib/analyze-action.js.map
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 7 additions & 2 deletions src/analyze-action.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,7 +183,10 @@ async function run() {
} catch (origError) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that if an error isn't thrown, we need to set the job to a failure if this input is set to true.

I'm a little confused: if an error isn't thrown, then why do we need to set the job to a failure? In that case, the job should succeed and we want that, right? Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

It's a little more complicated than that since I see there are two places where core.setFailed is being called. Do we need to call both of them? It looks like the one in runWrapper is higher up the call chain. Maybe we don't need the one in run.

I'm not sure either. I can play around with it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or do you mean that if we expect-error that there must be an error, otherwise there was a problem? That's interesting. I had interpreted it more like force-success

That's what I mean. If we expect an error and there is none, then that's an error. :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added this logic at the end of the outer try block (indicating that an error was not thrown). Thanks!!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly, I found that the outer try block wasn't very useful. This catch block with origError is where I was able to throw/catch/stop errors appropriately. I removed the duplicate logic from the outer try block because it wasn't doing anything.

const error =
origError instanceof Error ? origError : new Error(String(origError));
core.setFailed(error.message);
if (!actionsUtil.getOptionalInput("expect-error")) {
core.setFailed(error.message);
}

console.log(error);

if (error instanceof CodeQLAnalysisError) {
Expand Down Expand Up @@ -214,7 +217,9 @@ async function runWrapper() {
try {
await runPromise;
} catch (error) {
core.setFailed(`analyze action failed: ${error}`);
if (!actionsUtil.getOptionalInput("expect-error")) {
core.setFailed(`analyze action failed: ${error}`);
}
console.log(error);
}
}
Expand Down