Update the skip as it was not working correctly #685

fokion · 2023-06-19T16:45:47Z

It could not evaluate a variable that was defined on previous steps and during processing.

In the venom.log we would see something like that in the comparison

[skip-something] evaluating out ShouldEqual '1'
[skip-something] failed to evaluate : expected: 1  got: <nil>

which means that it does not get the value of out

now it is only [skip-something] evaluating out ShouldEqual '1' which means that it evaluates it correctly.

furthermore the test is working correctly

 • Testing skip (skip/skip.yml)
        • skip-something
                •  PASS
                  [info] Set a variable to 1
                • exec PASS
                •  PASS
                  [info] do not skip it as the value of out is different to 2
                • skip-this SKIP
                • exec PASS
                •  SKIP
                •  SKIP

for

name : "Testing skip"
testcases:
  - name : "skip something"
    steps:
      - info:
          - Set a variable to 1
      - script : "echo '1'"
        vars :
          out:
            from : result.systemout
      - info:
          - "do not skip it as the value of out is different to 2"
        skip :
          - out ShouldEqual '2'
      - name: skip-this
        skip:
          - out ShouldEqual '1'
      - script: "echo 'false'"
        vars:
          outAsBool:
            from: result.systemout
      - info:
          - "Fail should be skipped"
        skip :
          - out ShouldEqual 1
      - info:
          - "Fail should be skipped as false"
        skip:
          - outAsBool ShouldBeFalse

fokion · 2023-06-19T17:01:10Z

@yesnault , @fsamin any thoughts?

fokion · 2023-06-19T19:52:00Z

#651

yesnault · 2023-06-23T14:10:51Z

thank you @fokion for this proposal, we will check that

ovh-cds · 2023-06-23T14:31:54Z

CDS Report build-venom-a#64.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2023-07-19T12:10:52Z

CDS Report build-venom-a#70.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

yesnault

the current output (with your code) is:

❯ go run main.go run -v skip.yml
          [trac] writing venom.3.log
 • Testing skip (skip.yml)
        • skip-something
                •  PASS
                  [info] Set a variable to 1 (skip.yml:11)
                • exec PASS
                •  SKIP
                • skip-this PASS
                • exec PASS
                •  PASS
                  [info] Fail should be skipped
                •  PASS
                  [info] Fail should be skipped as false

I'm not sure to understand why it's so different with your output.

fokion · 2023-07-25T11:31:24Z

the current output (with your code) is:

❯ go run main.go run -v skip.yml
          [trac] writing venom.3.log
 • Testing skip (skip.yml)
        • skip-something
                •  PASS
                  [info] Set a variable to 1 (skip.yml:11)
                • exec PASS
                •  SKIP
                • skip-this PASS
                • exec PASS
                •  PASS
                  [info] Fail should be skipped
                •  PASS
                  [info] Fail should be skipped as false

I'm not sure to understand why it's so different with your output.

missed some commits from a rebase see commit 0d1dfe3

          [trac] writing venom.3.log
 • Skip testsuite (/Users/fokionsotiropoulos/GolandProjects/venom/tests/skip.yml)
        • init
                • exec PASS
        • do-not-skip-this
                • exec PASS
        • skip-this
                • (all steps were skipped)
        • skip-one-of-these
                • do-not-skip-this PASS
                • skip-this SKIP
        • skip-all-of-steps
                • skip-this SKIP
final status: PASS

ovh-cds · 2023-07-25T15:50:24Z

CDS Report build-venom-a#75.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

ovh-cds · 2023-07-25T16:04:17Z

CDS Report build-venom-a#75.1 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

yesnault · 2023-07-27T15:55:13Z

@fokion Same output.

Full log:

❯ git checkout master && git pull
❯ git checkout -b ye-test
Switched to a new branch 'ye-test'
❯ git pull [email protected]:fokion/venom.git feature/skip
From github.com:fokion/venom
 * branch            feature/skip -> FETCH_HEAD
Updating 5b319da..0d1dfe3
Fast-forward
 assertion.go        |  4 ++--
 assertion_test.go   | 13 +++++++++++++
 process_testcase.go | 45 +++++++++++++++++++++++++--------------------
 process_teststep.go | 21 ++++++++++++---------
 tests/skip/skip.yml | 29 +++++++++++++++++++++++++++++
 5 files changed, 81 insertions(+), 31 deletions(-)
 create mode 100644 tests/skip/skip.yml
❯ go run main.go run -v ../../tests/skip/skip.yml
          [trac] writing venom.5.log
 • Testing skip (../../tests/skip/skip.yml)
        • skip-something
                •  PASS
                  [info] Set a variable to 1 (skip.yml:11)
                • exec PASS
                •  SKIP
                • skip-this PASS
                • exec PASS
                •  PASS
                  [info] Fail should be skipped
                •  PASS
                  [info] Fail should be skipped as false
final status: PASS

fokion · 2023-07-27T16:02:52Z

rebasing it scratch

It could not evaluate a variable that was defined on previous steps and during processing we should check that the failures are empty Signed-off-by: Fokion Sotiropoulos <[email protected]>

Signed-off-by: Fokion Sotiropoulos <[email protected]>

ovh-cds · 2023-07-28T08:05:19Z

CDS Report build-venom-a#76.0 ✘

Build
- Build ✔
- Unit Tests ✔
Tests
- Acceptance Tests ✘

yesnault · 2023-07-28T09:05:24Z

@fokion 2 tests in error

❯ go run main.go run -v ../../tests/skip.yml
          [trac] writing venom.30.log
 • Skip testsuite (../../tests/skip.yml)
        • init
                • exec PASS
        • do-not-skip-this
                • (all steps were skipped)
        • skip-this
                • exec FAIL
                  Testcase "skip-this", step #0-0: Assertion "result.code ShouldEqual 0" failed. expected: 0  got: 127 (skip.yml:0)
        • skip-one-of-these
                • do-not-skip-this PASS
                • skip-this SKIP
        • skip-all-of-steps
                • skip-this SKIP

and error on SMTP testsuite / IMAP-Retrieve-sent-mail

Testcase "IMAP - Retrieve sent mail", step #0-0: retry conditions not fulfilled, skipping 2 remaining retries (smtp.yml:41)
Testcase "IMAP - Retrieve sent mail", step #0-0: Assertion "result.commands.commands0.err ShouldBeEmpty" failed. expected 'error while retrieving mail: empty mailbox' to be empty but it wasn't (smtp.yml:59)
Testcase "IMAP - Retrieve sent mail", step #0-0: Assertion "result.commands.commands0.search.from ShouldEqual [email protected]" failed. expected: [email protected]  got:  (smtp.yml:60)
Testcase "IMAP - Retrieve sent mail", step #0-0: Assertion "result.commands.commands0.search.to ShouldEqual [email protected]" failed. expected: [email protected]  got:  (smtp.yml:61)
Testcase "IMAP - Retrieve sent mail", step #0-0: Assertion "result.commands.commands0.search.subject ShouldEqual Venom SMTP tests" failed. expected: Venom SMTP tests  got:  (smtp.yml:62)
Testcase "IMAP - Retrieve sent mail", step #0-0: Assertion "result.commands.commands0.search.body ShouldEqual Hi, I am Venom SMTP Executor!" failed. expected: Hi, I am Venom SMTP Executor!  got:  (smtp.yml:63)

fokion · 2023-08-04T09:19:03Z

@yesnault I have updated the code in here as it was due to the variables not being passed for the assertion #708

fokion force-pushed the feature/skip branch 2 times, most recently from f15a042 to 6de083c Compare June 19, 2023 17:00

fokion mentioned this pull request Jun 19, 2023

Skip a step in range/loop doesn't work #651

Open

fokion marked this pull request as ready for review June 20, 2023 08:34

fokion force-pushed the feature/skip branch from fac28ca to 0681abf Compare June 23, 2023 19:03

yesnault force-pushed the feature/skip branch from 0681abf to 47a455f Compare July 25, 2023 07:21

yesnault requested changes Jul 25, 2023

View reviewed changes

fokion force-pushed the feature/skip branch 3 times, most recently from 45da6f5 to 0d1dfe3 Compare July 25, 2023 11:29

yesnault mentioned this pull request Jul 26, 2023

bug: Skip functionality is unreliable in tests #661

Closed

fokion requested a review from yesnault July 27, 2023 09:10

fokion force-pushed the feature/skip branch 2 times, most recently from 6f5a9d8 to 1c9f7d6 Compare July 27, 2023 18:04

fokion added 2 commits July 27, 2023 19:06

Update the skip as it was not working correctly

b857210

It could not evaluate a variable that was defined on previous steps and during processing we should check that the failures are empty Signed-off-by: Fokion Sotiropoulos <[email protected]>

adding the check in the testcase as well

eed4e71

Signed-off-by: Fokion Sotiropoulos <[email protected]>

fokion force-pushed the feature/skip branch from 1c9f7d6 to eed4e71 Compare July 27, 2023 18:06

adding one more test for the testcases

ef1ce9a

update retry in test step

9b96a79

fokion closed this Aug 4, 2023

yesnault mentioned this pull request Sep 7, 2023

update variables and executor loading #708

Closed

berlic mentioned this pull request Mar 14, 2024

[bug] skip doesn't work with variables from previous steps #777

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the skip as it was not working correctly #685

Update the skip as it was not working correctly #685

fokion commented Jun 19, 2023 •

edited

Loading

fokion commented Jun 19, 2023

fokion commented Jun 19, 2023

yesnault commented Jun 23, 2023

ovh-cds commented Jun 23, 2023

ovh-cds commented Jul 19, 2023

yesnault left a comment •

edited

Loading

fokion commented Jul 25, 2023

ovh-cds commented Jul 25, 2023

ovh-cds commented Jul 25, 2023

yesnault commented Jul 27, 2023

fokion commented Jul 27, 2023 •

edited

Loading

ovh-cds commented Jul 28, 2023

yesnault commented Jul 28, 2023 •

edited

Loading

fokion commented Aug 4, 2023

Update the skip as it was not working correctly #685

Update the skip as it was not working correctly #685

Conversation

fokion commented Jun 19, 2023 • edited Loading

fokion commented Jun 19, 2023

fokion commented Jun 19, 2023

yesnault commented Jun 23, 2023

ovh-cds commented Jun 23, 2023

ovh-cds commented Jul 19, 2023

yesnault left a comment • edited Loading

Choose a reason for hiding this comment

fokion commented Jul 25, 2023

ovh-cds commented Jul 25, 2023

ovh-cds commented Jul 25, 2023

yesnault commented Jul 27, 2023

fokion commented Jul 27, 2023 • edited Loading

ovh-cds commented Jul 28, 2023

yesnault commented Jul 28, 2023 • edited Loading

fokion commented Aug 4, 2023

fokion commented Jun 19, 2023 •

edited

Loading

yesnault left a comment •

edited

Loading

fokion commented Jul 27, 2023 •

edited

Loading

yesnault commented Jul 28, 2023 •

edited

Loading