Skip to content

Commit

Permalink
Allow for --retry option in configuration
Browse files Browse the repository at this point in the history
Add retry_attempts method to configuration base class

Add Retry formatter and specs for same

Remove files attempting to implement --retry with a formatter - bad idea

Add wip tag to prevent --retry features from killing the build

Actually run Travis build against active branch

Fix tiny grammatical error. Closes cucumber#914.
  • Loading branch information
danascheider authored and danascheider committed May 12, 2016
1 parent d459c6e commit 06aa0d7
Show file tree
Hide file tree
Showing 5 changed files with 86 additions and 26 deletions.
1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ rvm:
branches:
only:
- master
- resolve-issue-882
- v1.3.x-bugfix

before_install:
Expand Down
96 changes: 70 additions & 26 deletions features/docs/cli/retry_failing_tests.feature
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
@wip
Feature: Retry failing tests

Retry gives you a way to get through flaky tests usually pass after a few runs.
Retry gives you a way to get through flaky tests that usually pass after a few runs.
This gives a development team a way forward other than disabling a valuable test.

- Specify max retry count in option
Expand All @@ -11,10 +12,17 @@ Feature: Retry failing tests
use a tag for flaky tests? Global option to retry any test that fails?

Background:
Given a flaky step definition
Given a file named "features/step_definitions/flaky_steps.rb" with:
"""
Given(/^a flaky step$/) do
$answer ||= 0
$answer += 1
expect(@answer).to eq 2
end
"""

Scenario: run a flaky test
Given a file named "features/fail.feature" with:
Scenario: Run a flaky test
Given a file named "features/flaky.feature" with:
"""
Feature: Fails
Expand All @@ -26,47 +34,83 @@ Feature: Retry failing tests
Then it should fail with:
"""
Feature: Fails
Scenario: flaky test
Given a flaky step
(RuntimeError)
./features/step_definitions/steps.rb:4:in `flaky_pass'
./features/step_definitions/steps.rb:1:in `/^a flaky step$/'
features/fail.feature:4:in `Given a flaky step'
expected: 1
got: 0
(compared using ==)
(RSpec::Expectations::ExpectationNotMetError)
./features/step_definitions/flaky_steps.rb:3:in `/^a flaky step$/'
features/flaky.feature:4:in `Given a flaky step'
Failing Scenarios:
cucumber features/fail.feature:3
1 scenario (1 failed)
1 step (1 failed)
"""
@wip
Scenario: rerun a flaky test
Given a file named "features/fail.feature" with:

Scenario: Retry a flaky test
Given a file named "features/flaky.feature" with:
"""
Feature: Fails
Feature: Passes
Scenario: flaky test
Given a flaky step
"""
When I run `cucumber features --rerun_flakes=1 -q`
When I run `cucumber features --retry=1 -q`
Then it should pass with:
"""
Feature: Fails
Feature: Passes
Scenario: flaky test
Given a flaky step
(RuntimeError)
./features/step_definitions/steps.rb:4:in `flaky_pass'
./features/step_definitions/steps.rb:1:in `/^a flaky step$/'
features/fail.feature:4:in `Given a flaky step'
Failing Scenarios:
cucumber features/fail.feature:3
1 scenario (1 passed, 1 flake)
1 scenario (1 passed)
1 step (1 passed)
"""

Scenario: One failure, one flake
Given a file named "features/step_definitions/failing_steps.rb" with:
"""
Given(/^a failing step$/) do
expect(true).not_to be true
end
"""
And a file named "features/flaky.feature" with:
"""
Feature: Fails
Scenario: Flaky
Given a flaky step
Scenario: Failing
Given a failing step
"""
When I run `cucumber features --retry=1 -q`
Then it should fail with:
"""
Feature: Fails
Scenario: Flaky
Given a flaky step
(RuntimeError)
./features/step_definitions/steps.rb:4 in `flaky_pass'
./features/step_definitions/steps.rb:1 in `/^a flaky step$/'
features/fail.feature:4:in `Given a flaky step`
Scenario: Failing
Given a failing step
(RuntimeError)
./features/step_definitions/failing_steps.rb:4 in `expect'
./features/step_definitions/failing_steps.rb:1 in `/^a failing step$/'
features/fail.feature:7:in `Given a failing step'
2 scenarios (1 passed, 1 failed)
2 steps (1 passed, 1 failed)
"""
4 changes: 4 additions & 0 deletions lib/cucumber/cli/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,10 @@ def fail_fast?
!!@options[:fail_fast]
end

def retry_attempts
@options[:retry]
end

def snippet_type
@options[:snippet_type] || :regexp
end
Expand Down
4 changes: 4 additions & 0 deletions lib/cucumber/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ def fail_fast?
@options[:fail_fast]
end

def retry_attempts
@options[:retry]
end

def guess?
@options[:guess]
end
Expand Down
7 changes: 7 additions & 0 deletions spec/cucumber/cli/configuration_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,13 @@ def reset_config
expect(config.snippet_type).to eq :regexp
end
end

describe "#retry_attempts" do
it "returns the specified number of retries" do
config.parse!(['--retry=3'])
expect(config.retry_attempts).to eql 3
end
end
end
end
end

0 comments on commit 06aa0d7

Please sign in to comment.