Skip to content

Commit

Permalink
Start getting acceptance tests for retry working
Browse files Browse the repository at this point in the history
There's a puzzle here with the output on the second scenario, with two
retries. I've detailed what I think the summary should be, but there
seems to be a passing scenario missing from the results totals.

Did I make a mistake, or is the code still wrong?
  • Loading branch information
mattwynne committed Jul 2, 2016
1 parent ec57acf commit a812fd9
Show file tree
Hide file tree
Showing 3 changed files with 33 additions and 11 deletions.
42 changes: 31 additions & 11 deletions features/docs/cli/retry_failing_tests.feature
Original file line number Diff line number Diff line change
@@ -1,32 +1,52 @@
@wip
Feature: Retry failing tests

Retry gives you a way to get through flaky tests that usually pass after a few runs.
This gives a development team a way forward other than disabling a valuable test.

- Specify max retry count in option
- Output information to the screen
- Output retry information in test report

Questions:
use a tag for flaky tests? Global option to retry any test that fails?

Background:
Given a scenario "Flakey" that fails once, then passes
And a scenario "Shakey" that fails twice, then passes
And a scenario "Solid" that passes
And a scenario "No Dice" that fails

Scenario:

@wip
Scenario: Retry once, so Flakey starts to pass
When I run `cucumber -q --retry 1`
Then it should fail with:
"""
4 scenarios (2 passed, 2 failed)
7 scenarios (5 failed, 2 passed)
"""

Scenario:
@wip
Scenario: Retry twice, so Shakey starts to pass too

First try:
Flacey x
Shakey x
Solid ✔︎
No Dice x
=> 4 scenarios (1 passed, 3 failed)

First retry:
Flakey ✔︎
Shakey x
No Dice x
=> 7 scenarios (2 passed, 5 failed)

Second retry:
Shakey ✔︎
No Dice x
=> 9 scenarios (3 passed, 6 failed)

When I run `cucumber -q --retry 2`
Then it should pass with:
Then it should fail with:
"""
9 scenarios (6 failed, 3 passed)
"""
4 scenarios (3 passed, 1 failed)
"""
1 change: 1 addition & 0 deletions lib/cucumber/filters.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@
require 'cucumber/filters/prepare_world'
require 'cucumber/filters/quit'
require 'cucumber/filters/randomizer'
require 'cucumber/filters/retry'
require 'cucumber/filters/tag_limits'
1 change: 1 addition & 0 deletions lib/cucumber/runtime.rb
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ def filters
filters << Cucumber::Core::Test::LocationsFilter.new(filespecs.locations)
filters << Filters::Randomizer.new(@configuration.seed) if @configuration.randomize?
filters << Filters::Quit.new
filters << Filters::Retry.new(@configuration)
# TODO: can we just use RbLanguages's step definitions directly?
step_match_search = StepMatchSearch.new(@support_code.ruby.method(:step_matches), @configuration)
filters << Filters::ActivateSteps.new(step_match_search, @configuration)
Expand Down

0 comments on commit a812fd9

Please sign in to comment.