Skip to content

Commit

Permalink
Finish off retry
Browse files Browse the repository at this point in the history
- changed the acceptance test to use more descriptive names for the test
  scenarios (I kept getting confused)
- fixed a bug in the acceptance tests caused by our scenarios all running
  in one process, and using global variables (I added init scripts to
  reset them to zero at the start of a test run).
- tidied up the docs a bit in the acceptance test
- removed the @wip tag
- added the retry filter into the filterchain in runtime

There's a puzzle here with the output on the second scenario, with two
retries. I've detailed what I think the summary should be, but there
seems to be a passing scenario missing from the results totals.

Did I make a mistake, or is the code still wrong?
  • Loading branch information
mattwynne committed Aug 5, 2016
1 parent d3baaa4 commit 6532374
Show file tree
Hide file tree
Showing 4 changed files with 84 additions and 28 deletions.
67 changes: 51 additions & 16 deletions features/docs/cli/retry_failing_tests.feature
Original file line number Diff line number Diff line change
@@ -1,32 +1,67 @@
@wip
Feature: Retry failing tests

Retry gives you a way to get through flaky tests that usually pass after a few runs.
This gives a development team a way forward other than disabling a valuable test.

- Specify max retry count in option
- Output information to the screen
- Output retry information in test report

Questions:
use a tag for flaky tests? Global option to retry any test that fails?

Background:
Given a scenario "Flakey" that fails once, then passes
And a scenario "Shakey" that fails twice, then passes
Given a scenario "Fails-once" that fails once, then passes
And a scenario "Fails-twice" that fails twice, then passes
And a scenario "Solid" that passes
And a scenario "No Dice" that fails

Scenario:
When I run `cucumber -q --retry 1`
And a scenario "Fails-forever" that fails

Scenario: Retry once, so Fails-once starts to pass
When I run `cucumber -q --retry 1 --format summary`
Then it should fail with:
"""
7 scenarios (5 failed, 2 passed)
"""
And it should fail with:
"""
Fails-forever
Fails-forever ✗
Fails-forever ✗
Solid
Solid ✓
Fails-once feature
Fails-once ✗
Fails-once ✓
Fails-twice feature
Fails-twice ✗
Fails-twice ✗
"""

Scenario: Retry twice, so Fails-twice starts to pass too
When I run `cucumber -q --retry 2 --format summary`
Then it should fail with:
"""
4 scenarios (2 passed, 2 failed)
9 scenarios (6 failed, 3 passed)
"""
And it should fail with:
"""
Fails-forever
Fails-forever ✗
Fails-forever ✗
Fails-forever ✗
Solid
Solid ✓
Fails-once feature
Fails-once ✗
Fails-once ✓
Scenario:
When I run `cucumber -q --retry 2`
Then it should pass with:
Fails-twice feature
Fails-twice ✗
Fails-twice ✗
Fails-twice ✓
"""
4 scenarios (3 passed, 1 failed)
"""
43 changes: 31 additions & 12 deletions features/lib/step_definitions/retry_steps.rb
Original file line number Diff line number Diff line change
@@ -1,35 +1,54 @@
Given /^a scenario "([^\"]*)" that fails once, then passes$/ do |name|
Given /^a scenario "([^\"]*)" that fails once, then passes$/ do |full_name|
name = snake_case(full_name)
write_file "features/#{name}.feature",
<<-FEATURE
Feature: #{name}
Scenario: #{name}
Feature: #{full_name} feature
Scenario: #{full_name}
Given it fails once, then passes
FEATURE

write_file "features/step_defnitions/#{name}_steps.rb",
<<-STEPS
Given(/^it fails once, then passes$/) do
$#{name.downcase} ||= 0
$#{name.downcase} += 1
expect($#{name.downcase}).to eql 2
$#{name} += 1
expect($#{name}).to be > 1
end
STEPS

write_file "features/support/#{name}_init.rb",
<<-INIT
$#{name} = 0
INIT
end

Given /^a scenario "([^\"]*)" that fails twice, then passes$/ do |name|
Given /^a scenario "([^\"]*)" that fails twice, then passes$/ do |full_name|
name = snake_case(full_name)
write_file "features/#{name}.feature",
<<-FEATURE
Feature: #{name}
Scenario: #{name}
Feature: #{full_name} feature
Scenario: #{full_name}
Given it fails twice, then passes
FEATURE

write_file "features/step_definitions/#{name}_steps.rb",
<<-STEPS
Given(/^it fails twice, then passes$/) do
$#{name.downcase} ||= 0
$#{name.downcase} += 1
expect($#{name.downcase}).to eql 3
$#{name} ||= 0
$#{name} += 1
expect($#{name}).to be > 2
end
STEPS

write_file "features/support/#{name}_init.rb",
<<-INIT
$#{name} = 0
INIT
end

module SnakeCase
def snake_case(name)
name.downcase.gsub(/\W/, '_')
end
end

World(SnakeCase)
1 change: 1 addition & 0 deletions lib/cucumber/filters.rb
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@
require 'cucumber/filters/prepare_world'
require 'cucumber/filters/quit'
require 'cucumber/filters/randomizer'
require 'cucumber/filters/retry'
require 'cucumber/filters/tag_limits'
1 change: 1 addition & 0 deletions lib/cucumber/runtime.rb
Original file line number Diff line number Diff line change
Expand Up @@ -229,6 +229,7 @@ def filters
filters << Cucumber::Core::Test::LocationsFilter.new(filespecs.locations)
filters << Filters::Randomizer.new(@configuration.seed) if @configuration.randomize?
filters << Filters::Quit.new
filters << Filters::Retry.new(@configuration)
# TODO: can we just use RbLanguages's step definitions directly?
step_match_search = StepMatchSearch.new(@support_code.ruby.method(:step_matches), @configuration)
filters << Filters::ActivateSteps.new(step_match_search, @configuration)
Expand Down

0 comments on commit 6532374

Please sign in to comment.