diff --git a/features/docs/cli/retry_failing_tests.feature b/features/docs/cli/retry_failing_tests.feature index 1df3a1ece4..583eb4f4ed 100644 --- a/features/docs/cli/retry_failing_tests.feature +++ b/features/docs/cli/retry_failing_tests.feature @@ -1,32 +1,52 @@ -@wip Feature: Retry failing tests Retry gives you a way to get through flaky tests that usually pass after a few runs. This gives a development team a way forward other than disabling a valuable test. - + - Specify max retry count in option - Output information to the screen - Output retry information in test report - + Questions: use a tag for flaky tests? Global option to retry any test that fails? - + Background: Given a scenario "Flakey" that fails once, then passes And a scenario "Shakey" that fails twice, then passes And a scenario "Solid" that passes And a scenario "No Dice" that fails - - Scenario: + + @wip + Scenario: Retry once, so Flakey starts to pass When I run `cucumber -q --retry 1` Then it should fail with: """ - 4 scenarios (2 passed, 2 failed) + 7 scenarios (5 failed, 2 passed) """ - Scenario: + @wip + Scenario: Retry twice, so Shakey starts to pass too + + First try: + Flacey x + Shakey x + Solid ✔︎ + No Dice x + => 4 scenarios (1 passed, 3 failed) + + First retry: + Flakey ✔︎ + Shakey x + No Dice x + => 7 scenarios (2 passed, 5 failed) + + Second retry: + Shakey ✔︎ + No Dice x + => 9 scenarios (3 passed, 6 failed) + When I run `cucumber -q --retry 2` - Then it should pass with: + Then it should fail with: + """ + 9 scenarios (6 failed, 3 passed) """ - 4 scenarios (3 passed, 1 failed) - """ \ No newline at end of file diff --git a/lib/cucumber/filters.rb b/lib/cucumber/filters.rb index 30d252957a..a2b9c300fa 100644 --- a/lib/cucumber/filters.rb +++ b/lib/cucumber/filters.rb @@ -6,4 +6,5 @@ require 'cucumber/filters/prepare_world' require 'cucumber/filters/quit' require 'cucumber/filters/randomizer' +require 'cucumber/filters/retry' require 'cucumber/filters/tag_limits' diff --git a/lib/cucumber/runtime.rb b/lib/cucumber/runtime.rb index 98b39bae5d..e9cbc1e460 100644 --- a/lib/cucumber/runtime.rb +++ b/lib/cucumber/runtime.rb @@ -229,6 +229,7 @@ def filters filters << Cucumber::Core::Test::LocationsFilter.new(filespecs.locations) filters << Filters::Randomizer.new(@configuration.seed) if @configuration.randomize? filters << Filters::Quit.new + filters << Filters::Retry.new(@configuration) # TODO: can we just use RbLanguages's step definitions directly? step_match_search = StepMatchSearch.new(@support_code.ruby.method(:step_matches), @configuration) filters << Filters::ActivateSteps.new(step_match_search, @configuration)