Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fail fast switch #906

Merged
merged 13 commits into from
Sep 11, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,4 @@ branches:
only:
- master
- v1.3.x-bugfix

notifications:
email:
- [email protected]
irc:
- "irc.freenode.org#cucumber"
- add-fail-fast-switch
46 changes: 46 additions & 0 deletions features/docs/cli/fail_fast.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
@spawn
Feature: Fail fast

The --fail-fast flag causes Cucumber to exit immediately after the first
scenario fails.

Scenario: When a scenario fails
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The annotation @spawn needs to be added to this scenario so that the cucumber instance started by the When step is not executed in the same process as the cucumber instance that rake has started. This problem reminds me of "singletons considered harmful" and a quote from one of the pages return from a search of that topic: "everything that can be done with Singletons, can be done better with dependency injection". @mattwynne Cucumber.wants_to_quit really?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah totally agreed about the use of that global state @brasmusson - it's a legacy we're living with for now but I'd love to see a PR that gets rid of it. Should be much easier now that the behaviour is all centred in the QuitFilter.

Given a file named "features/bad.feature" with:
"""
Feature: Bad
Scenario: Failing
Given this step fails
"""
And a file named "features/good.feature" with:
"""
Feature: Good
Scenario: Passing
Given this step passes
"""
And the standard step definitions
When I run `cucumber --fail-fast`
Then it should fail
And the output should contain:
"""
1 scenario (1 failed)
"""

Scenario: When all the scenarios pass
Given a file named "features/first.feature" with:
"""
Feature: first feature
Scenario: foo first
Given this step passes
Scenario: bar first
Given this step passes
"""
And a file named "features/second.feature" with:
"""
Feature: second
Scenario: foo second
Given this step passes
Scenario: bar second
Given this step passes
"""
When I run `cucumber --fail-fast`
Then it should pass
8 changes: 5 additions & 3 deletions lib/cucumber/cli/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ def expand?
@options[:expand]
end

def fail_fast?
!!@options[:fail_fast]
end

def snippet_type
@options[:snippet_type] || :regexp
end
Expand Down Expand Up @@ -133,8 +137,6 @@ def arrange_formats
@options[:formats].uniq!
@options.check_formatter_stream_conflicts()
end


end
end
end
end
2 changes: 1 addition & 1 deletion lib/cucumber/cli/main.rb
Original file line number Diff line number Diff line change
Expand Up @@ -102,4 +102,4 @@ def trap_interrupt
end
end
end
end
end
6 changes: 5 additions & 1 deletion lib/cucumber/cli/options.rb
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ class Options
NO_PROFILE_SHORT_FLAG = '-P'
PROFILE_LONG_FLAG = '--profile'
NO_PROFILE_LONG_FLAG = '--no-profile'
FAIL_FAST_FLAG = '--fail-fast'
OPTIONS_WITH_ARGS = ['-r', '--require', '--i18n', '-f', '--format', '-o', '--out',
'-t', '--tags', '-n', '--name', '-e', '--exclude',
PROFILE_SHORT_FLAG, PROFILE_LONG_FLAG,
Expand Down Expand Up @@ -126,6 +127,9 @@ def parse!(args)
list_keywords_and_exit(lang)
end
end
opts.on(FAIL_FAST_FLAG, "Exit immediately following the first failing scenario") do |v|
options[:fail_fast] = true
end
opts.on("-f FORMAT", "--format FORMAT",
"How to format features (Default: pretty). Available formats:",
*FORMAT_HELP) do |v|
Expand Down Expand Up @@ -425,4 +429,4 @@ def default_options
end

end
end
end
5 changes: 5 additions & 0 deletions lib/cucumber/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,10 @@ def dry_run?
@options[:dry_run]
end

def fail_fast?
@options[:fail_fast]
end

def guess?
@options[:guess]
end
Expand Down Expand Up @@ -182,6 +186,7 @@ def default_options
:strict => false,
:require => [],
:dry_run => false,
:fail_fast => false,
:formats => [],
:excludes => [],
:tag_expressions => [],
Expand Down
21 changes: 21 additions & 0 deletions lib/cucumber/formatter/fail_fast.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
require 'cucumber/formatter/io'
require 'cucumber/formatter/console'

module Cucumber
module Formatter
class FailFast
def initialize(configuration)
@configuration = configuration
end

def after_test_case(test_case, result)
Cucumber.wants_to_quit = true unless result.ok? @configuration.strict?
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this line. Is it a typo? If I remove the @configuration.strict? bit all the tests pass.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That because there are no test with a test case with an undefined step, then the test case result is "not ok" only in strict mode, and only then shall the execution be stopped.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh silly me! I'm old-school and I couldn't read the Ruby without parentheses.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's OK, I just remembered recently that you didn't have to use them, so I stopped using them just for the sake of novelty, really. I meant to use them in this code.

end

def done; end
def before_test_case *args; end
def before_test_step *args; end
def after_test_step *args; end
end
end
end
11 changes: 9 additions & 2 deletions lib/cucumber/runtime.rb
Original file line number Diff line number Diff line change
Expand Up @@ -180,9 +180,12 @@ def set_encoding
require 'cucumber/formatter/legacy_api/runtime_facade'
require 'cucumber/formatter/legacy_api/results'
require 'cucumber/formatter/ignore_missing_messages'
require 'cucumber/formatter/fail_fast'
require 'cucumber/core/report/summary'
def report
@report ||= Formatter::Fanout.new([summary_report] + formatters)
args = [summary_report] + formatters
args << fail_fast_report if @configuration.fail_fast?
@report ||= Formatter::Fanout.new(args)
end

def summary_report
Expand All @@ -200,6 +203,10 @@ def formatters
}
end

def fail_fast_report
@fail_fast_report ||= Formatter::FailFast.new configuration
end

def failure?
if @configuration.wip?
summary_report.test_cases.total_passed > 0
Expand Down Expand Up @@ -248,4 +255,4 @@ def log

end

end
end
100 changes: 100 additions & 0 deletions spec/cucumber/formatter/fail_fast_spec.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
require 'cucumber/formatter/fail_fast'
require 'cucumber/core'
require 'cucumber/core/gherkin/writer'
require 'cucumber/core/test/result'
require 'cucumber/core/filter'
require 'cucumber/core/ast'
require 'cucumber'

module Cucumber::Formatter
describe FailFast do
include Cucumber::Core
include Cucumber::Core::Gherkin::Writer

class WithStepsFake < Cucumber::Core::Filter.new
def test_case(test_case)
test_steps = test_case.test_steps.map do |step|
case step.name
when /fail/
step.with_action { raise Failure }
when /pass/
step.with_action {}
else
step
end
end

test_case.with_steps(test_steps).describe_to(receiver)
end
end

let(:report) { FailFast.new(double.as_null_object) }

context 'failing scenario' do
before(:each) do
@gherkin = gherkin('foo.feature') do
feature do
scenario do
step 'failing'
end

scenario do
step 'failing'
end
end
end
end

after(:each) do
Cucumber.wants_to_quit = false
end

it 'sets Cucumber.wants_to_quit' do
execute([@gherkin], report, [WithStepsFake.new])
expect(Cucumber.wants_to_quit).to be true
end
end

context 'passing scenario' do
before(:each) do
@gherkin = gherkin('foo.feature') do
feature do
scenario do
step 'passing'
end
end
end
end

it 'doesn\'t set Cucumber.wants_to_quit' do
execute([@gherkin], report, [WithStepsFake.new])
expect(Cucumber.wants_to_quit).to be_falsey
end
end

describe 'after_test_case method' do
context 'failing scenario' do
it 'sets Cucumber.wants_to_quit' do
result = Cucumber::Core::Test::Result::Failed.new(double('duration'), double('exception'))

test_case = double('test_case')
allow(test_case).to receive(:location) { Cucumber::Core::Ast::Location.new('foo.feature')}
report.after_test_case(test_case, result)
expect(Cucumber.wants_to_quit).to be true
end
end

context 'passing scenario' do
let(:result) { Cucumber::Core::Test::Result::Passed.new(double) }

it 'doesn\'t raise an error' do
expect{ report.after_test_case(double, result) }.not_to raise_error
end

it 'returns nil' do
expect(report.after_test_case(double, result)).to eql nil
end
end
end
end
end