-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add fail fast switch #906
Add fail fast switch #906
Changes from 11 commits
6634d25
f253745
78c1aa9
ff6891b
5ffdb6c
a08bff4
1ba97c9
748ce3b
0949c55
65dca21
e7e52d1
ef93747
b6579a3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,9 +10,4 @@ branches: | |
only: | ||
- master | ||
- v1.3.x-bugfix | ||
|
||
notifications: | ||
email: | ||
- [email protected] | ||
irc: | ||
- "irc.freenode.org#cucumber" | ||
- add-fail-fast-switch |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,45 @@ | ||
Feature: Fail fast | ||
|
||
The --fail-fast flag causes Cucumber to exit immediately after the first | ||
scenario fails. | ||
|
||
Scenario: When a scenario fails | ||
Given a file named "features/bad.feature" with: | ||
""" | ||
Feature: Bad | ||
Scenario: Failing | ||
Given this step fails | ||
""" | ||
And a file named "features/good.feature" with: | ||
""" | ||
Feature: Good | ||
Scenario: Passing | ||
Given this step passes | ||
""" | ||
And the standard step definitions | ||
When I run `cucumber --fail-fast` | ||
Then it should fail | ||
And the output should contain: | ||
""" | ||
1 scenario (1 failed) | ||
""" | ||
|
||
Scenario: When all the scenarios pass | ||
Given a file named "features/first.feature" with: | ||
""" | ||
Feature: first feature | ||
Scenario: foo first | ||
Given this step passes | ||
Scenario: bar first | ||
Given this step passes | ||
""" | ||
And a file named "features/second.feature" with: | ||
""" | ||
Feature: second | ||
Scenario: foo second | ||
Given this step passes | ||
Scenario: bar second | ||
Given this step passes | ||
""" | ||
When I run `cucumber --fail-fast` | ||
Then it should pass |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -102,4 +102,4 @@ def trap_interrupt | |
end | ||
end | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
require 'cucumber/formatter/io' | ||
require 'cucumber/formatter/console' | ||
|
||
module Cucumber | ||
module Formatter | ||
class FailFast | ||
def initialize(configuration) | ||
@configuration = configuration | ||
end | ||
|
||
def after_test_case(test_case, result) | ||
Cucumber.wants_to_quit = true unless result.ok? @configuration.strict? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't understand this line. Is it a typo? If I remove the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That because there are no test with a test case with an undefined step, then the test case result is "not ok" only in strict mode, and only then shall the execution be stopped. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh silly me! I'm old-school and I couldn't read the Ruby without parentheses. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's OK, I just remembered recently that you didn't have to use them, so I stopped using them just for the sake of novelty, really. I meant to use them in this code. |
||
end | ||
|
||
def done; end | ||
def before_test_case *args; end | ||
def before_test_step *args; end | ||
def after_test_step *args; end | ||
end | ||
end | ||
end |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,100 @@ | ||
require 'cucumber/formatter/fail_fast' | ||
require 'cucumber/core' | ||
require 'cucumber/core/gherkin/writer' | ||
require 'cucumber/core/test/result' | ||
require 'cucumber/core/filter' | ||
require 'cucumber/core/ast' | ||
require 'cucumber' | ||
|
||
module Cucumber::Formatter | ||
describe FailFast do | ||
include Cucumber::Core | ||
include Cucumber::Core::Gherkin::Writer | ||
|
||
class WithStepsFake < Cucumber::Core::Filter.new | ||
def test_case(test_case) | ||
test_steps = test_case.test_steps.map do |step| | ||
case step.name | ||
when /fail/ | ||
step.with_action { raise Failure } | ||
when /pass/ | ||
step.with_action {} | ||
else | ||
step | ||
end | ||
end | ||
|
||
test_case.with_steps(test_steps).describe_to(receiver) | ||
end | ||
end | ||
|
||
let(:report) { FailFast.new(double.as_null_object) } | ||
|
||
context 'failing scenario' do | ||
before(:each) do | ||
@gherkin = gherkin('foo.feature') do | ||
feature do | ||
scenario do | ||
step 'failing' | ||
end | ||
|
||
scenario do | ||
step 'failing' | ||
end | ||
end | ||
end | ||
end | ||
|
||
after(:each) do | ||
Cucumber.wants_to_quit = false | ||
end | ||
|
||
it 'sets Cucumber.wants_to_quit' do | ||
execute([@gherkin], report, [WithStepsFake.new]) | ||
expect(Cucumber.wants_to_quit).to be true | ||
end | ||
end | ||
|
||
context 'passing scenario' do | ||
before(:each) do | ||
@gherkin = gherkin('foo.feature') do | ||
feature do | ||
scenario do | ||
step 'passing' | ||
end | ||
end | ||
end | ||
end | ||
|
||
it 'doesn\'t set Cucumber.wants_to_quit' do | ||
execute([@gherkin], report, [WithStepsFake.new]) | ||
expect(Cucumber.wants_to_quit).to be_falsey | ||
end | ||
end | ||
|
||
describe 'after_test_case method' do | ||
context 'failing scenario' do | ||
it 'sets Cucumber.wants_to_quit' do | ||
result = Cucumber::Core::Test::Result::Failed.new(double('duration'), double('exception')) | ||
|
||
test_case = double('test_case') | ||
allow(test_case).to receive(:location) { Cucumber::Core::Ast::Location.new('foo.feature')} | ||
report.after_test_case(test_case, result) | ||
expect(Cucumber.wants_to_quit).to be true | ||
end | ||
end | ||
|
||
context 'passing scenario' do | ||
let(:result) { Cucumber::Core::Test::Result::Passed.new(double) } | ||
|
||
it 'doesn\'t raise an error' do | ||
expect{ report.after_test_case(double, result) }.not_to raise_error | ||
end | ||
|
||
it 'returns nil' do | ||
expect(report.after_test_case(double, result)).to eql nil | ||
end | ||
end | ||
end | ||
end | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The annotation
@spawn
needs to be added to this scenario so that the cucumber instance started by theWhen
step is not executed in the same process as the cucumber instance that rake has started. This problem reminds me of "singletons considered harmful" and a quote from one of the pages return from a search of that topic: "everything that can be done with Singletons, can be done better with dependency injection". @mattwynneCucumber.wants_to_quit
really?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah totally agreed about the use of that global state @brasmusson - it's a legacy we're living with for now but I'd love to see a PR that gets rid of it. Should be much easier now that the behaviour is all centred in the
QuitFilter
.