Cucumber returns the same exit code when specs are failing and when command execution is failing

[ciembor]

When all cucumber scenarios pass it returns 0 exit code. When some of scenarios fails, cucumber returns 1 exit code. When there is wrong argument passed, or something else crashes cucumber, the exit code is 1 as well. Because of that it's not possible to distinguish if cucumber run all specs and some are failing or if some shit happend.

Real life usage of this feature? Recording failing specs and reruning them on continuous integration system.

[aslakhellesoy]

Aren't you able to detect the reason by looking at the output? Do you have any suggestions about how to improve this?

[ciembor]

I can parse the output to detect my specific problem, but it isn't generic solution. Here is more info about exit statuses:

http://www.gnu.org/software/libc/manual/html_node/Exit-Status.html

Cucumber is very specific application, which should fail even if it actually works. So I don't know if it is possible to follow any standard convention. However status 0 is always success and every other is an error. So the simplest thing is to define at least 3 exit statuses: 0 - success, 1 - cucumber error, 2 - failing specs. But there might be some conventions I don't know. I'm curious what other people think about it.

[mattwynne]

Programs that perform comparison use a different convention: they use status 1 to indicate a mismatch, and status 2 to indicate an inability to compare

In cucumber’s context, that sounds like 1 to indicate a test failure, and 2 to indicate an inability to run the tests.

As well as bad CLI arguments exit status 2 could represent, for example, undefined steps in strict mode, or exceptions raised when loading support code / step defs.

I would accept a PR for this, but I can’t see it being prioritised by the core team, sorry.

[ciembor]

From implementation point of view it would be cleaner to have 1 for application error (this is default in Ruby raise) and exit 2 when tests are failing. But maybe there is some way to force returning 2 bu default.

// edit

Oh, I looked at the code and it looks easier than I thought :). I will create some pull request soon.

[ciembor]

As you can see I made a pull request. I don't know your workflow, I hope it should go to the master branch. I would be glad to hear your opinions.

cucumber/common#845

[mattwynne]

Thanks @ciembor! I'll close this ticket in favour of the PR.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.