--require not being honoured with --dry-run

[dpsi]

Summary

Cucumber does not honour the -r flag, and Cucumber provides no way load *.env.rb files during dry runs. From what I understand, cucumber ignores env files due to interactions with Rails. Why this option is not configurable is a mystery as not every Ruby project is a Rails project.

The --require option has the following help text: (emphasis mine)

Require files before executing the features. If this option is not specified, all *.rb files that are siblings or below the features will be loaded auto-matically. Automatic loading is disabled when this option is specified, and all loading becomes explicit. Files under directories named "support" are always loaded first.

I am not sure how much more explicit I can be. Sure I could add another rb file and require my env file, but that brings up 2 more problems:

1. I need to then keep track of every env file and its location in the directory structure.
2. Cucumber uses load instead of require, so I will get spammed with re-declaration warnings.

Expected Behavior

Given I start cucumber is started with the following:
cucumber -d -r features/support/env.rb
Then env.rb should be loaded.

Current Behavior

Cucumber does not honour the -r option:

cucumber -d -r features/support/env.rb
Code:

Features:
  * features/alarms.feature
  * features/.... etc.

Possible Solution

Cucumber should honour the -r option.
Cucumber should allow a configurable option for the env.rb loading behaviour for dry runs.
If neither of the above is acceptable, then Cucumber should use requires instead of loads as per #1043 as a work around.

Steps to Reproduce (for bugs)

1. Run cucumber with the dry run option, and the -r option set to an env.rb file
2. See cucumber not loading the file.

Context & Motivation

I just want to do a dry run so I can find unused steps. I don't see why Rails specific behaviour should affect non-Rails usage.
< Providing context helps us come up with a solution that is most useful in the real world>
Cucumber should be a general purpose BDD testing suite, not a Rails specific one.

Your Environment

• Version used: 3.1.2
• Operating System and version: Ubuntu 14.04 x64

Other notes

In general the loading of files by cucumber is not very well defined. We know that files in support folders are loaded first, but beyond that it is a mystery. I have had issues with tests randomly starting to fail because cucumber decided to load xyz.rb before env.rb. I can't require env.rb because of #1043

[danascheider]

@dpsi Can I get a few more details on your directory structure? Anything under /features should be loaded automatically and you shouldn't need the --require flag, so I'm not sure I understand what you're trying to do. If that functionality isn't working there may be something you're missing, but it's hard to tell what that might be without more information.

[dpsi]

Here is an example: https://github.com/dpsi/cucumber-test

~/cucumber-test$ cucumber
You should see this when env.rb is loaded
Feature: This feature should pass

  Scenario: This Scenario should pass # features/test.feature:3
      This is a before hook
    Given I should pass               # features/step_definitions/my_steps.rb:1
      I am going to pass
      This is an after hook

1 scenario (1 passed)
1 step (1 passed)
0m0.037s
~/cucumber-test$ cucumber -d
Feature: This feature should pass

  Scenario: This Scenario should pass # features/test.feature:3
    Given I should pass               # features/step_definitions/my_steps.rb:1

1 scenario (1 skipped)
1 step (1 skipped)
~/cucumber-test$ cucumber -d -r features
Feature: This feature should pass

  Scenario: This Scenario should pass # features/test.feature:3
    Given I should pass               # features/step_definitions/my_steps.rb:1

1 scenario (1 skipped)
1 step (1 skipped)
~/cucumber-test$ cucumber -d -r features -r features/support/env.rb
Feature: This feature should pass

  Scenario: This Scenario should pass # features/test.feature:3
    Given I should pass               # features/step_definitions/my_steps.rb:1

1 scenario (1 skipped)
1 step (1 skipped)

So is cucumber's behaviour wrong or is the help text wrong? As you can see, -r is not honoured during dry runs.

In the first one, the output is as expected, env.rb was loaded, and it put "You should see this when env.rb is loaded" to the screen.
In the second run, with just -d, the output is expected, env.rb was not loaded.
In the third run, with -d -r features, the output is expected, env.rb was not loaded.
In the fourth run, with -d -r features -r features/support/env.rb, the output is not expected, env.rb was not loaded even when cucumber was explicitly told to load it.

[danascheider]

@dpsi Offhand I'm not sure. I was able to replicate this locally on my machine, so it's definitely not just you. Thinking about it, this seems like expected behaviour but I'll double check on that and let you know.

[danascheider]

Yes, @dpsi, this is expected behaviour. Env files are explicitly excluded in this code. They are filtered out after required directories/files are identified, so the --require flag won't affect this behaviour. What do you have in your env.rb file that you need to run even on dry runs?

[dpsi]

@danascheider First, thanks for verifying this on your end.

The env.rb would be the one requiring libraries, setting constants, among other things. All in all, it sets up the environment. Without it, other files will end up with uninitialized constants, no method errors, etc.

While doing googling for my issue, I came across this: https://groups.google.com/d/msg/cukes/Kucz1T_VBss/VkEUBpY6MQAJ

So the dry run surprises have been affecting people for a while now. I'm not sure if there is a willingness to change in time for cucumber 4, but waiting till cucumber 5 isn't practical either.

Is there any other way to change this behaviour other than monkey-patching configuration.rb? Can the help text be changed to reflect the actual behaviour, or can we make --require actually load things?

As a side note: If cucumber decides to ditch this dry run behaviour, you could still maintain the existing behaviour with the --exclude param.

[danascheider]

@dpsi What I'm not sure I understand is why you need to set constants and require libraries if the steps aren't running. Could you clarify that for me?

[dpsi]

I don't need them at all, but cucumber still has to load any non-env files, and any code not inside a step or method would still get executed, even during a dry-run. So if env,rb sets @config then when my_setup_helper.rb tries to access @config[:some_var] it will get a no method error.

I understand that perhaps my issue is unusual, but I am working on a fairly old codebase (we just upgraded from cucumber 2.4 to 3.1 last month, and we have been using cucumber since at least 1.3) so there are some instances where we might be doing something that is a code smell.

Cleaning up the .rb files is in the works, but removing unused step defs is a bigger priority since they take up majority of the execution time during cucumber's startup. After monkey patching cucumber to work around the env problem, it looks like I have 239 unused step defs spread out over dozens of files. You can maybe grasp the size of repo I am working on.

Not sure if you have already past feature cut for cucumber 4, but adding env files into dry runs along with a note about using exclude to maintain previous behaviour seems like a small change. If the only logic around loading is only controlled by that one file you mentioned, then the change would be 1 line of code.

[danascheider]

I definitely understand the challenges of working with a legacy code base. It would definitely be a small change, but I would need to get the input of other maintainers before I could make such a change. In the Google group link you shared, @mattwynne appears to suggest he'd be open to this kind of change in the API but I'd also be curious what @aslakhellesoy has to say about it, especially since we try to keep changes consistent across all Cucumber implementations.

[mattwynne]

I'd like to see us do something to support your use-case in 4.0 @dpsi, I don't think it's too late. If you think it's only a minor change @danascheider why don't we go for it?

As you've recognised, @dpsi, the current behaviour around requiring files is kinda crufty and incoherent. I'd love this to get some attention for 5.0 if you were interested in helping us spec it out in a more holistic / comprehensive manner.

[danascheider]

Sure thing. I'll go ahead and make a PR for that today @mattwynne.

[dpsi]

@mattwynne @danascheider sounds good to me. There are a couple areas I can think of for cucumber 5. Looking forward to the future.

Thanks for the help!

[aslakhellesoy]

Sorry I’m late to join the discussion. When we added dry runs back in the day, I think the purpose was to validate that all the feature files are syntactically correct. Nothing more.

Validating that feature files are correct only requires parsing. No user code needs to be loaded or executed.

I’m curious to understand what additional behaviour you are expecting from a dry run. Or put differently: To you - what is the purpose of a dry run?

I see there is a PR in flight, but let’s not rush into solutions just yet.

[aslakhellesoy]

I’m reopening this issue as it’s not resolved yet. We’ll close it when a fix is merged to master, or if the suggestion is rejected altogether.

My position is that no files should be loaded during a dry run, and that —require should either cause an immediate error or be ignored altogether.

[dpsi]

@aslakhellesoy I'm trying to both validate feature files, and to find un-used step-defs. I think the issue is that Cucumber uses Kernel#load, which causes code to be executed. Even if we instead require the file, code would still be executed. dry-run is somewhat misleading, when it loads all the files it never calls any of the step definitions you wrote. Therefore it is a 'dry-run' in the sense that Cucumber never actually runs through the Scenarios. The caveat is that Ruby by design interprets and executes code immediately if it's not guarded inside a class, method, or module. This is comparable to what happens when you source a file in a shell such as bash.

[aslakhellesoy]

I just realised (by reading the changelog) that the purpose of dry run is also to print information about what step definitions are implemented/undefined.

I was a bit surprised about this (I never use this feature myself), but I can see how it can be useful.

With this insight I think it makes sense to load exactly the same files as during a “wet” run. The —dry-run option should not affect what’s loaded or the order in which isnloaded.

The only effect should be that step definitions are not executed.

If this is what the PR does I’m happy with it!

[danascheider]

@aslakhellesoy Cool, that is what the PR does.

[SeanFelipe]

I use the dry-run feature exactly to find what definitions are unimplemented. We also use some metaprogramming to define our page models at runtime, so we do need the env.rb support to make a dry run work.

I'm having issues with dry run so I'll bump our cucumber version and hopefully pick up this change 👍

[luke-hill]

@SeanFelipe if you have an issue specifically with cucumber and dry-runs. It's worth opening a separate issue.

Be sure to include a github repo to a small reproducible case. See SSCCE.org

[SeanFelipe]

thanks @luke-hill . it has to do with tools like SitePrism, https://github.com/natritmeyer/site_prism , which dynamically create page models at runtime. I do have a demo repo I'm working on as well so I'll see if I can get that code to a state where the context is right for this bug, and file an issue.

If you'd like a look right away, here's a variation of SitePrism I'm working on for mobile and Appium testing. You can see the metaprogramming where I use class_eval to dynamically add constant methods. Not great for production, but this is a test framework for mobile, the raw string evals are quite useful and won't see a public facing environment. https://github.com/SeanFelipe/Appiumario/blob/master/features/screen_models/screen_models.rb#L17

Basically, cucumber -d fails in this use case because we need the env.rb code to make our page models. The step definitions reference the page models, which don't exist, and we get nil errors.

I'll put in a bit more context into the code and write something up. Peace ~~

[luke-hill]

Ironically I am the sole maintainer of site_prism. So if you think the issue is there, open an issue there, we can work in a cucumber stack

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.