Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cucumber 2.x locations_filter slow as hell (unlike 1.3.19) #901

Closed
akostadinov opened this issue Aug 6, 2015 · 4 comments
Closed

cucumber 2.x locations_filter slow as hell (unlike 1.3.19) #901

akostadinov opened this issue Aug 6, 2015 · 4 comments

Comments

@akostadinov
Copy link
Contributor

I did some more investigation. It seems like with a huge number of locations set on the command line, the location filter is slow as hell. At first it takes a few seconds to process each location and becoming increasingly slower.

I am looking at it by inserting Kernel.puts("#{location.file}:#{location.line}") at cucumber/core/test/filters/locations_filter.rb:30. Can that be done somehow more efficient? Otherwise we are stuck with obsoleted and flaky cucumber 1.3 :/

------ Original report ------
We've just hit a bad issue with cucumber 2.x (tested 2.0.0 and 2.0.2). Some of our runs hang forever in seemingly an infinite loop. Other selections do not hang in the same way. It's a huge command line with file:num pairs.

I'll paste below a couple of backtraces I've obtained with gdb. (read backwards to what normally expected). It seems to me Core::Test::Step#match_locations? might be producing an infinite loop somehow. Any idea or at least debugging suggestions?

The third backtrace looks interesting with the method_missing call.

from /usr/local/bin/cucumber:23:in `<main>'
    from /usr/local/bin/cucumber:23:in `load'
    from /home/slave1/.gem/ruby/gems/cucumber-2.0.2/bin/cucumber:9:in `<top (required)>'
    from /home/slave1/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/cli/main.rb:38:in `execute!'
    from /home/slave1/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/runtime.rb:70:in `run!'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:18:in `compile'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:29:in `parse'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/gherkin/parser.rb:31:in `done'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/compiler.rb:23:in `done'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/filter.rb:61:in `done'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `done'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `sorted_test_cases'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `map'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `block in sorted_test_cases'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `test_cases_matching'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `select'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:31:in `block in test_cases_matching'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `match_locations?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `any?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `each'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `block in match_locations?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `match_locations?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `any?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `each'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `block in match_locations?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `match_locations?'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `attributes'
    from /home/slave1/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `flatten'


from /usr/local/bin/cucumber:23:in `<main>'
    from /usr/local/bin/cucumber:23:in `load'
    from /home/slave2/.gem/ruby/gems/cucumber-2.0.2/bin/cucumber:9:in `<top (required)>'
    from /home/slave2/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/cli/main.rb:38:in `execute!'
    from /home/slave2/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/runtime.rb:70:in `run!'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:18:in `compile'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:29:in `parse'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/gherkin/parser.rb:31:in `done'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/compiler.rb:23:in `done'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/filter.rb:61:in `done'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `done'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `sorted_test_cases'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `map'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `block in sorted_test_cases'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `test_cases_matching'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `select'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:31:in `block in test_cases_matching'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `any?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `each'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `block in match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `any?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `each'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `block in match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/step.rb:62:in `match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:126:in `match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:126:in `any?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:126:in `each'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:126:in `block in match_locations?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:27:in `match?'
    from /usr/share/ruby/forwardable.rb:171:in `include?'
    from /home/slave2/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:75:in `include?'
    from /usr/share/ruby/set.rb:215:in `subset?'
    from /usr/share/ruby/set.rb:215:in `all?'


    from /usr/local/bin/cucumber:23:in `<main>'
    from /usr/local/bin/cucumber:23:in `load'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/bin/cucumber:9:in `<top (required)>'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/cli/main.rb:38:in `execute!'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/runtime.rb:70:in `run!'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:18:in `compile'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:29:in `parse'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/gherkin/parser.rb:31:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/compiler.rb:23:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/filter.rb:61:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `sorted_test_cases'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `map'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `block in sorted_test_cases'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `test_cases_matching'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `select'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:31:in `block in test_cases_matching'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `any?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `each'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `block in match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `match_locations?'

    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `any?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `each'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `block in match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `attributes'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `flatten'
    from /usr/local/share/gems/gems/rspec-expectations-2.14.4/lib/rspec/matchers/method_missing.rb:9:in `method_missing'



from /usr/local/bin/cucumber:23:in `<main>'
    from /usr/local/bin/cucumber:23:in `load'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/bin/cucumber:9:in `<top (required)>'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/cli/main.rb:38:in `execute!'
    from /home/slave3/.gem/ruby/gems/cucumber-2.0.2/lib/cucumber/runtime.rb:70:in `run!'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:18:in `compile'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core.rb:29:in `parse'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/gherkin/parser.rb:31:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/compiler.rb:23:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/filter.rb:61:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `done'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `sorted_test_cases'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `map'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:26:in `block in sorted_test_cases'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `test_cases_matching'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:30:in `select'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/filters/locations_filter.rb:31:in `block in test_cases_matching'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `any?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `each'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/case.rb:77:in `block in match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `any?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `each'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/test/step.rb:52:in `block in match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `any?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `each'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `block in match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:125:in `match_locations?'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `attributes'
    from /home/slave3/.gem/ruby/gems/cucumber-core-1.2.0/lib/cucumber/core/ast/location.rb:130:in `flatten'
@akostadinov akostadinov changed the title cucumber 2.x hanging on source selection (unlike 1.3.19) cucumber 2.x locations_filter slow as hell (unlike 1.3.19) Aug 6, 2015
akostadinov added a commit to akostadinov/cucumber-ruby-core that referenced this issue Aug 6, 2015
to make it 2 orders of magnitude faster two things were needed:
1. do not try all locations, only those with matching file name
2. do not order test cases after the fact, put them in proper place from the beginning using matching locations index
3. cache locations lists by file name (this gives little improvement but may help better with even larger number of files)

This is achieved by adding a Test::Case method to return index of matching location so we don't need to figure out later which one exactly the matching location was.
akostadinov added a commit to akostadinov/cucumber-ruby-core that referenced this issue Aug 6, 2015
to make it 2 orders of magnitude faster two things were needed:
1. do not try all locations, only those with matching file name
2. do not order test cases after the fact, put them in proper place from the beginning using matching locations index
3. cache locations lists by file name (this gives little improvement but may help better with even larger number of files)

This is achieved by adding a Test::Case method to return index of matching location so we don't need to figure out later which one exactly the matching location was.
akostadinov added a commit to akostadinov/cucumber-ruby-core that referenced this issue Aug 6, 2015
to make it 2 orders of magnitude faster two things were needed:
1. do not try all locations, only those with matching file name
2. do not order test cases after the fact, put them in proper place from the beginning

The above also removes duplicates for good or for bad.

This is achieved by adding a Test::Case method to return index of matching location so we don't need to figure out later which one exactly the matching location was.
@mattwynne mattwynne modified the milestone: 2.0.x Aug 7, 2015
@xuanzhaopeng
Copy link

I confirm it to run the test case within cucumber/cucumber-ruby-core#97

has got amazing improvment !

Thanks @akostadinov

@erran-r7
Copy link

@xuanzhaopeng 👍 I also see the improvement by running with cucumber/cucumber-ruby-core#99 in one of team's suites.

@akostadinov
Copy link
Contributor Author

I believe issue is fixed better upstream now, closing.

@lock
Copy link

lock bot commented Oct 25, 2018

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked as resolved and limited conversation to collaborators Oct 25, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants