-
-
Notifications
You must be signed in to change notification settings - Fork 53
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support Prism as a Ruby parser #277
Conversation
Prism requires Ruby 2.7 or higher at runtime. The most straightforward solution would be to drop Ruby 2.6 from RuboCop AST's runtime support. I plan to open a PR to drop Ruby 2.6 from the runtime. Do you have any thoughts on this? |
That's fine by me. Ruby 2.6's usage is quite low already and the upgrade that to 2.7 is trivial. |
lib/rubocop/ast/processed_source.rb
Outdated
"RuboCop found unknown Ruby version: #{ruby_version.inspect}" | ||
def parser_class(ruby_version, parser_engine) | ||
case parser_engine | ||
when :parser |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we shouldn't name this :parser_whitequark
, given that parser
is a very generic name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
parser_whitequark
is a unique name, but the context might become distant if the org of whitequark/parser
changes to something other than whitequark. (Well, that's unlikely.)
Maybe something like parser_gem
can be suggested, but I'm not that confident about that. So, parser_whitequark
is fine too, if @bbatsov prefers that :-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For now, I've renamed it to parser_whitequark
. Please let me know if you'd like the name updated further.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that name is fine and we can always provide some aliases down the road.
In general, I'm guessing that at some point native Prism (without the parser_whitequark
) translation will be the only parsing engine we'll be supporting. I guess that once most tools switch to Prism maintaining the older parsers will become pointless. It's amazing how quickly Prism is getting traction in the Ruby community these days https://blog.jruby.org/2024/02/23/jruby-prism-parser.html
I like the PR overall - no concerns from me regarding the overall approach and the implementation. @rubocop/rubocop-core please take a look as well, given this is a major development. @koic Did you get the chance to compare the performance with the new old and the new parser engine? |
Here is an example of benchmark for my local development: # bench.rb
require 'bundler/inline'
gemfile(true) do
source 'https://rubygems.org'
gem 'benchmark-ips'
gem 'rubocop-ast', path: '/Users/koic/src/github.com/rubocop/rubocop-ast'
end
require 'rubocop-ast'
source = File.read(__FILE__)
Benchmark.ips do |x|
x.report('Parser') do
RuboCop::AST::ProcessedSource.new(source, 3.3, parser_engine: :parser)
end
x.report('Prism') do
RuboCop::AST::ProcessedSource.new(source, 3.3, parser_engine: :parser_prism)
end
x.compare!
end Although results may depending on the code analyzed, it has been observed that Prism tends to be faster: $ ruby bench.rb
Fetching gem metadata from https://rubygems.org/..
Resolving dependencies...
ruby 3.4.0dev (2024-02-24T04:19:37Z master e9e752c7ef) [x86_64-darwin23]
Warming up --------------------------------------
Parser 62.000 i/100ms
Prism 107.000 i/100ms
Calculating -------------------------------------
Parser 607.475 (± 4.9%) i/s - 3.038k in 5.013690s
Prism 1.040k (± 3.0%) i/s - 5.243k in 5.047621s
Comparison:
Prism: 1039.7 i/s
Parser: 607.5 i/s - 1.71x slower |
Follow up rubocop#277 (comment). This PR drops Ruby 2.6 runtime support to prepare to support Prism.
Follow up rubocop#277 (comment). This PR drops Ruby 2.6 runtime support to prepare to support Prism.
Follow up #277 (comment). This PR drops Ruby 2.6 runtime support to prepare to support Prism.
This PR introduces the `parser_engine` option to `ProcessedSource` to support Prism, as part of the RuboCop AST side effort towards addressing rubocop/rubocop#12600. ## Configuration By default, analysis is performed using the Parser gem, so the default value for the newly added `parser_engine` is `parser_whitequark`: ```ruby ProcessedSource.new(@options[:stdin], ruby_version, file, parser_engine: :parser_whitequark) ``` This code maintains compatibility, meaning the traditional behavior is preserved: ```ruby ProcessedSource.new(@options[:stdin], ruby_version, file) ``` To perform analysis using Prism, specify `parser_engine: :parser_prism`: ```ruby ProcessedSource.new(@options[:stdin], ruby_version, file, parser_engine: :parser_prism) ``` The parameter name `parser_prism` reflects the original parser_prism which was the basis for `Prism::Translation::Parser` (now integrated into Prism): https://github.com/kddnewton/parser-prism This is an experimental introduction, and some incompatibilities still remain. > [!NOTE] > As initially mentioned in rubocop/rubocop#12600 (comment), > the plan was to set `parser_engine: prism`. > > However, the parser engine used in this PR is `Prism::Translation::Parser`, not `Prism`: > ruby/prism#2419 > > `Prism::Translation::Parser` and `Prism` have different ASTs, so their migration will definitely cause incompatibility. > So, considering the possibility of further replacing `Prism::Translation::Parser` with `Prism` in the future, > it has been decided that it might be better not to use `ParserEngine: prism` for the time being. > `ParserEngine: prism` is reserved for `Prism`, not `Prism::Translation::Parser`. > > Therefore, the parameter value has been set to `parser_engine: parser_prism` specifically for > `Prism::Translation::Parser`. > > This means that the planned way to specify Prism in .rubocop.yml file will be `ParserEngine: parser_prism`, > not `ParserEngine: prism`. ## Compatibility The compatibility issues between Prism and the Parser gem have not been resolved. The failing tests will be skipped with `broken_on: :prism`: - ruby/prism#2454 has been resolved but not yet released. - ruby/prism#2467 is still unresolved. Issues that will be resolved in several upcoming releases of Prism are being skipped with `broken_on: :prism`. Anyway, RuboCop AST can be released independently of the resolution and release of Prism. > [!NOTE] > The hack in `Prism::Translation::Parser` for `ProcessedSource` needs to be fixed: > https://github.com/ruby/prism/blob/v0.24.0/lib/prism/translation/parser/rubocop.rb > > If the above interface is accepted, a fix will be proposed on the Prism side. ## Test Tests for RuboCop AST with Prism as the backend can be run as follows: ```console bundle exec rake prism_spec ``` The above is the shortcut alias for: ```console PARSER_ENGINE=parser_prism TARGET_RUBY_VERSION=3.3 rake spec ``` RuboCop AST works on Ruby versions 2.6+, but since Prism only targets analysis for Ruby 3.3+, `internal_investigation` Rake task will not be executed. This task is only run with the Parser gem, which can analyze Ruby versions 2.0+.
One step closer! Great work, @koic! 🔥 |
With the introduction of `Prism::Translation::Parser` to RuboCop (RuboCop AST), the number of arguments for `RuboCop::AST::ProcessedSource#parser_class` internal API will be changed: rubocop/rubocop-ast#277 ## Before As a result, the following error will occur starting from the next release of RuboCop AST (< 1.30.0) : ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/rubocop.rb:25:in `parser_class': wrong number of arguments (given 2, expected 1) (ArgumentError) from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:309:in `create_parser' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:219:in `parse' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:47:in `initialize' from -e:1:in `new' from -e:1:in `<main>' ``` ## After This PR prevents the above error by updating the monkey patch to support the new argument: ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:int, 42) ``` Moreover, to ensure compatibility with the existing RuboCop AST, conditional logic has been implemented to maintain backward compatibility.
Looks very to me 🚀 |
With the introduction of `Prism::Translation::Parser` to RuboCop (RuboCop AST), the number of arguments for `RuboCop::AST::ProcessedSource#parser_class` internal API will be changed: rubocop/rubocop-ast#277 ## Before As a result, the following error will occur starting from the next release of RuboCop AST (< 1.30.0) : ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision ruby/prism@5124f9ac75) [x86_64-darwin22] /Users/koic/src/github.com/ruby/prism/lib/prism/translation/parser/rubocop.rb:25:in `parser_class': wrong number of arguments (given 2, expected 1) (ArgumentError) from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:309:in `create_parser' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:219:in `parse' from /Users/koic/src/github.com/rubocop/rubocop-ast/lib/rubocop/ast/processed_source.rb:47:in `initialize' from -e:1:in `new' from -e:1:in `<main>' ``` ## After This PR prevents the above error by updating the monkey patch to support the new argument: ```console $ bundle exec ruby -rrubocop/ast -rprism -rprism/translation/parser/rubocop -ve \ "p RuboCop::AST::ProcessedSource.new('42', 80_82_73_83_77.33).ast" ruby 3.3.0 (2023-12-25 revision ruby/prism@5124f9ac75) [x86_64-darwin22] s(:int, 42) ``` Moreover, to ensure compatibility with the existing RuboCop AST, conditional logic has been implemented to maintain backward compatibility. ruby/prism@8a6909f4d5
This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
Resolves rubocop#12600. This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
@bbatsov @marcandre Can you release this feature to move forward with rubocop/rubocop#12724? |
Resolves rubocop#12600. This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
Sure, I'll do this right away. |
Resolves rubocop#12600. This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
Resolves rubocop#12600. This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
Resolves rubocop#12600. This is the initial basic implementation to support multiple parser engines. It is still in an experimental phase. There are still incompatibilities with Prism compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support for Prism independently within the RuboCop core. > [!IMPORTANT] > To work this feature, the following patch for RuboCop AST needs to be released: > rubocop/rubocop-ast#277 ## Setting the parser engine RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`. Here are the parsers used as backends for each value: - `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser - `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`) By default, `parser_whitequark` is implicitly used. `parser_whitequark` can analyze source code from Ruby 2.0 and above: ```yaml AllCops: ParserEngine: parser_whitequark ``` `parser_prism` can analyze source code from Ruby 3.3 and above: ```yaml AllCops: ParserEngine: parser_prism TargetRubyVersion: 3.3 ``` `parser_prism` tends to perform analysis faster than `parser_whitequark`. However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ## Incompatibility At the time of opening this PR, the following cops have incompatibility with Prism: - `Gemspec/RequiredRubyVersion` - `Internal_affairs/location_line_equality_comparison` - `Layout/class_structure` - `Layout/empty_lines` - `Layout/end_of_line` - `Layout/heredoc_indentation` - `Layout/indentation_style` - `Layout/line_length` - `Layout/space_around_operators` - `Layout/space_inside_hash_literal_braces` - `Layout/trailing_whitespace` - `Lint/ambiguous_operator` - `Lint/ambiguous_regexp_literal` - `Lint/circular_argument_reference` - `Lint/deprecated_class_methods` - `Lint/deprecated_constants` - `Lint/erb_new_arguments` - `Lint/non_deterministic_require_order` - `Lint/numbered_parameter_assignment` - `Lint/parentheses_as_grouped_expression` - `Lint/redundant_dir_glob_sort` - `Lint/redundant_require_statement` - `Lint/refinement_import_methods` - `Lint/require_parentheses` - `Lint/syntax` - `Lint/unified_integer` - `Lint/useless_else_without_rescue` - `Naming/block_forwarding` - `Naming/heredoc_delimiter_case` - `Naming/heredoc_delimiter_naming` - `Naming/variable_number` - `Security/yaml_load` - `Style/arguments_forwarding` - `Style/array_intersect` - `Style/block_delimiters` - `Style/collection_compact` - `Style/command_literal` - `Style/comparable_clamp` - `Style/conditional_assignment_assign_in_condition` - `Style/conditional_assignment_assign_to_condition` - `Style/data_inheritance` - `Style/dir_empty` - `Style/file_empty` - `Style/frozen_string_literal_comment` - `Style/guard_clause` - `Style/hash_except` - `Style/hash_syntax` - `Style/hash_transform_keys` - `Style/hash_transform_values` - `Style/if_with_boolean_literal_branches` - `Style/multiline_ternary_operator` - `Style/multiline_when_then` - `Style/mutable_constant` - `Style/nested_file_dirname` - `Style/numeric_literals` - `Style/numeric_predicate` - `Style/object_then` - `Style/quoted_symbols` - `Style/redundant_begin` - `Style/redundant_freeze` - `Style/redundant_heredoc_delimiter_quotes` - `Style/redundant_parentheses` - `Style/rescue_modifier` - `Style/safe_navigation` - `Style/select_by_regexp` - `Style/single_line_methods` - `Style/slicing_with_range` - `Style/string_literals` - `Style/yaml_file_read` - `Style/yoda_condition` Some cop incompatibilities have been resolved in the Prism development line. For known issues, please refer to the Prism issues: https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop ### `Lint/Syntax` cop The messages generated by Lint/Syntax depend on the parser engine used. Parser gem: ```console $ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: unexpected token $end ``` Displays `unexpected token $end`. Prism: ```console $ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] (string):1:2: error: expected a matching `)` ``` Displays `expected a matching )`. There are differences in the messages between Parser gem and Prism, but since Prism can provide clearer messages in some cases, this incompatibility is accepted. In other words, the messages may vary depending on the parser engine. ## Test To run tests with Prism, the command `bundle exec rake prism_spec` is provided. This task is also executed in GitHub Actions. To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`: ```console $ PARSER_ENGINE=parser_prism path/to/test_spec.rb ``` In the context of testing with Prism, two options for test cases are provided: `broken_on: :prism` and `unsupported_on: :prism`. Both options are utilized to skip tests specifically for Prism. ### `broken_on: :prism` Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`. This indicates an expectation for the issue to be resolved within Prism. ### `unsupported_on: :prism` Prism is designed to parse Ruby versions 3.3+, which means that features unique to Ruby versions 2.0 through 3.2 are not supported. Test cases falling into this category are marked with `unsupported_on: :prism`. This marker is used for cases that are testable with the Parser gem but not with Prism. > [!NOTE] > With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`. > The `ascii_spec` task has not been failing for a while, so it will not be run by default. > However, `ascii_spec` task will continue to be checked in CI. If there are any failures > originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
Follow up rubocop/rubocop-ast#277 Internally, the Parser gem API is used directly to parse db/schema.rb. This has been replaced with `RuboCop::AST::ProcessedSource` from RuboCop AST 1.31, which is compatible with the Prism parser API of the Parser gem API. This change resolves error (e.g., `uninitialized constant Parser::Ruby33`) related to db/schema.rb parsing when Prism is specified as the parser engine, enabling paring with Prism.
Follow up rubocop/rubocop-ast#277 Internally, the Parser gem API is used directly to parse db/schema.rb. This has been replaced with `RuboCop::AST::ProcessedSource` from RuboCop AST 1.31, which is compatible with the Prism parser API of the Parser gem API. This change resolves error (e.g., `uninitialized constant Parser::Ruby33`) related to db/schema.rb parsing when Prism is specified as the parser engine, enabling paring with Prism.
Follow up rubocop/rubocop-ast#277 Internally, the Parser gem API is used directly to parse db/schema.rb. This has been replaced with `RuboCop::AST::ProcessedSource` from RuboCop AST 1.31, which is compatible with the Prism parser API of the Parser gem API. This change resolves error (e.g., `uninitialized constant Parser::Ruby33`) related to db/schema.rb parsing when Prism is specified as the parser engine, enabling paring with Prism.
Follow up rubocop/rubocop-ast#277 There are no changes to the implementation, but the required dependency version of RuboCop AST will be updated. It would be more rational than adding new dependency on `gem 'rubocop-ast', '>= 1.31.1'` in the Gemfile just to bump up the minor version. RuboCop Minitest now ensures that `RuboCop::AST::ProcessedSource` used in tests is aware of `parser_engine` parameter. Tests for RuboCop AST with Prism as the backend can be run as follows: ```console bundle exec rake prism_test ``` The above is the shortcut alias for: ```console PARSER_ENGINE=parser_prism TARGET_RUBY_VERSION=3.3 bundle exec rake test ``` RuboCop works on Ruby versions 2.7+, but since Prism only targets parsing for Ruby 3.3+, `internal_investigation` Rake task will not be executed. This task is only run with the Parser gem, which can parse Ruby versions 2.0+.
Follow up rubocop/rubocop-ast#277 Internally, the Parser gem API is used directly to parse db/schema.rb. This has been replaced with `RuboCop::AST::ProcessedSource` from RuboCop AST 1.31, which is compatible with the Prism parser API of the Parser gem API. This change resolves error (e.g., `uninitialized constant Parser::Ruby33`) related to db/schema.rb parsing when Prism is specified as the parser engine, enabling paring with Prism.
Follow up rubocop/rubocop-ast#277 Internally, the Parser gem API is used directly to parse db/schema.rb. This has been replaced with `RuboCop::AST::ProcessedSource` from RuboCop AST 1.31, which is compatible with the Prism parser API of the Parser gem API. This change resolves error (e.g., `uninitialized constant Parser::Ruby33`) related to db/schema.rb parsing when Prism is specified as the parser engine, enabling paring with Prism.
Follow up rubocop/rubocop-ast#277 There are no changes to the implementation, but the required dependency version of RuboCop AST will be updated. It would be more rational than adding new dependency on `gem 'rubocop-ast', '>= 1.31.1'` in the Gemfile just to bump up the minor version. RuboCop Minitest now ensures that `RuboCop::AST::ProcessedSource` used in tests is aware of `parser_engine` parameter. Tests for RuboCop AST with Prism as the backend can be run as follows: ```console bundle exec rake prism_test ``` The above is the shortcut alias for: ```console PARSER_ENGINE=parser_prism TARGET_RUBY_VERSION=3.3 bundle exec rake test ``` RuboCop works on Ruby versions 2.7+, but since Prism only targets parsing for Ruby 3.3+, `internal_investigation` Rake task will not be executed. This task is only run with the Parser gem, which can parse Ruby versions 2.0+.
Follow up rubocop/rubocop-ast#277 There are no changes to the implementation, but the required dependency version of RuboCop AST will be updated. It would be more rational than adding new dependency on `gem 'rubocop-ast', '>= 1.31.1'` in the Gemfile just to bump up the minor version. RuboCop Minitest now ensures that `RuboCop::AST::ProcessedSource` used in tests is aware of `parser_engine` parameter. Tests for RuboCop AST with Prism as the backend can be run as follows: ```console bundle exec rake prism_test ``` The above is the shortcut alias for: ```console PARSER_ENGINE=parser_prism TARGET_RUBY_VERSION=3.3 bundle exec rake test ``` RuboCop works on Ruby versions 2.7+, but since Prism only targets parsing for Ruby 3.3+, `internal_investigation` Rake task will not be executed. This task is only run with the Parser gem, which can parse Ruby versions 2.0+.
Follow up rubocop/rubocop-ast#277 There are no changes to the implementation, but the required dependency version of RuboCop AST will be updated. It would be more rational than adding new dependency on `gem 'rubocop-ast', '>= 1.31.1'` in the Gemfile just to bump up the minor version. RuboCop Minitest now ensures that `RuboCop::AST::ProcessedSource` used in tests is aware of `parser_engine` parameter. Tests for RuboCop AST with Prism as the backend can be run as follows: ```console bundle exec rake prism_test ``` The above is the shortcut alias for: ```console PARSER_ENGINE=parser_prism bundle exec rake test ``` RuboCop works on Ruby versions 2.7+, but since Prism only targets parsing for Ruby 3.3+, `internal_investigation` Rake task will not be executed. This task is only run with the Parser gem, which can parse Ruby versions 2.0+.
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
Follow up rubocop/rubocop-ast#277 In Prism (`Prism::Translation::Parser`), `match-with-lvasgn` can be distinctly differentiated. It is unclear whether to conform to the current behavior of the Parser gem, but initially, `def_node_matcher` has been updated to accept the following incompatibilities for `Performance/EndWith`, `Performance/StringInclude`, and `Performance/StartWith` cops to ensure it works with Prism 0.24.0 as well. ## Parser gem Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "foo"), s(:regopt)), s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ``` This lvar-injecting feature appears to have not been supported by Parser gem for a long time: whitequark/parser#69 (comment) ## Prism Returns an `send` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:send, s(:regexp, s(:str, "foo"), s(:regopt)), :=~, s(:send, nil, :bar)) ``` Returns an `match_with_lvasgn` node: ```console $ bundle exec ruby -Ilib -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("/(?<foo>)foo/ =~ bar")' ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22] s(:match_with_lvasgn, s(:regexp, s(:str, "(?<foo>)foo"), s(:regopt)), s(:send, nil, :bar)) ```
This PR introduces the
parser_engine
option toProcessedSource
to support Prism, as part of the RuboCop AST side effort towards addressing rubocop/rubocop#12600.Configuration
By default, analysis is performed using the Parser gem, so the default value for the newly added
parser_engine
isparser_whitequark
:This code maintains compatibility, meaning the traditional behavior is preserved:
To perform analysis using Prism, specify
parser_engine: :parser_prism
:The parameter name
parser_prism
reflects the original parser_prism which was the basis forPrism::Translation::Parser
(now integrated into Prism): https://github.com/kddnewton/parser-prismThis is an experimental introduction, and some incompatibilities still remain.
Note
As initially mentioned in rubocop/rubocop#12600 (comment),
the plan was to set
parser_engine: prism
.However, the parser engine used in this PR is
Prism::Translation::Parser
, notPrism
:ruby/prism#2419
Prism::Translation::Parser
andPrism
have different ASTs, so their migration will definitely cause incompatibility.So, considering the possibility of further replacing
Prism::Translation::Parser
withPrism
in the future,it has been decided that it might be better not to use
ParserEngine: prism
for the time being.ParserEngine: prism
is reserved forPrism
, notPrism::Translation::Parser
.Therefore, the parameter value has been set to
parser_engine: parser_prism
specifically forPrism::Translation::Parser
.This means that the planned way to specify Prism in .rubocop.yml file will be
ParserEngine: parser_prism
,not
ParserEngine: prism
.Compatibility
The compatibility issues between Prism and the Parser gem have not been resolved. The failing tests will be skipped with
broken_on: :prism
:Prism::Translation::Parser
incompatibility withParser::Diagnostic
ruby/prism#2454 has been resolved but not yet released.Prism::Translation::Parser
incompatibility with token type of left brace ruby/prism#2467 is still unresolved.Issues that will be resolved in several upcoming releases of Prism are being skipped with
broken_on: :prism
.Anyway, RuboCop AST can be released independently of the resolution and release of Prism.
Note
The hack in
Prism::Translation::Parser
forProcessedSource
needs to be fixed:https://github.com/ruby/prism/blob/v0.24.0/lib/prism/translation/parser/rubocop.rb
If the above interface is accepted, a fix will be proposed on the Prism side.
Test
Tests for RuboCop AST with Prism as the backend can be run as follows:
bundle exec rake prism_spec
The above is the shortcut alias for:
PARSER_ENGINE=parser_prism TARGET_RUBY_VERSION=3.3 rake spec
RuboCop AST works on Ruby versions 2.6+, but since Prism only targets analysis for Ruby 3.3+,
internal_investigation
Rake task will not be executed. This task is only run with the Parser gem, which can analyze Ruby versions 2.0+.