[Fix rubocop#12600] Support Prism as a Ruby parser

Resolves rubocop#12600.

This is the initial basic implementation to support multiple parser engines.
It is still in an experimental phase. There are still incompatibilities with Prism
compared to the Parser gem, but as the RuboCop core, it is possible to begin providing support
for Prism independently within the RuboCop core.

> [!IMPORTANT]
> To work this feature, the following patch for RuboCop AST needs to be released:
> rubocop/rubocop-ast#277

## Setting the parser engine

RuboCop allows switching the backend parser by specifying either
`parser_whitequark` or `parser_prism` for the `ParserEngine`.

Here are the parsers used as backends for each value:

- `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser
- `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`)

By default, `parser_whitequark` is implicitly used.

`parser_whitequark` can analyze source code from Ruby 2.0 and above:

```yaml
AllCops:
  ParserEngine: parser_whitequark
```

`parser_prism` can analyze source code from Ruby 3.3 and above:

```yaml
AllCops:
  ParserEngine: parser_prism
  TargetRubyVersion: 3.3
```

`parser_prism` tends to perform analysis faster than `parser_whitequark`.

However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental.
For known issues, please refer to the Prism issues:
https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop

## Incompatibility

At the time of opening this PR, the following cops have incompatibility with Prism:

- `Gemspec/RequiredRubyVersion`
- `Internal_affairs/location_line_equality_comparison`
- `Layout/class_structure`
- `Layout/empty_lines`
- `Layout/end_of_line`
- `Layout/heredoc_indentation`
- `Layout/indentation_style`
- `Layout/line_length`
- `Layout/space_around_operators`
- `Layout/space_inside_hash_literal_braces`
- `Layout/trailing_whitespace`
- `Lint/ambiguous_operator`
- `Lint/ambiguous_regexp_literal`
- `Lint/circular_argument_reference`
- `Lint/deprecated_class_methods`
- `Lint/deprecated_constants`
- `Lint/erb_new_arguments`
- `Lint/non_deterministic_require_order`
- `Lint/numbered_parameter_assignment`
- `Lint/parentheses_as_grouped_expression`
- `Lint/redundant_dir_glob_sort`
- `Lint/redundant_require_statement`
- `Lint/refinement_import_methods`
- `Lint/require_parentheses`
- `Lint/syntax`
- `Lint/unified_integer`
- `Lint/useless_else_without_rescue`
- `Naming/block_forwarding`
- `Naming/heredoc_delimiter_case`
- `Naming/heredoc_delimiter_naming`
- `Naming/variable_number`
- `Security/yaml_load`
- `Style/arguments_forwarding`
- `Style/array_intersect`
- `Style/block_delimiters`
- `Style/collection_compact`
- `Style/command_literal`
- `Style/comparable_clamp`
- `Style/conditional_assignment_assign_in_condition`
- `Style/conditional_assignment_assign_to_condition`
- `Style/data_inheritance`
- `Style/dir_empty`
- `Style/file_empty`
- `Style/frozen_string_literal_comment`
- `Style/guard_clause`
- `Style/hash_except`
- `Style/hash_syntax`
- `Style/hash_transform_keys`
- `Style/hash_transform_values`
- `Style/if_with_boolean_literal_branches`
- `Style/multiline_ternary_operator`
- `Style/multiline_when_then`
- `Style/mutable_constant`
- `Style/nested_file_dirname`
- `Style/numeric_literals`
- `Style/numeric_predicate`
- `Style/object_then`
- `Style/quoted_symbols`
- `Style/redundant_begin`
- `Style/redundant_freeze`
- `Style/redundant_heredoc_delimiter_quotes`
- `Style/redundant_parentheses`
- `Style/rescue_modifier`
- `Style/safe_navigation`
- `Style/select_by_regexp`
- `Style/single_line_methods`
- `Style/slicing_with_range`
- `Style/string_literals`
- `Style/yaml_file_read`
- `Style/yoda_condition`

Some cop incompatibilities have been resolved in the Prism development line.

For known issues, please refer to the Prism issues:
https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop

### `Lint/Syntax` cop

The messages generated by Lint/Syntax depend on the parser engine used.

Parser gem:

```console
$ ruby -rparser/ruby33 -ve 'p Parser::Ruby33.parse("(")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:2: error: unexpected token $end
```

Displays `unexpected token $end`.

Prism:

```console
$ ruby -rprism -rprism/translation/parser33 -ve 'p Prism::Translation::Parser33.parse("(")'
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [x86_64-darwin22]
(string):1:2: error: expected a matching `)`
```

Displays `expected a matching )`.

There are differences in the messages between Parser gem and Prism,
but since Prism can provide clearer messages in some cases, this incompatibility is accepted.
In other words, the messages may vary depending on the parser engine.

## Test

To run tests with Prism, the command `bundle exec rake prism_spec` is provided.
This task is also executed in GitHub Actions.

To run tests with Prism specifying files, set the environment variable `PARSER_ENGINE=parser_prism`:

```console
$ PARSER_ENGINE=parser_prism path/to/test_spec.rb
```

In the context of testing with Prism, two options for test cases are provided:
`broken_on: :prism` and `unsupported_on: :prism`.
Both options are utilized to skip tests specifically for Prism.

### `broken_on: :prism`

Test cases failing due to Prism incompatibilities are marked with `broken_on: :prism`.
This indicates an expectation for the issue to be resolved within Prism.

### `unsupported_on: :prism`

Prism is designed to parse Ruby versions 3.3+, which means that features unique to
Ruby versions 2.0 through 3.2 are not supported.
Test cases falling into this category are marked with `unsupported_on: :prism`.
This marker is used for cases that are testable with the Parser gem but not with Prism.

> [!NOTE]
> With `bundle exec rake`, `prism_spec` will be run instead of `ascii_spec`.
> The `ascii_spec` task has not been failing for a while, so it will not be run by default.
> However, `ascii_spec` task will continue to be checked in CI. If there are any failures
> originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.

.github/workflows/rubocop.yml

@@ -76,6 +76,22 @@ jobs:
       - name: internal_investigation
         run: bundle exec rake internal_investigation
 
+  prism:
+    runs-on: ubuntu-latest
+    name: Prism
+    steps:
+      - uses: actions/checkout@v4
+      - name: set up Ruby
+        uses: ruby/setup-ruby@v1
+        with:
+          # Specify the minimum Ruby version 2.7 required for Prism to run.
+          ruby-version: 2.7
+          bundler-cache: true
+      - name: spec
+        env:
+          PARSER_ENGINE: parser_prism
+        run: bundle exec rake prism_spec
+
   rspec4:
     runs-on: ubuntu-latest
     name: RSpec 4

.rubocop_todo.yml

@@ -13,7 +13,7 @@ InternalAffairs/NodeDestructuring:
 # Offense count: 55
 # Configuration parameters: CountComments, CountAsOne.
 Metrics/ClassLength:
-  Max: 191
+  Max: 192
 
 # Offense count: 235
 # Configuration parameters: CountComments, CountAsOne, ExcludedMethods, IgnoredMethods.

Rakefile

@@ -24,7 +24,10 @@ Dir['tasks/**/*.rake'].each { |t| load t }
 desc 'Run RuboCop over itself'
 RuboCop::RakeTask.new(:internal_investigation)
 
-task default: %i[documentation_syntax_check spec ascii_spec internal_investigation]
+# The `ascii_spec` task has not been failing for a while, so it will not be run by default.
+# However, `ascii_spec` task will continue to be checked in CI. If there are any failures
+# originating from `ascii_spec` in CI, please run `bundle exec ascii_spec` to investigate.
+task default: %i[documentation_syntax_check spec prism_spec internal_investigation]
 
 require 'yard'
 YARD::Rake::YardocTask.new

changelog/new_support_prism.md

@@ -0,0 +1 @@
+* [#12600](https://github.com/rubocop/rubocop/issues/12600): Support Prism as a Ruby parser (experimental). ([@koic][])

config/default.yml

@@ -144,6 +144,12 @@ AllCops:
   # Ruby version is still unresolved, RuboCop will use the oldest officially
   # supported Ruby version (currently Ruby 2.7).
   TargetRubyVersion: ~
+  # You can specify the parser engine. There are two options available:
+  # - `parser_whitequark` ... https://github.com/whitequark/parser
+  # - `parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`)
+  # By default, `parser` is used. For the `TargetRubyVersion` value, `parser` can be specified for versions `2.0` and above.
+  # `parser_prism` can be specified for versions `3.3` and above. `parser_prism` is faster but still considered experimental.
+  ParserEngine: parser_whitequark
   # Determines if a notification for extension libraries should be shown when
   # rubocop is run. Keys are the name of the extension, and values are an array
   # of gems in the Gemfile that the extension is suggested for, if not already

docs/modules/ROOT/pages/compatibility.adoc

@@ -37,7 +37,9 @@ The following table is the runtime support matrix.
 | 3.3 | -
 |===
 
-RuboCop targets Ruby 2.0+ code analysis since RuboCop 1.30. It restored code analysis support that had been removed earlier by mistake, together with dropping runtime support for unsupported Ruby versions.
+RuboCop targets Ruby 2.0+ code analysis with Parser gem as a parser since RuboCop 1.30. It restored code analysis support that had been removed earlier by mistake, together with dropping runtime support for unsupported Ruby versions.
+
+Starting from RuboCop 1.62, support for Prism's `Prism::Translation::parser` will enable analysis of Ruby 3.3+. For more details, please refer to the xref:configuration.adoc#setting-the-parser-engine[setting `ParserEngine`].
 
 NOTE: The compatibility xref:configuration.adoc#setting-the-target-ruby-version[setting `TargetRubyVersion`] is about code analysis (what RuboCop can analyze), not runtime (is RuboCop capable of running on some Ruby or not).
 

docs/modules/ROOT/pages/configuration.adoc

@@ -645,12 +645,50 @@ AllCops:
   TargetRubyVersion: 2.5
 ----
 
+NOTE: When `ParserEngine: parser_prism` is specified, the values that can be assigned to `TargetRubyVersion` must be `3.3` or higher.
+
 Otherwise, RuboCop will then check your project for a series of files where
 the version may be specified already. The files that will be looked for are
 `*.gemspec`, `.ruby-version`, `.tool-versions`, and `Gemfile.lock`.
 If Gemspec file has an array for `required_ruby_version`, the lowest version will be used.
 If none of the files are found a default version value will be used.
 
+== Setting the parser engine
+
+NOTE: The parser engine configuration was introduced in RuboCop 1.62. This experimental feature has been under consideration for a while.
+
+RuboCop allows switching the backend parser by specifying either `parser_whitequark` or `parser_prism` for the `ParserEngine`.
+
+Here are the parsers used as backends for each value:
+
+- `ParserEngine: parser_whitequark` ... https://github.com/whitequark/parser
+- `ParserEngine: parser_prism` ... https://github.com/ruby/prism (`Prism::Translation::Parser`)
+
+By default, `parser_whitequark` is implicitly used.
+
+`parser_whitequark` can analyze source code from Ruby 2.0 and above:
+
+[source,yaml]
+----
+AllCops:
+  ParserEngine: parser_whitequark
+----
+
+`parser_prism` can analyze source code from Ruby 3.3 and above:
+
+[source,yaml]
+----
+AllCops:
+  ParserEngine: parser_prism
+  TargetRubyVersion: 3.3
+----
+
+`parser_prism` tends to perform analysis faster than `parser_whitequark`.
+
+However, there are some incompatibilities with `parser_whitequark`, making it still considered experimental.
+For known issues, please refer to the Prism issues:
+https://github.com/ruby/prism/issues?q=is%3Aissue+is%3Aopen+label%3Arubocop
+
 == Automatically Generated Configuration
 
 If you have a code base with an overwhelming amount of offenses, it can

docs/modules/ROOT/pages/development.adoc

@@ -372,6 +372,26 @@ This works because the correcting a file is implemented by repeating investigati
 
 Note that `expect_correction` in `Cop` specs only asserts the result after one pass.
 
+=== Run tests
+
+RuboCop supports two parser engines: the Parser gem and Prism. By default, tests are executed with the Parser:
+
+```console
+$ bundle exec rake spec
+```
+
+To run all tests with the experimental support for Prism, use `bundle exec prism_spec`, and to execute tests for individual files,
+specify the environment variable `PARSER_ENGINE=parser_prism`.
+
+e.g., `PARSER_ENGINE=parser_prism spec/rubocop/cop/style/hash_syntax_spec.rb`
+
+`bundle exec rake` runs tests for both Parser gem and Prism parsers.
+
+But `ascii_spec` rake task does not run by default. Because `ascii_spec` task has not been failing for a while.
+However, `ascii_spec` task will continue to be checked in CI.
+
+In CI, all tests required for merging are executed. Please investigate if anything fails.
+
 === Configuration
 
 Each cop can hold a configuration and you can refer to `cop_config` in the

lib/rubocop/config.rb

@@ -62,7 +62,7 @@ def validate_after_resolution
 
     def_delegators :@hash, :[], :[]=, :delete, :dig, :each, :key?, :keys, :each_key,
                    :fetch, :map, :merge, :replace, :to_h, :to_hash, :transform_values
-    def_delegators :@validator, :validate, :target_ruby_version
+    def_delegators :@validator, :validate, :target_ruby_version, :parser_engine
 
     def to_s
       @to_s ||= @hash.to_s

lib/rubocop/config_validator.rb

@@ -63,6 +63,10 @@ def target_ruby_version
       target_ruby.version
     end
 
+    def parser_engine
+      for_all_cops.fetch('ParserEngine', :parser_whitequark).to_sym
+    end
+
     def validate_section_presence(name)
       return unless @config.key?(name) && @config[name].nil?
 

lib/rubocop/cop/base.rb

@@ -232,6 +232,10 @@ def target_ruby_version
         @config.target_ruby_version
       end
 
+      def parser_engine
+        @config.parser_engine
+      end
+
       def target_rails_version
         @config.target_rails_version
       end
@@ -254,7 +258,7 @@ def excluded_file?(file)
 
       # There should be very limited reasons for a Cop to do it's own parsing
       def parse(source, path = nil)
-        ProcessedSource.new(source, target_ruby_version, path)
+        ProcessedSource.new(source, target_ruby_version, path, parser_engine: parser_engine)
       end
 
       # @api private

lib/rubocop/rspec/cop_helper.rb

@@ -6,7 +6,11 @@
 module CopHelper
   extend RSpec::SharedContext
 
-  let(:ruby_version) { RuboCop::TargetRuby::DEFAULT_VERSION }
+  let(:ruby_version) do
+    # The minimum version Prism can parse is 3.3.
+    ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : RuboCop::TargetRuby::DEFAULT_VERSION
+  end
+  let(:parser_engine) { ENV.fetch('PARSER_ENGINE', :parser_whitequark).to_sym }
   let(:rails_version) { false }
 
   def inspect_source(source, file = nil)
@@ -28,7 +32,9 @@ def parse_source(source, file = nil)
       file = file.path
     end
 
-    processed_source = RuboCop::ProcessedSource.new(source, ruby_version, file)
+    processed_source = RuboCop::ProcessedSource.new(
+      source, ruby_version, file, parser_engine: parser_engine
+    )
     processed_source.config = configuration
     processed_source.registry = registry
     processed_source

lib/rubocop/rspec/shared_contexts.rb

@@ -139,49 +139,64 @@ def source_range(range, buffer: source_buffer)
 end
 
 RSpec.shared_context 'ruby 2.0' do
-  let(:ruby_version) { 2.0 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.0 }
 end
 
 RSpec.shared_context 'ruby 2.1' do
-  let(:ruby_version) { 2.1 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.1 }
 end
 
 RSpec.shared_context 'ruby 2.2' do
-  let(:ruby_version) { 2.2 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.2 }
 end
 
 RSpec.shared_context 'ruby 2.3' do
-  let(:ruby_version) { 2.3 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.3 }
 end
 
 RSpec.shared_context 'ruby 2.4' do
-  let(:ruby_version) { 2.4 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.4 }
 end
 
 RSpec.shared_context 'ruby 2.5' do
-  let(:ruby_version) { 2.5 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.5 }
 end
 
 RSpec.shared_context 'ruby 2.6' do
-  let(:ruby_version) { 2.6 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.6 }
 end
 
 RSpec.shared_context 'ruby 2.7' do
-  let(:ruby_version) { 2.7 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 2.7 }
 end
 
 RSpec.shared_context 'ruby 3.0' do
-  let(:ruby_version) { 3.0 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 3.0 }
 end
 
 RSpec.shared_context 'ruby 3.1' do
-  let(:ruby_version) { 3.1 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 3.1 }
 end
 
 RSpec.shared_context 'ruby 3.2' do
-  let(:ruby_version) { 3.2 }
+  # Prism supports parsing Ruby 3.3+.
+  let(:ruby_version) { ENV['PARSER_ENGINE'] == 'parser_prism' ? 3.3 : 3.2 }
 end
 
 RSpec.shared_context 'ruby 3.3' do
   let(:ruby_version) { 3.3 }
 end
+
+RSpec.shared_context 'ruby 3.4' do
+  let(:ruby_version) { 3.4 }
+end

lib/rubocop/rspec/support.rb

@@ -27,4 +27,5 @@
   config.include_context 'ruby 3.1', :ruby31
   config.include_context 'ruby 3.2', :ruby32
   config.include_context 'ruby 3.3', :ruby33
+  config.include_context 'ruby 3.4', :ruby34
 end

lib/rubocop/runner.rb

@@ -467,15 +467,21 @@ def minimum_severity_to_fail
       end
     end
 
+    # rubocop:disable Metrics/MethodLength
     def get_processed_source(file)
       config = @config_store.for_file(file)
       ruby_version = config.target_ruby_version
+      parser_engine = config.parser_engine
 
       processed_source = if @options[:stdin]
-                           ProcessedSource.new(@options[:stdin], ruby_version, file)
+                           ProcessedSource.new(
+                             @options[:stdin], ruby_version, file, parser_engine: parser_engine
+                           )
                          else
                            begin
-                             ProcessedSource.from_file(file, ruby_version)
+                             ProcessedSource.from_file(
+                               file, ruby_version, parser_engine: parser_engine
+                             )
                            rescue Errno::ENOENT
                              raise RuboCop::Error, "No such file or directory: #{file}"
                            end
@@ -484,6 +490,7 @@ def get_processed_source(file)
       processed_source.registry = mobilized_cop_classes(config)
       processed_source
     end
+    # rubocop:enable Metrics/MethodLength
 
     # A Cop::Team instance is stateful and may change when inspecting.
     # The "standby" team for a given config is an initialized but

lib/rubocop/target_ruby.rb

@@ -4,7 +4,7 @@ module RuboCop
   # The kind of Ruby that code inspected by RuboCop is written in.
   # @api private
   class TargetRuby
-    KNOWN_RUBIES = [2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1, 3.2, 3.3].freeze
+    KNOWN_RUBIES = [2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 3.0, 3.1, 3.2, 3.3, 3.4].freeze
     DEFAULT_VERSION = 2.7
 
     OBSOLETE_RUBIES = {

lib/rubocop/version.rb

@@ -5,7 +5,7 @@ module RuboCop
   module Version
     STRING = '1.60.2'
 
-    MSG = '%<version>s (using Parser %<parser_version>s, ' \
+    MSG = '%<version>s (using %<parser_version>s, ' \
           'rubocop-ast %<rubocop_ast_version>s, ' \
           'running on %<ruby_engine>s %<ruby_version>s)%<server_mode>s [%<ruby_platform>s]'
 
@@ -20,7 +20,7 @@ module Version
     # @api private
     def self.version(debug: false, env: nil)
       if debug
-        verbose_version = format(MSG, version: STRING, parser_version: Parser::VERSION,
+        verbose_version = format(MSG, version: STRING, parser_version: parser_version,
                                       rubocop_ast_version: RuboCop::AST::Version::STRING,
                                       ruby_engine: RUBY_ENGINE, ruby_version: RUBY_VERSION,
                                       server_mode: server_mode,
@@ -39,6 +39,19 @@ def self.version(debug: false, env: nil)
       end
     end
 
+    # @api private
+    def self.parser_version
+      config_path = ConfigFinder.find_config_path(Dir.pwd)
+      yaml = YAML.safe_load_file(config_path, permitted_classes: [Regexp, Symbol], aliases: true)
+
+      if yaml.dig('AllCops', 'ParserEngine') == 'parser_prism'
+        require 'prism'
+        "Prism #{Prism::VERSION}"
+      else
+        "Parser #{Parser::VERSION}"
+      end
+    end
+
     # @api private
     def self.extension_versions(env)
       features = Util.silence_warnings do

rubocop.gemspec

@@ -35,10 +35,11 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency('language_server-protocol', '>= 3.17.0')
   s.add_runtime_dependency('parallel', '~> 1.10')
   s.add_runtime_dependency('parser', '>= 3.3.0.2')
+  s.add_runtime_dependency('prism', '>= 0.24.0')
   s.add_runtime_dependency('rainbow', '>= 2.2.2', '< 4.0')
   s.add_runtime_dependency('regexp_parser', '>= 1.8', '< 3.0')
   s.add_runtime_dependency('rexml', '>= 3.2.5', '< 4.0')
-  s.add_runtime_dependency('rubocop-ast', '>= 1.30.0', '< 2.0')
+  s.add_runtime_dependency('rubocop-ast', '>= 1.31.0', '< 2.0')
   s.add_runtime_dependency('ruby-progressbar', '~> 1.7')
   s.add_runtime_dependency('unicode-display_width', '>= 2.4.0', '< 3.0')
 end

spec/rubocop/cop/alignment_corrector_spec.rb

@@ -69,7 +69,8 @@
         it_behaves_like 'heredoc indenter', '<<DOC', 20
       end
 
-      context 'with heredoc in backticks (<<``)' do
+      # FIXME: https://github.com/ruby/prism/issues/2498
+      context 'with heredoc in backticks (<<``)', broken_on: :prism do
         it_behaves_like 'heredoc indenter', '<<`DOC`', 20
       end
     end