Add new ruby parser that uses Prism #1144

tompng · 2024-07-22T14:13:40Z

Adds a new ruby parser rdoc/parser/prism_ruby.rb that uses Prism.
The added new ruby parser is enabled by ENV['RDOC_USE_PRISM_PARSER']

Background

RDoc should use Prism as a parser to fix parsing bugs someday. These are few examples.

# endless method is not supported
def foo() = 1
# include ends with semicolon is ignored, making hard to write RDoc's test
class A; include M; end

Ruby parser should be refactored. Even small bugs are hard to fix because parser and comment handling is tightly coupled.
So I think rdoc/parser/ruby.rb needs to be completely rewritten.

This pull request completely rewrites lib/rdoc/parser/ruby.rb using Prism.
Note that this change will fix many bugs but might introduce unintentional changes or bugs.

Implementation details

Comments

There are two kind of comments. consecutive comments and modifier comments.

Consecutive comment

Consecutive comment is a set of continual comments written in blank line. They are linked to the next non-blank line.

01| # consecutive comment 1
02| # consecutive comment 1, linked to line 4
03|
04| # consecutive comment 2
05| # consecutive comment 2, linked to line 7
06|
07| puts(:hello)
08|
09| # consecutive comment 3, linked to line 11
10|
11| # consecutive comment 4
12| # consecutive comment 4, linked to line 14
13|
14| end

Consecutive comment that starts with ##\n is a metaprogramming method comment, but this rule is not strictly followed. RDoc sourcecode itself does not follow this rule.

Modifier comment

Modifier comment is a comment written in non-blank line.

puts :hello # modifier comment 1
def foo # modifier comment 2
  42
end # modifier comment 3

AST Traversing

Prism::Visitor is used to traverse AST node and to collect documentable comments and AST nodes.
When visitor visits a documentable node, we need to first process consecutive comments before start_line of the node which is not processed yet.
These unprocessed consecutive comment might be a metaprogramming comment that starts with ##\n.
After that, we can document the node itself with consecutive comment linked to node.start_line.
In this phase, RDoc also looks modifier comment on node.start_line and on node.end_line for some node types.

# standalone consecutive comment 2

# consecutive comment linked to undocumentable node
1 + 2

# consecutive comment to documentable node
def foo # modifier comment
  42
end # modifier comment

When visitor enters class/module node, we need to push that class/module to module_nesting stack.
Before leaving class/module node, we need to process unprocessed consecutive comments because these comments might be a metaprogramming comment that defines a method to the current class/module node. After that, we can pop class/module from module_nesting.

# consecutive comment to module A
module A
  # consecutive comment to class B
  class B
    # consecutive comment to def f
    def f; end
    ##
    # standalone consecutive comment
    # :method: method_of_class_b
  end
  ##
  # standalone consecutive comment 2
  # :method: method_of_module_a
end

Test

A new test test_rdoc_parser_prism_ruby.rb is added. Some ruby code passed to util_parser are extracted from test_rdoc_parser_ruby.rb.
To ensure there is no bad difference between the original RDoc::Parser::Ruby and the new RDoc::Parser::PrismRuby, test is run with both parsers.
Unfortunately, there are many difference that I think it's a bug. Assertion for these cases are skipped by unless accept_legacy_bug?

Output HTML file differences

These are the difference of the generated html files between master branch and this pull request.
Most file are identical except for newlines between html tags.

Removed files, these are not linked from any another html files.

RDoc/Markdown/Literals/MemoEntry.html
RDoc/Markdown/Literals/ParseError.html
RDoc/Markdown/Literals/RuleInfo.html
RDoc/Markdown/MemoEntry.html
RDoc/Markdown/ParseError.html
RDoc/Markdown/RuleInfo.html
RDoc/Parser/RipperStateLex/InnerStateLex.html
Rake.html

Link in html file changed

RDoc/Encoding.html
  Link changed, improved
    - <a href="RDoc.html"><code>RDoc</code></a>
    + <a href="../RDoc.html"><code>RDoc</code></a>

RDoc/Generator.html
  Link changed, improved
    - <a href="RDoc.html"><code>RDoc</code></a>
    + <a href="../RDoc.html"><code>RDoc</code></a>
  Link is gone
    - <h2>Adding <a href="Options.html"><code>Options</code></a> to <code>rdoc</code>
    + <h2>Adding Options to <code>rdoc</code>

RDoc/Generator/Markup.html
  Link is gone
    - It allows RDoc’s <a href="../CodeObject.html"><code>CodeObject</code></a> tree to avoid
    + It allows RDoc’s CodeObject</a> tree to avoid

Other changes

RDoc/Markup.html
  Constant `Heading` is added, bugfix

RDoc/Markup/Rule.html
  Superclass is changed from `Struct.new :weight` to `Object`
  RDoc does not support Struct as a superclass yet. It should be blank, unknown, or `Object` as a fallback.

RDoc/MarkupReference.html
  space is added after `<pre>:call-seq:`

RDoc/Parser/C.html
  `@enclosure_dependencies.extend TSort` is removed from extended module list. bugfix

RDoc/Parser/ChangeLog/Git.html
  Constant `LogEntry` is added, bugfix

RDoc/Parser/Ruby.html
  space is added after `:call-seq:`. Comment is bad because `:call-seq:` is written inside meta-comment style comment that start with `##\n`
  + <p>... either the :call-seq: , :arg: or :args: directives.</p>
  - <p>... either the :call-seq:, :arg: or :args: directives.</p>

RDoc/RI/ErRDoc/RI/Error.htmlror.html
  bugfix
  + <p class="link">RDoc::Error
  - <p class="link">RDoc::RDoc::Error

RDoc/Stats.html
  `def report; extend RDoc::Text; end` is excluded from extended module list. bugfix

RDoc/TopLevel.html
  wrong source of `def add_include` is fixed
  alias of name is added, (influence of the above source bug)

kddnewton · 2024-07-22T14:45:28Z

lib/rdoc/parser/ruby.rb

-  # Creates a module alias in +container+ at +rhs_name+ (or at the top-level
-  # for "::") with the name from +constant+.
+  def scan
+    @tokens = RDoc::Parser::RipperStateLex.parse(@content).sort_by { |t| [t.line_no, t.char_no] }


If you wanted, you could use Prism.parse_lex, which will give you a parse result with both the AST and the tokens. Alternatively, you could use Prism.lex or Prism.lex_compat to get back a list of tokens in a separate pass.

Thank you for letting me know.
Ideally, we should use it, but RDoc::Parser::RipperStateLex is doing some concatenation of tokens and RDoc's source code colorization seems to depends on it.

I want to make that part out of scope of this pull request.

That makes sense!

st0012 · 2024-07-22T16:11:35Z

lib/rdoc/parser/ruby.rb

@@ -145,21 +136,8 @@ class RDoc::Parser::Ruby < RDoc::Parser

  parse_files_matching(/\.rbw?$/)


To make it easier to rollback or compare behaviour between the 2 parsers, can we create a new parser/prism_ruby.rb file to host the new parser, and do the same for the test file too?
And to switch we simply remove this line from the current parser and add it to the new one.

Updated 👍

Added parser Parser::PrismRuby can be enabled by setting ENV['RDOC_USE_PRISM_PARSER']
Added test test_rdoc_parser_prism_ruby.rb tests both Parser::PrismRuby and Parser::Ruby (the above env is not needed for this test)

…lementation

hsbt · 2024-07-24T01:13:40Z

In my understanding, Prism only support Ruby 2.7+.

https://github.com/ruby/rdoc/blob/master/rdoc.gemspec#L233

This PR drop to support Ruby 2.6 from RDoc.

kddnewton · 2024-07-24T01:46:14Z

2.6 has been EOL for 2 years now. That being said, if it's a blocker, I can add 2.6 support to Prism.

st0012 · 2024-07-24T22:18:34Z

Personally, I'm fine with dropping Ruby 2.6 support because irb, reline, and debug have all dropped it. Also, the community should be encouraged to upgrade their Ruby versions in pursuit of better tools that are built on better infrastructure. Expanding Prism support to EOL Ruby versions will be sending the opposite message IMO.

st0012 · 2024-07-27T11:42:38Z

lib/rdoc/markup/pre_process.rb


      blankline
    else
-      result = yield directive, param if block_given?
+      result = yield directive, param, line if block_given?


What are these changes for? Looks like they're fixing issues that'd also affect the default Ruby parser?

In the default ruby parser, the third parameter line is just ignored.

The line where :method: is written is the source line number of ghost method.

1| class A 2| ## 3| # this is a ghost method `foo(x, y). Source location is line 4 4| # :method: foo 5| # :args: x, y 6| end

The original code is calculating line number by this code.

rdoc/lib/rdoc/parser/ruby.rb

Lines 1094 to 1095 in 1bb3bec

if (comment.text = comment.text.sub(/^# +:?method: *(\S*).*?\n/i, '')) && !!$~ then

line_no += $`.count("\n")

pre_process.rb parses comment directives, parse_comment also parses some comment directives.
To avoid re-implementing these kind of thing, I changed PreProcess#handle to also provide line_number for each directive.

This is not the best way. pre_process has problem and I think it should be fixed in the future. pre_process needs code_object already created, but we need to parse the comment first to identify code_object type which can be either attribute or method.

st0012 · 2024-07-27T11:45:21Z

lib/rdoc/parser/prism_ruby.rb

+#   def regular_method() end
+#
+# Note that by default, the :method: directive will be ignored if there is a
+# standard rdocable item following it.


How much do these comments still hold?

All of them, I think. And we should.
I removed these comments and add a short description instead because it is a duplication of lib/rdoc/parser/ruby.rb's comment. We can restore it to this file when removing lib/rdoc/parser/ruby.rb

st0012 · 2024-07-27T12:01:28Z

lib/rdoc/parser/prism_ruby.rb

+      @yields = []
+      @calls_super = false
+      @params = "(#{def_node.parameters&.slice})"
+      def_node.body&.accept(self)


I think for readability visitors should NOT invoke itself, so we should move this out of initialize and do this in visit_def_node instead

if node.body signature = MethodSignature.new(node) node.body.accept(signature) end

lib/rdoc/parser/prism_ruby.rb

rdoc.gemspec

lib/rdoc/parser/prism_ruby.rb

…s_super

st0012 · 2024-07-28T22:10:56Z

lib/rdoc/parser/prism_ruby.rb

+  end
+
+  class RDocVisitor < Prism::Visitor # :nodoc:
+    DSL = {


Can we make these visit_call methods more easily distinguishable from the visitor's external interface? I think adding _ as prefix + moving them to the private section should be enough.

I added _ 👍
Visibility is still public because the code below is calling method through v. so we can't change the visibility to private.

-> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'R') }

Of course we can use instance_eval or instance_exec to remove v. part, but I think not using instance_eval/exec is more straightforward.

I think expanding these conditions in visit_call_node doesn't take much space but improves the readability quite significantly:

diff --git a/lib/rdoc/parser/prism_ruby.rb b/lib/rdoc/parser/prism_ruby.rb index 5ea596b4..c49cafbc 100644 --- a/lib/rdoc/parser/prism_ruby.rb +++ b/lib/rdoc/parser/prism_ruby.rb @@ -670,25 +670,6 @@ class RDoc::Parser::PrismRuby < RDoc::Parser end class RDocVisitor < Prism::Visitor # :nodoc: - DSL = { - attr: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'R') }, - attr_reader: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'R') }, - attr_writer: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'W') }, - attr_accessor: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'RW') }, - include: -> (v, call_node) { v._visit_call_include(call_node) }, - extend: -> (v, call_node) { v._visit_call_extend(call_node) }, - public: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :public, &block) }, - private: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :private, &block) }, - protected: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :protected, &block) }, - private_constant: -> (v, call_node) { v._visit_call_private_constant(call_node) }, - public_constant: -> (v, call_node) { v._visit_call_public_constant(call_node) }, - require: -> (v, call_node) { v._visit_call_require(call_node) }, - alias_method: -> (v, call_node) { v._visit_call_alias_method(call_node) }, - module_function: -> (v, call_node, &block) { v._visit_call_module_function(call_node, &block) }, - public_class_method: -> (v, call_node, &block) { v._visit_call_public_private_class_method(call_node, :public, &block) }, - private_class_method: -> (v, call_node, &block) { v._visit_call_public_private_class_method(call_node, :private, &block) }, - } - def initialize(scanner, top_level, store) @scanner = scanner @top_level = top_level @@ -697,8 +678,43 @@ class RDoc::Parser::PrismRuby < RDoc::Parser def visit_call_node(node) @scanner.process_comments_until(node.location.start_line - 1) - if node.receiver.nil? && (dsl_proc = DSL[node.name]) - dsl_proc.call(self, node) { super } + if node.receiver.nil? + case node.name + when :attr, :attr_reader + _visit_call_attr_reader_writer_accessor(node, 'R') + when :attr_writer + _visit_call_attr_reader_writer_accessor(node, 'W') + when :attr_accessor + _visit_call_attr_reader_writer_accessor(node, 'RW') + when :include + _visit_call_include(node) + when :extend + _visit_call_extend(node) + when :public, :private, :protected + _visit_call_public_private_protected(node, node.name) do + super + end + when :private_constant + _visit_call_private_constant(node) + when :public_constant + _visit_call_public_constant(node) + when :require + _visit_call_require(node) + when :alias_method + _visit_call_alias_method(node) + when :module_function + _visit_call_module_function(node) do + super + end + when :public_class_method + _visit_call_public_private_class_method(node, :public) do + super + end + when :private_class_method + _visit_call_public_private_class_method(node, :private) do + super + end + end else super end

This way the reader can immediately know what special calls we handle without jumping to DSL, and the relationships between them will also be clearer.

Seems good and this way we can make it private!
done

st0012 · 2024-07-28T22:16:16Z

lib/rdoc/parser/prism_ruby.rb

+    # 4|
+    # 5|   def f; end # comment linked to this line
+    # 6| end
+    @unprocessed_comments = consecutive_comments.reject(&:empty?).map do |comments|


We could avoid allocating multiple arrays by changing this to:

Suggested change

@unprocessed_comments = consecutive_comments.reject(&:empty?).map do |comments|

consecutive_comments.reject!(&:empty?)

@unprocessed_comments = consecutive_comments.map! do |comments|

I normally consider this micro-optimization. But given projects could have long consecutive comments, I feel this may help in some cases.

lib/rdoc/parser/prism_ruby.rb

…isit_[NODE_TYPE]_node

…t method

st0012

I think we're very close to merging this 👍

lib/rdoc/parser/prism_ruby.rb

st0012 · 2024-07-30T13:49:03Z

lib/rdoc/parser/prism_ruby.rb

+  end
+
+  class RDocVisitor < Prism::Visitor # :nodoc:
+    DSL = {


I think expanding these conditions in visit_call_node doesn't take much space but improves the readability quite significantly:

diff --git a/lib/rdoc/parser/prism_ruby.rb b/lib/rdoc/parser/prism_ruby.rb index 5ea596b4..c49cafbc 100644 --- a/lib/rdoc/parser/prism_ruby.rb +++ b/lib/rdoc/parser/prism_ruby.rb @@ -670,25 +670,6 @@ class RDoc::Parser::PrismRuby < RDoc::Parser end class RDocVisitor < Prism::Visitor # :nodoc: - DSL = { - attr: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'R') }, - attr_reader: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'R') }, - attr_writer: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'W') }, - attr_accessor: -> (v, call_node) { v._visit_call_attr_reader_writer_accessor(call_node, 'RW') }, - include: -> (v, call_node) { v._visit_call_include(call_node) }, - extend: -> (v, call_node) { v._visit_call_extend(call_node) }, - public: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :public, &block) }, - private: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :private, &block) }, - protected: -> (v, call_node, &block) { v._visit_call_public_private_protected(call_node, :protected, &block) }, - private_constant: -> (v, call_node) { v._visit_call_private_constant(call_node) }, - public_constant: -> (v, call_node) { v._visit_call_public_constant(call_node) }, - require: -> (v, call_node) { v._visit_call_require(call_node) }, - alias_method: -> (v, call_node) { v._visit_call_alias_method(call_node) }, - module_function: -> (v, call_node, &block) { v._visit_call_module_function(call_node, &block) }, - public_class_method: -> (v, call_node, &block) { v._visit_call_public_private_class_method(call_node, :public, &block) }, - private_class_method: -> (v, call_node, &block) { v._visit_call_public_private_class_method(call_node, :private, &block) }, - } - def initialize(scanner, top_level, store) @scanner = scanner @top_level = top_level @@ -697,8 +678,43 @@ class RDoc::Parser::PrismRuby < RDoc::Parser def visit_call_node(node) @scanner.process_comments_until(node.location.start_line - 1) - if node.receiver.nil? && (dsl_proc = DSL[node.name]) - dsl_proc.call(self, node) { super } + if node.receiver.nil? + case node.name + when :attr, :attr_reader + _visit_call_attr_reader_writer_accessor(node, 'R') + when :attr_writer + _visit_call_attr_reader_writer_accessor(node, 'W') + when :attr_accessor + _visit_call_attr_reader_writer_accessor(node, 'RW') + when :include + _visit_call_include(node) + when :extend + _visit_call_extend(node) + when :public, :private, :protected + _visit_call_public_private_protected(node, node.name) do + super + end + when :private_constant + _visit_call_private_constant(node) + when :public_constant + _visit_call_public_constant(node) + when :require + _visit_call_require(node) + when :alias_method + _visit_call_alias_method(node) + when :module_function + _visit_call_module_function(node) do + super + end + when :public_class_method + _visit_call_public_private_class_method(node, :public) do + super + end + when :private_class_method + _visit_call_public_private_class_method(node, :private) do + super + end + end else super end

This way the reader can immediately know what special calls we handle without jumping to DSL, and the relationships between them will also be clearer.

colby-swandale · 2024-07-31T14:23:37Z

test/rdoc/test_rdoc_parser_ruby.rb

@@ -2,6 +2,8 @@

 require_relative 'helper'

+return if ENV['RDOC_USE_PRISM_PARSER']


I would like to see us running the test for prism and classic ruby parser without requiring these environment variables. This would keep our CI relatively simple and would ensure we're catching any bugs early.

I was planning to update CI setups myself after the PR is merged as it's easier to test those with write access.

The added test test_rdoc_parser_prism_ruby.rb already runs with both RDoc::Parser::Ruby and RDoc::Parser::PrismRuby on CI.
I don't have a good idea to run other test with RDoc::Parser::Ruby without using this environment variable.

st0012

Thanks for the impressive work!
I'll merge this PR now and add a temporary CI step for it.
Let's start refactoring parsing process to make this new parser less coupled with legacy components before replacing the old one completely.

(ruby/rdoc#1144) * Add a new ruby parser RDoc::Parser::PrismRuby * Add a new ruby parser testcase independent from parser's internal implementation * unknown meta method * Use MethodSignatureVisitor only to scan params, block_params and calls_super * Add calls_super test * Drop ruby 2.6. Prism requires ruby >= 2.7 * Remove duplicated documentation comment from prism_ruby.rb * Add test for wrong argument passed to metaprogramming method * Rename visit_call_[DSL_METHOD_NAME] to make it distinguishable from visit_[NODE_TYPE]_node * Method receiver switch of true/false/nil to a case statement * Extract common part of add_method(by def keyword) and add meta_comment method * Reuse consecutive comments array when collecting comments * Simplify DSL call_node handling * Refactor extracting method visibility arguments ruby/rdoc@fde99f1be6

hsbt · 2024-08-01T08:42:07Z

@st0012 @tompng https://github.com/ruby/rdoc/blob/master/rdoc.gemspec#L233 and https://github.com/ruby/rdoc/blob/master/rdoc.gemspec#L236 are bit of aggressive changes.

The release model of rdoc is branch cut from master HEAD. Now, If we have a plan to release 6.7.1, that teeny release contains required_ruby_version change and add C ext dependency. So, gem i rdoc -v 6.7.1 needs C compiler. It's not useful.

We should keep prism is optional in 6.7.x at least.

st0012 · 2024-08-01T12:13:53Z

After discussing with other maintainers, we agreed to hold off making prism an official dependency of RDoc until 2025. We decided to be more conservative on this because no other Ruby default gems have used it as a dependency and the gem also hasn't reached 1.0 release yet.
Also, in 2025 with Ruby 3.5's development, RDoc will become a bundled gem, which should make such change easier too.

That being said, I think it's still great that we have this PR merged as we can use the Prism parser to evaluate all the upcoming refactorings 👍

kddnewton reviewed Jul 22, 2024

View reviewed changes

tompng force-pushed the ruby_parser_prism branch from 24df3db to 1fef14c Compare July 22, 2024 14:48

st0012 reviewed Jul 22, 2024

View reviewed changes

tompng added 2 commits July 24, 2024 02:47

Add a new ruby parser RDoc::Parser::PrismRuby

61e7613

Add a new ruby parser testcase independent from parser's internal imp…

76d7786

…lementation

tompng force-pushed the ruby_parser_prism branch from 1fef14c to 76d7786 Compare July 23, 2024 18:28

tompng changed the title ~~Rewrite rdoc/parser/ruby.rb using Prism~~ Add new ruby parser rdoc/parser/prism_ruby.rb that uses Prism Jul 23, 2024

tompng changed the title ~~Add new ruby parser rdoc/parser/prism_ruby.rb that uses Prism~~ Add new ruby parser that uses Prism Jul 23, 2024

st0012 reviewed Jul 27, 2024

View reviewed changes

tompng added 6 commits July 29, 2024 02:00

unknown meta method

021d547

Use MethodSignatureVisitor only to scan params, block_params and call…

c44bd1f

…s_super

Add calls_super test

83b0fde

Drop ruby 2.6. Prism requires ruby >= 2.7

aca20e1

Remove duplicated documentation comment from prism_ruby.rb

ec8ba9e

Add test for wrong argument passed to metaprogramming method

6531724

st0012 reviewed Jul 28, 2024

View reviewed changes

tompng added 4 commits July 29, 2024 20:04

Rename visit_call_[DSL_METHOD_NAME] to make it distinguishable from v…

f8a7c8f

…isit_[NODE_TYPE]_node

Method receiver switch of true/false/nil to a case statement

c6341ec

Extract common part of add_method(by def keyword) and add meta_commen…

00e164d

…t method

Reuse consecutive comments array when collecting comments

24ce586

st0012 reviewed Jul 30, 2024

View reviewed changes

tompng added 2 commits July 31, 2024 13:38

Simplify DSL call_node handling

0a8634c

Refactor extracting method visibility arguments

1ce2634

colby-swandale reviewed Jul 31, 2024

View reviewed changes

st0012 approved these changes Jul 31, 2024

View reviewed changes

st0012 merged commit fde99f1 into ruby:master Jul 31, 2024
23 checks passed

tompng deleted the ruby_parser_prism branch August 1, 2024 07:06

st0012 added this to the v6.8.0 milestone Oct 19, 2024

st0012 added the enhancement label Oct 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new ruby parser that uses Prism #1144

Add new ruby parser that uses Prism #1144

tompng commented Jul 22, 2024 •

edited

Loading

kddnewton Jul 22, 2024

tompng Jul 22, 2024

kddnewton Jul 22, 2024

st0012 Jul 22, 2024

tompng Jul 23, 2024

hsbt commented Jul 24, 2024

kddnewton commented Jul 24, 2024

st0012 commented Jul 24, 2024

st0012 Jul 27, 2024

tompng Jul 28, 2024

st0012 Jul 27, 2024

tompng Jul 28, 2024 •

edited

Loading

st0012 Jul 27, 2024

st0012 Jul 28, 2024

tompng Jul 29, 2024

st0012 Jul 30, 2024

tompng Jul 31, 2024

st0012 Jul 28, 2024 •

edited

Loading

st0012 left a comment

st0012 Jul 30, 2024

colby-swandale Jul 31, 2024

st0012 Jul 31, 2024

tompng Jul 31, 2024

st0012 left a comment

hsbt commented Aug 1, 2024

st0012 commented Aug 1, 2024

		@@ -145,21 +136,8 @@ class RDoc::Parser::Ruby < RDoc::Parser

		parse_files_matching(/\.rbw?$/)

	if (comment.text = comment.text.sub(/^# +:?method: (\S).*?\n/i, '')) && !!$~ then
	line_no += $`.count("\n")

	@unprocessed_comments = consecutive_comments.reject(&:empty?).map do \|comments\|
	consecutive_comments.reject!(&:empty?)
	@unprocessed_comments = consecutive_comments.map! do \|comments\|

		@@ -2,6 +2,8 @@

		require_relative 'helper'

		return if ENV['RDOC_USE_PRISM_PARSER']

Add new ruby parser that uses Prism #1144

Add new ruby parser that uses Prism #1144

Conversation

tompng commented Jul 22, 2024 • edited Loading

Background

Implementation details

Comments

Consecutive comment

Modifier comment

AST Traversing

Test

Output HTML file differences

Removed files, these are not linked from any another html files.

Link in html file changed

Other changes

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hsbt commented Jul 24, 2024

kddnewton commented Jul 24, 2024

st0012 commented Jul 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tompng Jul 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

st0012 Jul 28, 2024 • edited Loading

Choose a reason for hiding this comment

st0012 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

st0012 left a comment

Choose a reason for hiding this comment

hsbt commented Aug 1, 2024

st0012 commented Aug 1, 2024

tompng commented Jul 22, 2024 •

edited

Loading

tompng Jul 28, 2024 •

edited

Loading

st0012 Jul 28, 2024 •

edited

Loading