Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it loose coupling between RubyGems and RDoc #1171

Open
wants to merge 16 commits into
base: master
Choose a base branch
from

Conversation

mterada1228
Copy link
Contributor

@mterada1228 mterada1228 commented Sep 5, 2024

Problems

There are following problems because of tight coupling between RubyGems and RDoc.

  1. If there are braking changes in RDoc, RubyGems is also broken.
  2. When we maintain RDoc, we have to change RubyGems.

The reason why they are happened is that RubyGems creates documents about a gem with installing it.

Note that RubyGems uses functions of RDoc to create documents. Specifically,

  • Creating documents is executed by rubygems/lib/rubygems/rdoc.rb.
  • ::RDoc::RubygemsHook which is defined by RDoc is called by the file.

Solution

RubyGems has the plugin system.

If a gem includes rubygems_plugin.rb, RubyGems loads it. RubyGems executes a process defined in it while installing gems, uninstalling gems or other events.

We can use the system to solve the problems.

The root cause is RubyGems directly references the class of RDoc.

We can remove the root cause by making RDoc RubyGems plugin.

Alternatively rubygems_plugin.rb creates documents about gems.

FAQ

Q1. Do we need to change codes of RubyGems?

A.

No, we don't.

This change keeps compatibility of API used from RubyGems.

Q2. Is it better to delete existing codes related to RDoc in RubyGems?

No, it isn't.

If we change codes of RubyGems,
we can't keep a compatibility.

Example:

If we delete codes that uses RDoc::RubygemsHook in rubygems/lib/rubygems/rdoc.rb, documentations are not created with old RDoc.

Q3. When can we delete rubygems/lib/rubygems/rdoc.rb?

A.

We can delete it when all users use RDoc including rubygems_plugin.

Default gems and old RDoc just don't use rubygems_plugins.
So, When following conditions are completed, we can delete rubygems/lib/rubygems/rdoc.rb.

Q4. Is it a breaking change that Rubygems creates documents with rubygems_plugin not RDoc::RubygemsHook?

A.

No, it isn't.

If we simply implement this approach,
we move the implementation from rdoc/lib/rdoc/rubygems_hook.rb to rubygems_plugin.rb.

This way can be breaking change.

It seems to be fine that we just need to delete rdoc/rubygems_hook.rb but it doesn't work. It generates multiple documents.

rubygems/lib/rubygems/rdoc.rb has the following code.

begin
  require "rdoc/rubygems_hook"
  # ...
rescue LoadError
end

This code ignores RDoc related processes when rdoc/rubygems_hook can't be required. But, this 'require' is not failed.

This is because Ruby installs Rdoc as a default gem.

So, Rdoc installed as a default gem generates documents and one installed as a normal gem does it too.

If you think that this behavior is accectable,
we can just delete rdoc/rubygems_hook.rb.

What do you think about this approach?

In this change, we take another approach to solve the problem that creates multiple documents.

If Gem.done_installing(&Gem::RDoc.method(:generation_hook)) in rubygems/rdoc.rb doesn't create documents, we can solve the problem.

We have some options.

  • We change rubygems/rdoc.rb and then don't execute Gem.done_installing. (This is a change on RubyGems.)
  • We change rdoc/rubygems_hook.rb and then make generation_hook a method that create documents when there is not rubygems_plugin.rb. (This is a change on RDoc.)

We choose the latter to avoid changing on RubyGems.

Test

Preparation

Install Rdoc which including our changes by executing rake install.

❯ rake install

We confirmed that Rdoc which including our changes was installed.

❯ gem list | grep rdoc
rdoc (6.6.0, default: 6.4.0)

Check point

We tested to check compatibility.

How to chack the compatibility?

We tested creating same documents by our RDoc and old RDoc with latest RubyGems.

We used following versions to test.

❯ ruby -v
ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [arm64-darwin22]

❯ gem list | grep rdoc
rdoc (default: 6.4.0)

❯ ruby -I rubygems/lib rubygems/exe/gem --version
3.5.14

Here is a result of test with old RDoc.
We can see that the document is created correctlly with Parsing... and Done installing....

❯ ruby -I rubygems/lib rubygems/exe/gem install pkg-config
Successfully installed pkg-config-1.5.6
Parsing documentation for pkg-config-1.5.6
Done installing documentation for pkg-config after 0 seconds
1 gem installed

Here is a result of test with our RDoc.
We can see that the document is created correctlly with Parsing... and Done installing....

❯ ruby -I rubygems/lib rubygems/exe/gem install pkg-config
Successfully installed pkg-config-1.5.6
Parsing documentation for pkg-config-1.5.6
Done installing documentation for pkg-config after 0 seconds
1 gem installed

As you can see we got the same results, our RDoc keeps compatibility.

Test for RDoc as default gem

Setup

# For backupmv $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc_bkupmv $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb.bkup

# Copy sources to directory of default gemscp -r ./lib/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoccp -r ./lib/rdoc.rb $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb

# Checkgem list | grep rdoc
rdoc (default: 6.6.3.1)

Result

gem install pkg-config
Successfully installed pkg-config-1.5.7
Parsing documentation for pkg-config-1.5.7
Done installing documentation for pkg-config after 0 seconds
1 gem installed

\### Problems

There are following problems because of tight coupling between RubyGems and RDoc.

1. If there are braking changes in RDoc, RubyGems is also broken.
2. When we maintain RDoc, we have to change RubyGems.

The reason why they are happened is that RubyGems creates documents about a gem with installing it.

Note that RubyGems uses functions of RDoc to create documents.
Specifically,

- Creating documents is executed by `rubygems/lib/rubygems/rdoc.rb`.
- `::RDoc::RubygemsHook` which is defined by RDoc is called by the file.

\### Solution

RubyGems has the plugin system.

If a gem includes `rubygems_plugin.rb`, RubyGems loads it.
RubyGems executes a process defined in it while installing gems, uninstalling gems or other events.

We can use the system to solve the problems.

The root cause is RubyGems directly references the class of RDoc.

We can remove the root cause by making RDoc RubyGems plugin.

Alternatively `rubygems_plugin.rb` creates documents about gems.

\### FAQ

Q1. Do we need to change codes of RubyGems?

A.

No, we don't.

This change keeps compatibility of API used from RubyGems.

Q2. Is it better to delete existing codes related to RDoc in RubyGems?

No, it isn't.

If we change codes of RubyGems,
we can't keep a compatibility.

Example:

If we delete codes that uses `RDoc::RubygemsHook` in `rubygems/lib/rubygems/rdoc.rb`,
documentations are not created with old RDoc.

Q3. When can we delete `rubygems/lib/rubygems/rdoc.rb`?

A.

We can delete it when all users use RDoc including `rubygems_plugin`.

Next ruby version is 3.4.
If it includes the RDoc including `rubygems_plugin`,
we can delete `rubygems/lib/rubygems/rdoc.rb` after ruby 3.3 is EOL.

Q4. Is it a breaking change that Rubygems creates documents with
rubygems_plugin not RDoc::RubygemsHook?

A.

No, it isn't.

If we simply implement this approach,
we move the implementation from `rdoc/lib/rdoc/rubygems_hook.rb` to
`rubygems_plugin.rb`.

This way can be breaking change.

It seems to be fine that we just need to delete `rdoc/rubygems_hook.rb` but it doesn't work.
It generates multiple documents.

`rubygems/lib/rubygems/rdoc.rb` has the following code.

```
begin
  require "rdoc/rubygems_hook"
  # ...
rescue LoadError
end
```

This code ignores RDoc related processes when `rdoc/rubygems_hook` can't be required.
But, this 'require' is not failed.

This is because Ruby installs Rdoc as a default gem.

So, Rdoc installed as a default gem generates documents and one installed as a normal gem does it too.

If you think that this behavior is accectable,
we can just delete `rdoc/rubygems_hook.rb`.

What do you think about this approach?

In this change, we take another approach to solve the problem that creates multiple documents.

If `Gem.done_installing(&Gem::RDoc.method(:generation_hook))` in `rubygems/rdoc.rb` doesn't create documents,
we can solve the problem.

We have some options.

* We change `rubygems/rdoc.rb` and then don't execute `Gem.done_installing`.
  (This is a change for RubyGems.)
* We change `rdoc/rubygems_hook.rb` and then make `generation_hook` a no-op method.
  (This is a change for RDoc.)

We choose the latter to avoid changing for RubyGems.

\### Test

\#### Preparation

Install Rdoc which including our changes by executing `rake install`.

❯ rake install

We confirmed that Rdoc which including our changes was installed.

❯ gem list | grep rdoc
rdoc (6.6.0, default: 6.4.0)

\#### Check point

We tested to check compatibility.

How to chack the compatibility?

We tested creating same documents by our RDoc and old RDoc with latest RubyGems.

We used following versions to test.

```
❯ ruby -v
ruby 3.1.0p0 (2021-12-25 revision fb4df44d16) [arm64-darwin22]

❯ gem list | grep rdoc
rdoc (default: 6.4.0)

❯ ruby -I rubygems/lib rubygems/exe/gem --version
3.5.14
```

Here is a result of test with old RDoc.
We can see that the document is created correctlly with `Parsing...` and `Done installing...`.

```
❯ ruby -I rubygems/lib rubygems/exe/gem install pkg-config
Successfully installed pkg-config-1.5.6
Parsing documentation for pkg-config-1.5.6
Done installing documentation for pkg-config after 0 seconds
1 gem installed
```

Here is a result of test with our RDoc.
We can see that the document is created correctlly with `Parsing...` and `Done installing...`.

```
❯ ruby -I rubygems/lib rubygems/exe/gem install pkg-config
Successfully installed pkg-config-1.5.6
Parsing documentation for pkg-config-1.5.6
Done installing documentation for pkg-config after 0 seconds
1 gem installed
```

As you can see we got the same results, our RDoc keeps compatibility.
@mterada1228 mterada1228 marked this pull request as draft September 5, 2024 23:52
@mterada1228 mterada1228 marked this pull request as ready for review September 7, 2024 05:49
@kou
Copy link
Member

kou commented Sep 11, 2024

FYI: RubyGems wants to remove RDoc integration code if possible:

@st0012
Copy link
Member

st0012 commented Sep 12, 2024

Thank you @mterada1228 @kou. My current understanding is: this PR will allow RubyGems to remove RDoc specific code, because RDoc will start using RubyGems plugin (its official API) to generate documentation for gems. Is this correct?

@mterada1228
Copy link
Contributor Author

Thank you @mterada1228 @kou. My current understanding is: this PR will allow RubyGems to remove RDoc specific code, because RDoc will start using RubyGems plugin (its official API) to generate documentation for gems. Is this correct?

@st0012 Cc: @kou

Yes, it is. Your understanding is correct.

@kou
Copy link
Member

kou commented Sep 13, 2024

Correct as @mterada1228 said.

Note:
But we can't remove the RDoc integration code from RubyGems immediately as the PR description mentioned. Because:

  • We can't use the RubyGems plugin feature for a default gem
  • RDoc that doesn't have this RubyGems plugin support exist

RDoc will be a bundled gem (not a default gem) since Ruby 3.5: https://bugs.ruby-lang.org/issues/20309
We can remove the RDoc integration code from RubyGems when Ruby 3.4 reaches EOL.
We can assume that all maintained RDocs are installed as a normal gem and they have this RubyGems plugin support at the time.

@deivid-rodriguez
Copy link
Contributor

Hello! From RubyGems maintainers team perspective, I think this is a nice thought out PR, and I'm very happy it will allow us to eventually remove RDoc code from RubyGems ❤️. Thank you!

@kou
Copy link
Member

kou commented Oct 17, 2024

Are there people who want to review this PR?
If nobody wants to this, I'll merge this in this month. (I'll review this again before I merge this.)

@st0012 st0012 added this to the v6.8.0 milestone Oct 17, 2024
@hsbt
Copy link
Member

hsbt commented Oct 18, 2024

I'm +1 to this change. But I have some concerns. Is this working with fresh installation of ruby package?

In my understanding, the current rubygems plugin system could work with GEM_HOME/plugins/rubygems-generate_index_plugin.rb and GEM_HOME/plugins/rubygems-generate_index-1.1.3/lib/rubygems_plugin.rb with latest rubygems.

How generate GEM_HOME/plugins/rdoc_plugin.rb with fresh installation of Ruby 3.4?

@kou
Copy link
Member

kou commented Oct 18, 2024

Does "fresh installation" mean that RDoc is installed as a default gem?
This solution (plugin based RDoc generation) isn't used for the case. RDoc hook in RubyGems is used. It's the current approach.
This solution is only used when RDoc is installed as a bundled gem or a normal gem.

@hsbt
Copy link
Member

hsbt commented Oct 18, 2024

Does "fresh installation" mean that RDoc is installed as a default gem?

Yes.

This solution is only used when RDoc is installed as a bundled gem or a normal gem.

I see. Thanks. We make document generation is disabled with "RDoc is installed as a default gem" case like Ruby 3.4?

@kou
Copy link
Member

kou commented Oct 18, 2024

We make document generation is disabled with "RDoc is installed as a default gem" case like Ruby 3.4?

Does this mean the following?

If we ship this change in Ruby 3.4, whether gem install XXX generates documentation of XXX or not?

The answer is that gem install XXX still generates documentation of XXX. This doesn't change the current behavior. In the case, https://github.com/rubygems/rubygems/blob/465bfaf530c6712487495419e8ce9f5a0a0385e9/lib/rubygems/rdoc.rb#L11 generates documentation of XXX. (This solution isn't used.)

@hsbt
Copy link
Member

hsbt commented Oct 18, 2024

The answer is that gem install XXX still generates documentation of XXX.

Really? https://github.com/ruby/rdoc/pull/1171/files#diff-57670ce1156b5670b904c97f8a01d658a0b5a458ffe71f2fcc7a7eefd4254d2cR15 seems empty.

Could you explain how work lib/rubygems/rdoc.rb and this change without plugin installation?

@kou
Copy link
Member

kou commented Oct 18, 2024

Ah, wait. It may not work.
@mterada1228 Did we check the case?

@deivid-rodriguez
Copy link
Contributor

deivid-rodriguez commented Oct 21, 2024

@hsbt raised a good point. I think ruby-core default gem installer may need to make sure to generate plugins for default gems too, because as of now, no default gem includes rubygems plugins and I believe the mechanism is not going to work by default.

@mterada1228
Copy link
Contributor Author

mterada1228 commented Oct 25, 2024

@hsbt, @kou

I checked the behavior in the following case.
#1171 (comment)

When RDoc includes this changes was used as default gem,
documents weren't created as you say.

How to check

  1. Copy codes from this directory to the directory of default gems library.
mv $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc_bkup # for backupcp -r ./lib/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc
  1. Execute gem install.
gem install foo
Fetching foo.gem
Successfully installed foo
1 gem installed

Documents was not created.

Solutions

One solution is that If RDoc::RubygemsHook can find rubygems_plugin.rb RDoc creates documents by rubygems plugins, otherwise RDoc does so by current process.
Another solution is that I wait for Ruby 3.5 will make RDoc from default gems to bundled gems.

Although It is possible that I make default gems able to handle rubygems plugins, this approach is maybe too much.

@kou
Copy link
Member

kou commented Oct 26, 2024

Thanks for checking the case that RDoc is installed as a default gem.
It didn't work...

I agree that merging this after Ruby 3.4 is an option. We'll migrate RDoc to a bundled gem from a default gem for Ruby 3.5: https://bugs.ruby-lang.org/issues/20309
If RDoc is installed as a bundled gem, we don't need to care about the default gem case.

For merging this before Ruby 3.4, mterada1228#1 may work. (Sorry. I didn't test it yet. It's a concept implementation.)
The followings are important parts:

It's not complex. If this works well, this approach may be an option.
@mterada1228 Do you want to check whether this approach work with the following cases?

@mterada1228
Copy link
Contributor Author

@kou

Thank you for your PR.
I think that solution is so nice.

I want to try whether it may work or not, but the branch have to merge some fix in the master branch to try it.
Can you merge the branch of add-rubygems-hook ?

@kou
Copy link
Member

kou commented Nov 1, 2024

@mterada1228 You want to try mterada1228#1 on https://github.com/ruby/rdoc/tree/master , right?
If mterada1228#1 doesn't work as expected, we need to revert the change from https://github.com/ruby/rdoc/tree/master . So can we avoid the approach?

Can we try mterada1228#1 without merging it to https://github.com/ruby/rdoc/tree/master ?
It seems that you merged the current master to https://github.com/mterada1228/rdoc/tree/add-rubygems-hook . So I rebased mterada1228#1 on the current https://github.com/mterada1228/rdoc/tree/add-rubygems-hook .
Can we try mterada1228#1 by the following now?

$ git close https://github.com/kou/rdoc.git rdoc.kou
$ cd rdoc.kou
$ git switch add-rubygems-hook-default-gem
$ ...

@mterada1228
Copy link
Contributor Author

mterada1228 commented Nov 1, 2024

@kou

Sorry, My explanation was so bad...
What I wanted is same as one you say.

Thank you for your operation.
I'll test mterada1228#1 later.

kou and others added 3 commits November 7, 2024 06:07
Co-authored-by: mterada1228 <[email protected]>
Add support for the case that RDoc is installed as a default gem
@mterada1228
Copy link
Contributor Author

mterada1228 commented Nov 7, 2024

@hsbt @kou

I merged @kou's proposal mterada1228#1.
So, in case RDoc is installed as default gem, documents can be created.

Test

1. RDoc is installed as normal gem.

Setup

rake install
rdoc 6.7.0 built to pkg/rdoc-6.7.0.gem.
rdoc (6.7.0) installed.gem list | grep rdoc
rdoc (6.7.0, default: 6.6.3.1)

Execution

gem install pkg-config
Fetching pkg-config-1.5.7.gem
Successfully installed pkg-config-1.5.7
Parsing documentation for pkg-config-1.5.7
Installing ri documentation for pkg-config-1.5.7
Done installing documentation for pkg-config after 0 seconds
1 gem installed

2. RDoc is installed as default gem

Setup

# For backupmv $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc_bkupmv $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb.bkup

# Copy sources to directory of default gemscp -r ./lib/rdoc $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoccp -r ./lib/rdoc.rb $(ruby -e 'print Gem::RUBYGEMS_DIR')/rdoc.rb

# Checkgem list | grep rdoc
rdoc (default: 6.6.3.1)

Execution

gem install pkg-config
Successfully installed pkg-config-1.5.7
Parsing documentation for pkg-config-1.5.7
Done installing documentation for pkg-config after 0 seconds
1 gem installed

test/rdoc/test_rdoc_rubygems_hook.rb Outdated Show resolved Hide resolved
test/rdoc/test_rdoc_rubygems_hook.rb Outdated Show resolved Hide resolved
lib/rdoc/rubygems_hook.rb Outdated Show resolved Hide resolved
lib/rdoc/rubygems_hook.rb Outdated Show resolved Hide resolved
lib/rdoc/rubygems_hook.rb Outdated Show resolved Hide resolved
@kou
Copy link
Member

kou commented Nov 10, 2024

This is ready again.

This should be worked when RDoc is installed as a default gem too. In the case, the existing integration logic in RubyGems (not RubyGems plugin logic) is used.

Are there any other concerns?

@st0012 st0012 modified the milestones: v6.8.0, v7.0.0 Nov 12, 2024
# If generate and remove are executed at that time, an error will occur.
# So, we can't register generate and remove to Gem at that time.
begin
require_relative 'rdoc/markdown'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rdoc/rd/block_parser and rdoc/rd/inline_parser have similar problems to markdown too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right.
We can use one of rdoc/markdown, rdoc/rd/block_parser or rdoc/rd/inline_parser here.
This just wants to fix a bootstrap problem.

We want to run bundle install when we start developing cloned ruby/rdoc.
In the time, this lib/rubygems_plugin.rb is used. But RDoc isn't ready yet because rdoc/markdown.rb and so on aren't generated yet. So bundle install is failed.

This disables this lib/rubygems_plugin.rb only when the initial bundle install to fix the bootstrap problem.

# To install dependency libraries of RDoc, you need to run bundle install.
# At that time, rdoc/markdown is not generated.
# If generate and remove are executed at that time, an error will occur.
# So, we can't register generate and remove to Gem at that time.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand what this comment intends to express. Do you mind elaborate it a bit more?

@@ -0,0 +1,15 @@
# frozen_string_literal: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add comments to explain when and by what this file will be used?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added comment.
Can you check that?

@st0012
Copy link
Member

st0012 commented Nov 13, 2024

@mterada1228 can you also update the description to reflect the latest changes? Thanks 😄

@mterada1228
Copy link
Contributor Author

@st0012

@mterada1228 can you also update the description to reflect the latest changes? Thanks 😄

Thank you for your comment.

I changed a description of this PR.
Following points are summary of changes.

1
Q3. When can we delete rubygems/lib/rubygems/rdoc.rb?
-> I added a condition.

RDoc will be a bundled gem (not a default gem). (cf: https://bugs.ruby-lang.org/issues/20309)

2
Q4. Is it a breaking change that Rubygems creates documents with rubygems_plugin not RDoc::RubygemsHook?
-> I changed description about the approach we took.

We change rdoc/rubygems_hook.rb and then make generation_hook a method that create documents when there is not rubygems_plugin.rb. (This is a change on RDoc.)

3
I added a chapter of Test for RDoc as default gem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

5 participants