Backport latest_specs api for private gems to the 1.0 stable branch

[smellsblue]

I also pulled in #113 as it was used by #131, and it is just adding some more tests, so should be fine to pull in to the 1.0 branch.

[smellsblue]

@randycoulman would you mind reviewing my backport of your PR? I had to cherry pick another branch because of some conflicts, but it was purely spec code, so I think it should be fine to pull in. I also had to tweak the code a tad due to some refactorings that have happened in master.

I implemented the gem install integration spec, however when I put the original implementation for serve_latest_specs, the test passed... so I'm not quite sure how to replicate the failures you had seen before with installing private gems. I temporarily had Gemstash output every path that was requested, and it used the /api/v1/dependencies endpoint rather than the /latest_specs.4.8.gz endpoint. Perhaps I wasn't invoking the command the way you were, or perhaps you had some config somewhere that might have changed rubygems' behavior? The integration spec you had already implemented failed with the old implementation, so I'm comfortable keeping it as is, though it wouldn't hurt to add other uses of /latest_specs.4.8.gz.

Thanks again for your work! I'm hoping to get a new release with your code out within the week.

[randycoulman]

@smellsblue Your backport looks good to me. I'll run some experiments with gem install on my setup and report back with the results.

[randycoulman]

@smellsblue I can no longer reproduce the issue with gem install in my setup. I know I reproduced it initially, but now it's behaving differently. I'm not sure what might have changed.

I've asked the person who originally reported the problem to me to try again; I'll let you know if his results are any different.

[smellsblue]

@randycoulman awesome! Thanks a bunch! Regardless of the result, I think this will be great to get out ASAP. I'll wait to push a new release until I hear from you though, as I'd like to include any additional tests that might be useful beforehand :-)

[randycoulman]

@smellsblue My colleague can still reproduce the issue with gem install. He's using the same version of gem as I am, but there are some minor differences in our .gemrc files. As soon as I'm able, I'll make my configuration the same as his and see what happens.

[randycoulman]

@smellsblue OK, I figured out what was going on.

On my system, I had a fallback entry to our old gem server. gem was querying both servers at the same time, and it got a response from the old one before it had a chance to send the latest_specs4.8.gz request to the new one. I was running with --clear-sources --source <new gem server>, but it seemed to be ignoring those arguments.

I edited my .gemrc to take out the old server and I started getting the failures again.

I started duplicating what gem was doing using curl commands and I figured out the problem. I have my gem server configured as http://server:9292/private, but the gem install command was not including the /private part in its api requests (the gemstash log did not have any "Rewriting" entries in it.

I edited my .gemrc to have a trailing slash (so http://server:9292/private/) and gem install started working.

All of the documentation I can see says not to use a trailing slash, but gem install wouldn't work until I added it in. It looks like this is a bug in gem and not gemstash.

I also ran gem search, and it also strips the /private part unless there is a trailing slash.

This is all with gem version 2.6.8, BTW.

[smellsblue]

@randycoulman thanks for the great investigation! And I have actually run into this exact same problem, and even found the part of rubygems code that was doing it (oops... I probably should have opened a ticket for them...). I have a test skipped because of this exact same issue: https://github.com/bundler/gemstash/blob/master/spec/integration_spec.rb#L140-L147

[smellsblue]

@randycoulman This has been released with 1.0.4 😄

[randycoulman]

Awesome! Thanks @smellsblue! I'll test it out early next week.