[BUG] 429 Too Many Requests #836
Comments
|
Same here, but with |
|
#805 similar isseue - random 403 or 429 |
|
Having same issue on our pipelines. Responses vary between |
|
We see this in any of our CI tasks running in AWS |
|
Also for me on bamboo build: |
|
Centralized infrastructure :~( |
|
It'd be useful to have a list of (verified) public registry mirrors. I found some but I can't trust them. |
|
Same, both locally and on Circle CI |
|
Also seeing the same using Circle CI and locally
|
|
I'm seeing errors like.. "The owner of this website (registry.npmjs.org) has banned you temporarily from accessing this website" and "You are being rate limited" I'm guessing this is all related? |
|
We are also having this issue when deploying on Heroku. |
|
Having the same errors when deploying on heroku. |
|
same here with AWS CodeBuild and npm i -g aws-cdk
general server issue? |
|
I also have same problem |
|
Same here when installing packages locally. |
|
Yep, I'm seeing this on Travis too for |
|
Same thing happening over here. Getting the error when doing |
|
Same issue here. We are using bamboo ci. Own installation. |
|
Facing this issue too, is this a global thing or maybe region related? We just had something similar last year in Germany. |
|
Same here running on Gitlab CI |
|
Same here in The Netherlands. (AWS Codebuild from Ireland) |
|
Russia to |
|
Istanbul here |
|
This appears to be a Cloudflare related issue to the registry.npmjs.org site. got the following html response on update: |
|
Same issue happening with AWS Codebuild us-east-1. Was broken locally up til about 30 minutes ago but back working now (locally from Ireland) |
Is there any mirror not using cloudflare? |
|
Same problem! Build pipelines are failing :( |
|
Same: |
|
That's it. Internet is done. Good bye everyone. |
|
|
|
@aeleuterio Any chance we can get a post-mortem on this? |
|
Working now! |
|
It's not working again. I guess, issue has not been fixed by your partners. |
Memes and jokes drown out the conversation that may be vital to fixing whatever the issue actually is, though. |
|
Working now 😓 |
Indeed, I had to scroll through about 200 memes just to see the actual status update from NPM. |
|
Just for your info: yarn was able to load the packages without triggering the rate limit :1) yarn is great! (and safe) |
|
Hello and profuse apologies from Cloudflare, a post-mortem of sorts directly in your issue comments. I am the engineering manager for the DDoS protection team and this morning at 11:06 UTC we tweaked a rule that affected one of our signals. The signal relates to the HTTP referer header, and we have a piece of code that looks at invalid referer headers. In this case we tweaked it to include not just "obvious garbage" but "anything that does not conform to the HTTP specification"... i.e. is the referer a URI? If not then it contributes to knowledge about bad traffic. So... why did this impact npmjs.org? It turns out that a lot of NPM traffic sends the referer as "install" which is invalid according to the HTTP specification. As NPM is also a heavily trafficked site this resulted in the DDoS systems picking this up and treating the traffic as a HTTP flood and determining that a rate-limit should be applied. When we noticed that NPM was seeing an increase in HTTP 429s (as seen on Twitter) we contacted NPM and started an internal investigation. As soon as we identified the root cause we reverted the change, which was at 13:00 UTC. We'll note that NPM and 1 other site use the referer for purposes outside the HTTP spec and we'll update our systems to ensure that this does not happen again. Additionally we'll improve our monitoring around changes of this nature so that we can discover impact sooner and roll back automatically. |
|
Thanks for the explaination @buro9 Hopefully you will have some explicit tests for NPM, given the importance to the developer community. We (and many others I'm sure) were unable to deploy a number of projects for 2 hours this morning, during EU business hours. This should also serve as a reminder for us all to have better continuity measures in place for when these events happen. |
|
In my opinion, it would be best to make sure that the requests from the NPM installer comply with the HTTP specification. |
Referer should be blank, installer should be user agent |
|
Thanks, I am able to download all my 5464950 dependencies every 15min for another build again. |
|
@buro9 we would appreciate it if you respond to our tickets and our internal slack comms, before posting to a public issue, we still have not gotten a post-mortem report for the last two outages. as for pointing at HTTP specifications, considering this behaviour has been in place for years, I would ask to review what change was pushed in CF today that caused this sudden "compliance with HTTP Specification" result? I'll ask again to please follow up with our open tickets and report back to to us on the post-mortem for the last two outages, we'd rather learn about this from you directly, than seeing it in an issue on github .. |
I don't think you have to apologize. npm obviously messed up the referrer field and you did nothing wrong. Just because it accidentally worked out in the past doesn't mean it should stay that way. Who shall guarantee that something like this doesn't happen again in the future, because someone is respecting the spec? |
That's called BC breaks and should not occur in the same 'version'. |
Yeah I give you that point. But hopefully the decision will not be "it stays forever that way and everyone must comply". |
Doesn't the
Without probing the server for existence of the URI, how to you distinguish arbitrary, urlencoded text from an actual partial-URI? If I follow down the deductions of the specification then an https://tools.ietf.org/html/rfc7230#section-2.7
https://tools.ietf.org/html/rfc3986#section-4.2
https://tools.ietf.org/html/rfc3986#section-3.3
So if I'm not terribly misreading the specification, I have to wonder: Why the heck has that referrer header been treated as invalid? Does Cloudflare check the existence of that URI upon request (and then caches it), or is this yet again a shoot fast-and-loose "whoopsie"? |
|
@datenwolf syntactically it should be possible to have a partial URI like This imho applies here. This doesn't, however, answer your last question. But it seems that the topic is a bit more complex and splits into two parts: 1) is npm conforming to the http spec, and 2) does CF use reliable detection rules. And maybe, just maybe, the answer to both questions is: no. But I leave this to others to discuss. I just wanted to point out that prematurely apologizing for "something" may leave potential bugs unfixed, so the wording might have been a bit unfortunate towards a real "solution" to the problem - whatever it is. |
|
Instead of implementing |
Yes, it does. Maybe I've not made it clear enough, that I'm fully aware of this. But that's entirely besides the point. As you've already pointed out
and I already pointed out, that without an explicit check for the existence of the Does npm violate the specification? Sure. Can CF correctly detect that? Only by performing an explicit check of the URI. Does CF perform that check? I don't know… yet (but I might set up a testing ground, just for that). However, without further information I just have to assume that CF again did as CF does and for ill-founded reasons broke a part of the Internet… yet again. |
|
It's worth keeping in mind in all this that there's very little that a CDN can do which will meaningfully improve their service and which does not carry the risk of "breaking a part of the internet". The nature of the service that CF provides, and their well-deserved popularity as one to provide that service, means that they are basically always playing with fire, and likely to upset quite a lot of people when they make otherwise forgivable and well-intended mistakes. Having done this npm thing for quite some time now, I have no shortage of sympathy for their position, and I believe that it is inappropriate to heap scorn on them for this. They're doing a great job, protecting npm (and thus, the whole JS community) from a lot of bad actors and outages, and making all of our builds much faster and more reliable. We love and appreciate Cloudflare. So let's be kind here. That being said, I do believe that npm is neither in violation of the letter nor the spirit of the relevant HTTP specifications in our use of the Referer header to indicate the command that caused a given request to be made. I hope that anyone following this discussion find the following flood of HTTP pedantry useful or at least enjoyable. If that's not your kind of thing, please go do something else, you won't have a good time reading this :) CF deployed a change that treated unusual use of HTTP headers as a heuristic to be marked as a malicious request. When dealing primarily with traffic from web browsers, the Referer header will generally always be either missing, However: The specification says "URI". It does not say "URL". It certainly does not say "Fully qualified URL". Given the typically meticulous usage of "URI" vs "URL" in IETF documentation, the hair-splitting discussions that often ensue over matters like this, and the fact that Referer first appeared in rfc1945 (though it was in use long before then), and that the HTTP specs have kept "URI" through multiple rounds of updates and revisions, I have to conclude that the intent here is for Referer to be a URI rather than required to be a URL per se, as well as the letter of the spec. A URI and a URL are not the same thing. Both the linked RFCs have been updated and obsoleted (in part) since their inception by subsequent RFCs, and I strongly encourage anyone still reading at this point to follow the links and learn about how Uniform Resource Location and Identification standards have changed and expanded in subtle and interesting ways over the years. The bottom line is this: the HTTP A URI explicitly does not need to be locatable, fetchable, or resolveable by any given network agent, or over any given protocol. So, As URI semantics and syntax are defined by their scheme, it is impossible, in absence of a scheme, to say that That is exactly what happens with the npm client and the npm registry. It sends a Referer header containing the command being run. (When the command contains positional arguments, anything containing And just to reiterate, I certainly don't think Cloudflare is a bad actor here, and I'm disappointed to see how quick so many people have been to "pick sides" as if it's npm vs Cloudflare in a battle over this. We were affected by a mistake they made, but we're sometimes going to be affected by mistakes that they make, because we're their customer, and of course they're going to make mistakes from time to time, because humans and machines aren't perfect. That's just how the world is. By and large, we're very happy with the response we got, and we've all improved our monitoring and response systems in light of this hiccup. |
|
FWIW, the "referer" is not defined as URI. See the spec: https://greenbytes.de/tech/webdav/rfc7231.html#rfc.section.5.5.2. It's a URI reference. "install" would be interpreted as path, relative to the request URI. |
|
@reschke Even on that reading, it's still perfectly valid. That is: from Even if I'm reading the spec more broadly than intended, it's certainly not a "violation" of the spec to use Referer in this way, and it's a (completely forgivable!) mistaken overreach to block or rate-limit requests that include Referer headers that are not fully-qualified URLs. In light of this, however, it probably would be worthwhile to put a scheme on the Referer headers that the npm cli sends. We'd have to research this to see if that makes it more or less likely that requests will be mangled by proxies. Another option, of course, is to accept that some proxies will just be overeager in filtering resulting in slightly less ideal data, but use a custom header with a more definitive meaning, like |
and others
What / Why
When
npm ci(since today at least)Where
Current Behavior
npm cicommand returns E429 error (Too Many Requests) and doesn't complete packages installationSteps to Reproduce
npm ciExpected Behavior
The text was updated successfully, but these errors were encountered: