Expose returned status code in verbose / Rate Limit inference engine feature

[neilwillgettoit]

It would be nice to expose the returned status code while scraping is underway with the -v flag to spot for rate limits being triggered.

Secondly, I would enjoy having a rate limit inference engine as part of katana to spot for when I am too aggressively scraping/crawling a site and have the rate limit auto adjust down similar to the drop packets inference in nmap. It would also be a feature to just pass a flag of ‘discover rate limit’ where it purposely tries to trigger a 429 code back, and then tests at a fuzzy logic interval to determine what the rate limit rules are. This way I could test for a rate limit from a disposable IP/source and then perform my tuned low and slow scrape/crawl from an IP I would rather not burn.