HPHosts Collector Error

[mf370]

I configured the collector for the feed HPHosts with the following settings:

"hphosts-collector": {
        "description": "Generic URL Fetcher is the bot responsible to get the report from an URL.",
        "group": "Collector",
        "module": "intelmq.bots.collectors.http.collector_http",
        "name": "Generic URL Fetcher",
        "parameters": {
            "error_log_message": false,
            "feed": "HPHosts",
            "http_password": null,
            "http_proxy": null,
            "http_url": "http://hosts-file.net/download/hosts.txt",
            "http_username": null,
            "https_proxy": null,
            "provider": "HPHosts",
            "rate_limit": 3600,
            "ssl_client_certificate": null
        }
    }

I think it is well configured, however everytime I run it, it throws the following error:

2017-06-26 14:41:11,214 - hphosts-collector - INFO - Downloading report from http://hosts-file.net/download/hosts.txt
2017-06-26 14:41:11,519 - hphosts-collector - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/intelmq/lib/bot.py", line 145, in start
    self.process()
  File "/usr/lib/python3.4/site-packages/intelmq/bots/collectors/http/collector_http.py", line 54, in process
    ''.format(resp.status_code))
ValueError: HTTP response status code was 999.
2017-06-26 14:41:11,520 - hphosts-collector - INFO - Bot will continue in 15 seconds.

Does anyone knows how to solve this?

[ghost]

Status code 999 is interesting. Is it reproducible? Did you check the proxy settings? Can you get the page with other tools from this host (e.g. curl, wget)?

Please consider using the intelmq-users list for support, not the bug tracker. Thanks!

[mf370]

Yes, I can download the report using curl and wget. I have used the same bot configuration in different operating systems and they all return the same error.

Please consider using the intelmq-users list for support, not the bug tracker. Thanks!

Alright!

[ghost]

Some sources imply that this means Request denied and link it with scraping, i.e. lots of requests. Beside querying less often they suggest setting a user agent.

[navtej]

"http_password": null, is the problem. The UI automatically adds it. HPHosts firewall rules get triggered because of empty auth header and it generates 999. @wagner-certat perhaps it should be fixed on UI side, if no auth is supplied dont add it to the json. Alternatively, http collector can be modified to fix it.

For a temp workaround, remove it manually by editing /opt/intelmq/etc/runtime.conf

[ghost]

Either a bug in requests or at hosts-file.de:

>>> r = requests.get('http://hosts-file.net/download/hosts.txt')
>>> r.status_code
200
>>> r = requests.get('http://hosts-file.net/download/hosts.txt', auth=None)
>>> r.status_code
200
>>> r = requests.get('http://hosts-file.net/download/hosts.txt', auth=(None, None))
>>> r.status_code
999
>>> r = requests.get('http://example.com/', auth=(None, None))
>>> r.status_code
200

As workaround we can check if http_username and http_password are not null in Bot.set_request_parameters.

[navtej]

>>> r = requests.get('http://hosts-file.net/download/hosts.txt', auth=(None, None))
>> r.status_code
999

It is hosts-file.de firewall rule which gets triggered, aggressive yes but not a bug.

[ghost]

When using auth=(None, None) this results in this header:

Authorization: Basic Tm9uZTpOb25l

And Tm9uZTpOb25l is None:None.
Fixed in 1e75ef0