FileNotFoundError: [Errno 2] No such file or directory: 'default.txt'

[blshkv]

Hi, I have installed the tool as system module using provided setup.py. It installed .txt files into "/usr" prefix as below:

/usr/lists/20000.txt
/usr/lists/5000.txt
/usr/lists/default.txt

The tool is unable to find that location and fails with the following error:

bash$ fierce --domain facebook.com --wide
NS: b.ns.facebook.com. a.ns.facebook.com.
SOA: a.ns.facebook.com. (69.171.239.12)
Zone: failure
Wildcard: failure
Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.5/fierce", line 11, in <module>
    load_entry_point('fierce==1.2.0', 'console_scripts', 'fierce')()
  File "/usr/lib64/python3.5/site-packages/fierce.py", line 405, in main
    fierce(**vars(args))
  File "/usr/lib64/python3.5/site-packages/fierce.py", line 278, in fierce
    kwargs["subdomain_file"]
  File "/usr/lib64/python3.5/site-packages/fierce.py", line 201, in get_subdomains
    return get_stripped_file_lines(subdomain_filename)
  File "/usr/lib64/python3.5/site-packages/fierce.py", line 189, in get_stripped_file_lines
    return [line.strip() for line in open(filename).readlines()]
FileNotFoundError: [Errno 2] No such file or directory: 'default.txt'

The problem seems with find_subdomain_list_file function as it tries to find that location in the current directly only.

[blshkv]

Based on the FHS, the directory should be something like /usr/share/fierce/lists and the tool should be able to find it.

[blshkv]

FYI, https://stackoverflow.com/questions/4519127/setuptools-package-data-folder-location

Option 1: Install as package data
Option 2: Install to fixed location

[mschwager]

What commands did you use to install fierce?

Typically it's installed via pip with the PyPI package: https://pypi.python.org/pypi/fierce

I haven't tried using setup.py before, but we may be able to debug it. The find_subdomain_list_file function looks in, 1.) the current directory, then 2.) in the directory it was installed as a package in. Note that the package includes the files under the data_files attribute in https://github.com/mschwager/fierce/blob/master/setup.py.

Alternatively, you could just specify the full path of the list file you're trying to use, like:

$ fierce --subdomain-file /usr/lists/default.txt ...

Does that work?

[blshkv]

I already mentioned that I installed the tool using setup.py.
As you can see, I'm have ported the tool to Pentoo (Gentoo) and had to patch it in order it to work by default. We report about such problems to upstream usually so that you can fix it once and forever.
I think the expected logic (1 current 2 installed) did not work for me. Please double check.

+	sed -i -e "s|os.path.dirname(__file__)|\"/usr/share/fierce/\"|" fierce.py || die
 +	sed -i -e "s|'lists', data|'share/fierce/lists', data|" setup.py || die

P.S. Please help to reply faster so I would remember about it.

[mschwager]

This project is just a hobby of mine, so I maintain it as I have the time and motivation.

I'm curious about the exact command you ran with setup.py because it works for me in an Alpine Docker container:

$ git clone https://github.com/mschwager/fierce.git
$ cd fierce
$ python3 setup.py install
$ fierce --domain facebook.com --wide
NS: b.ns.facebook.com. a.ns.facebook.com.
SOA: a.ns.facebook.com. (69.171.239.12)
Zone: failure
Wildcard: failure
Found: 0.facebook.com. (157.240.18.36)
...

So I'm curious why this works for most flavors of Linux, but not Gentoo.

Regardless, I've pushed a new commit (7c666e4) that hopefully fixes your issue. Could you pull down the latest code and see if it works as you'd expect?

[blshkv]

somehow, files under "lists/*" are not longer getting installed. Any ideas?

[mschwager]

It appears as though Python has some severe deficiencies when using data_files OR package_data:

• neither package_data or data_files get installed pypa/sampleproject#30
• https://stackoverflow.com/questions/7522250/how-to-include-package-data-with-setuptools-distribute

I'll continue to investigate.

It still seems weird to me that python3 setup.py install works for me and not you, though.

[blshkv]

I'm installing it in a sandbox, could it be an issue?

 * fierce-1.2.0.tar.gz SHA256 SHA512 WHIRLPOOL size ;-) ...                                                                                   [ ok ]
>>> Unpacking source...
>>> Unpacking fierce-1.2.0.tar.gz to /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work
>>> Source unpacked in /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work
>>> Preparing source in /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work/fierce-1.2.0 ...
 * Applying 7c666e452b8a1b26efc72c7f5444b7bfe17af44a.patch ...                                                                                [ ok ]
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work/fierce-1.2.0 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work/fierce-1.2.0 ...
 * python3_5: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python3.5 setup.py build
running build
running build_py
copying fierce.py -> /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work/fierce-1.2.0-python3_5/lib
warning: build_py: byte-compiling is disabled, skipping.

>>> Source compiled.
>>> Test phase [not enabled]: net-analyzer/fierce-1.2.0-r2

>>> Install fierce-1.2.0-r2 into /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/ category net-analyzer
 * python3_5: running distutils-r1_run_phase distutils-r1_python_install
/usr/bin/python3.5 setup.py install --root=/var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5
running install
running build
running build_py
running install_lib
creating /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5
creating /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr
creating /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64
creating /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64/python3.5
creating /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64/python3.5/site-packages
copying /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work/fierce-1.2.0-python3_5/lib/fierce.py -> /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64/python3.5/site-packages
byte-compiling /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64/python3.5/site-packages/fierce.py to fierce.cpython-35.pyc
writing byte-compilation script '/var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmp7lj1v3qa.py'
/usr/bin/python3.5 -O /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmp7lj1v3qa.py
removing /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmp7lj1v3qa.py
writing byte-compilation script '/var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmpjloatwf8.py'
/usr/bin/python3.5 -OO /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmpjloatwf8.py
removing /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/temp/tmpjloatwf8.py
running install_egg_info
running egg_info
creating fierce.egg-info
writing requirements to fierce.egg-info/requires.txt
writing entry points to fierce.egg-info/entry_points.txt
writing fierce.egg-info/PKG-INFO
writing top-level names to fierce.egg-info/top_level.txt
writing dependency_links to fierce.egg-info/dependency_links.txt
writing manifest file 'fierce.egg-info/SOURCES.txt'
reading manifest file 'fierce.egg-info/SOURCES.txt'
writing manifest file 'fierce.egg-info/SOURCES.txt'
Copying fierce.egg-info to /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib64/python3.5/site-packages/fierce-1.2.0-py3.5.egg-info
running install_scripts
Installing fierce script to /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/_python3.5/usr/lib/python-exec/python3.5
 * python3_5: running distutils-r1_run_phase distutils-r1_python_install_all
>>> Completed installing fierce-1.2.0-r2 into /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/image/

 * Final size of build directory: 332 KiB
 * Final size of installed tree:   80 KiB

ecompressdir: bzip2 -9 /usr/share/doc

[mschwager]

Hmm, maybe. What type of sandbox are you using?

Also, it looks like you're trying to install fierce-1.2.0.tar.gz:

...
>>> Unpacking fierce-1.2.0.tar.gz to /var/tmp/portage/net-analyzer/fierce-1.2.0-r2/work
...

Where does that file come from? I've been distributing .whl packages, not .tar.gz at https://pypi.python.org/pypi/fierce.

If you can obtain the wheel distribution you should be able to install that with pip via:

$ /usr/bin/python3.5 -m pip install fierce-1.2.0-py3-none-any.whl

I suspect the .tar.gz installation may not be working correctly because this package has been created to be distributed as a wheel file, not a source distribution. So package_data and data_files may not work as expected.

Also, previously you mentioned "FHS":

Based on the FHS, the directory should be something like /usr/share/fierce/lists and the tool should be able to find it.

What's "FHS"?

Thanks for helping me debug this :)

[mschwager]

Okay, I was finally able to reproduce! It'll be much easier to debug now!

[blshkv]

ok, good to hear.

P.S.

• It is Gentoo sandbox;
• We do not use .WHL files because we compile it from sources, tar.gz is the source file from github "release" section obviously.
• FHS is Filesystem Hierarchy Standard, but it is not entirely relevant to python. You are working in the right direction: just use standard python installer (setup.py file)

[mschwager]

I had to revert back the change from data_files -> package_data because it broke my binary wheel distributions. However, adding a MANIFEST.in file with the subdomain lists included fixed the issue in my testing. Can you confirm?

[blshkv]

Can you post the updated patch please?

[mschwager]

Just pushed: https://github.com/mschwager/fierce/releases/tag/1.2.1-rc0

[blshkv]

Ok, so we are back to FHS discussion. The files are getting installed into /usr/lists/* which is wrong.
Could you place it them somewhere /usr/share/fierce/lists/* instead?

[mschwager]

I'm sorry to hear that installation is not working on your machine. However, I'm unable to reproduce your issue on standard Linux distributions. All distros I've tested have installed the files in a predictable place:

Ubuntu 16.04:

$ find / -name default.txt
/usr/local/lib/python3.5/dist-packages/fierce-1.2.0-py3.5.egg/lists/default.txt

Debian 8:

$ find / -name default.txt
/usr/local/lib/python3.4/dist-packages/fierce-1.2.0-py3.4.egg/lists/default.txt

Centos 7:

$ find / -name default.txt
/usr/lib/python3.6/site-packages/fierce-1.2.0-py3.6.egg/lists/default.txt

Further, running fierce after setup.py build and setup.py install works as expected.

If you'd like to debug your specific environment further and post a pull request I'd be happy to review it. If not, I'd recommend you follow the official installation procedure and install via the wheel file.

[blshkv]

Please re-open the issue, here is the latest log:

ebuild fierce-9999.ebuild manifest clean install 
>>> Creating Manifest for /data/pentoo/pentoo/net-analyzer/fierce
>>> Unpacking source...
 * Fetching https://github.com/mschwager/fierce.git ...
git fetch https://github.com/mschwager/fierce.git +HEAD:refs/git-r3/HEAD
git symbolic-ref refs/git-r3/net-analyzer/fierce/0/__main__ refs/git-r3/HEAD
 * Checking out https://github.com/mschwager/fierce.git to /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999 ...
git checkout --quiet refs/git-r3/HEAD
GIT update -->
   repository:               https://github.com/mschwager/fierce.git
   at the commit:            ce8e08f0941993c8980ee2ee839a8874c06155b9
>>> Source unpacked in /var/tmp/portage/net-analyzer/fierce-9999/work
>>> Preparing source in /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999 ...
>>> Source prepared.
>>> Configuring source in /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999 ...
>>> Source configured.
>>> Compiling source in /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999 ...
 * python3_5: running distutils-r1_run_phase distutils-r1_python_compile
/usr/bin/python3.5 setup.py build
running build
running build_py
copying fierce.py -> /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999-python3_5/lib
warning: build_py: byte-compiling is disabled, skipping.

>>> Source compiled.
>>> Test phase [not enabled]: net-analyzer/fierce-9999

>>> Install fierce-9999 into /var/tmp/portage/net-analyzer/fierce-9999/image/ category net-analyzer
 * python3_5: running distutils-r1_run_phase distutils-r1_python_install
/usr/bin/python3.5 setup.py install --root=/var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5
running install
running build
running build_py
running install_lib
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64/python3.5
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64/python3.5/site-packages
copying /var/tmp/portage/net-analyzer/fierce-9999/work/fierce-9999-python3_5/lib/fierce.py -> /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64/python3.5/site-packages
byte-compiling /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64/python3.5/site-packages/fierce.py to fierce.cpython-35.pyc
writing byte-compilation script '/var/tmp/portage/net-analyzer/fierce-9999/temp/tmpmv0d75gv.py'
/usr/bin/python3.5 -O /var/tmp/portage/net-analyzer/fierce-9999/temp/tmpmv0d75gv.py
removing /var/tmp/portage/net-analyzer/fierce-9999/temp/tmpmv0d75gv.py
writing byte-compilation script '/var/tmp/portage/net-analyzer/fierce-9999/temp/tmpg5975dkz.py'
/usr/bin/python3.5 -OO /var/tmp/portage/net-analyzer/fierce-9999/temp/tmpg5975dkz.py
removing /var/tmp/portage/net-analyzer/fierce-9999/temp/tmpg5975dkz.py
running install_data
creating /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lists
copying lists/default.txt -> /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lists
copying lists/5000.txt -> /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lists
copying lists/20000.txt -> /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lists
running install_egg_info
running egg_info
creating fierce.egg-info
writing top-level names to fierce.egg-info/top_level.txt
writing entry points to fierce.egg-info/entry_points.txt
writing requirements to fierce.egg-info/requires.txt
writing dependency_links to fierce.egg-info/dependency_links.txt
writing fierce.egg-info/PKG-INFO
writing manifest file 'fierce.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'fierce.egg-info/SOURCES.txt'
Copying fierce.egg-info to /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib64/python3.5/site-packages/fierce-1.2.0-py3.5.egg-info
running install_scripts
Installing fierce script to /var/tmp/portage/net-analyzer/fierce-9999/image/_python3.5/usr/lib/python-exec/python3.5
 * python3_5: running distutils-r1_run_phase distutils-r1_python_install_all
>>> Completed installing fierce-9999 into /var/tmp/portage/net-analyzer/fierce-9999/image/

 * Final size of build directory: 448 KiB
 * Final size of installed tree:  276 KiB

ecompressdir: bzip2 -9 /usr/share/doc

[blshkv]

I'm reading documentation and here it is:
https://packaging.python.org/tutorials/distributing-packages/

Although configuring package_data is sufficient for most needs, in some cases you may need to place data files outside of your packages. The data_files directive allows you to do that.
...
setuptools allows absolute “data_files” paths, and pip honors them as absolute, when installing from sdist. This is not true when installing from wheel distributions. Wheels don’t support absolute paths, and they end up being installed relative to “site-packages”

So it looks like you are facing the issue with wheel, pypa/wheel#92

However, it is clear from the documentation that files should not be installed there.

[mschwager]

Okay, what's your purposed solution?

It appears the problem here (among the many problems with Python's packaging system) is the usage of --root. When running install with --root I can reproduce your issue, when running without --root everything is fine.

Further, I re-attempted to include this files with package_data instead of data_files, but that didn't correctly install the files either.

[blshkv]

I'm not a Python expert, but the issue seems with the absolute path based on the description.
You may want to try "os.path.relpath" with os.path.join, line 17 in the setup.py?

[mschwager]

Okay, it appears that package_data wasn't working because the list files weren't under the fierce package. cf0e41e should hopefully fix your issue. I've tested that setup.py install should now work with the --root flag. The latest tag should have the changes: https://github.com/mschwager/fierce/releases/tag/1.2.1-rc1. Can you confirm this fixes your issue?

With that being said, there was a reason I wasn't distributing the fierce.py script under a particular Python package directory. I wanted to keep the script simple to download, and self-sufficient, so users don't have to install the full Python package if they don't want to. E.g. a simple wget of the fierce.py script. In order to ensure this isn't a problem I'm going to sit on these changes for a month or so to let anyone voice a concern. If there are no problems I'll push the changes in a new version on PyPI. You'll still be free to pull down the release-candidate tag, the package just won't be on PyPI yet.

Sound good?

[blshkv]

I have tested the latest git and confirm that the issue is fixed finally
Thank you

[mschwager]

Just a heads up, I've officially released 1.2.1: https://github.com/mschwager/fierce/releases/tag/1.2.1

👍

[blshkv]

Please excuse me for bothering you with all these unrelated bugs but it seems paid off.

The last project (wifite2) have managed to resolve the issue better. Now they install shared data into /usr/share using data_files once only. Would you consider fixing it in the same way?

Thanks

[blshkv]

ping?

[blshkv]

and ping