Generating python files from setup.py?

[paugier]

I try to improve an old project (https://foss.heptapod.net/mercurial/mercurial-devel/) with a pyproject.toml and a setup.py using setuptools.

I'd like to know what are the good practices for generation of files at different stages.

• Before the sdist build (actually before setuptools uses MANIFEST.in because this is for dynamic generation of this file)
• During sdist build (translation, __version__.py, hgext/__index__.py)
• During wheel build (mercurial/__modulepolicy__.py)

Currently, these file generations are called through setuptools commands/subcommands in setup.py and also with Makefile targets.

I'd like to know if this is still the best methods when using setuptools through PEP 517.

I'd like them to work fine in different cases:

• pip install .
• pip install --editable .
• python -m build (produce sdist and wheel from the sdist)

I'm a bit lost with the association between setuptools commands/subcommands (build_py, install, etc.) and the PEP 517 functions (defined for setuptools in https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py).

I'm a bit lost where the generated files have to be generated. For example, currently, mercurial/__modulepolicy__.py (produced during build_py) is produced in the wheel but not for the editable install.

Indeed, mercurial/__modulepolicy__.py seems to be produced in this case in a temporary directory and not in the source tree.

[abravalheri]

Hi @paugier this customisation is trick to implement, I will try my best to give some pointers, but I don't have all the answers:

I'm a bit lost with the association between setuptools commands/subcommands (build_py, install, etc.) and the PEP 517 functions (defined for setuptools in https://github.com/pypa/setuptools/blob/main/setuptools/build_meta.py).

In general it looks like the following:

get_requires_for_* => Internally setuptools will run the egg_info/dist_info command objects
prepare_metadata_for_* => Internally setuptools will run the egg_info/dist_info command objects
build_sdist => Internally setuptools will run the sdist command object
build_wheel => Internally setuptools will run the bdist_wheel command object
build_editable => Internally setuptools will run the egg_info/dist_info/editable_wheel command objects

In turn, each command has subcommands and dependencies on other commands, so setuptools will run a bunch of stuff in the process (e.g. sdist depends on build_py, dist_info depends on egg-info, bdist_wheel depends on build, which in turn has subcommands for build_py, build_ext, etc...).

Before the sdist build (actually before setuptools uses MANIFEST.in because this is for dynamic generation of this file).

Setuptools uses MANIFEST.in in all the build hooks, including get_requires_for_build_wheel and prepare_metadata_for_build_wheel.

I think you can get away with ignoring get_requires_for_build_wheel if you simply define all your build dependencies statically in pyproject.toml [build-system] requires = [...] and don't use the setup_requires keyword in setup(...).

So assuming the file has to be generated in prepare_metadata_for_build_wheel, egg_info may be a good starting point.

I'm a bit lost where the generated files have to be generated. For example, currently, mercurial/modulepolicy.py (produced during build_py) is produced in the wheel but not for the editable install.

The problem with editable installs and "regular" installs is that for editable installs to work, the files need to be produced inside the source tree. On the other hand regular installs try as much as possible to produce transient files outside of the source tree (this is intentional). This means that the custom commands need to be aware of that difference. build_* commands have a boolean editable_mode property that can be used during build time to tell if the project is being build in the editable mode or not.

Finally, the build process may run different PEP 517 using different directories as working directory. E.g. the "recommended" (less error prone) frontend implementation is to run prepare_metadata_for_build_wheel, get_requires_for_sdist and build_sdist from the directory containing the source code, but then get_requires_for_wheel, build_wheel from a temporary directory created by extracting/inflating the sdist. The diagram bellow is a rough/incomplete attempt of representing this:

graph LR
    src[fa:fa-laptop-code source code] --> b1(fa:fa-hammer)
    
    subgraph build process
      b1 --> sdist(fa:fa-file-code sdist) --> b2(fa:fa-hammer)
      b2 --> wheel(fa:fa-box wheel)
       
      build([build dependencies]) --> b1
      build([build dependencies]) --> b2
    end

    subgraph installation process
        wheel --> pkg
        pkg(fa:fa-box-open) --> inst[fa:fa-python installed packages]
        runtime([runtime dependencies]) --> inst
    end

    classDef deps fill:#2aa198,stroke:#333
    classDef env fill:#6c71c4,stroke:#333
    build:::deps
    runtime:::deps

Loading

[paugier]

Thanks a lot for this great answer. This is very useful for me. I start to understand how setuptools works internally, and I realize that for advanced tasks, one still needs to understand this. I also see that setuptools documentation is not oriented towards such applications (or I didn't find the right page). For example a graph showing the command dependencies would be useful. Of course, I understand that it needs work to be done and maintained and that it would be useful only to few people.

Another quick question: do you think one can get the path towards the source tree from build_py when called from pip install --editable (i.e. from editable_wheel and from a temporary directory created by extracting the sdist)? In this case, self.build_lib is not in the source tree. Is there another variable that one could use to get the path towards the source?

[abravalheri]

If you define your custom build_py inside of setup.py, then I suppose you can also use os.path.dirname(__file__).

[paugier]

It works! Thanks. I was worried that in this case setup.py would be the one in the temporary directory but no. I guess the PEP 517 frontend (here pip) imports setup.py only once. Or a least it uses only the original setup.py.

[abravalheri]

In the case of build_editable all the hooks are called with the original source tree as working directory (if I am not mistaken).

The hooks regarding the wheel are the ones that can be invoked from a temporary working directory (created by extracting the sdist).

[webknjaz]

I think that there's also a difference between how build frontends operate in non-editable mode. IIRC pypa/pip copies things into a temporary directory before invoking the backend, while pypa/build doesn't.