prepare_metadata_for_build_wheel yields different SOURCES.txt depending on environment

[penguinpee]

I came across some unexpected behavior of prepare_metadata_for_build_wheel yielding a different SOURCES.txt depending on whether the function is called inside a repository checkout or not.

Steps to reproduce:

1. cd /tmp
2. git clone https://github.com/AFM-analysis/igor2.git
3. cd igor2
4. SETUPTOOLS_SCM_PRETEND_VERSION="0.0.7" python3 -c 'import setuptools.build_meta as sbm; sbm.prepare_metadata_for_build_wheel(".")'

This results in igor2.egg-info/SOURCES.txt including everything below igor2/. That is all subpackages.

Clean up for the next iteration:

1. git clean -dxf
2. rm -rf .git/

This results in a situation very similar to installing a release tarball from upstream. Now re-run step 4 from above.

The produced igor2.egg-info/SOURCES.txt only contains the toplevel package igor2, but none of the subpackages. This is the result of upstream's broken pyproject.toml, which has:

[tool.setuptools]
packages = ["igor2"]

An error (listing git files failed - pretending there aren't any) is shown when not inside a git repo. That seems to suggest the difference in behavior is intentional.

Can that be turned off somehow? In other words, do users have any control over the git specific behavior?

When using pip to build a wheel from the git checkout, I do get a BIG warning that packages would be ignored due to ambiguous configuration, but not when building the wheel from the extracted tar archive. I think in the latter case it would be rather helpful showing a warning regarding ambiguous configuration and (sub)packages being ignored.

[penguinpee]

An error (listing git files failed - pretending there aren't any) is shown when not inside a git repo. That seems to suggest the difference in behavior is intentional.

It has been brought to my attention that above error is coming from setuptools_scm. The full message is: ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any.

So this is an issue with setuptools_scm. The behavior is described in https://setuptools.pypa.io/en/latest/userguide/extension.html#adding-support-for-revision-control-systems.

I still think it's undesirable for a broken config to be ignored just because it happens to reside in a repo.

[webknjaz]

It's not setuptools that's posting the warnings. It's a limitation of the setuptools-scm plugin. Since it's a Python-native warning, a standard warning suppression mechanism works. Here's my workaround: pypa/setuptools-scm#987 (comment).

[penguinpee]

Thanks for the suggestion and for pointing me to a similar issue. However, I think what I'm seeing is different in that setuptools_scm silently accepts a broken pyproject.toml because it uses a different mechanism to determine what files are part of the package and should end up in SOURCES.txt.

On the other hand, after having poked around a bit more, our situation¹ is a bit of a corner case. It works as intended in the following situation:

1. Build wheel from git checkout
2. Build wheel from sdist (includes .egg_info/ and PKG-INFO)

It fails when downloading an archive from GitHub because that lacks .git/ and any of the Python package metadata. I think in the end it boils down to the question:

Note

Should setuptools_scm / setuptools fail by default if it detects a broken pyproject.toml?

I have a hunch this question has been asked before and is likely answered very differently depending on whom you ask.

Footnotes

1. That is Fedora's build environment, where we build everything from scratch. Even the metadata is generated freshly from scratch, removing shipped egg_info/. ↩

[webknjaz]

How is pyproject.toml broken? FWIW, the maintainers (of pure-python projects) should run python -m build without the --wheel and --sdist args. This builds an sdist and then builds a wheel from that and not from Git. Just like pip install would and downstream packagers mostly do the same. setuptools-scm generates metadata from Git for sdists (and auto-includes everything git-tracked). It's not supposed to recompute the version when building wheels — it just picks up the computed metadata in sdist (and in case of Git archives, it sources it from .git_archival.txt as a fallback, which is populated by Git itself).

[penguinpee]

Upstream has:

[tool.setuptools]
packages = ["igor2"]

That leads to the following error warning:

        ********************************************************************************
        ############################
        # Package would be ignored #
        ############################
        Python recognizes 'igor2.record' as an importable package[^1],
        but it is absent from setuptools' `packages` configuration.

        This leads to an ambiguous overall configuration. If you want to distribute this
        package, please make sure that 'igor2.record' is explicitly added
        to the `packages` configuration field.

        Alternatively, you can also rely on setuptools' discovery methods
        (for example by using `find_namespace_packages(...)`/`find_namespace:`
        instead of `find_packages(...)`/`find:`).

        You can read more about "package discovery" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

        If you don't want 'igor2.record' to be distributed and are
        already explicitly excluding 'igor2.record' via
        `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
        you can try to use `exclude_package_data`, or `include-package-data=False` in
        combination with a more fine grained `package-data` configuration.

        You can read more about "package data files" on setuptools documentation page:

        - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


        [^1]: For Python, any directory (with suitable naming) can be imported,
              even if it does not contain any `.py` files.
              On the other hand, currently there is no concept of package data
              directory, all directories are treated like packages.
        ********************************************************************************

It's an easy fix.

FWIW, the maintainers (of pure-python projects) should run python -m build without the --wheel and --sdist args.

That doesn't make a difference in this situation. Both sdist and wheel are incomplete due to above error.