feat(bzlmod): support cross-platform whl setups within the main hub repo (#1837)

With this change we add support for platform-specific wheel registration
and
doing the selection of which wheel is used at build time.

This supports:
- Different package versions for different platforms.
- Use string_flags to configure what to fetch/select:
    - only whls, only sdist or auto mode.
    - libc version and `musl` vs `glibc` selection.
    - universal2 vs arch wheels for mac.
    - target osx version selection.

Summary of changes:
- The `uv pip install` would only warn the user of yanked packages but
  would not refuse to install them. Update our implementation to better
  match the same behaviour.
- A previous PR has added the support for passing it in the
`requirements_by_platform` and this just add the necessary code to make
  sure that we can also do the dependency management when parsing the
  `whl` `METADATA` files.
- Only configure `dev_pip` deps for `linux` and `osx` platforms to not
raise
  issues later.
- Add a function for generating a `whl_library` name from a `filename`.
- Add a macro for generating all config settings for a particular set of
parameters.
- Update `render_pkg_aliases` to also use those config settings.
- Update the toolchain selection `target_settings` to use the
`py_linux_libc`
config setting. With this the user can register a `musl` linux toolchain
if
  needed. We can also use similar `flag_values` to resolve #1876.
- Integrate everything in the `pip` extension and setup cross-platform
  select statements.

Work towards #1357, #260

Stacked on #1937

Author: aignas

CHANGELOG.md

@@ -36,10 +36,20 @@ A brief description of the categories of changes:
 * (bzlmod): The `python` and internal `rules_python` extensions have been
   marked as `reproducible` and will not include any lock file entries from now
   on.
-
 * (gazelle): Remove gazelle plugin's python deps and make it hermetic.
   Introduced a new Go-based helper leveraging tree-sitter for syntax analysis.
   Implemented the use of `pypi/stdlib-list` for standard library module verification.
+* (pip.parse): Do not ignore yanked packages when using `experimental_index_url`.
+  This is to mimic what `uv` is doing. We will print a warning instead.
+* (pip.parse): Add references to all supported wheels when using `experimental_index_url`
+  to allowing to correctly fetch the wheels for the right platform. See the
+  updated docs on how to use the feature. This is work towards addressing
+  [#735](https://github.com/bazelbuild/rules_python/issues/735) and
+  [#260](https://github.com/bazelbuild/rules_python/issues/260). The spoke
+  repository names when using this flag will have a structure of
+  `{pip_hub_prefix}_{wheel_name}_{py_tag}_{abi_tag}_{platform_tag}_{sha256}`,
+  which is an implementation detail which should not be relied on and is there
+  purely for better debugging experience.
 
 ### Fixed
 * (gazelle) Remove `visibility` from `NonEmptyAttr`.
@@ -63,6 +73,13 @@ A brief description of the categories of changes:
   the `experimental_index_url` feature which will fetch metadata from PyPI or a
   different private index and write the contents to the lock file. Fixes
   [#1643](https://github.com/bazelbuild/rules_python/issues/1643).
+* (pip.parse): Install `yanked` packages and print a warning instead of
+  ignoring them. This better matches the behaviour of `uv pip install`.
+* (toolchains): Now matching of the default hermetic toolchain is more robust
+  and explicit and should fix rare edge-cases where the host toolchain
+  autodetection would match a different toolchain than expected. This may yield
+  to toolchain selection failures when the python toolchain is not registered,
+  but is requested via `//python/config_settings:python_version` flag setting.
 
 ### Added
 * (rules) Precompiling Python source at build time is available. but is
@@ -104,6 +121,14 @@ A brief description of the categories of changes:
   {obj}`PyRuntimeInfo.zip_main_template`.
 * (toolchains) A replacement for the Bazel-builtn autodetecting toolchain is
   available. The `//python:autodetecting_toolchain` alias now uses it.
+* (pip): Support fetching and using the wheels for other platforms. This
+  supports customizing whether the linux wheels are pulled for `musl` or
+  `glibc`, whether `universal2` or arch-specific MacOS wheels are preferred and
+  it also allows to select a particular `libc` version. All of this is done via
+  the `string_flags` in `@rules_python//python/config_settings`. If there are
+  no wheels that are supported for the target platform, `rules_python` will
+  fallback onto building the `sdist` from source. This behaviour can be
+  disabled if desired using one of the available string flags as well.
 
 [precompile-docs]: /precompiling
 

MODULE.bazel

@@ -96,7 +96,9 @@ dev_pip.parse(
     },
     hub_name = "dev_pip",
     python_version = "3.11",
-    requirements_lock = "//docs/sphinx:requirements.txt",
+    requirements_by_platform = {
+        "//docs/sphinx:requirements.txt": "linux_*,osx_*",
+    },
 )
 dev_pip.parse(
     hub_name = "pypiserver",

docs/sphinx/api/python/config_settings/index.md

@@ -5,11 +5,18 @@
 
 # //python/config_settings
 
-:::{bzl:flag} precompile
+:::{bzl:flag} python_version
+Determines the default hermetic Python toolchain version. This can be set to
+one of the values that `rules_python` maintains.
+:::
+
+::::{bzl:flag} precompile
 Determines if Python source files should be compiled at build time.
 
-NOTE: The flag value is overridden by the target level `precompile` attribute,
+:::{note}
+The flag value is overridden by the target level `precompile` attribute,
 except for the case of `force_enabled` and `forced_disabled`.
+:::
 
 Values:
 
@@ -27,22 +34,24 @@ Values:
 * `force_disabled`: Like `disabled`, except overrides target-level setting. This
   is useful useful for development, testing enabling precompilation more
   broadly, or as an escape hatch if build-time compiling is not available.
-:::
+::::
 
-:::{bzl:flag} precompile_source_retention
+::::{bzl:flag} precompile_source_retention
 Determines, when a source file is compiled, if the source file is kept
 in the resulting output or not.
 
-NOTE: This flag is overridden by the target level `precompile_source_retention`
+:::{note}
+This flag is overridden by the target level `precompile_source_retention`
 attribute.
+:::
 
 Values:
 
 * `keep_source`: Include the original Python source.
 * `omit_source`: Don't include the orignal py source.
 * `omit_if_generated_source`: Keep the original source if it's a regular source
   file, but omit it if it's a generated file.
-:::
+::::
 
 :::{bzl:flag} precompile_add_to_runfiles
 Determines if a target adds its compiled files to its runfiles.
@@ -59,15 +68,80 @@ Values:
   incrementally enabling precompilation on a per-binary basis.
 :::
 
-:::{bzl:flag} pyc_collection
+::::{bzl:flag} pyc_collection
 Determine if `py_binary` collects transitive pyc files.
 
-NOTE: This flag is overridden by the target level `pyc_collection` attribute.
+:::{note}
+This flag is overridden by the target level `pyc_collection` attribute.
+:::
 
 Values:
 * `include_pyc`: Include `PyInfo.transitive_pyc_files` as part of the binary.
 * `disabled`: Don't include `PyInfo.transitive_pyc_files` as part of the binary.
+::::
+
+::::{bzl:flag} py_linux_libc
+Set what libc is used for the target platform. This will affect which whl binaries will be pulled and what toolchain will be auto-detected. Currently `rules_python` only supplies toolchains compatible with `glibc`.
+
+Values:
+* `glibc`: Use `glibc`, default.
+* `muslc`: Use `muslc`.
+:::{versionadded} 0.33.0
+:::
+::::
+
+::::{bzl:flag} pip_whl
+Set what distributions are used in the `pip` integration.
+
+Values:
+* `auto`: Prefer `whl` distributions if they are compatible with a target
+  platform, but fallback to `sdist`. This is the default.
+* `only`: Only use `whl` distributions and error out if it is not available.
+* `no`: Only use `sdist` distributions. The wheels will be built non-hermetically in the `whl_library` repository rule.
+:::{versionadded} 0.33.0
 :::
+::::
+
+::::{bzl:flag} pip_whl_osx_arch
+Set what wheel types we should prefer when building on the OSX platform.
+
+Values:
+* `arch`: Prefer architecture specific wheels.
+* `universal`: Prefer universal wheels that usually are bigger and contain binaries for both, Intel and ARM architectures in the same wheel.
+:::{versionadded} 0.33.0
+:::
+::::
+
+::::{bzl:flag} pip_whl_glibc_version
+Set the minimum `glibc` version that the `py_binary` using `whl` distributions from a PyPI index should support.
+
+Values:
+* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
+* `X.Y`: The string representation of a `glibc` version. The allowed values depend on the `requirements.txt` lock file contents.
+:::{versionadded} 0.33.0
+:::
+::::
+
+::::{bzl:flag} pip_whl_muslc_version
+Set the minimum `muslc` version that the `py_binary` using `whl` distributions from a PyPI index should support.
+
+Values:
+* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
+* `X.Y`: The string representation of a `muslc` version. The allowed values depend on the `requirements.txt` lock file contents.
+:::{versionadded} 0.33.0
+:::
+::::
+
+::::{bzl:flag} pip_whl_osx_version
+Set the minimum `osx` version that the `py_binary` using `whl` distributions from a PyPI index should support.
+
+Values:
+* `""`: Select the lowest available version of each wheel giving you the maximum compatibility. This is the default.
+* `X.Y`: The string representation of the MacOS version. The allowed values depend on the `requirements.txt` lock file contents.
+
+:::{versionadded} 0.33.0
+:::
+::::
 
 ::::{bzl:flag} bootstrap_impl
 Determine how programs implement their startup process.

docs/sphinx/pip.md

@@ -1,168 +1,4 @@
 (pip-integration)=
 # Pip Integration
 
-To pull in dependencies from PyPI, the `pip_parse` function is used, which
-invokes `pip` to download and install dependencies from PyPI.
-
-In your WORKSPACE file:
-
-```starlark
-load("@rules_python//python:pip.bzl", "pip_parse")
-
-pip_parse(
-    name = "pip_deps",
-    requirements_lock = ":requirements.txt",
-)
-
-load("@pip_deps//:requirements.bzl", "install_deps")
-
-install_deps()
-```
-
-For `bzlmod` an equivalent `MODULE.bazel` would look like:
-```starlark
-pip = use_extension("//python/extensions:pip.bzl", "pip")
-pip.parse(
-    hub_name = "pip_deps",
-    requirements_lock = ":requirements.txt",
-)
-use_repo(pip, "pip_deps")
-```
-
-You can then reference installed dependencies from a `BUILD` file with:
-
-```starlark
-load("@pip_deps//:requirements.bzl", "requirement")
-
-py_library(
-    name = "bar",
-    ...
-    deps = [
-        "//my/other:dep",
-        "@pip_deps//requests",
-        "@pip_deps//numpy",
-    ],
-)
-```
-
-The rules also provide a convenience macro for translating the entries in the
-`requirements.txt` file (e.g. `opencv-python`) to the right bazel label (e.g.
-`@pip_deps//opencv_python`). The convention of bazel labels is lowercase
-`snake_case`, but you can use the helper to avoid depending on this convention
-as follows:
-
-```starlark
-load("@pip_deps//:requirements.bzl", "requirement")
-
-py_library(
-    name = "bar",
-    ...
-    deps = [
-        "//my/other:dep",
-        requirement("requests"),
-        requirement("numpy"),
-    ],
-)
-```
-
-If you would like to access [entry points][whl_ep], see the `py_console_script_binary` rule documentation.
-
-[whl_ep]: https://packaging.python.org/specifications/entry-points/
-
-(per-os-arch-requirements)=
-## Requirements for a specific OS/Architecture
-
-In some cases you may need to use different requirements files for different OS, Arch combinations. This is enabled via the `requirements_by_platform` attribute in `pip.parse` extension and the `pip_parse` repository rule. The keys of the dictionary are labels to the file and the values are a list of comma separated target (os, arch) tuples.
-
-For example:
-```starlark
-    # ...
-    requirements_by_platform = {
-        "requirements_linux_x86_64.txt": "linux_x86_64",
-        "requirements_osx.txt": "osx_*",
-        "requirements_linux_exotic.txt": "linux_exotic",
-        "requirements_some_platforms.txt": "linux_aarch64,windows_*",
-    },
-    # For the list of standard platforms that the rules_python has toolchains for, default to
-    # the following requirements file.
-    requirements_lock = "requirements_lock.txt",
-```
-
-In case of duplicate platforms, `rules_python` will raise an error as there has
-to be unambiguous mapping of the requirement files to the (os, arch) tuples.
-
-An alternative way is to use per-OS requirement attributes.
-```starlark
-    # ...
-    requirements_windows = "requirements_windows.txt",
-    requirements_darwin = "requirements_darwin.txt",
-    # For the remaining platforms (which is basically only linux OS), use this file.
-    requirements_lock = "requirements_lock.txt",
-)
-```
-
-(vendoring-requirements)=
-## Vendoring the requirements.bzl file
-
-In some cases you may not want to generate the requirements.bzl file as a repository rule
-while Bazel is fetching dependencies. For example, if you produce a reusable Bazel module
-such as a ruleset, you may want to include the requirements.bzl file rather than make your users
-install the WORKSPACE setup to generate it.
-See https://github.com/bazelbuild/rules_python/issues/608
-
-This is the same workflow as Gazelle, which creates `go_repository` rules with
-[`update-repos`](https://github.com/bazelbuild/bazel-gazelle#update-repos)
-
-To do this, use the "write to source file" pattern documented in
-https://blog.aspect.dev/bazel-can-write-to-the-source-folder
-to put a copy of the generated requirements.bzl into your project.
-Then load the requirements.bzl file directly rather than from the generated repository.
-See the example in rules_python/examples/pip_parse_vendored.
-
-
-(credential-helper)=
-## Credential Helper
-
-The "use Bazel downloader for python wheels" experimental feature includes support for the Bazel
-[Credential Helper][cred-helper-design].
-
-Your python artifact registry may provide a credential helper for you. Refer to your index's docs
-to see if one is provided.
-
-See the [Credential Helper Spec][cred-helper-spec] for details.
-
-[cred-helper-design]: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md
-[cred-helper-spec]: https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md
-
-
-### Basic Example:
-
-The simplest form of a credential helper is a bash script that accepts an arg and spits out JSON to
-stdout. For a service like Google Artifact Registry that uses ['Basic' HTTP Auth][rfc7617] and does
-not provide a credential helper that conforms to the [spec][cred-helper-spec], the script might
-look like:
-
-```bash
-#!/bin/bash
-# cred_helper.sh
-ARG=$1  # but we don't do anything with it as it's always "get"
-
-# formatting is optional
-echo '{'
-echo '  "headers": {'
-echo '    "Authorization": ["Basic dGVzdDoxMjPCow=="]'
-echo '  }'
-echo '}'
-```
-
-Configure Bazel to use this credential helper for your python index `example.com`:
-
-```
-# .bazelrc
-build --credential_helper=example.com=/full/path/to/cred_helper.sh
-```
-
-Bazel will call this file like `cred_helper.sh get` and use the returned JSON to inject headers
-into whatever HTTP(S) request it performs against `example.com`.
-
-[rfc7617]: https://datatracker.ietf.org/doc/html/rfc7617
+See [PyPI dependencies](./pypi-dependencies).