Poetry cannot be installed on Github Actions runner using latest installer and Python 3.8.

[KelSolaar]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• [N/A] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows
• Poetry version: Latest
• Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

After changing from the old installer process, i.e. curl -L https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py -o get-poetry.py to the new installer process, i.e. curl -sSL https://install.python-poetry.org/ | POETRY_HOME=$HOME/.poetry python3 -, my builds are failing on Github Actions and Python 3.8: Poetry cannot be installed:

022-09-17T01:37:56.7561313Z ##[group]Run curl -sSL https://install.python-poetry.org | python3 -
2022-09-17T01:37:56.7561741Z �[36;1mcurl -sSL https://install.python-poetry.org | python3 -�[0m
2022-09-17T01:37:56.7572615Z shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
2022-09-17T01:37:56.7572971Z env:
2022-09-17T01:37:56.7573137Z   CI_PYTHON_VERSION: 3.8
2022-09-17T01:37:56.7573406Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-17T01:37:56.7573650Z ##[endgroup]
2022-09-17T01:37:59.4702925Z Retrieving Poetry metadata
2022-09-17T01:37:59.4703126Z 
2022-09-17T01:37:59.4703425Z # Welcome to Poetry!
2022-09-17T01:37:59.4703691Z 
2022-09-17T01:37:59.4703874Z This will download and install the latest version of Poetry,
2022-09-17T01:37:59.4704250Z a dependency and package manager for Python.
2022-09-17T01:37:59.4704413Z 
2022-09-17T01:37:59.4704801Z It will add the `poetry` command to Poetry's bin directory, located at:
2022-09-17T01:37:59.4705053Z 
2022-09-17T01:37:59.4705200Z C:\Users\runneradmin\AppData\Roaming\Python\Scripts
2022-09-17T01:37:59.4705368Z 
2022-09-17T01:37:59.4705644Z You can uninstall at any time by executing this script with the --uninstall option,
2022-09-17T01:37:59.4705956Z and these changes will be reverted.
2022-09-17T01:37:59.4706100Z 
2022-09-17T01:37:59.4706188Z Installing Poetry (1.2.1)
2022-09-17T01:37:59.4706429Z Installing Poetry (1.2.1): Creating environment
2022-09-17T01:37:59.4706740Z Installing Poetry (1.2.1): An error occurred. Removing partial environment.
2022-09-17T01:37:59.4963567Z Traceback (most recent call last):
2022-09-17T01:37:59.4970320Z   File "<stdin>", line 935, in <module>
2022-09-17T01:37:59.4970900Z   File "<stdin>", line 914, in main
2022-09-17T01:37:59.4971289Z   File "<stdin>", line 545, in run
2022-09-17T01:37:59.4971657Z   File "<stdin>", line 566, in install
2022-09-17T01:37:59.4972175Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\contextlib.py", line 113, in __enter__
2022-09-17T01:37:59.4972602Z     return next(self.gen)
2022-09-17T01:37:59.4972971Z   File "<stdin>", line 638, in make_env
2022-09-17T01:37:59.4973331Z   File "<stdin>", line 624, in make_env
2022-09-17T01:37:59.4973679Z   File "<stdin>", line 314, in make
2022-09-17T01:37:59.4974144Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 68, in create
2022-09-17T01:37:59.4974571Z     self._setup_pip(context)
2022-09-17T01:37:59.4975025Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 289, in _setup_pip
2022-09-17T01:37:59.4975508Z     subprocess.check_output(cmd, stderr=subprocess.STDOUT)
2022-09-17T01:37:59.4976022Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 415, in check_output
2022-09-17T01:37:59.4976513Z     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-09-17T01:37:59.4977012Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 493, in run
2022-09-17T01:37:59.4977453Z     with Popen(*popenargs, **kwargs) as process:
2022-09-17T01:37:59.4977935Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 858, in __init__
2022-09-17T01:37:59.4978405Z     self._execute_child(args, executable, preexec_fn, close_fds,
2022-09-17T01:37:59.4978916Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 1311, in _execute_child
2022-09-17T01:37:59.4979397Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2022-09-17T01:37:59.4979834Z FileNotFoundError: [WinError 2] The system cannot find the file specified
2022-09-17T01:37:59.5142470Z ##[error]Process completed with exit code 1.
2022-09-17T01:37:59.5272597Z Cleaning up orphan processes

An exception is raised in the new installer make definition:

    @classmethod
    def make(cls, target: Path) -> "VirtualEnvironment":
        try:
            # on some linux distributions (eg: debian), the distribution provided python
            # installation might not include ensurepip, causing the venv module to
            # fail when attempting to create a virtual environment
            # we import ensurepip but do not use it explicitly here
            import ensurepip  # noqa: F401
            import venv

            builder = venv.EnvBuilder(clear=True, with_pip=True, symlinks=False)
            context = builder.ensure_directories(target)

            if (
                WINDOWS
                and hasattr(context, "env_exec_cmd")
                and context.env_exe != context.env_exec_cmd
            ):
                target = target.resolve()

            builder.create(target)
        except ImportError:
            # fallback to using virtualenv package if venv is not available, eg: ubuntu
            python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
            virtualenv_bootstrap_url = (
                f"https://bootstrap.pypa.io/virtualenv/{python_version}/virtualenv.pyz"
            )

            with tempfile.TemporaryDirectory(prefix="poetry-installer") as temp_dir:
                virtualenv_pyz = Path(temp_dir) / "virtualenv.pyz"
                request = Request(
                    virtualenv_bootstrap_url, headers={"User-Agent": "Python Poetry"}
                )
                virtualenv_pyz.write_bytes(urlopen(request).read())
                cls.run(
                    sys.executable, virtualenv_pyz, "--clear", "--always-copy", target
                )

        # We add a special file so that Poetry can detect
        # its own virtual environment
        target.joinpath("poetry_env").touch()

        env = cls(target)

        # we do this here to ensure that outdated system default pip does not trigger older bugs
        env.pip("install", "--disable-pip-version-check", "--upgrade", "pip")

        return env

Specifically, the venv._setup_pip definition seems to not find the Python interpreter for the Poetry environment:

    def _setup_pip(self, context):
        """Installs or upgrades pip in a virtual environment"""
        # We run ensurepip in isolated mode to avoid side effects from
        # environment vars, the current directory and anything else
        # intended for the global Python environment
        cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade',
                                                         '--default-pip']
        subprocess.check_output(cmd, stderr=subprocess.STDOUT)

I have a repo with a small repro case here: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3071692472/jobs/4962601087

[neersighted]

This looks like an interesting edge case with Python 3.8 and Git Bash (or maybe MSYS bash in general?) triggering a bug in venv. See our CI for 3.8 here not using a pipe. Powershell also works very well (and is what is officially documented):

https://github.com/python-poetry/install.python-poetry.org/actions/runs/3005610039/jobs/4826098405

I suppose we could probably work around this by using the virtualenv zipapp in this case, but I wonder if it's worth fixing as we do not officially suggest using anything but Powershell on Windows (CMD and Git Bash are not tested and best effort only as you can always use Powershell). We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

[KelSolaar]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

That sound sensible to me!

[Secrus]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

I am pretty sure Python 3.8 will be supported for the next 2 years (source). It's Python 3.7 that is EOL in 9 months (not 6).

[KelSolaar]

From a pure selfish standpoint and because I tend to follow Numpy schedule, it is April 2023:

On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)

[neersighted]

Ah, looks like I got the Numpy schedule in my head. @KelSolaar, could you try reproducing with Python 3.7 just for giggles?

[KelSolaar]

I just did: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3108121099/jobs/5036970221

[neersighted]

Good to know it's the same on 3.7 -- we'll probably just declare this a known issue with Python 3.7 & 3.8 + MSYS Bash.

[Secrus]

So, the issue is only present when using the bash shell on Windows runner?

[neersighted]

That is correct, yes. We haven't drilled down deep enough to figure out if it's truly MSYS bash in general, or just the version shipped by Git (bash on GHA actions is Git Bash), but I think that's mostly unnecessary.

[sondrelg]

Hi 👋 I maintain a Github action for setting up Poetry (this one), and we've had a few reports of broken 3.8 workflows after switching to this installer.

I'm fine with this not being fixed, but wanted to ask: would you recommend using the old script from the main Poetry repo for these targets? That's what I'm planning to do, unless you have a better idea? 🙂