Error installing `tar.gz` with python 3.13.4-3.14.0(with pip >=24.1) when %TMP% folder is in an ImDisk created RAMDISK volume

[barkoder]

Bug report

Bug description:

Please see discussion at adang1345/PythonVista#56 (comment)

python versions 3.13.4 - 3.14.0 don't like it when %TMP% folder is in ImDisk created RAMDISK volumes.

I've confirmed Python 3.13.3 works when %TMP% folder is in ImDisk created RAMDISKs.

To reproduce.
Run Windows 11.
Install https://sourceforge.net/projects/imdisk-toolkit/files/20220301/ (20220301 is the version I'm using. There are newer versions as well but I've not attempted to reproduce the bug in those versions).
Create an NTFS or exFAT RAMDISK of 1024MB. Say drive letter X:

First to confirm it used to work with 3.13.3

Install https://www.python.org/ftp/python/3.13.3/python-3.13.3-amd64.exe
Run cmd.

X:
md tmp
set TMP=X:\tmp
set TEMP=X:\tmp
python --version && python.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Python 3.13.3
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz (998 kB)
     ---------------------------------------- 998.5/998.5 kB 1.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: gallery_dl
  Building wheel for gallery_dl (pyproject.toml) ... done
  Created wheel for gallery_dl: filename=gallery_dl-1.31.0.dev0-py3-none-any.whl size=703184 sha256=cdac923250f61b1c563328449657c7f642c9536a179900695c6c85d8baa8f549
  Stored in directory: X:\tmp\pip-ephem-wheel-cache-77tc2m6j\wheels\25\ff\1b\fd8a6b035f92eb9808314dfe7832003d04eaf0c61ba8e4c900
Successfully built gallery_dl
Installing collected packages: gallery_dl
Successfully installed gallery_dl

[notice] A new release of pip is available: 25.0.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip

Now to confirm it doesn't work with 3.13.4

Install https://www.python.org/ftp/python/3.13.4/python-3.13.4-amd64.exe
Run cmd.

X:
md tmp
set TMP=X:\tmp
set TEMP=X:\tmp
python --version && python.exe -m pip --no-cache-dir install --upgrade --force-reinstall --no-deps https://github.com/mikf/gallery-dl/archive/master.tar.gz
Python 3.13.4
Collecting https://github.com/mikf/gallery-dl/archive/master.tar.gz
  Downloading https://github.com/mikf/gallery-dl/archive/master.tar.gz
     - 998.5 kB 1.0 MB/s 0:00:01
ERROR: Could not install packages due to an OSError: [WinError 1] Incorrect function: 'X:\\tmp\\pip-req-build-5rbqqbqd'


[notice] A new release of pip is available: 25.1.1 -> 25.2
[notice] To update, run: python.exe -m pip install --upgrade pip

I tested every version 3.13.4 onwards. They all fail.
Something changed between 3.13.3 and 3.13.4 that is causing this issue.

The reason I use RAMDISKs is to minimize wear on my disks. So I would appreciate any help. Thanks!

From @adang1345 in that discussion

I also did some testing with different versions of pip, and it seems that the error occurs when you have pip 24.1 or higher and the Python instance contains #135034

Basically,

	PYTHON 3.13.3+ImDisk	PYTHON 3.13.4-3.14.0 + ImDisk
PIP 24.0	✓	✓
PIP >=24.1	✓	☓

CPython versions tested on:

3.14

Operating systems tested on:

Windows

[ChrisDenton]

This is likely a problem with ImDisk specifically, not RAM disks in general.

ImDisk does not support volumes properly. E.g. it "fakes" the X: drive instead of creating a proper mount. This breaks Windows API functions such as GetFinalPathNameByHandle, which is used by Path.resolve in strict mode.

[barkoder]

I'm aware of ImDisk's peculiarities, as the issue doesn't exist in SoftPerfect's inferior RamDisk solution.
But I don't think this particular issue is with ImDisk, as Python 3.13.3 works fine(with ImDisk) with all pip versions. See the table above, and the discussion thread I've linked.
Something changed between py 3.13.3 and 3.13.4 that's causing this.

[emmatyping]

But I don't think this particular issue is with ImDisk, as Python 3.13.3 works fine(with ImDisk) with all pip versions

I expect what's changed is usage of an API that breaks with ImDisk in a bugfix change. Searching online it seems like it is a common issue to get Incorrect function with ImDisk. So it does seem like an issue with ImDisk compatibility that surfaced because of a bugfix change.

[adang1345]

I ran a simple test where I created the file X:\test.txt, where X:\ is a RAM disk created with ImDisk. Then I created and ran a small C program that calls GetFinalPathNameByHandleW() with a handle to X:\test.txt. It indeed failed with GetLastError() equal to 1 (Incorrect function). So this is an ImDisk bug. What likely happened is that a change in Python 3.13.4 resulted in GetFinalPathNameByHandleW() getting called in your situation, while this function was not called in previous versions of Python.

[emmatyping]

Yeah GetFinalPathNameByHandleW is called by os.path.realpath, which is part of a security fix (#135064) for tarfile that was done in June.

I don't think there's much for us to do here.

[zooba]

It might be worth adding a fallback in realpath (perhaps with a noisy warning, if there's actual security impact). "Incorrect function" really does mean "the driver doesn't implement this", which is a legitimate result that's independent of the incoming data (as in, a malicious archive can't cause winerror=1 here).

I wouldn't trust ImDisk to identify themselves in any specific way we can detect, but there's probably a really slow path we can fall back to in realpath that will at least work.

[barkoder]

there's probably a really slow path we can fall back to in realpath that will at least work.

I would appreciate that please. Thanks!