Python 3.10 compatiblity

[sashkab]

Questions

• Which operating system are you using (if Linux, please provide flavour of it, i.e RedHat, CentOS or Ubuntu), is it 32-bit, or 64-bit?

  macOS 11.6, 64-bit
  

• Which version of PyQ are you running? Please provide output of pyq --versions, if PyQ isn't operational, please provide Python interpreter version and PyQ version python -V; python3 -V; pip list | grep pyq:

  5.00
  

• Which version of kdb+ are you using, is it 32-bit or 64-bit?

  64-bit
  

  • If on 64-bit, is your QLIC set? Please provide output env | grep QLIC on linux/macOS, or set|grep QLIC on Windows.

    $ env | grep QLIC
    QLIC=/Users/user/.lic
    

• Did you use virtual environment to install PyQ? If not, why?

  • yes

• Where is your QHOME? Please provide output env | grep QHOME on linux/macOS, or set|grep QHOME on Windows.

  $ env | grep QHOME
  $
  

• Do you use Conda? If so, what version?

  • no

Steps to reproduce the issue

$ brew install python@3.10
$  /usr/local/opt/python@3.10/bin/python3 -mvenv .virtualenvs/310
$ source .virtualenvs/310/bin/activate
$ python3.10 -mpip install -U pip setuptools wheel
$ unzip ~/Downloads/m64.zip -d "${VIRTUAL_ENV}/q"
$ (unset PIP_NO_INDEX; python3.10 -mpip install pyq)
...
Successfully built pyq
Installing collected packages: pyq
Successfully installed pyq-5.0.0
$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Expected result

pyq --versions will output versions installed on system

Actual result

$ pyq --versions
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)
'2021.10.06T12:38:40.403 python

Workaround

$ cp -pv /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{,.orig}
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py -> /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py.orig
$ vi /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py
$ diff /Users/user/.virtualenvs/310/lib/python3.10/site-packages/pyq/__init__.py{.orig,}
4c4,7
< from collections import Mapping as _Mapping
---
> try:
>     from collections import Mapping as _Mapping
> except ImportError:
>     from collections.abc import Mapping as _Mapping
$ pyq --versions
PyQ 5.0.0
KDB+ 3.5 (2019.01.31) m64
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)]

[sashkab]

After first problem fixed, there is second problem.

$ python3.10 -mpip list
Package    Version
---------- -------
attrs      21.2.0
coverage   6.0
iniconfig  1.1.1
numpy      1.21.2
packaging  21.0
pip        21.2.4
pluggy     1.0.0
py         1.10.0
pyparsing  2.4.7
pyq        5.0.0
pytest     6.2.5
pytest-cov 3.0.0
pytest-pyq 1.2.0
setuptools 58.2.0
toml       0.10.2
tomli      1.2.1
wheel      0.37.0
$ pyq -mpip list
/Users/user/.virtualenvs/310/bin/pyq: No module named pip

I believe we saw this kind of issues before, but can't recall if it was addressed. It was related to the virutalenv and venv modules, how they are now implemented... which breaks logic PyQ relies on.

[sashkab]

Interestingly:

$ QHOME=$VIRTUAL_ENV/q $VIRTUAL_ENV/q/m64/q
...
q)p)import numpy
q)p)import pytest
q)p)import pip
q)p)print(pip.__file__)
/Users/user/.virtualenvs/310/lib/python3.10/site-packages/pip/__init__.py
q)p)print(sys.executable)
/Users/user/.virtualenvs/310/bin/pyq
q)p)print(sys.path)
['/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload', '/Users/user/.virtualenvs/310/lib/python3.10/site-packages']
q)p)print(sys.prefix)
/Users/user/.virtualenvs/310
q)p)print(sys.exec_prefix)
/Users/user/.virtualenvs/310

but

$ pyq
Python 3.10.0 (default, Oct  6 2021, 01:11:32) [Clang 13.0.0 (clang-1300.0.29.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/Users/user/.virtualenvs/310/bin/pyq'
>>> sys.path
['', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python310.zip', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10', '/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload']
>>> sys.prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'
>>> sys.exec_prefix
'/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10'

[sashkab]

NB: mapping import was fixed in the internal MR 746, which is planned to be released as 5.0.1...

[MurrMack]

Hi Aleks - is there a plan for 5.0.1 to be released any time soon? the default version of python on ubuntu 22.04 LTS is 3.10 so others may start running into this issue as upgrades begin to occur. Cheers!

[b-nln]

There are no plans to release 5.0.1, as there is no full python 3.10 support yet.

[MurrMack]

Is there anyway to force pyq to use a different version of python? My default version of python is 3.9 using update-alternatives however when I run pyq it tries to still use 3.10

CPUS=24 pyq                                                                                                                                                                         
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/murray/.local/lib/python3.10/site-packages/pyq/__init__.py", line 4, in <module>
    from collections import Mapping as _Mapping
ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)
'2022.08.15T08:58:06.931 python
  [5]  /home/murray/q/p.k:7: p)from pyq import q
                             ^
  [2]  /home/murray/q/python.q:1: \l p.k

python --version  
Python 3.9.13

python3 --version
Python 3.9.13

[b-nln]

Use virtualenv with python3.9 and install PyQ in there as described in the installation guide.

[sashkab]

After spending hours earlier this week in order to rewrite Python initialization for 3.10 and failing, I came up with a quick workaround for the importing modules installed in the virtual environment when using Python 3.10 and PyQ:

$ pyq
Python 3.10.5 (main, Jul 25 2022, 15:52:08) [GCC 11.2.1 20220219] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import markdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/.virtualenvs/3/lib/python3.10/site-packages/pyq/__init__.py", line 820, in __import__
    m = _imp(name, globals, locals, fromlist, level)
ModuleNotFoundError: No module named 'markdown'

Unfortunately, due to Python Initialization changes in 3.10, sys.path is not properly initialized when Python is loaded from kdb+. Dirty workaround is to extend sys.path to what it should have been:

>>> import sys
>>> sys.path.extend(['/home/user/.virtualenvs/3/lib/python3.10/site-packages', '/usr/lib/python3.10/site-packages'])
>>> import markdown
>>> markdown.__version__
'3.3.4'

Basically, first path is your $VIRTUAL_ENV/lib/python3.10/site-packages, and second path is where system python 3 packages are installed.