gcc backend does not install anymore on newer platforms

[ksakmann]

Hi @Trinkle23897. Awesome work. I came across the following environment issue.
On an older platform the installation proceeds to install gcc backend

$ fpie --check-backend
[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.9.7
['numpy', 'numba', 'taichi-cpu', 'taichi-gpu', 'gcc', 'openmp']
$ gcc --version
gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)

However, on a newer platform the gcc backend does not install.

$ fpie --check-backend
[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12
['numpy', 'numba', 'taichi-cpu', 'taichi-gpu']
$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Do you have any ideas what the issue could be?

[Trinkle23897]

Could you try to inspect which error it raises in this line via python setup.py install or pip install -e .?

Fast-Poisson-Image-Editing/setup.py

Line 56 in 6a607f1

print(f"{type(e)}: {e}")

[ksakmann]

Hi @Trinkle23897 The solution is to comment out cmake in setup.py. I don't get how that affects the build, but it solves my problem. For reference, I just checked what you suggested, run pip install -e .. I use the code on master, but just to be sure to not deal with CUDA issues I commented out the CUDA section in CMakeLists.txt. pip install -e . then spits out the following exception. It looks like some path is wrong

...
    [ 83%] Linking CXX shared module /home/XXXX/Fast-Poisson-Image-Editing/build/lib.linux-x86_64-cpython-310/fpie/core_mpi.cpython-310-x86_64-linux-gnu.so
    lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
    [ 91%] Linking CXX shared module /home/XXXX/Fast-Poisson-Image-Editing/build/lib.linux-x86_64-cpython-310/fpie/core_gcc.cpython-310-x86_64-linux-gnu.so
    [100%] Linking CXX shared module /home/XXXX/Fast-Poisson-Image-Editing/build/lib.linux-x86_64-cpython-310/fpie/core_openmp.cpython-310-x86_64-linux-gnu.so
    lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
    lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
    [100%] Built target core_mpi
    [100%] Built target core_gcc
    [100%] Built target core_openmp
    error: can't copy 'build/lib.linux-x86_64-cpython-310/fpie.cpython-310-x86_64-linux-gnu.so': doesn't exist or not a regular file
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

Running pip install -e . after commenting out "cmake>=3.4", in setup.py still returns the exact same exception. However, when I run pip install . I finally get what I want:

XXX@XXXX:~/Fast-Poisson-Image-Editing$ fpie --check-backend
[Taichi] version 1.6.0, llvm 15.0.4, commit f1c6fbbd, linux, python 3.10.12
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
Authorization required, but no authorization protocol specified
['numpy', 'numba', 'taichi-cpu', 'taichi-gpu', 'gcc', 'openmp', 'mpi']

I can't say I understand what happend, but I'm happy now. For reference:

XXX@XXXX:~/Fast-Poisson-Image-Editing$ gcc --version
gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

XXX@XXXX:~/Fast-Poisson-Image-Editing$ cmake --version
cmake version 3.27.7

CMake suite maintained and supported by Kitware (kitware.com/cmake).

[atarss]

Same issue here.

I found that building the GCC backend fails when using cmake versions >= 4.0.

What’s a bit funny in my case is that during the first installation, the build used the system cmake (3.x) and everything worked fine. However, during the pip installation process, a cmake 4.x Python package was also installed. After I uninstalled and reinstalled the package, the build started using cmake 4.x (which may be incompatible with the build process), and then it failed.

So keeping cmake at a lower version seems to be a temporary workaround:

pip install fpie "cmake<4"

That said, I’m not sure this is a good long-term fix. Some users may need a newer version of cmake in their environment. Relying on the Python-packaged cmake might not be the best choice here. I’m not very familiar with cmake internals, but I suspect that fixing the cmake configuration files to support newer versions would be a better solution.