Regenerate modulefiles on update (fixes #1601) #1984
Conversation
opoplawski
force-pushed
the
3.x-module-regen
branch
from
2a3f6ef to
bd44035
Compare
opoplawski
force-pushed
the
3.x-module-regen
branch
from
bd44035 to
f211c92
Compare
opoplawski
force-pushed
the
3.x-module-regen
branch
from
f211c92 to
cac80a4
Compare
Yes, for each OS there should be an RPM attached to the GitHub Actions. The RPMs are only kept for 24 hours however, previously we reached space limits keeping them for a longer time. Thanks for your PR. I will need at least one week before being able to look closer at this PR. |
|
I would like to run new shell scripts through shellcheck. We have a https://github.com/openhpc/ohpc/blob/3.x/tests/ci/Makefile which does that for us. Could you add the new shellscript to the shellcheck, whitespace and shfmt sections there. If you prefer to not do these changes I can also do them later. There is a similar script in the intel MPI compatibility package. I guess we should do the same changes there, right? |
opoplawski
force-pushed
the
3.x-module-regen
branch
from
79144f2 to
e221fee
Compare
|
Am I right that shmft want TAB characters for indentation? |
We just use the defaults that Just running |
|
A friendly reminder that this PR had no activity for 30 days. |
|
Any update on the PR? Still cannot upgrade oneAPI smoothly. |
|
A friendly reminder that this PR had no activity for 30 days. |
|
(This is just a message to prevent expiration. Please ignore it.) |
opoplawski
force-pushed
the
3.x-module-regen
branch
6 times, most recently
from
757c67a to
387f31d
Compare
opoplawski
force-pushed
the
3.x-module-regen
branch
from
387f31d to
6ab39e8
Compare
|
I'll just note that in a previous CI run this was the output: This indicated a problem in the package, but apparently it was ignored. It this intentional? |
opoplawski
changed the title
|
I think this is ready for review now |
Yes and no. When it comes to testing things in CI with the Intel compiler we are not yet there. The testing still has a couple of places where the compiler family is hardcoded. If you look at https://github.com/openhpc/ohpc/blob/3.x/tests/ci/spec_to_test_mapping.py#L235 (for example) there is still a lot of Then there is also this line in https://github.com/openhpc/ohpc/blob/3.x/tests/ci/setup_slurm_and_run_tests.sh#L35 # Install rebuilt packages (if any)
# shellcheck disable=SC2046 # (we want the words to be split)
"${PKG[@]}" install $(find /home/"${USER}"/rpmbuild/RPMS/ -name "*rpm") || trueThe idea is, as the comment says, to install the rebuilt packages (if any). If we are running without any RPM rebuilt we want to skip installing the packages, thus So the current behaviour is not intentional but historical. It is based on how this script evolved and the script needs to be adapted to better handle possible situations. |
|
You need to add |
Signed-off-by: Orion Poplawski <orion@nwra.com>
opoplawski
force-pushed
the
3.x-module-regen
branch
from
6ab39e8 to
2f870b5
Compare
good catch - I got thrown off by the ordering of sections. |
Yes, it is confusing. |
|
I think we can remove all the psxe sub package as we do not mention them anywhere. I will add this to today's TSC agenda to see if anyone thinks we still need them. |
|
@opoplawski If you are motivated please remove all the sections concerning psxe from the compatibility RPMs. Let's just drop it. The TSC also agreed that it is not needed any more. If you do not want to do it, I can do it later. |
|
I'd like to leave this cleaner and am very strapped for time, so I'd prefer to leave it to you if that's okay. Thank you for your work on this project, I find it very helpful. |
Sure, no problem. I will take another look at this in the next days and test it some more. But so far it looks ready. Thanks for helping out. I will remove the |
|
Running the mpi script I still see a couple of errors: # /opt/ohpc/pub/bin/ohpc-update-modules-mpi
Generating new oneAPI modulefiles
/opt/intel/oneapi/modulefiles-setup.sh: line 119: cd: /opt/intel/oneapi/compiler/2024.0/modulefiles/../opt/oclfpga/modulefiles: No such file or directory
Creating OpenHPC-style modulefiles for local oneAPI MPI installation(s).
--> Installing modulefile for version=2021.11
Lmod has detected the following error: The following module(s) are unknown: "mpi/"2021.11""
Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
$ module --ignore_cache load "mpi/"2021.11""
Also make sure that all modulefiles written in TCL start with the string #%Module
/opt/ohpc/pub/moduledeps/intel/impi/2021.11 /opt/ohpc/pub/moduledeps/intel/impi/2021.11.rpmnew differ: byte 889, line 28
/opt/ohpc/pub/moduledeps/gnu/impi/2021.11 /opt/ohpc/pub/moduledeps/gnu/impi/2021.11.rpmnew differ: byte 714, line 21
/opt/ohpc/pub/moduledeps/gnu14/impi/2021.11 /opt/ohpc/pub/moduledeps/gnu14/impi/2021.11.rpmnew differ: byte 714, line 21
cp: cannot stat '/opt/ohpc/pub/moduledeps/gnu/impi/.version.rpmnew': No such file or directory
cmp: /opt/ohpc/pub/moduledeps/gnu14/impi/.version.rpmnew: No such file or directory
md5sum: /opt/ohpc/pub/moduledeps/gnu14/impi/.version.rpmnew: No such file or directoryThe unknown module error seems to be because of the quotes you added (probably to make ShellCheck happy). You could disable that check for that line. Not sure about the other messages. Any ideas how to handle those? |
This is as yet completely untested. But the idea is:
Does the CI generate rpms that can be tested?