Compile scorpio with adios2, got error on PIO_MAX_CACHED_STEPS_FOR_ADIOS

[halehawk]

Hello,

I tried to compile scorpio with adios2 at derecho.ucar.edu, I got the following error:
/glade/derecho/scratch/haiyingx/scorpio/src/clib/pioc_support.c:360:58: error: expected expression
file->max_step_calls = PIO_MAX_CACHED_STEPS_FOR_ADIOS;

It looks like PIO_MAX_CACHED_STEPS_FOR_ADIOS is set by during cmake as the follows, do I miss anything in the cmake options?

Thanks, Haiying

Here is the partial cmake log:
CC=mpicc CXX=mpicxx FC=mpif90 cmake -DNetCDF_C_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.25/oneapi/2023.0.0/wzol/
-DNetCDF_Fortran_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cray-mpich/8.1.25/oneapi/2023.0.0/wzol/
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr/
-DWITH_ADIOS2=ON -DADIOS2_DIR=/glade/work/haiyingx/ADIOS2/installintelde/lib64/cmake/adios2
-DCMAKE_INSTALL_PREFIX=/glade/derecho/scratch/haiyingx/scorpio/install
-DPIO_ENABLE_TESTS=ON
..

-- ===== Configuring SCORPIO... =====
-- Enabling SCORPIO I/O performance statistics collection (default)
-- Using BGET to allocate memory for caching data in SCORPIO
-- Disabling debug logging in SCORPIO (default)
-- Disabling use/check of the MPI serial library (default)
-- Disabling saving I/O decompositions (default)
-- No limit on the number of cached I/O regions (default)
-- Limit on the number of Lustre OSTs, PIO_MAX_LUSTRE_OSTS, is not set (default)
-- Filesystem striping unit is not set (default)
-- Using PnetCDF independent data mode to read variables in SCORPIO (default)
-- Reserving some extra space in the header when creating NetCDF files, requested bytes = 10240 (default)
-- Setting the maximum number of I/O decompositions registered with ADIOS type to PIO_MAX_ADIOS_DECOMPS = 65536 (default)
-- Setting the maximum number of cached application steps for ADIOS type to PIO_MAX_CACHED_STEPS_FOR_ADIOS = 128 (default)
-- Disabling code coverage... (use -DPIO_ENABLE_COVERAGE:BOOL=ON to enable coverage, only GNU is supported for now)

[jayeshkrishna]

Can you include your complete configure/make log (you can copy it in a github gist and include the link to gist here)?

[halehawk]

I found this error caused by the previously compiled libgptl.a of scorpio is reused. So this problem is gone, but after the successful compilation, I got a lot of fails on ctest. So I still share the cmake log here and ctest log. Could you please tell me how I can pass all ctest? Thanks! https://gist.github.com/halehawk/1acad191f06c3d805edf2a6cf9e020b2
LastTest.log

[jayeshkrishna]

Can you try launching one of the tests in a batch file (without using ctest, directly using mpiexec/aprun etc)?

It looks like ctest support for this machine (derecho?) is not added in SCORPIO yet. It should be a small patch that we can provide to you for testing that should get the ctest testing working for you. Please try running the tests manually in a batch file and let us know the results (Also include the batch file in the issue).

[halehawk]

I requested an interactive node on derecho, ran the ctest and single test as well. Some tests can pass now, but more cannot and even stuck at test #71.
LastTest_0103.log

[dqwu]

@halehawk Your ADIOS2 lib is quite old (version 2.7.1)
-- Found ADIOS2: /glade/work/haiyingx/ADIOS2/installintelde/lib64/cmake/adios2/adios2-config.cmake (found suitable version "2.7.1.745", minimum required is "2.7.0") found components: C CXX Fortran MPI

For latest ADIOS read support we do require ADIOS2 2.9.0 or higher versions. I think you can install ADIOS2 2.9.1 for testing (we have not tested latest 2.9.2 or 2.10.0-rc1 so far).

FYI, you can install ADIOS2 2.9.1 with the commands below (please change /path/to/your/adios/installation accordingly)

wget https://github.com/ornladios/ADIOS2/archive/refs/tags/v2.9.1.tar.gz
tar zxf v2.9.1.tar.gz

cd ADIOS2-2.9.1

mkdir build
cd build

CC=mpicc CXX=mpicxx FC=mpif90 \
CFLAGS="-g -O2" CPPFLAGS="-g -O2" CXXFLAGS="-g -O2" FCFLAGS="-g -O2" \
cmake \
-DCMAKE_INSTALL_PREFIX=/path/to/your/adios/installation \
-DBUILD_SHARED_LIBS=OFF -DADIOS2_BUILD_EXAMPLES=OFF -DBUILD_TESTING=OFF \
-DADIOS2_USE_Blosc2=OFF \
-DADIOS2_USE_BZip2=OFF \
-DADIOS2_USE_ZFP=OFF \
-DADIOS2_USE_SZ=OFF \
-DADIOS2_USE_MGARD=OFF \
-DADIOS2_USE_PNG=OFF \
-DADIOS2_USE_DataMan=OFF \
-DADIOS2_USE_DataSpaces=OFF \
-DADIOS2_USE_MHS=OFF \
-DADIOS2_USE_ZeroMQ=OFF \
-DADIOS2_USE_HDF5=OFF \
-DADIOS2_USE_Python=OFF \
-DADIOS2_USE_Fortran=OFF \
-DADIOS2_USE_Profiling=OFF \
..

make -j4

make install

[dqwu]

@halehawk It seems that you configured SCORPIO with NetCDF 4.9.2:
-- Checking NetCDF version - 4.9.2./*!<

Our nightly builds use an older but more stable NetCDF 4.8.0 lib for testing (higher versions have some known issues for the unit tests). In fact, you do not need NetCDF lib for testing ADIOS type (PnetCDF lib is sufficient).

FYI, you can use "-DWITH_NETCDF=OFF" to configure SCORPIO without NetCDF support (you can safely remove -DNetCDF_C_PATH=XXXX and -DNetCDF_Fortran_PATH=XXXX in this case).

[dqwu]

@halehawk As a sanity check on that machine, I would suggest that you first run the full SCORPIO unit tests with PnetCDF type only (SCORPIO is configured without NetCDF or ADIOS support):

git clone https://github.com/E3SM-Project/scorpio.git
cd scorpio

mkdir build
cd build

CC=mpicc CXX=mpicxx FC=mpif90 cmake -Wno-dev \
-DWITH_NETCDF=OFF \
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr \
-DPIO_USE_MALLOC=ON \
-DPIO_ENABLE_TESTS=ON \
-DPIO_ENABLE_EXAMPLES=ON \
..

make -j4

make tests

ctest

PIO_ENABLE_EXAMPLES is set to ON to enable testing some C examples.
PIO_USE_MALLOC is set to ON (recommended) to use native malloc (instead of bget package).

[halehawk]

Sure, I will try your last suggestion first.Sent from my iPhoneOn Jan 4, 2024, at 9:24 AM, dqwu ***@***.***> wrote: @halehawk As a sanity check, I would suggest that you run the full SCORPIO unit tests with PnetCDF type only (SCORPIO is configured without NetCDF or ADIOS support): git clone https://github.com/E3SM-Project/scorpio.git cd scorpio mkdir build cd build CC=mpicc CXX=mpicxx FC=mpif90 cmake -Wno-dev \ -DWITH_NETCDF=OFF \ -DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr \ -DPIO_USE_MALLOC=ON \ -DPIO_ENABLE_TESTS=ON \ -DPIO_ENABLE_EXAMPLES=ON \ .. make -j4 make tests ctest PIO_ENABLE_EXAMPLES is set to ON to enable testing some C examples. PIO_USE_MALLOC is set to ON (recommended) to use native malloc (instead of bget package). —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[dqwu]

@halehawk ADIOS IO type in SCORPIO has some known limitations, see #553 for detailed information.

For ADIOS type, you should not run all of the unit tests (C and Fortran). In fact, our nightly builds only test a subset of the Fortran unit tests (C unit tests do not support ADIOS type so far), plus C example test_adios.c.

After you install a custom ADIOS 2.9.1 lib on that machine, I would suggest that you test ADIOS write feature first (ADIOS read support has more restrictions).

FYI, below are the commands to test ADIOS write feature:

git clone https://github.com/E3SM-Project/scorpio.git
cd scorpio

mkdir build
cd build

ADIOS2_DIR=/path/to/your/adios2/2.9.1/installation \
CC=mpicc CXX=mpicxx FC=mpif90 cmake -Wno-dev \
-DWITH_NETCDF=OFF \
-DWITH_ADIOS2=ON \
-DADIOS_BP2NC_TEST=ON \
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr \
-DPIO_USE_MALLOC=ON \
-DPIO_ENABLE_TESTS=ON \
-DPIO_ENABLE_EXAMPLES=ON \
..

make -j4

make tests

ctest -R "pio_unit_test|^init|pio_file\
|ncdf_get_put|ncdf_inq|ncdf_simple_tests|pio_rearr\
|pio_decomp|pio_sync_tests|pio_buf_lim_tests|pio_iodesc_tests\
|pio_iosystem_tests|examplePio|example1|darray_no_async|test_adios"

Note that we only run a subset of the Fortran unit tests and test_adios example with the ctest command.

Also, ADIOS_BP2NC_TEST is set to ON to perform implicit file conversion (ADIOS to NetCDF) such that we actually read converted .nc files for testing.

[dqwu]

@halehawk For testing ADIOS read feature, we only run test_adios.c example so far (the shorter list of applicable Fortran unit tests has not been determined yet). Simply remove "-DADIOS_BP2NC_TEST=ON" and only run test_adios with the ctest command:

git clone https://github.com/E3SM-Project/scorpio.git
cd scorpio

mkdir build
cd build

ADIOS2_DIR=/path/to/your/adios2/2.9.1/installation \
CC=mpicc CXX=mpicxx FC=mpif90 cmake -Wno-dev \
-DWITH_NETCDF=OFF \
-DWITH_ADIOS2=ON \
-DPnetCDF_PATH=/glade/u/apps/derecho/23.06/spack/opt/spack/parallel-netcdf/1.12.3/cray-mpich/8.1.25/oneapi/2023.0.0/blyr \
-DPIO_USE_MALLOC=ON \
-DPIO_ENABLE_TESTS=ON \
-DPIO_ENABLE_EXAMPLES=ON \
..

make -j4

make tests

ctest -R test_adios

[halehawk]

I tested on pnetcdf with scorpio. Until test#69, all tests passed.
LastTest_0104.log

[halehawk]

I tried to compile scorpio with esmf. I included scorpio include path and adios2 include path. But I still got this error:
In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_IO_Handler.C:36:
In file included from /glade/.../esmf/src/Infrastructure/IO/src/../include/ESMCI_PIO_Handler.h:45:
/glade/.../scorpio/install/include/pio.h:71:10: fatal error: In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_PIO_Handler.C:'spio_hash.h' file not found27:

It looks like spio_hash.h is not in my scorpio include path, how can I change cmake option to install this header file?

[dqwu]

I tried to compile scorpio with esmf. I included scorpio include path and adios2 include path. But I still got this error: In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_IO_Handler.C:36: In file included from /glade/.../esmf/src/Infrastructure/IO/src/../include/ESMCI_PIO_Handler.h:45: /glade/.../scorpio/install/include/pio.h:71:10: fatal error: In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_PIO_Handler.C:'spio_hash.h' file not found27:

It looks like spio_hash.h is not in my scorpio include path, how can I change cmake option to install this header file?

It might take some time for us to fix this issue. If you manually copy spio_hash.h to /glade/.../scorpio/install/include, does it work?

[halehawk]

Yes, this error is gone. Then I got this error /glade/.../esmf/src/Infrastructure/Mesh/src/ESMCI_UGRID_Util.C:357:11: error: use of undeclared identifier 'PIOc_InitDecomp_ReadOnly' piorc = PIOc_InitDecomp_ReadOnly(pioSystemDesc, PIO_INT, Is there any macro I should define to avoid this error?

…

On Wed, Jan 10, 2024 at 3:53 PM dqwu ***@***.***> wrote: I tried to compile scorpio with esmf. I included scorpio include path and adios2 include path. But I still got this error: In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_IO_Handler.C:36: In file included from /glade/.../esmf/src/Infrastructure/IO/src/../include/ESMCI_PIO_Handler.h:45: /glade/.../scorpio/install/include/pio.h:71:10: fatal error: In file included from /glade/.../esmf/src/Infrastructure/IO/src/ESMCI_PIO_Handler.C:'spio_hash.h' file not found27: It looks like spio_hash.h is not in my scorpio include path, how can I change cmake option to install this header file? It might take some time for us to fix this issue. If you manually copy spio_hash.h to /glade/.../scorpio/install/include, does it work? — Reply to this email directly, view it on GitHub <#554 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACAPEFAZC63EKDM745JTX6DYN4LV7AVCNFSM6AAAAABBKVPKWOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBVHA3TGMZRGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>