sparta execution fails on matrix

[rfhaque]

Describe the bug
sparta givens the following error on the matrix system,

ERROR on proc 0: Failed to reallocate 4325376 bytes for array surf2grid:sbuf2 (/tmp/haque1/spack-stage/spack-stage-sparta-snl-master-gem7xm5wvjrfqnfkpbemzobm5sbeqtaq/spack-src/src/memory.cpp:91)
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[cli_0]: aborting job:
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

To Reproduce
Steps to reproduce the behavior:

1. bin/benchpark system init --dest=matrix llnl-matrix compiler=gcc cuda=12.6.0
2. bin/benchpark experiment init --dest sparta-cuda --system=matrix sparta-snl +cuda~openmp fft_kokkos=cufft fft=fftw3
3. bin/benchpark setup sparta sparta-snl <workspace_dir>
4. . <workspace_dir>/setup.sh
5. ramble --workspace-dir <workspace_dir>/sparta-snl/matrix/workspace workspace setup
6. ramble --workspace-dir <workspace_dir>/sparta-snl/matrix/workspace on

Supercomputer (please complete the following information):

• system: matrix
• system parameters: llnl-matrix compiler=gcc cuda=12.6.0
• experiment: sparta-snl
• experiment parameters: --system=matrix sparta-snl +cuda~openmp fft_kokkos=cufft fft=fftw3

[stanmoore1]

@rfhaque it looks like you are running out of memory. Do you know how many particles per GPU, or have a SPARTA logfile from the failed run?