Specfem3D Crashing Halfway

[athira981]

I've been running into an issue with my recent Specfem3D cartesian simulations. When running larger models (over 2 million elements and 2L time steps), the simulation crashes midway without showing any error message. Strangely, if I rerun the exact same setup, the simulation sometimes completes successfully on the 4th or 5th attempt.

The partial outputs seem usable up to the point where it stops. I've tested both the latest master and developer versions, and the problem persists in both. I've also double-checked my scripts and model setup, and I'm almost certain there's nothing wrong on my end.

Has anyone encountered something similar? Any ideas or suggestions to resolve this would be really appreciated.

[danielpeter]

this sounds like your machine is running out of memory for these larger model runs.

just try to run the simulation on more compute nodes to see if that solves the memory problem. you will need to figure out how to modify your job script accordingly, but that's the easiest solution.

following the user manual, we also recommend that you add ulimit -S -s unlimited to your .bash_profile file and/or limit stacksize unlimited to your .cshrc file to suppress any potential limit to the size of the Unix stack. this assumes that you're running it on a single machine though. in case this doesn't help, you'll have to go with the multiple compute node approach.

[athira981]

Thank you for your quick response.
However, I don't believe this is a memory issue. I'm running the simulation on a computing server with 3TB of RAM, and I’ve been monitoring the memory usage during the runs. The system never runs out of memory, and even the swap memory remains largely unused.
What I don't understand is that I’ve successfully run even larger simulations on the same machine without any issues until about two weeks ago. That’s why this behaviour seems unusual.
That said, I’ll definitely try the commands you suggested to see if they make any difference.
Thanks again for your support.