Some ID tasks are unresponsive and stuck when EVALUATE INITIAL MISFIT

[zhangchengang]

Describe the bug/feature/issue
After I submit, I can successfully generate synthetic data. However, when I evaluate the initial misfit, some IDs compute normally while others freeze without responding. What could be causing this? I'm quite puzzled. My number of Receivers is 5000, the number of shots is 100, and the maximum ntask_max is set to 100.

To Reproduce
ntask: 100
nproc: 7
tasktime: 10000
rerun: 0
mpiexec: mpirun
array: null
log_level: DEBUG
verbose: False
title: scratch_acoustic
ntask_max: 100
walltime: 1000
environs:

Expected behavior
I hope he can successfully calculate the misfit without getting stuck.

Versions
seisflows3.5.3, ubantu22.04, openmpi, cluster,

Data
The folder “logs” and the files “sflog.txt”, parameters.yaml,specfem2d_workdir/OUTPUT_FILES_INIT, and specfem2d_workdir/OUTPUT_FILES_TRUE in https://github.com/zhangchengang/test. My question is:

https://github.com/zhangchengang/test/blob/main/logs/0002_60.log) completes successfully with the parameter “FINISH QUANTIFY MISFIT: 061”, but the process gets stuck indefinitely in https://github.com/zhangchengang/test/blob/main/logs/0002_31.log. I'm unsure what's causing this.

[bch0w]

Hi @zhangchengang, thanks for sharing, that's strange that some are finishing and others are not, but with no error message.
Is there a smaller number of sources of receivers that does work? I also wonder if the filesystem on your cluster is being overwhelmed, perhaps scaling back ntask_max would help?

[zhangchengang]

Yes. It works and completes normally when I use fewer receivers, but with more receivers the program sometimes stalls, even when the CPU isn’t fully utilized.