🐛 [BUG] - Significant difference in the values of the Hessian for high-valued points between GPU and CPU #1785
Description
Description
A significant difference in the values of the Hessian for high-valued points is observed between simulation run on GPU and on CPU. Thus, while the median, 99th, and 99.9th percentile are still the same, a difference is already observed at the 99.99th percentile (2%) and keeps growing from there - 16% at the 99.999th percentile, and 11 order of magnitude difference between the absolute maximum of the two. This difference was not observed for the sensitivity kernels for alpha, beta and rho. While this issue can be overcome by clipping, it still seems like a problem.
Affected SPECFEM3D version
Latest development version
Your software and hardware environment
gcc 10.1.0, slurm 24.05.05, cluster with gpu
Reproduction steps
Model: 80x80x80 elements, 20 km x 20 km x 20 km, dt = 3 ms, NSTEP = 5000.
CPU run on 50 cores, GPU on 2 GPUs.Screenshots
Logs
OS
Linux