GROMACS patched DEEPMD simulation -nan energy

[stfalxndria]

Hi All,
I am trying to run an MLMD simulation with GROMACS patched DP for water diffusion in MOFs. I have trained my model using DeepMD for 100K steps (i am aware this not a lot but I just want to test that everything is working). I should also note that my training data comes from CP2K DFT single point energy calulations.

I have gotten this error whilst running my MLMD:

Fatal error:
Step 50: The total potential energy is -nan, which is not finite. The LJ and
electrostatic contributions to the energy are 0 and 0, respectively. A
non-finite potential energy can be caused by overlapping interactions in
bonded interactions or very large or Nan coordinate values. Usually this is
caused by a badly- or non-equilibrated initial configuration, incorrect
interactions or parameters in the topology.

and I am wondering what can be causing this. I have tried both inserting the water molecules into my MOFs using gmx_insert and also using my final XTB structure (where I generated my training data). I am also wondering whether the bonds and angles specified in my itp files and also force field files would have anything to do with this.

I have attached the mlmd.log file, If anyone can give me advice on it, I would greatly appreciate it! Thank you in advance!

mlmd.log.txt

[Ericwang6]

Hi, there are a couple of thoughts to help you fix this:

1. The DeepMD-Gromacs interface was only tested on gmx 2020.2 and deepmd-kit 2.0.3 and for other versions it may not work properly.
2. According to my experience, if you trained a model from scratch it may not directly applicable to yield a stable MD simulation. A way to fix that is you either fine-tuning the model from pre-trained models or use DP-GEN to iteratively train the model by keeping adding new structures into training.
3. Are there any specific reason that you want to use DeepMD-Gromacs interface? The special feature of this interface is to enable DP/MM simulation, otherwise you can use DP/LAMMPS which is more stable and MUCH more faster. If you still want to run DP/MM simulation, I also recommend to try the openmm-deepmd https://github.com/JingHuangLab/openmm_deepmd_plugin, which should yield better performance and stability than DP/GMX.

Thanks!

[stfalxndria]

hi! thanks for the reply, I'm using the gromacs patched deepmd for consistency purposes since it's what I have been doing since last year. So I was originally using V3 but i have downgraded deepmd to V2 whilst all the energies looked fine the simulation completely 'blew up'. I have also teste the model and MLMD which I was able to successfully do last year but the same thing is observed. I should also note that the simulation was incredibly fast (1000ps within 10 minutes for a structure with 2000 atoms)

[stfalxndria]

Hi Eric,
Thanks for the reply! I have decided to delete everything and try to recompile dp patched gromacs v2.0.3 from scratch, but i am getting the error:

-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "1.8.3.1") 
-- Supported model version: 1.0 
-- Will not build nv GPU support
-- Will not build AMD GPU support
CMake Error at cmake/Findtensorflow.cmake:68 (message):
  Not found 'tensorflow/core/public/session.h' directory in path
  '/../tensorflow_core;/../tensorflow_core;/usr/;/usr/local/' You can
  manually set the tensorflow install path by -DTENSORFLOW_ROOT
Call Stack (most recent call first):
  CMakeLists.txt:82 (find_package)

I have tried installing different versions of tensorflow that were released around the same time as v2.0.3 but nothing has worked.

[MornightBlue]

Hi, I was using GROMACS with DeepMD-kit v3.1.0, but didn't come across the same error like yours, would you mind providing more informations such as topol.top?
I was using the pretrained model. Just like what you said, all the energies looked fine the simulation completely 'blew up'. But I didn't downgrade deepmd. I had tried five reps, every rep when it came to step 13 the second atom will suddenly fly away. And it was always this atom, always the 13th step, which has made me confused.

[stfalxndria]

hi! Sorry for the late reply. I managed to get it working in the end using v3.1.0 on both LAMMPS and GROMACS. I did not use a pretrained model and continued to finetune my model and actually that fixed the issue for me.

[locard00]

I got the completely consistent situation! I have tried deepmd-kit from v2.2.5 to v2.2.11 with gromacs 2020.2 for the water system in examples fold, it's very quite strange to run smoothly on my own computer while failed on the node on cluster.....On the cluster it will appear -nan energy....I promise they have the same or very similar compiling like gcc, cmake, python for gromacs. And for the deepmd-kit, I have tried install by Conda and off-line package, it's all useless. And I found it's "dener" in "enerd->term[F_EPOT] += dener * e_dp2gmx * deepmdPlugin->lmd;" in sim_util.cpp, the "dener" become -nan through "deepmdPlugin->nnp->compute(dener, dforce, dvirial, dcoord, deepmdPlugin->dtype, dbox);" ! I have no idea about this now. I think it's likely to be a bug.

[stfalxndria]

I managed to get it working with v3.0.1 in the end. That is strange that it worked on your own computer but not the cluster... are you sure it was patched correctly on your cluster? I fixed my issue by finetunning the model (i think my issue was the model itself was not good and had high forces for each atoms), and I suspect that maybe on your cluster GROMACS was not able to read the deepmd potentials correctly? Have you tried running one step and printing the energy and forces for each atoms?

[locard00]

Appreciate for your reply! I'm sure it was patched correctly on cluster. My model is no problem as well 'cause it gives the right RDF on my own computer. But you may be right that it's possible failed to read the model. Something info like "·······/gcc/lib" may be hid by "······/deepmd/lib" in my gromacs installing process, which I doubt leading to the problem. And I wonder two questions : 1、Does your new model work on other versions of DeePMD-kit ? 2、How could I running one step and printing the energy and force for each atom like you say ? Thanks for your help !!