About using Custom datasets to training the model.

[chengfengke]

Hi, congrats to the novelty work!
I am training GCDM model using my own datasets,
but when I was going to train the model, I found that the loss become convergence in only a few epoch.
I use almost same hyperparameters settings of QM9 property conditional generation model, and max_natoms is also about 30, it has about 50k molecules in the dataset.
I wonder if I set some inappropriate hyperparameters.
Besides, is the "n_nodes", "atom_types" in dataset_config important? I barely not change them.

[amorehead]

Hello. Have you kept training the model past epoch 5? Sometimes, generative neural networks can get stuck in local minima for a few epochs before finding a new lower minima to explore, so it doesn't hurt to keep running the network until it's clear it's stuck in its current minimum.

[chengfengke]

when I trained 950epoch, I still got [2025-06-17 23:16:04,653][src.datamodules.components.edm.rdkit_functions][INFO] - Validity over 500 molecules: 25.40%
[2025-06-17 23:16:04,655][src.datamodules.components.edm.rdkit_functions][INFO] - Uniqueness over 127 valid molecules: 86.61%, is it common or something wrong?

[chengfengke]

and My loss has maintained for about 900 epoch without any change

[amorehead]

Hi, @chengfengke. How large are your new dataset's molecules?

[chengfengke]

it's about 50000, and max n atoms is 30.> Hi, @chengfengke. How large are your new dataset's molecules?

[chengfengke]

I notice that you only use one gpu in your experiment settings，may the problem by 2 gpus in training？

[chengfengke]

and I still got clip_gradient after the first several epoch.

[amorehead]

@chengfengke, I wouldn't worry too much about the gradients being clipped. That's to be expected with this type of model. However, you mentioned your molecules are larger than e.g., those in QM9, and you are using 2 GPUs for data parallel training. That means your effective batch size is larger than the QM9 or GEOM-Drugs model configs were designed for. As such, you may need to try out some different learning rates and model sizes to better fit the complexities of this new dataset.

[chengfengke]

@chengfengke, I wouldn't worry too much about the gradients being clipped. That's to be expected with this type of model. However, you mentioned your molecules are larger than e.g., those in QM9, and you are using 2 GPUs for data parallel training. That means your effective batch size is larger than the QM9 or GEOM-Drugs model configs were designed for. As such, you may need to try out some different learning rates and model sizes to better fit the complexities of this new dataset.

thx a lot, I will try some different hyper parameters.