loss issue?

[ZephyrGo]

Thank you for this amazing project! I'm training a voice model but encountered an issue where several loss values suddenly become NaN during training.
During training, I observe the following loss pattern:

Normal training state:
INFO:mi-test:loss_disc=3.322, loss_gen=3.670, loss_fm=10.674, loss_mel=24.860, loss_kl=9.000

After some steps, losses suddenly become NaN:
INFO:mi-test:loss_disc=nan, loss_gen=nan, loss_fm=nan, loss_mel=23.440, loss_kl=9.000

Is this a training collapse? Should I rollback to the previous checkpoint?
Could this be caused by training data quality issues?
What's the recommended solution?Should I:

• Lower the learning rate?
• Reduce batch size?
• Disable FP16 training?
• Clean the training dataset?

[fumiama]

Of course you can roll back and try it again. By the way, what's your graphic card brand? The quality of dataset may not make it be NAN because it has already been normalized.

[ZephyrGo]

Thank you for the response!

I'm using an NVIDIA L40 GPU. I should mention that I was running the fork from github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI when this happened, not this repo - this might be related to version differences.

I am now retraining with this repository version. I'll report back if the issue persists.

Thanks for confirming that normalized data shouldn't cause NaN - that's helpful to know!

[fumiama]

I should mention that I was running the fork from github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI when this happened, not this repo - this might be related to version differences.

Of course there's some difference. You can try this repo because github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI has not been actively developing.