[maybe a bug] loss nan

[xmy0916]

https://github.com/yitu-opensource/T2T-ViT/blob/main/models/token_performer.py#L18
My code has turned on fp16, so the 1e-8 on this line to prevent division by 0 is not enough for my code... the loss of the network calculation appears nan due to this code :
https://github.com/yitu-opensource/T2T-ViT/blob/main/models/token_performer.py#L50

[yuanli2333]

Yes, you may be right, we can try to change 1e-8 to a large one, did you try it?

[xmy0916]

@yuanli2333 I have tested 1e-4 but also can't fix the problem.

[jiawangbai]

@xmy0916 In my implementation, 1e-6 can fix this problem, with bs=2048 and lr=1e-3.

[xmy0916]

Thanks!