Model Training ‐ Comparison ‐ [Bias Correction]
Models | Logs | Graphs | Configs
The penultimate optimizer parameter in the comparison.
ChatGPT:
Adam bias correction is a technique used in the Adam optimization algorithm to reduce bias in the estimated first and second moments of gradients during early training iterations. It involves scaling the moving averages of gradients and squared gradients by correction terms that depend on the iteration number. This correction helps to provide more accurate and stable updates to model parameters, enhancing optimization performance.
Compared values:
-
true-B, -
false-D.
I have a separate story related to this parameter. I found two guides where almost identical settings were presented for model training, but in one case, the models turned out to be good, while in the other, they quickly overtrain and started producing heavily distorted results. By eliminating parameters one by one, I came to the conclusion that this particular parameter had an impact on the results. However...
DLR(step)
Loss(epoch)
...However, if we look at the graphs and grids now, we won't see any difference in the results. Perhaps at the time of my experiment there was some bug that resulted in incorrect results, but now everything is fine.
Again, default value is a simpler choice.
It happened again at a later trainings. Maybe bug came back with another update. Or it is triggered by random. Or it is triggered by another parameter. Anyway, if your results looks like this, use use_bias_correction=True optimizer argument.
