val_check_interval and check_val_every_n_epoch appear redundant?

[turian]

val_check_interval and check_val_every_n_epoch appear redundant, except val_check_interval can be a float and operate within training epochs.

Is there a difference between the two parameters and can they be used simultaneously? If so, can that be documented clearly?

If not, can check_val_every_n_epoch be marked as deprecated and users pointed towards val_check_interval?

[SkafteNicki]

They are different:

• Use val_check_interval if you want to do validation more than 1 time per training epoch
• Use check_val_every_n_epoch if you want to do validation less often

The serve different scenarios:

• Users will often use val_check_interval when their training data is so large that they are only performing 1 or 2 epochs maybe. Then it is beneficial to do validation more often because your model is changing significantly over a single training epoch
• Users will use check_val_every_n_epoch when either their training data is very small, meaning that the model does not change that much over a single epoch and/or the validation dataset is very large so it may take a long time to validate.

[turian]

Thinking through this a bit more, my proposal so as not to break backwards compatability would be:

check_val_every_n_epoch can now take a float value.

val_check_interval is deprecated, removed from the documentation, and a warning is issued if it is used.

[SkafteNicki]

There is no way to not break backwards compatability by only having one argument. Your new proposal would remove the possibility to validate after x number of training batches.

[turian]

It just bugs me that val_check_interval has different semantics for float and int, and is confusing.

check_val_every_n_epoch = x allows you to check every x batches. If it's really required to check by number of batches (steps) and we don't want people to divide by the size of data loader, then check_val_every_n_step could be added and only check_val_every_n_epoch or check_val_every_n_step could be non-None

[SkafteNicki]

Fair point, that you find them confusing.
I does seem that we have an old issue that is related to this: #4409.
It seems that we will be keeping val_check_interval and remove check_val_every_n_epoch. Then val_check_interval=2.0 will be interpret as validation every second epoch.

[jkaniewski-tii]

Hi Everyone, what is the status of these two parameters? They both seems to still be present, but their descriptions are not very clear. Is there any place in the documentation where they are clearly explained? Also, is there an option somewhere to force validation (and hence saving the model) at the end of training?