Add a variant of Adam called "PAdam"

[4SAnalyticsnModelling]

Add a variant of Adam optimizer called "partially adaptive momentum estimation method (PAdam)"

[CarloLucibello]

why did you open so many PRs?

[mcabbott]

When I looked the most meat was in #2276 . The others should be commits on that branch, not indep. PRs.

[4SAnalyticsnModelling]

@CarloLucibello, I apologize for creating confusion with so many PRs! As @mcabbott mentioned, they are all related PRs. While #2276 has most of the meat, the other commits are also required to make the changes work (i.e. they are not independent). I wasn't sure how to include all of those changes in one PR (since the proposed changes were in different files), that's why I went to create so many of them.

[mcabbott]

Somehow you need to push more commits to the same branch, which will update the PR. It's generally easiest to do things on your computer, rather than on github's website... I'm not sure I know how to move commits using the website, but this is called git cherry-pick locally. These things are confusing at first but soon become second nature.

I like this guide but not sure it's the ideal reference:

https://kshyatt.github.io/post/firstjuliapr/

[4SAnalyticsnModelling]

Thanks @mcabbott! I'll follow the article and give it a try to see if I can make all related commits to one branch.