-
Notifications
You must be signed in to change notification settings - Fork 2
Open
Description
See ctgnnlib/training.py
:
def init_weights_xavier(m: nn.Module):
"Usage: `model.apply(init_weights_xavier)`"
# XXX For ReLU (as opposed to tanh and sigmoid)
# XXX He initialization is more appropriate
# XXX
# XXX <https://arxiv.org/abs/1502.01852>
# XXX Delving Deep into Rectifiers: Surpassing
# XXX Human-Level Performance on ImageNet Classification
# XXX
# XXX Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun
if isinstance(m, nn.Linear):
init.xavier_uniform_(m.weight)
m.bias.data.fill_(0.01)
So a He initialization should be used instead.
Metadata
Metadata
Assignees
Labels
No labels