Skip to content

Use He weight initialization instead of Xavier #2

@tail-call

Description

@tail-call

See ctgnnlib/training.py:

def init_weights_xavier(m: nn.Module):
    "Usage: `model.apply(init_weights_xavier)`"
    # XXX For ReLU (as opposed to tanh and sigmoid)
    # XXX He initialization is more appropriate
    # XXX
    # XXX <https://arxiv.org/abs/1502.01852>
    # XXX     Delving Deep into Rectifiers: Surpassing
    # XXX     Human-Level Performance on ImageNet Classification
    # XXX
    # XXX     Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

    if isinstance(m, nn.Linear):
        init.xavier_uniform_(m.weight)
        m.bias.data.fill_(0.01)

So a He initialization should be used instead.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions