calculate_gain¶

paddle.nn.initializer. calculate_gain ( nonlinearity, param=None ) [source]

Get the recommended gain value of some nonlinearity function. gain value can be used in some paddle.nn.initializer api to adjust the initialization value.

Parameters

nonlinearity (str) – name of nonlinearity activation function. If it is a linear function, such as: linear/conv1d/conv2d/conv3d/conv1d_transpose/conv2d_transpose/conv3d_transpose , 1.0 will be returned.
param (bool|int|float, optional) – optional parameter for somme nonlinearity function. Now, it only applies to ‘leaky_relu’. Default: None, it will be calculated as 0.01 in the formula.

Returns

A float value, which is the recommended gain for this nonlinearity function.

Examples

           >>> import paddle

>>> gain = paddle.nn.initializer.calculate_gain('tanh')
>>> print(gain)
1.6666666666666667
>>> # 5.0 / 3
>>> gain = paddle.nn.initializer.calculate_gain('leaky_relu', param=1.0)
>>> print(gain)
1.0
>>> # math.sqrt(2.0 / (1+param^2))
>>> initializer = paddle.nn.initializer.Orthogonal(gain)

          