rrelu¶

paddle.nn.functional. rrelu ( x, lower=0.125, upper=0.3333333333333333, training=True, name=None ) [source]

rrelu activation.

Applies the randomized leaky rectified liner unit function to improve generalization performance, as described in the paper: Empirical Evaluation of Rectified Activations in Convolutional Network

During training, randomly samples the negative slope for activation values as described below:

\[\begin{split}rrelu(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ a * x, & & otherwise \\ \end{array} \right.\end{split}\]

where \(x\) is the input tensor, \(a\) is randomly sampled from uniform distribution in range (\(lower\), \(upper\)),

In the test phase, the negative slope will take the average value of \(lower\) and \(upper\):

\[\begin{split}rrelu(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ (lower + upper) * 0.5 * x, & & otherwise \\ \end{array} \right.\end{split}\]

where \(x\) is the input tensor, \(lower\) and \(upper\) are the bounds of uniform distribution.

Parameters

• x (Tensor) – The input Tensor with data type float16, float32, float64.

• lower (float, optional) – The lower bound of uniform distribution. Default: 0.125.

• upper (float, optional) – The upper bound of uniform distribution. Default: 0.3333333333333333.

• training (bool, optional) – Current mode is in training or others. Default is True.

• name (str, optional) – For details, please refer to Name. Generally, no setting is required. Default: None.

Returns

A Tensor with the same data type and shape as x .

Examples

>>> import paddle
>>> import paddle.nn.functional as F
>>> paddle.seed(1)
>>> input_tensor = paddle.to_tensor([[[[-2.0,  3.0, -4.0,  5.0],
...                                    [ 3.0, -4.0,  5.0, -6.0],
...                                    [-7.0, -8.0,  8.0,  9.0]],
...                                   [[ 1.0, -2.0, -3.0,  4.0],
...                                    [-5.0,  6.0,  7.0, -8.0],
...                                    [ 6.0,  7.0,  8.0,  9.0]]]], dtype='float32')
>>> out = F.rrelu(input_tensor, 0.1, 0.3)
>>> print(out)
Tensor(shape=[1, 2, 3, 4], dtype=float32, place=Place(cpu), stop_gradient=True,
[[[[-0.20715050,  3.        , -1.01193857,  5.        ],
   [ 3.        , -0.94084597,  5.        , -0.65544695],
   [-1.24268556, -2.34339547,  8.        ,  9.        ]],
  [[ 1.        , -0.44942653, -0.68969047,  4.        ],
   [-1.03736508,  6.        ,  7.        , -0.95799232],
   [ 6.        ,  7.        ,  8.        ,  9.        ]]]])