RReLU¶

class paddle.nn. RReLU ( lower=0.125, upper=0.3333333333333333, name=None ) [source]

RReLU activation layer.

Applies the randomized leaky rectified liner unit function to improve generalization performance, as described in the paper: Empirical Evaluation of Rectified Activations in Convolutional Network

During training, randomly samples the negative slope for activation values as described below:

\[\begin{split}RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ a * x, & & otherwise \\ \end{array} \right.\end{split}\]

where \(x\) is the input tensor, \(a\) is randomly sampled from uniform distribution in range (\(lower\), \(upper\)),

In the test phase, the negative slope will take the average value of \(lower\) and \(upper\):

\[\begin{split}RReLU(x)= \left\{ \begin{array}{rcl} x, & & if \ x >= 0 \\ (lower + upper) * 0.5 * x, & & otherwise \\ \end{array} \right.\end{split}\]

where \(x\) is the input tensor, \(lower\) and \(upper\) are the bounds of uniform distribution.

Parameters

lower (float, optional) – The lower bound of uniform distribution. Default: 1.0/8.0.
upper (float, optional) – The upper bound of uniform distribution. Default: 1.0/3.0.
name (str, optional) – Name for the operation (optional, default is None). For more information, please refer to Name.

Shape:

input: Tensor with any shape. Default dtype is float32.
output: Tensor with the same shape as input.

Examples

           >>> import paddle
>>> paddle.seed(2023)

>>> input_tensor = paddle.to_tensor([[[[-2.0,  3.0, -4.0,  5.0],
...                                    [ 3.0, -4.0,  5.0, -6.0],
...                                    [-7.0, -8.0,  8.0,  9.0]],
...                                   [[ 1.0, -2.0, -3.0,  4.0],
...                                    [-5.0,  6.0,  7.0, -8.0],
...                                    [ 6.0,  7.0,  8.0,  9.0]]]], dtype='float32')
...
>>> rrelu_layer = paddle.nn.RReLU(0.1, 0.3)
>>> out = rrelu_layer(input_tensor)
>>> print(out)
Tensor(shape=[1, 2, 3, 4], dtype=float32, place=Place(cpu), stop_gradient=True,
[[[[-0.54633451,  3.        , -0.81611776,  5.        ],
   [ 3.        , -0.60768753,  5.        , -1.68630385],
   [-1.29360127, -1.45026064,  8.        ,  9.        ]],
  [[ 1.        , -0.58808362, -0.74662417,  4.        ],
   [-1.01785135,  6.        ,  7.        , -1.97268605],
   [ 6.        ,  7.        ,  8.        ,  9.        ]]]])

          

forward ( x ) forward¶

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

*inputs (tuple) – unpacked tuple arguments
**kwargs (dict) – unpacked dict arguments

extra_repr ( ) extra_repr¶: Extra representation of this layer, you can have custom implementation of your own layer.

RReLU¶

forward¶

extra_repr¶