inplace_abn

paddle.fluid.layers.nn. inplace_abn ( input, act=None, is_test=False, momentum=0.9, epsilon=1e-05, param_attr=None, bias_attr=None, data_layout='NCHW', name=None, moving_mean_name=None, moving_variance_name=None, do_model_average_for_mean_and_var=True, use_global_stats=False, act_alpha=1.0 ) [source]

In-place Activation Batch Normalization Layer

This layer calculates batch normalization and activation with in-place memory. For batch normalization calculations, see fluid.layers.batch_norm. For in-place activation batch normalization, see In-Place Activated BatchNorm for Memory-Optimized Training of DNNs

inplace_abn only support activation type as None, identity, leaky_relu, elu currently. inplace_abn only support data type as float32, float64 currently.

Note

if build_strategy.sync_batch_norm=True, the batch_norm in network will use sync_batch_norm automatically. is_test = True can only be used in test program and inference program, is_test CANNOT be set to True in train program, if you want to use global status from pre_train model in train program, please set use_global_stats = True.

Parameters

  • input (Variable) – The rank of input variable can be 2, 3, 4, 5. The data type is float16 or float32 or float64.

  • act (string, Default None) – Activation type, linear|relu|prelu|…

  • is_test (bool, Default False) – A flag indicating whether it is in test phrase or not.

  • momentum (float|Variable, Default 0.9) – The value used for the moving_mean and moving_var computation. This should be a float number or a Variable with shape [1] and data type as float32. The updated formula is: \(moving\_mean = moving\_mean * momentum + new\_mean * (1. - momentum)\) \(moving\_var = moving\_var * momentum + new\_var * (1. - momentum)\) Default is 0.9.

  • epsilon (float, Default 1e-05) – A value added to the denominator for numerical stability. Default is 1e-5.

  • param_attr (ParamAttr|None) – The parameter attribute for Parameter scale of inplace_abn. If it is set to None or one attribute of ParamAttr, inplace_abn will create ParamAttr as param_attr, the name of scale can be set in ParamAttr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default: None.

  • bias_attr (ParamAttr|None) – The parameter attribute for the bias of inplace_abn. If it is set to None or one attribute of ParamAttr, inplace_abn will create ParamAttr as bias_attr, the name of bias can be set in ParamAttr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default: None.

  • data_layout (str, optional) – Specify the data format of the input, and the data format of the output will be consistent with that of the input. An optional string from: “NCHW”, “NHWC”. The default is “NCHW”. When it is “NCHW”, the data is stored in the order of: [batch_size, input_channels, input_height, input_width].

  • name (str|None) – For detailed information, please refer to Name. Usually name is no need to set and None by default.

  • moving_mean_name (str, Default None) – The name of moving_mean which store the global Mean. If it is set to None, inplace_abn will save global mean with a random name, otherwise, inplace_abn will save global mean with the string.

  • moving_variance_name (str, Default None) – The name of the moving_variance which store the global Variance. If it is set to None, inplace_abn, will save global variance with a random name, otherwise, inplace_abn will save global variance with the string.

  • do_model_average_for_mean_and_var (bool, Default True) – Whether parameter mean and variance should do model average when model average is enabled.

  • use_global_stats (bool, Default False) – Whether to use global mean and variance. In inference or test mode, set use_global_stats to true or is_test to true, and the behavior is equivalent. In train mode, when setting use_global_stats True, the global mean and variance are also used during train period.

  • act_alpha (float, Default 1.0) – when activation is in [‘elu’, ‘identity’, ‘leaky_relu’], inplace activative batch normalization will be used, and alpha parameter for activation can be given by this parameter.

Returns

A Variable holding Tensor which is the result after applying batch normalization and activation on the input, has same shape and data type with input.

Examples

import paddle.fluid as fluid
x = fluid.data(name='x', shape=[3, 7, 3, 7], dtype='float32')
hidden1 = fluid.layers.fc(input=x, size=200, param_attr='fc1.w')
hidden2 = fluid.layers.inplace_abn(input=hidden1)
hidden3 = fluid.layers.inplace_abn(input=hidden2, act='leaky_relu', act_alpha=0.2)