data_norm

paddle.static.nn. data_norm ( input, act=None, epsilon=1e-05, param_attr=None, data_layout='NCHW', in_place=False, name=None, moving_mean_name=None, moving_variance_name=None, do_model_average_for_mean_and_var=True, slot_dim=- 1, sync_stats=False, summary_decay_rate=0.9999999, enable_scale_and_shift=False ) [source]
Api_attr

Static Graph

Data Normalization Layer

This op can be used as a normalizer function for conv2d and fully_connected operations. The required data format for this layer is one of the following:

  1. NHWC [batch, in_height, in_width, in_channels]

  2. NCHW [batch, in_channels, in_height, in_width]

\(input\) is the input features over a mini-batch.

\[\begin{split}\\mu_{\\beta} &\\gets \\frac{1}{m} \\sum_{i=1}^{m} x_i \\qquad &//\\ \ mini-batch\ mean \\\\ \\sigma_{\\beta}^{2} &\\gets \\frac{1}{m} \\sum_{i=1}^{m}(x_i - \\ \\mu_{\\beta})^2 \\qquad &//\ mini-batch\ variance \\\\ \\hat{x_i} &\\gets \\frac{x_i - \\mu_\\beta} {\\sqrt{\\ \\sigma_{\\beta}^{2} + \\epsilon}} \\qquad &//\ normalize \\\\ y_i &\\gets \\gamma \\hat{x_i} + \\beta \\qquad &//\ scale\ and\ shift\end{split}\]
Parameters

  • input (Tensor) – The input Tensor.

  • act (string, Default None) – Activation type, linear|relu|prelu|…

  • epsilon (float, Default 1e-05) –

  • param_attr (ParamAttr) – The parameter attribute for Parameter scale.

  • data_layout (str, optional) – Specify the data format of the input, and the data format of the output will be consistent with that of the input. An optional string from: “NCHW”, “NHWC”. The default is “NCHW”. When it is “NCHW”, the data is stored in the order of: [batch_size, input_channels, input_height, input_width].

  • in_place (bool, Default False) – Make the input and output of batch norm reuse memory.

  • name (string, Default None) – A name for this layer(optional). If set None, the layer will be named automatically.

  • moving_mean_name (string, Default None) – The name of moving_mean which store the global Mean.

  • moving_variance_name (string, Default None) – The name of the moving_variance which store the global Variance.

  • do_model_average_for_mean_and_var (bool, Default True) – Whether parameter mean and variance should do model average when model average is enabled.

  • slot_dim (int) – The embedding dimension of one slot. Slot is a set of one specific feature. In pslib mode, we distinguish feature ids by slot and pull their embeddings from parameter server (pslib). The first place of the embedding is the historical show number (occurence time of this feature id with a label 0). If the input of this op is concated by slot-wise embeddings, and the show number is zero when this slot is new or empty, the normalization result may be impractical. To avoid this, we add slot_dim to locate the show number and judge if the show number is zero. If so, we choose to skip normalization on this embedding.

  • sync_stats (bool, Default False) – When running with multiple GPU cards, using allreduce to sync the summary messages.

  • summary_decay_rate (float, Default 0.9999999) – The decay rate when updating summary.

  • enable_scale_and_shift (bool, Default False) – do scale&shift after normalization.

Returns

A tensor which is the result after applying data normalization on the input.

Return type

Tensor

Examples

import paddle
paddle.enable_static()

x = paddle.randn(shape=[32,100])
hidden2 = paddle.static.nn.data_norm(input=x)