HSigmoidLoss

class paddle.nn. HSigmoidLoss ( feature_size, num_classes, weight_attr=None, bias_attr=None, is_custom=False, is_sparse=False, name=None ) [source]

Hierarchical Sigmoid Layer.

The hierarchical sigmoid organizes the classes into a complete binary tree to reduce the computational complexity and speed up the model training, especially the training of language model. Each leaf node of the complete binary tree represents a class(word) and each non-leaf node acts as a binary classifier. For each class(word), there’s a unique path from root to itself, hsigmoid calculate the cost for each non-leaf node on the path, and sum them to get a total cost. Comparing to softmax, the OP can reduce the computational complexity from \(O(N)\) to \(O(logN)\), where \(N\) represents the number of classes or the size of word dict.

The OP supports default tree and custom tree. For the default tree, you can refer to Hierarchical Probabilistic Neural Network Language Model <http://www.iro.umontreal.ca/~lisa/pointeurs/hierarchical-nnlm-aistats05.pdf>_. For the custom tree, you need to set is_custom to True, and do the following steps (take the language model as an example):

  1. Using a custom word dict to build a binary tree, each leaf node should be an word in the word dict.

  2. Creating a dict map word_id -> path that from the word to the root node, we call it path_table.

  3. Creating a dict map word_id -> code of path that from the word to the root node, we call it path_code. Code means the label of each binary classifier, 1 indicate true, 0 indicate false.

  4. Now, each word should has its path and code along the path, you can pass a batch of path and code related to the same batch of inputs.

Parameters

  • feature_size (int) – The number of features.

  • num_classes (int) – The number of classes or the size of word dict, must be greater than 2. If the default tree is used (is_custom is set to False), num_classes should not be None. If the custom tree is used (is_custom is set to True), num_classes should be the number of non-leaf nodes, which indicates the num of classes using by the binary classifier.

  • weight_attr (ParamAttr, optional) – The parameter attribute for the learnable weights of hsigmoid. If it is set to None or one attribute of ParamAttr, hsigmoid will create a ParamAttr as param_attr. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. Default is None.

  • bias_attr (ParamAttr|bool, optional) – The parameter attribute for the bias of hsigmoid. If it is set to False, no bias will be added. If it is set to None or one attribute of ParamAttr, hsigmoid will create a ParamAttr as bias_attr. If the Initializer of the bias_attr is not set, the bias is initialized zero. Default is None.

  • is_custom (bool, optional) – Whether use custom binary tree. If it’s True, path_table and path_code should be passed to its forward method, otherwise path_table and path_code should not be passed to its forward method. Default is False.

  • is_sparse (bool, optional) – Whether use sparse updating instead of dense updating, if it’s True, the gradient of weight and input will be sparse. Default is False.

  • name (str, optional) – Name for the operation (optional, default is None). For more information, please refer to Name.

Shape:

input (Tensor): The input tensor. The shapes is [N, D], where N is batch size and D is feature size. It’s data type should be float32, float64. label (Tensor): It’s shapes is [N, 1]. It’s data type should be int64. output (Tensor): The HSigmoid Loss of input and label. Shape is [N, 1]

Examples

>>> import paddle
>>> paddle.set_device('cpu')
>>> paddle.seed(2023)
>>> input = paddle.uniform([4, 3])
>>> print(input)
Tensor(shape=[4, 3], dtype=float32, place=Place(cpu), stop_gradient=True,
[[ 0.73167229,  0.04029441, -0.48078126],
 [ 0.81050646, -0.15199822, -0.18717426],
 [ 0.94041789,  0.48874724,  0.03570259],
 [ 0.46585739,  0.95573163, -0.91368192]])
>>> label = paddle.to_tensor([0, 1, 4, 5])
>>> m = paddle.nn.HSigmoidLoss(3, 6)
>>> out = m(input, label)
>>> print(out)
Tensor(shape=[4, 1], dtype=float32, place=Place(cpu), stop_gradient=False,
[[1.94512916],
 [2.26129627],
 [2.36135936],
 [2.97453213]])
forward ( input, label, path_table=None, path_code=None )

forward

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments