AdaptiveLogSoftmaxWithLoss

class paddle.nn. AdaptiveLogSoftmaxWithLoss ( in_features, n_classes, cutoffs, weight_attr=None, bias_attr=None, div_value=4.0, head_bias=False, name=None ) [source]

Adaptive softmax is an approximate strategy for training models with large output spaces. It is most effective when the label distribution is highly imbalanced, for example in natural language modelling, where the word frequency distribution approximately follows the Zipf’s law.

Adaptive softmax partitions the labels into several clusters, according to their frequency. These clusters may contain different number of targets each. Additionally, clusters containing less frequent labels assign lower dimensional embeddings to those labels, which speeds up the computation. For each minibatch, only clusters for which at least one target is present are evaluated.

The idea is that the clusters which are accessed frequently (like the first one, containing most frequent labels), should also be cheap to compute – that is, contain a small number of assigned labels. We highly recommend taking a look at the original paper for more details.

For cutoffs should be an ordered Sequence of integers sorted in the increasing order. It controls number of clusters and the partitioning of targets into clusters. For example setting cutoffs = [10, 100, 1000] means that first 10 targets will be assigned to the ‘head’ of the adaptive softmax, targets 11, 12, ..., 100 will be assigned to the first cluster, and targets 101, 102, ..., 1000 will be assigned to the second cluster, while targets 1001, 1002, ..., n_classes - 1 will be assigned to the last, third cluster.

For div_value is used to compute the size of each additional cluster, which is given as follow:

\[\lfloor \frac{\text{in\_features}}{\text{div\_value}^{idx}} \rfloor\]

where \(idx\) is the cluster index (with clusters for less frequent words having larger indices, and indices starting from \(1\)).

For head_bias if set to True, adds a bias term to the ‘head’ of the adaptive softmax. See paper for details. Set to False in the official implementation.

Parameters

  • in_features (int) – Number of features in the input tensor.

  • n_classes (int) – Number of classes in the dataset.

  • cutoffs (Sequence) – Cutoffs used to assign targets to their buckets.

  • weight_attr (ParamAttr, optional) – The attribute for the learnable weight of this layer. The default value is None. If the Initializer of the param_attr is not set, the parameter is initialized with Xavier. For detailed information, please refer to ParamAttr.

  • bias_attr (ParamAttr|bool, optional) – The attribute for the learnable bias of this layer. If it is set to False, no bias will be added to the output. If it is set to None or one kind of ParamAttr, a bias parameter will be created according to ParamAttr. For detailed information, please refer to ParamAttr. The default value is None and the bias will be initialized to zero.

  • div_value (float, optional) – value used as an exponent to compute sizes of the clusters. Default: 4.0.

  • head_bias (bool, optional) – If True, adds a bias term to the ‘head’ of the adaptive softmax. Default: False.

  • name (str, optional) – Name for the operation (optional, default is None). For more information, please refer to Name.

Shape:

  • input (Tensor): The input tensor. The shapes is [N, in_features]. N is batch size.

  • label (Tensor): target. The shapes is [N]

  • output1 (Tensor): The shape is [N]

  • output2 (Scalar).

Returns

A callable object of AdaptiveLogSoftmaxWithLoss.

Examples

>>> import paddle
>>> import paddle.nn as nn
>>> paddle.seed(2024)

>>> input = paddle.randn([3, 5], dtype="float32")
>>> target = paddle.full((3,), 1, dtype='int64')
>>> asfm = nn.AdaptiveLogSoftmaxWithLoss(in_features=5, n_classes=3, cutoffs=[
                      2], div_value=2.0, head_bias=False)
>>> out, loss = asfm(input, target)
>>> print(out)
Tensor(shape=[3], dtype=float32, place=Place(cpu), stop_gradient=False,
[-1.04691017, -0.42341536, -1.16909981])
>>> print(loss)
Tensor(shape=[], dtype=float32, place=Place(cpu), stop_gradient=False,
0.87980843)
>>> out = asfm.log_prob(input)
>>> print(out)
Tensor(shape=[3, 3], dtype=float32, place=Place(cpu), stop_gradient=False,
[[-1.13710010, -1.04691017, -1.11403584],
[-1.51841831, -0.42341536, -2.07040048],
[-4.25405550, -1.16909981, -0.39282480]])
>>> out = asfm.predict(input)
>>> print(out)
Tensor(shape=[3], dtype=float32, place=Place(cpu), stop_gradient=True,
[1., 1., 2.])

Note

Labels passed as inputs to this module should be sorted according to their frequency. This means that the most frequent label should be represented by the index 0, and the least frequent label should be represented by the index n_classes - 1. To compute log-probabilities for all classes, the log_prob method can be used.

forward ( input, label )

forward

Defines the computation performed at every call. Should be overridden by all subclasses.

Parameters

  • *inputs (tuple) – unpacked tuple arguments

  • **kwargs (dict) – unpacked dict arguments