swiglu

paddle.incubate.nn.functional. swiglu ( x: Tensor, y: Tensor | None = None, name: str | None = None ) → Tensor [source]

This function performs SwiGLU activation to the input Tensor.

$$out = silu(x) * y when y is not None out = silu(xs[0]) * xs[1] when y is None, where xs = paddle.chunk(x, 2, axis=-1)$$

Parameters

• x (Tensor) – The first input Tensor of SwiGLU.

• y (Tensor, optional) – The second input Tensor of SwiGLU. Default: None.

• name (str, optional) – For details, please refer to Name. Generally, no setting is required. Default: None.

Returns

A Tensor with the same data type with x and y.

Examples

>>> import paddle
>>> import paddle.incubate.nn.functional as F
>>> x = paddle.to_tensor([1, 2], dtype='float32')
>>> out1, out2 = F.swiglu(x), F.swiglu(x, x)
>>> print(out1, out2)
Tensor(shape=[1], dtype=float32, place=Place(cpu), stop_gradient=True,
       [1.46211720]) Tensor(shape=[2], dtype=float32, place=Place(cpu), stop_gradient=True,
       [0.73105860, 3.52318811])