alltoall

paddle.distributed.communication.stream. alltoall ( out_tensor_or_tensor_list, in_tensor_or_tensor_list, group=None, sync_op=True, use_calc_stream=False ) [source]

Scatter a tensor (or a tensor list) across devices and gather outputs to another tensor (or a tensor list, respectively).

Parameters

  • out_tensor_or_tensor_list (Union[Tensor, List[Tensor]]) – The output. If it is a tensor, it should be correctly-sized.

  • list (If it is a) –

  • input. (it should be empty or contain correctly-sized tensors. Its data type should be the same as the) –

  • in_tensor_or_tensor_list (Union[Tensor, List[Tensor]]) – The input to scatter (must be specified on the source rank). If it is a tensor, it should be correctly-sized. If it is a list, it should contain correctly-sized tensors. Support float16, float32, float64, int32, int64, int8, uint8 or bool as the input data type.

  • group (Group, optional) – Communicate in which group. If none is given, use the global group as default.

  • sync_op (bool, optional) – Indicate whether the communication is sync or not. If none is given, use true as default.

  • use_calc_stream (bool, optional) – Indicate whether the communication is done on calculation stream. If none is given, use false as default. This option is designed for high performance demand, be careful to turn it on except you are clearly know its meaning.

Returns

Return a task object.

Examples

>>> 
>>> import paddle
>>> import paddle.distributed as dist

>>> dist.init_parallel_env()
>>> out_tensor_list = []
>>> if dist.get_rank() == 0:
...     data1 = paddle.to_tensor([[1, 2, 3], [4, 5, 6]])
...     data2 = paddle.to_tensor([[7, 8, 9], [10, 11, 12]])
>>> else:
...     data1 = paddle.to_tensor([[13, 14, 15], [16, 17, 18]])
...     data2 = paddle.to_tensor([[19, 20, 21], [22, 23, 24]])
>>> task = dist.stream.alltoall(out_tensor_list, [data1, data2], sync_op=False)
>>> task.wait()
>>> print(out_tensor_list)
>>> # [[[1, 2, 3], [4, 5, 6]], [[13, 14, 15], [16, 17, 18]]]    (2 GPUs, out for rank 0)
>>> # [[[7, 8, 9], [10, 11, 12]], [[19, 20, 21], [22, 23, 24]]] (2 GPUs, out for rank 1)