ParallelMode¶

class paddle.distributed. ParallelMode [source]

There are all the parallel modes currently supported:

• DATA_PARALLEL: Distribute input data to different devices.

• TENSOR_PARALLEL: Shards tensors in the network to different devices.

• PIPELINE_PARALLEL: Place different layers of the network on different devices.

• SHARDING_PARALLEL: Segment the model parameters, parameter gradients and optimizer states corresponding to the parameters to each device.

Examples

import paddle
parallel_mode = paddle.distributed.ParallelMode
print(parallel_mode.DATA_PARALLEL)  # 0