How to write a new operator¶

Defining ProtoMaker¶

Matrix Multiplication can be written as $Out = X * Y$, meaning that the operation consists of two inputs and one output.

First, define ProtoMaker to describe the Operator’s input, output, and additional comments:

class MulOpMaker : public framework::OpProtoAndCheckerMaker {
 public:
  MulOpMaker(OpProto *proto, OpAttrChecker *op_checker)
      : OpProtoAndCheckerMaker(proto, op_checker) {
    AddInput("X", "(Tensor), 2D tensor of size (M x K)");
    AddInput("Y", "(Tensor), 2D tensor of size (K x N)");
    AddOutput("Out", "(Tensor), 2D tensor of size (M x N)");
    AddComment(R"DOC(
Two Element Mul Operator.
The equation is: Out = X * Y
)DOC");
  }
};

MulOpMakeris inherited fromframework::OpProtoAndCheckerMaker, consisting of 2 variables in the constructor：

• framework::OpProto stores Operator input and variable attribute, used for generating Python API interfaces.

• framework::OpAttrChecker is used to validate variable attributes.

The constructor utilizes AddInput to add input parameter, AddOutput to add output parameter, and AddComment to add comments for the Op, so that the corresponding information will be added to OpProto.

The code above adds two inputs X and Y to MulOp, an output Out, and their corresponding descriptions. Names are given in accordance to Paddle’s naming convention.

An additional example ScaleOp is implemented as follows:

template <typename AttrType>
class ScaleOpMaker : public framework::OpProtoAndCheckerMaker {
 public:
  ScaleOpMaker(OpProto *proto, OpAttrChecker *op_checker)
      : OpProtoAndCheckerMaker(proto, op_checker) {
    AddInput("X", "(Tensor) Input tensor of scale operator.");
    AddOutput("Out", "(Tensor) Output tensor of scale operator.");
    AddComment(R"DOC(
Scale operator
$$Out = scale*X$$
)DOC");
    AddAttr<AttrType>("scale",
                      "(float, default 1.0)"
                      "The scaling factor of the scale operator.")
        .SetDefault(1.0);
  }
};

Note AddAttr<AttrType>("scale", "...").SetDefault(1.0); adds scaleconstant as an attribute, and sets the default value to 1.0.

Defining the GradProtoMaker class¶

Each Op must have a corresponding GradProtoMaker. If GradProtoMaker corresponding to the forward Op is not customized, Fluid provides DefaultGradProtoMaker. The default registration will use all input and output, including Input, Output, Output@Grad and so on. Using unnecessary variables will cause waste of memory. The following example defines ScaleOp’s GradProtoMaker.

class ScaleGradMaker : public framework::SingleGradOpDescMaker {
 public:
  using framework::SingleGradOpDescMaker::SingleGradOpDescMaker;

  std::unique_ptr<framework::OpDesc> Apply() const override {
    auto *grad_op = new framework::OpDesc();
    grad_op->SetType("scale");
    grad_op->SetInput("X", OutputGrad("Out"));
    grad_op->SetOutput("Out", InputGrad("X"));
    grad_op->SetAttr("scale", GetAttr("scale"));
    return std::unique_ptr<framework::OpDesc>(grad_op);
  }
};

Defining Operator¶

The following code defines the interface for MulOp:

class MulOp : public framework::OperatorWithKernel {
 public:
  using framework::OperatorWithKernel::OperatorWithKernel;

 protected:
  void InferShape(const framework::InferShapeContext &ctx) const override {
    //never use Input<Tensor> or Output<Tensor> if you want a to get a LoDTensor.
    auto dim0 = ctx.Input<LoDTensor>("X")->dims();
    auto dim1 = ctx.Input<LoDTensor>("Y")->dims();
    PADDLE_ENFORCE_EQ(dim0.size(), 2,
                      "input X(%s) should be a tensor with 2 dims, a matrix",
                      ctx.op_.Input("X"));
    PADDLE_ENFORCE_EQ(dim1.size(), 2,
                      "input Y(%s) should be a tensor with 2 dims, a matrix",
                      ctx.op_.Input("Y"));
    PADDLE_ENFORCE_EQ(
        dim0[1], dim1[0],
        "First matrix's width must be equal with second matrix's height.");
    ctx.Output<LoDTensor>("Out")->Resize({dim0[0], dim1[1]});
  }
};

MulOp is inherited from OperatorWithKernel. Its public member

using framework::OperatorWithKernel::OperatorWithKernel;

expresses an operator constructor using base class OperatorWithKernel, alternatively written as

MulOp(const std::string &type, const framework::VariableNameMap &inputs,
      const framework::VariableNameMap &outputs,
      const framework::AttributeMap &attrs)
  : OperatorWithKernel(type, inputs, outputs, attrs) {}

InferShape interface needs to be re-written.InferShape is a const method and cannot modify Op’s member variables. Its constant member const framework::InferShapeContext &ctx can be used to extract input, output, and attributes. Its functions are

• 1). validate and error out early: it checks input data dimensions and types.

• 2). configures the tensor shape in the output.

Usually OpProtoMaker and Op definitions are written in .cc files, which also include the registration methods introduced later.

Defining OpKernel¶

MulKernel is derived from framework::OpKernel, which includes the following templates:

• typename  DeviceContext denotes device context type. When different devices, namely the CPU and the CUDA, share the same kernel, this template needs to be added. If they don’t share kernels, this must not be added. An example of a non-sharing kernel is OnehotCrossEntropyOpKernel.

• typename T denotes data type, such as float or double.

MulKernel types need to rewrite the interface for Compute.

• Compute takes one input parameter: const framework::ExecutionContext& context.

• Compared with InferShapeContext, ExecutionContext includes device types, and can similarly extract input, output, and attribute variables.

• Compute function implements the computation logics of an OpKernel.

The input and output of Op can be obtained by ExecutionContext::Input<T>() and ExecutionContext::Output<T>() respectively.

Note: If the input/output variable type of op is LoDTensor (In Fluid, all Tensors are LoDTensor type by default), please write ExecutionContext::Input<LoDTensor>() and ExecutionContext:: Output<LoDTensor>(), do not write ExecutionContext::Input<Tensor>() and ExecutionContext::Output<Tensor>(). Because if the actual variable type is SelectedRows, the Input<Tensor>() and Output<Tensor>() methods will specialize the SelectedRows type to Tensor, causing a potential error.

MulKernel’s implementation of Compute is as follows:

template <typename DeviceContext, typename T>
class MulKernel : public framework::OpKernel {
public:
void Compute(const framework::ExecutionContext& context) const override {
  auto* X = context.Input<LoDTensor>("X");
  auto* Y = context.Input<LoDTensor>("Y");
  auto* Z = context.Output<LoDTensor>("Out");
  Z->mutable_data<T>(context.GetPlace());
  auto& device_context = context.template device_context<DeviceContext>();
  math::matmul<DeviceContext, T>(*X, false, *Y, false, 1, Z, 0, device_context);
}
};

Note that different devices (CPU, CUDA)share one Op definition; whether or not they share the same OpKernel depends on whether functions called by Computecan support both devices.

MulOp’s CPU and CUDA share the same Kernel. A non-sharing OpKernel example can be seen in OnehotCrossEntropyOpKernel.

To ease the writing of OpKernel compute, and for reusing code cross-device, Eigen-unsupported Tensor module is used to implement Compute interface. To learn about how the Eigen library is used in PaddlePaddle, please see usage document.

This concludes the forward implementation of an operator. Next its operation and kernel need to be registered in a .cc file.

The definition of its corresponding backward operator, if applicable, is similar to that of an forward operator. Note that a backward operator does not include a ProtoMaker.

Registering Operator and OpKernel¶

• In .cc files, register forward and backward operator classes and the CPU kernel.

  namespace ops = paddle::operators;
  REGISTER_OPERATOR(mul, ops::MulOp, ops::MulOpMaker,
                paddle::framework::DefaultGradOpDescMaker<true>)
  REGISTER_OPERATOR(mul_grad, ops::MulGradOp)
  REGISTER_OP_CPU_KERNEL(mul, ops::MulKernel<paddle::platform::CPUDeviceContext, float>);
  REGISTER_OP_CPU_KERNEL(mul_grad,
                ops::MulGradKernel<paddle::platform::CPUDeviceContext, float>);
  

  In that code block,

  • REGISTER_OPERATOR registers the ops::MulOp class, with the type named mul. Its ProtoMaker is ops::MulOpMaker. Register ops::MulOpGrad as type named mul_grad.

  • REGISTER_OP_CPU_KERNEL registers ops::MulKernel class and specializes template parameters as type paddle::platform::CPUPlace and float, and also registers ops::MulGradKernel.

• Registering CUDA Kernel in .cu files

  • Note that if CUDA Kernel is implemented using the Eigen unsupported module, then on top of .cu, a macro definition #define EIGEN_USE_GPU is needed, such as

  // if use Eigen unsupported module before include head files
  #define EIGEN_USE_GPU
  
  namespace ops = paddle::operators;
  REGISTER_OP_CUDA_KERNEL(mul, ops::MulKernel<paddle::platform::CUDADeviceContext, float>);
  REGISTER_OP_CUDA_KERNEL(mul_grad,
                         ops::MulGradKernel<paddle::platform::CUDADeviceContext, float>);
  

Compilation¶

In folder build/paddle/fluid/operators, run the following commands to compile.

make mul_op

Unit Test for Forward Operators¶

The Op unit test is inherited from OpTest. More specific unit tests are done in TestMulOp. To test the Operator, you need to:

1. Define input, output, and related property parameters in the setUp function.

2. Generate random input data.

3. Implement the same calculation logic as the forward operator in the Python script to get the output, which is to be compared with the output of the forward operator calculation.

4. The backward calculation has been automatically integrated into the test framework and the corresponding interface can be called directly.

import unittest
import numpy as np
from op_test import OpTest


class TestMulOp(OpTest):
    def setUp(self):
        self.op_type = "mul"
        self.inputs = {
            'X': np.random.random((32, 84)).astype("float32"),
            'Y': np.random.random((84, 100)).astype("float32")
        }
        self.outputs = {'Out': np.dot(self.inputs['X'], self.inputs['Y'])}

    def test_check_output(self):
        self.check_output()

    def test_check_grad_normal(self):
        self.check_grad(['X', 'Y'], 'Out', max_relative_error=0.5)

    def test_check_grad_ingore_x(self):
        self.check_grad(
            ['Y'], 'Out', max_relative_error=0.5, no_grad_set=set("X"))

    def test_check_grad_ingore_y(self):
        self.check_grad(
            ['X'], 'Out', max_relative_error=0.5, no_grad_set=set('Y'))

The code above first loads required packages. In addition, we have

• self.op_type = "mul" defines the type that is identical to what the operator’s registered type.

• self.inputs defines input, with type numpy.array and initializes it.

• self.outputs defines output and completes the same operator computation in the Python script, and returns its result from the Python script.

Unit Test for Backward Operators¶

In the backward operator test:

• check_grad is called in test_check_grad_normal to use numerical methods to detect gradient correctness and stability.

• The first parameter ["X", "Y"] : specifies gradient check for the input variables X, Y.

• The second parameter "Out" : specifies the final output target variable Out of the forward network.

• The third parameter max_relative_error: specifies the maximum error value that can be tolerated when checking gradients.

• The test_check_grad_ingore_x and test_check_grad_ingore_y branches are used to test cases where only one input gradient needs to be calculated.

Compiling and Running¶

Any new unit testing file of the format test_*.py added to the directory python/paddle/fluid/tests/unittests/ is automatically added to the project to compile.

Note that running unit tests requires compiling the entire project and requires compiling with flag WITH_TESTING on i.e. cmake paddle_dir -DWITH_TESTING=ON.

After successfully compiling the project, run the following command to run unit tests:

make test ARGS="-R test_mul_op -V"

Or,

ctest -R test_mul_op

PADDLE_ENFORCE Usage Note¶

To check the validity of data when implementing Op, you need to use macro definitions such as PADDLE_ENFORCE and PADDLE_ENFORCE_EQ. The basic format is as follows:

PADDLE_ENFORCE (expression, error message)
PADDLE_ENFORCE_EQ (comparison object A, comparison object B, error message)

If the expression is true, or the comparison object A=B, the check will be passed, otherwise the program will be terminated and the corresponding error message will be fed back to the user. In order to ensure that the feedbacks are user-friendly and easy to understand, developers need to pay attention to how to use them.

PADDLE_ENFORCE(ctx->HasInput("X"), "");

PADDLE_ENFORCE(i != nullptr, "i must be set"); // What is i?

PADDLE_ENFORCE(forward_pd != nullptr,
"Fail to find eltwise_fwd_pd in device context"); //eltwise_fwd_pduser may not be understood

auto *out = ctx.Output<framework::LoDTensor>("Out");
auto *in = ctx.Input<framework::LoDTensor>("X");
out->ShareDataWith(*in);