DataFeeder

class paddle.fluid.data_feeder. DataFeeder ( feed_list, place, program=None ) [source]
Api_attr

Static Graph

DataFeeder converts the data that returned by a reader into a data structure that can feed into Executor. The reader is usually a python generator that returns a list of mini-batch data entries.

Parameters

  • feed_list (list) – Variables or names of Variables that need to feed.

  • place (api_fluid_CPUPlace | api_fluid_CUDAPlace) – place indicates the device (CPU | GPU) the data will be fed into, if you want to feed data into GPU, please using fluid.CUDAPlace(i) (i represents the GPU id), or if you want to feed data into CPU, please using fluid.CPUPlace().

  • program (api_fluid_Program , optional) – The Program that will feed data into, if program is None, it will use default_main_program(). Default None.

Raises

ValueError –

Example

import numpy as np
import paddle
import paddle.fluid as fluid

place = fluid.CPUPlace()
def reader():
    for _ in range(4):
        yield np.random.random([4]).astype('float32'), np.random.random([3]).astype('float32'),

main_program = fluid.Program()
startup_program = fluid.Program()

with fluid.program_guard(main_program, startup_program):
    data_1 = fluid.data(name='data_1', shape=[None, 2, 2], dtype='float32')
    data_2 = fluid.data(name='data_2', shape=[None, 1, 3], dtype='float32')
    out = fluid.layers.fc(input=[data_1, data_2], size=2)
    # ...
feeder = fluid.DataFeeder([data_1, data_2], place)

exe = fluid.Executor(place)
exe.run(startup_program)

feed_data = feeder.feed(reader())

# print feed_data to view feed results
# print(feed_data['data_1'])
# print(feed_data['data_2'])

outs = exe.run(program=main_program,
                feed=feed_data,
                fetch_list=[out])
print(outs)
feed ( iterable )

feed

According to feed_list of DataFeeder and iterable , converts the input into a data structure that can feed into Executor.

Parameters

iterable (generator) – user defined python generator to read the raw input data

Returns

a dict that contains (variable name - converted tensor) pairs

Return type

dict

Example

# In this example, reader - generator will return a list of ndarray of 3 elements
# feed API will convert each ndarray input into a tensor
# the return result is a dict with keys: data_1, data_2, data_3
# result['data_1']  a LoD-Tensor with shape of  [5, 2, 1, 3]. 5 is batch size, and [2, 1, 3] is the real shape of data_1.
# result['data_2'], result['data_3'] are similar.
import numpy as np
import paddle.fluid as fluid

def reader(limit=5):
    for i in range(1, limit + 1):
        yield np.ones([6]).astype('float32') * i , np.ones([1]).astype('int64') * i, np.random.random([9]).astype('float32')

data_1 = fluid.data(name='data_1', shape=[None, 2, 1, 3])
data_2 = fluid.data(name='data_2', shape=[None, 1], dtype='int64')
data_3 = fluid.data(name='data_3', shape=[None, 3, 3], dtype='float32')
feeder = fluid.DataFeeder(['data_1','data_2', 'data_3'], fluid.CPUPlace())


result = feeder.feed(reader())
print(result['data_1'])
print(result['data_2'])
print(result['data_3'])
feed_parallel ( iterable, num_places=None )

feed_parallel

Similar with feed function, feed_parallel is used with multiple devices (CPU|GPU). Here iterable is a list of python generators. The data return by each generator in the list will be fed into a separate device.

Parameters

  • iterable (list|tuple) – list of user-defined python generators. The element number should match the num_places.

  • num_places (int, optional) – the number of devices. If not provided (None), all available devices on the machine will be used. Default None.

Returns

a generator that generate dict which contains (variable name - converted tensor) pairs, the total number of dicts will be generated matches with the num_places

Return type

generator

Note

The number of devices - num_places should equal to the generator (element of iterable ) number

Example

import numpy as np
import paddle.fluid as fluid

def generate_reader(batch_size, base=0, factor=1):
    def _reader():
        for i in range(batch_size):
            yield np.ones([4]) * factor + base, np.ones([4]) * factor + base + 5
    return _reader()

x = fluid.data(name='x', shape=[None, 2, 2])
y = fluid.data(name='y', shape=[None, 2, 2], dtype='float32')

z = fluid.layers.elementwise_add(x, y)

feeder = fluid.DataFeeder(['x','y'], fluid.CPUPlace())
place_num = 2
places = [fluid.CPUPlace() for x in range(place_num)]
data = []
exe = fluid.Executor(fluid.CPUPlace())
exe.run(fluid.default_startup_program())
program = fluid.CompiledProgram(fluid.default_main_program()).with_data_parallel(places=places)

# print sample feed_parallel r result
# for item in list(feeder.feed_parallel([generate_reader(5, 0, 1), generate_reader(3, 10, 2)], 2)):
#     print(item['x'])
#     print(item['y'])

reader_list = [generate_reader(5, 0, 1), generate_reader(3, 10, 2)]
res = exe.run(program=program, feed=list(feeder.feed_parallel(reader_list, 2)), fetch_list=[z])
print(res)
decorate_reader ( reader, multi_devices, num_places=None, drop_last=True )

decorate_reader

Decorate the reader (generator) to fit multiple devices. The reader generate multiple mini-batches. Each mini-batch will be fed into a single device.

Parameters

  • reader (generator) – a user defined python generator used to get mini-batch of data. A mini-batch can be regarded as a python generator that returns batches of input entities, just like the below _mini_batch in the code example.

  • multi_devices (bool) – indicate whether to use multiple devices or not.

  • num_places (int, optional) – if multi_devices is True, you can specify the number of devices(CPU|GPU) to use, if multi_devices is None, the function will use all the devices of the current machine. Default None.

  • drop_last (bool, optional) – whether to drop the last round of data if it is not enough to feed all devices. Default True.

Returns

a new generator which return converted dicts that can be fed into Executor

Return type

generator

Raises

ValueError – If drop_last is False and the data cannot fit devices perfectly.

Example

import numpy as np
import paddle
import paddle.fluid as fluid
import paddle.fluid.compiler as compiler

def reader():
    def _mini_batch(batch_size):
        for i in range(batch_size):
            yield np.random.random([16]).astype('float32'), np.random.randint(10, size=[1])

    for _ in range(10):
        yield _mini_batch(np.random.randint(1, 10))

place_num = 3
places = [fluid.CPUPlace() for _ in range(place_num)]

# a simple network sample
data = fluid.data(name='data', shape=[None, 4, 4], dtype='float32')
label = fluid.data(name='label', shape=[None, 1], dtype='int64')
hidden = fluid.layers.fc(input=data, size=10)

feeder = fluid.DataFeeder(place=places[0], feed_list=[data, label])
reader = feeder.decorate_reader(reader, multi_devices=True, num_places=3, drop_last=True)

exe = fluid.Executor(places[0])
exe.run(fluid.default_startup_program())
compiled_prog = compiler.CompiledProgram(
         fluid.default_main_program()).with_data_parallel(places=places)

for i,data in enumerate(reader()):
    # print data if you like
    # print(i, data)
    ret = exe.run(compiled_prog, feed=data, fetch_list=[hidden])
    print(ret)