DataFeedDesc¶

class paddle.fluid.data_feed_desc. DataFeedDesc ( proto_file ) [source]

Api_attr

Static Graph

Datafeed descriptor, describing input training data format. This class is currently only used for AsyncExecutor (See comments for class AsyncExecutor for a brief introduction)

DataFeedDesc shall be initialized from a valid protobuf message from disk.

See paddle/fluid/framework/data_feed.proto for message definition. A typical message might look like:

import paddle.fluid as fluid
f = open("data.proto", "w")
print >> f, 'name: "MultiSlotDataFeed"'
print >> f, 'batch_size: 2'
print >> f, 'multi_slot_desc {'
print >> f, '    slots {'
print >> f, '         name: "words"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '     }'
print >> f, '     slots {'
print >> f, '         name: "label"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '    }'
print >> f, '}'
f.close()
data_feed = fluid.DataFeedDesc('data.proto')

However, users usually shouldn’t care about the message format; instead, they are encouraged to use Data Generator as a tool to generate a valid data description, in the process of converting their raw log files to training files acceptable to AsyncExecutor.

DataFeedDesc can also be changed during runtime. Once you got familiar with what each field mean, you can modify it to better suit your need. E.g.:

import paddle.fluid as fluid
data_feed = fluid.DataFeedDesc('data.proto')
data_feed.set_batch_size(128)
data_feed.set_dense_slots('wd')  # The slot named 'wd' will be dense
data_feed.set_use_slots('wd')    # The slot named 'wd' will be used

Finally, the content can be dumped out for debugging purpose:

print(data_feed.desc())

Parameters

proto_file (string) – Disk file containing a data feed description.

set_batch_size ( batch_size )

set_batch_size¶

Set batch_size in api_fluid_DataFeedDesc . batch_size can be changed during training.

Example

import paddle.fluid as fluid
f = open("data.proto", "w")
print >> f, 'name: "MultiSlotDataFeed"'
print >> f, 'batch_size: 2'
print >> f, 'multi_slot_desc {'
print >> f, '    slots {'
print >> f, '         name: "words"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '     }'
print >> f, '     slots {'
print >> f, '         name: "label"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '    }'
print >> f, '}'
f.close()
data_feed = fluid.DataFeedDesc('data.proto')
data_feed.set_batch_size(128)

Parameters

batch_size (int) – The number of batch size.

Returns

None.

set_dense_slots ( dense_slots_name )

set_dense_slots¶

Set slots in dense_slots_name as dense slots. Note: In default, all slots are sparse slots.

Features for a dense slot will be fed into a Tensor, while those for a sparse slot will be fed into a LoDTensor.

Example

import paddle.fluid as fluid
f = open("data.proto", "w")
print >> f, 'name: "MultiSlotDataFeed"'
print >> f, 'batch_size: 2'
print >> f, 'multi_slot_desc {'
print >> f, '    slots {'
print >> f, '         name: "words"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '     }'
print >> f, '     slots {'
print >> f, '         name: "label"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '    }'
print >> f, '}'
f.close()
data_feed = fluid.DataFeedDesc('data.proto')
data_feed.set_dense_slots(['words'])

Parameters

dense_slots_name (list(str)) – a list of slot names which will be set dense.

Returns

None.

set_use_slots ( use_slots_name )

set_use_slots¶

Set if a specific slot will be used for training. A dataset shall contain a lot of features, through this function one can select which ones will be used for a specific model.

Example

import paddle.fluid as fluid
f = open("data.proto", "w")
print >> f, 'name: "MultiSlotDataFeed"'
print >> f, 'batch_size: 2'
print >> f, 'multi_slot_desc {'
print >> f, '    slots {'
print >> f, '         name: "words"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '     }'
print >> f, '     slots {'
print >> f, '         name: "label"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '    }'
print >> f, '}'
f.close()
data_feed = fluid.DataFeedDesc('data.proto')
data_feed.set_use_slots(['words'])

Parameters

use_slots_name – a list of slot names which will be used in training

Note

Default is not used for all slots

desc ( )

desc¶

Returns a protobuf message for this DataFeedDesc

Example

import paddle.fluid as fluid
f = open("data.proto", "w")
print >> f, 'name: "MultiSlotDataFeed"'
print >> f, 'batch_size: 2'
print >> f, 'multi_slot_desc {'
print >> f, '    slots {'
print >> f, '         name: "words"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '     }'
print >> f, '     slots {'
print >> f, '         name: "label"'
print >> f, '         type: "uint64"'
print >> f, '         is_dense: false'
print >> f, '         is_used: true'
print >> f, '    }'
print >> f, '}'
f.close()
data_feed = fluid.DataFeedDesc('data.proto')
print(data_feed.desc())

Returns

A string message