Initialize large table value randomly #9787
Conversation
…selected_rows_value
…ddle into random_selected_rows_value
Yancey1989
changed the title
| // Only allocate the memory of large table on CPU | ||
| auto cpu = platform::CPUPlace(); | ||
| float *data = tensor->mutable_data<float>(cpu); | ||
| VLOG(3) << "generate seed"; |
| auto tensor = out->mutable_value(); | ||
| tensor->Resize(framework::make_ddim(shape)); | ||
| // Only allocate the memory of large table on CPU | ||
| auto cpu = platform::CPUPlace(); |
There was a problem hiding this comment.
Do we need to enforce that dev_place is on CPU?
| for (int64_t idx = 1; idx < rows_size; ++idx) { | ||
| (*rows)[idx] = (*rows)[idx - 1] + shard_cnt; | ||
| } | ||
| out->set_height(max_id); |
There was a problem hiding this comment.
As discussed offline, with @Yancey1989 @jacquesqiao , need to implement this using an auto-growth SelectedRows, and init it randomly with a startup init buffer size, like 128M. This feature may also need to implement operations that when "Prefetch" runs.
There was a problem hiding this comment.
When lookup table op find that some id is not in the rows of the table, it should push this new id to the table.rows to init it.
There was a problem hiding this comment.
@jacquesqiao Yes, and I have an offline discussion with @typhoonzero, maybe we need a new class to represent the Table, and I create an issue #9841 for the details if you also agree with that, I will create a new PR to implement it.
There was a problem hiding this comment.
FROM @typhoonzero
and init it randomly with a startup init buffer size, like 128M
We have already had an attribute shape to represent the shape of the SelectedRows.value(), maybe we can only use the shape to determinate the memory size, use shape and memory size at the same time may be confused.
There was a problem hiding this comment.
And maybe we resue uniform_random_op, they have the same computing logic.
| tensor->Resize(framework::make_ddim(shape)); | ||
| // Only allocate the memory of large table on CPU | ||
| auto cpu = platform::CPUPlace(); | ||
| float *data = tensor->mutable_data<float>(cpu); |
There was a problem hiding this comment.
This float should be a template
| auto shape = Attr<std::vector<int>>("shape"); | ||
|
|
||
| auto tensor = out->mutable_value(); | ||
| tensor->Resize(framework::make_ddim(shape)); |
There was a problem hiding this comment.
The first dimension of the tensor is not sure, should be calculated by some attribute, such as id_num / shard_cnt + buffer_size. Do you mean that this calculation is done in Python ?
There was a problem hiding this comment.
Do you mean that this calculation is done in Python
I think so, but as the discussion just now #9787 (comment), I will update this PR according to it.
|
@jacquesqiao @typhoonzero I created an issue #9841 and will implement it ASAP. |
| tensor = out_var->GetMutable<framework::SelectedRows>()->mutable_value(); | ||
| tensor->Resize(framework::make_ddim(shape)); | ||
| } else { | ||
| PADDLE_THROW("Only support SelectedRows and Tensor"); |
There was a problem hiding this comment.
uniform_random_op's output only support...
| @@ -24,7 +24,17 @@ template <typename T> | |||
| class CPUUniformRandomKernel : public framework::OpKernel<T> { | |||
| public: | |||
| void Compute(const framework::ExecutionContext& ctx) const override { | |||
| auto* tensor = ctx.Output<framework::Tensor>("Out"); | |||
| framework::Tensor* tensor(nullptr); | |||
There was a problem hiding this comment.
framework::Tensor *tensor = nullptr;
Fixed #9777
issue: #9211