[AutoParallel] remove pyreader, use feed op in pipeline schedule #56511
Conversation
|
你的PR提交成功，感谢你对开源项目的贡献! |
zhaoyinglia
changed the title
| def _validate_feed(self, feed): | ||
| if feed is None: | ||
| return [None] | ||
| if self._strategy.pipeline.enable: |
There was a problem hiding this comment.
为何pipeline.enable的时候不需要处理？pipeline.enable是否代表的不是开启pipeline的字面意思？
There was a problem hiding this comment.
strategy.pipeline.enable=True 表示开启了调度优化，需要配合设置 schedule_mode 使用。如果该策略为 False，仍然可以走pipeline切图，但是不会有任何调度优化
| assert ( | ||
| batch_size % self._acc_steps == 0 | ||
| ), "Requires batch_size:[{}] to be divisible by acc_steps:[{}].".format( | ||
| batch_size, self._acc_steps | ||
| ) | ||
| return batch_size // self._acc_steps | ||
|
|
||
| def _validate_feed(self, feed): |
There was a problem hiding this comment.
建议添加注释示例说明从feed到micro_feeds的数据形式变化和切分逻辑
There was a problem hiding this comment.
fix in next pr.
| micro_feed = ( | ||
| _as_lodtensor( | ||
| micro_cur_feed[i], self.place, var.dtype | ||
| ) | ||
| if num_micro_batch > 1 | ||
| else micro_cur_feed[i] | ||
| ) |
There was a problem hiding this comment.
| micro_feed = ( | |
| _as_lodtensor( | |
| micro_cur_feed[i], self.place, var.dtype | |
| ) | |
| if num_micro_batch > 1 | |
| else micro_cur_feed[i] | |
| ) | |
| micro_feed = ( | |
| _as_lodtensor( | |
| micro_cur_feed[i], self.place, var.dtype | |
| ) | |
| if not isinstance(micro_cur_feed[i], core.LoDTensor) | |
| else micro_cur_feed[i] | |
| ) |
There was a problem hiding this comment.
fix in next pr.
| void SetColAttrForFeedFetchOps(std::shared_ptr<ProgramDesc> program_desc, | ||
| const int64_t micro_batch_num, | ||
| const int64_t micro_batch_id) { | ||
| const std::set<std::string>& valid_fetch_op_types = { |
There was a problem hiding this comment.
| const std::set<std::string>& valid_fetch_op_types = { | |
| const std::set<std::string>& valid_feed_fetch_op_types = { |
There was a problem hiding this comment.
fix in next pr.
| const std::set<std::string>& valid_feed_fetch_op_types = {"fetch", | ||
| "fetch_v2"}; | ||
|
|
||
| const std::vector<int> all_op_ids = job.AllFetchOpIds(); |
There was a problem hiding this comment.
这里的逻辑删除后，job相关的几个接口ColAttrForFetchOp、AllFetchOpIds和SetColAttrForFetchOp也应一并删除。
There was a problem hiding this comment.
delete in next pr.
zhaoyinglia
changed the title
…dlePaddle#56511) * modify feed_data for dataloader in pipline parallel mode * add pre-commit * remove read op, use feed op * fix validate batch_size * tiny fix * support catch EOFException * fix conflict * fix conflict * fix executor if cond --------- Co-authored-by: Frida-a <2624653516@qq.com>
PR types
Others
PR changes
Others
Description
PCard-71568