[AutoParallel] convert distensor for eager custom op #59137
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…b.com/wanghuancoder/Paddle into convert_disttensor_for_eager_custom_op
|
你的PR提交成功，感谢你对开源项目的贡献! |
…b.com/wanghuancoder/Paddle into convert_disttensor_for_eager_custom_op
jiahy0825
reviewed
Nov 20, 2023
| @@ -44,7 +44,7 @@ void ShareTensor(PyObject* src, PyObject* dst) { | |||
| } | |||
| } | |||
|
|
|||
| paddle::Tensor CastPyArg2Tensor(PyObject* obj, Py_ssize_t arg_pos) { | |||
| paddle::Tensor& CastPyArg2Tensor(PyObject* obj, Py_ssize_t arg_pos) { | |||
There was a problem hiding this comment.
因为需要对PyObject*里的Tensor做原位修改，如果返回复制对象，则不能做原位修改。
| paddle::Tensor tensor = | ||
| std::move(CastPyArg2Tensor(obj, i + 1)); // NOLINT | ||
| ctx.EmplaceBackInput(std::move(tensor)); | ||
| paddle::Tensor& tensor = CastPyArg2Tensor(obj, i + 1); // NOLINT |
There was a problem hiding this comment.
Suggested change
| paddle::Tensor& tensor = CastPyArg2Tensor(obj, i + 1); // NOLINT | |
| const paddle::Tensor& tensor = CastPyArg2Tensor(obj, i + 1); // NOLINT |
Comment on lines
587
to
613
| const phi::distributed::ProcessMesh* mesh = nullptr; | ||
| if (InputsContainDistTensor(&mesh, *(ctx.AllMutableInput()))) { | ||
| ctx.AllMutableInput()->clear(); | ||
| for (size_t i = 0; i < inputs.size(); ++i) { | ||
| const auto& input = inputs.at(i); | ||
| // Parse op_type first, so that use i + 1 | ||
| PyObject* obj = PyTuple_GET_ITEM(args, i + 1); | ||
| // Emplace Py_None from python, this means optional inputs passed to C++, | ||
| // use one un-initialized tensor to indicate both Tensor and | ||
| // vector<Tensor> inputs. | ||
| if (obj == Py_None) { | ||
| VLOG(7) << "Custom operator add input " << input | ||
| << " to CustomOpKernelContext. Add un-initialized tensor " | ||
| "because the optional input is None"; | ||
| ctx.EmplaceBackInput(std::move(paddle::Tensor())); | ||
| continue; | ||
| } | ||
| if (paddle::framework::detail::IsDuplicableVar(input)) { | ||
| std::vector<paddle::Tensor> tensors = | ||
| std::move(CastPyArg2VectorOfTensor(obj, i + 1, mesh)); // NOLINT | ||
| ctx.EmplaceBackInputs(std::move(tensors)); | ||
| VLOG(7) << "Custom operator add input " << input | ||
| << " to CustomOpKernelContext. Add vector<Tensor> size = " | ||
| << ctx.InputRangeAt(i).second - ctx.InputRangeAt(i).first; | ||
| } else { | ||
| paddle::Tensor& tensor = CastPyArg2Tensor(obj, i + 1); // NOLINT | ||
| ConvertAllInputsToDistTensor(mesh, tensor); |
There was a problem hiding this comment.
这里的代码和前面对 input 的处理，重复度很高，可以合在一起，或者写成一个函数吗？
这里也用了很多 Tensor&，建议改成 const Tensor&
There was a problem hiding this comment.
我想了一下，没办法合并或者写成函数。
不能合并的原因是：可能传入的参数中，中间的一个Tensor是DistTensor，我需要重新全部扫描一遍，如果Tensor不是DistTensor则转为DistTensor，如果是DistTensor，则要求他们的mesh相等。必须重新来一遍。
不能写成函数的原因是：两个代码的结构虽然一样，但到具体执行的时候有差别。不适合1个函数，强行写成1个函数只能是可读性更差。
… convert_disttensor_for_eager_backward
…b.com/wanghuancoder/Paddle into convert_disttensor_for_eager_custom_op
…b.com/wanghuancoder/Paddle into convert_disttensor_for_eager_custom_op
… convert_disttensor_for_eager_custom_op
… convert_disttensor_for_eager_custom_op
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Description
自定义算子，如果有1个Tensor为DistTensor，则全部转换为DistTensor
Pcard-73145