[WIP] C++ implementation of parallel executor #9035
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
March 14, 2018 00:11
… for record io reader; use double buffer reader
panyx0718
reviewed
Mar 27, 2018
|
|
||
| for (auto& op : ctx->ops_) { | ||
| // sgd should wait for allreduce finished | ||
| for (auto& param2argu : op->Inputs()) { |
There was a problem hiding this comment.
Instead of looping every input every time, perhaps the op can cache the param inputs and only wait for them
|
|
||
| PADDLE_ENFORCE( | ||
| cudaEventRecord(computation_event[argu], computation_stream)); | ||
| PADDLE_ENFORCE(cudaStreamWaitEvent(all_reduce_stream, |
There was a problem hiding this comment.
This seems to block the next computation op. We should profile and see how much it hurts
| @@ -86,6 +88,7 @@ class Scope { | |||
| mutable std::unordered_map<std::string, Variable*> vars_; | |||
| mutable std::list<Scope*> kids_; | |||
| Scope const* parent_{nullptr}; | |||
| std::vector<std::shared_ptr<Scope>> replicas_; | |||
There was a problem hiding this comment.
This seems a little strange. why would the Scope keeps all the replicas? It's not a singleton?
| @@ -457,12 +457,39 @@ class BatchNormGradKernel<platform::CPUDeviceContext, T> | |||
| } | |||
| }; | |||
|
|
|||
| class BatchNormGradMaker : public framework::SingleGradOpDescMaker { | |||
There was a problem hiding this comment.
Is this a fix? Should it be checked-in separately?
| if (!(ins[i].place() == dev_place)) { | ||
| LOG(INFO) << "Copy " << out_arg_names[i] << " from " << ins[i].place() | ||
| << " to " << dev_place; | ||
| framework::TensorCopy(ins[i], dev_place, out); |
There was a problem hiding this comment.
Does this ensure different gpu get different data?
|
|
||
| if not isinstance(program, Program): | ||
| raise TypeError() | ||
| if not isinstance(fetch_list, dict): |
| std::unordered_set<std::string>* param_grads_; | ||
| }; | ||
|
|
||
| class MultiGPUExecutor { |
There was a problem hiding this comment.
Can we extend it to be MultiCPUThreadExecutor??
command: CUDA_VISIBLE_DEVICES=3 python -m unittest test_parallel_executor.TestResnet
|
I am closing this pr since its alternative #9080 will be merged. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE THIS PR!
This PR will serve a baseline to #9080. The main difference is that:
In this PR, one thread is bound to one GPU. Each thread launches all Ops sequentially in the computation stream and launches AllReduce in the io stream. CudaEvent is used for coordination between streams.
In #9080, a dependency parsing is used for scheduling the ready Ops to a thread pool.
machine: 250
test_script: test_parallel_executor.py in this PR
test_command:
CUDA_VISIBLE_DEVICES=3 python -m unittest test_parallel_executor.TestResnetCUDA_VISIBLE_DEVICES=3,4,5,6 python -m unittest test_parallel_executor.TestResnetmodel: SE_ResNeXt152
batch_size: 16 per GPU
model size: 1382651904
peak memory: 7351879168
20.7775 Instance per second
60.8804 Instance per second