This repository has been archived by the owner on Sep 18, 2024. It is now read-only.
[quantization speedup]fix data device type bug #3856
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
QuanluZhang
reviewed
Jun 21, 2021
| @@ -326,6 +329,8 @@ def inference(self, test_data): | |||
| Model input tensor | |||
| """ | |||
| # convert pytorch tensor to numpy darray | |||
| if test_data.device != torch.device("cpu"): | |||
| test_data = test_data.to("cpu") | |||
There was a problem hiding this comment.
it is a little strange that inference can only be executed on cpu... why?
There was a problem hiding this comment.
Actually the inference executed on GPU after the memory copy from host memory to device memory which is done in our tool and the process of host2device memory copy is unavoidable. We use pycuda' s memcpy_htod_async api to do such copy. It is just a way of unifying that we choose to handle all host data and we will do memory copy in our engine before calibration and inference. We can also take another to handle it such as finding other way or API which can use torch' s cuda type data directly. However, it will be more complicated and no extra advantage.
QuanluZhang
approved these changes
Jun 21, 2021
J-shang
approved these changes
Jun 21, 2021
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is for fixing data's device type which has to be "cpu". Otherwise, it can not convert to numpy directly and will raise errors.