Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The GPU memory allocation policy is not reasonable #6158

Closed
QiJune opened this issue Dec 1, 2017 · 0 comments · Fixed by #6159
Closed

The GPU memory allocation policy is not reasonable #6158

QiJune opened this issue Dec 1, 2017 · 0 comments · Fixed by #6159
Assignees

Comments

@QiJune
Copy link
Member

QiJune commented Dec 1, 2017

Here is the method caculating GpuMaxChunkSize.

size_t GpuMaxChunkSize() {
  size_t total = 0;
  size_t available = 0;

  GpuMemoryUsage(available, total);

  // Reserving the rest memory for page tables, etc.
  size_t reserving = (1 - FLAGS_fraction_of_gpu_memory_to_use) * total;

  // If available less than minimum chunk size, no usable memory exists.
  available = std::max(available, GpuMinChunkSize()) - GpuMinChunkSize();

  // If available less than reserving, no usable memory exists.
  size_t usable = std::max(available, reserving) - reserving;

  return usable;
}

If we set certain fraction_of_gpu_memory_to_use, we must ensure available GPU memory is larger than reserving memory under this policy. It will always fail if we run many unit tests in parallel under this policy.
For example, we set FLAGS_fraction_of_gpu_memory_to_use=0.2. The first unit test occupy 0.2 GPU memory. When the second unit test comes in, the available GPU memory is 0.8, but the reserving memory is also 0.8. So, it will failed.
In fact, we should ensure our allocating memory is less than available memory. So, we need to change the allocation policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants