Replace use of API wrapper stream and event with plain CUDA, part 1 #389

makortel · 2019-09-17T16:46:19Z

PR description:

This PR is part of #386 and replaces the use of cuda::stream_t<> and cuda::event_t in the interfaces and in the user code. The "framework" part still uses them as replacing them in the stream and event caches requires cms-sw#28004. Anyway, this PR minimizes the impact of the later PR.

I also left HeterogeneousCore/Product and HeterogeneousCore/Producer out from this exercise as they will get nuked as soon as ClusterTPAssociationHeterogeneous is migrated away from those (#229 (comment)).

PR validation:

Unit tests run, profiling workflow runs. Code formatting was run.

fwyzard · 2019-10-23T17:44:19Z

Looks like something went very wrong during the validation ?

makortel · 2019-10-23T21:12:22Z

It certainly looks so. Would it be feasible to get full stack trace in case of crashes?

fwyzard · 2019-10-24T05:49:29Z

In theory we should have the full logs, in practice the latest changes to the validation script seem to have broken that... to be followed up :(

fwyzard · 2019-10-24T16:13:47Z

Validation summary

Reference release CMSSW_11_0_0_pre7 at 411b633
Development branch CMSSW_11_0_X_Patatrack at 2b9431d
Testing PRs:

Replace use of API wrapper stream and event with plain CUDA, part 1 #389 at 4d4f6b2

Validation plots

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

Throughput plots

/EphemeralHLTPhysics1/Run2018D-v1/RAW run=323775 lumi=53

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

Logs

The full log is available at https://patatrack.web.cern.ch/patatrack/validation/pulls/3a394a5257b10550f738491ea58e79943974ced0/log .

CUDADataFormats/Common/interface/CUDAProductBase.h

CUDADataFormats/TrackingRecHit/src/TrackingRecHit2DCUDA.cc

CUDADataFormats/TrackingRecHit/test/TrackingRecHit2DCUDA_t.cpp

fwyzard · 2019-10-25T08:35:57Z

HeterogeneousCore/CUDACore/interface/CUDAESProduct.h

@@ -86,7 +84,7 @@ class CUDAESProduct {
 private:
  struct Item {
    mutable std::mutex m_mutex;
-    CMS_THREAD_GUARD(m_mutex) mutable std::unique_ptr<cuda::event_t> m_event;
+    CMS_THREAD_GUARD(m_mutex) mutable std::shared_ptr<cuda::event_t> m_event;


why this becomes a shared_ptr ?

Because the event comes from the CUDAEventCache (instead of creating one explicitly), and that returns a shared_ptr.

HeterogeneousCore/CUDACore/interface/CUDAScopedContext.h

fwyzard · 2019-10-25T08:48:32Z

HeterogeneousCore/CUDAUtilities/src/getCachingDeviceAllocator.h

@@ -5,6 +5,8 @@
 #include "HeterogeneousCore/CUDAUtilities/interface/cudaCheck.h"
 #include "CachingDeviceAllocator.h"

+#include <cuda/api_wrappers.h>


why do we add this ?

Technically the #include should have been there already since the code below calls cuda::device::count() (which I'd rather deal with in another PR). Likely one/some of the files including this header lost the #include <cuda/api_wrappers.h> that it got from elsewhere.

fwyzard

See comments inline.

makortel · 2019-10-25T18:20:31Z

Thanks @fwyzard for the review, I addressed all your comments.

fwyzard · 2019-10-25T19:16:03Z

Validation summary

Reference release CMSSW_11_0_0_pre7 at 411b633
Development branch CMSSW_11_0_X_Patatrack at 55ce902
Testing PRs:

Replace use of API wrapper stream and event with plain CUDA, part 1 #389 at a5d004d

Validation plots

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.52

Throughput plots

/EphemeralHLTPhysics1/Run2018D-v1/RAW run=323775 lumi=53

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

/RelValZMM_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_realistic_v4-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

/RelValTTbar_13/CMSSW_10_6_0-PU25ns_106X_upgrade2018_design_v3-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
development release, workflow 10824.5
development release, workflow 10824.51
development release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 136.86452
testing release, workflow 10824.5
testing release, workflow 10824.51
testing release, workflow 10824.52
- ✔️ cuda-memcheck --tool initcheck (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 136.86452

Logs

The full log is available at https://patatrack.web.cern.ch/patatrack/validation/pulls/1edb07f93d2b4f3c404a400f2831c375ab7f19b4/log .

…389) Replace cuda::stream_t<> with cudaStream_t in client code Replace cuda::event_t with cudaEvent_t in the client code Clean up BuildFiles

makortel added 3 commits September 17, 2019 16:13

Replace cuda::stream_t<> with cudaStream_t in client code

3173923

Replace cuda::event_t with cudaEvent_t in the client code

b7f4f38

Clean up BuildFiles

4d4f6b2

makortel mentioned this pull request Sep 17, 2019

Remove the use of CUDA API wrappers #386

Closed

20 tasks

This comment has been minimized.

Sign in to view