bugfix - change treatment of GPU tracks not on CPU #284

felicepantaleo · 2019-03-13T09:46:40Z

In very rare cases a GPU track, which does not exist on the CPU, might be created.
This is due to the fact that the clusterizer is not exactly the same and a single hit difference in very rare cases could lead to a single track difference.
Currently the treatment for this track, which is not found on the CPU is to throw an assert.
This PR changes this assert into a skip of the track, fix by @VinInn

… CPU. If this is the case ignore the track in vertexing

fwyzard · 2019-03-13T09:54:17Z

IIUC the check is only on the number of tracks, not on the individual tracks themselves ?

fwyzard · 2019-03-13T09:55:17Z

What leads to the inconsistency ?
What I mean is, how do we end up comparing the number of tracks produced on the CPU and on the GPU ?

felicepantaleo · 2019-03-13T09:55:59Z

no, the id of the track is set to be a high number, higher than the number of tracks

felicepantaleo · 2019-03-13T10:26:12Z

Validation summary

Reference release CMSSW_10_5_0_pre2 at 4fd4ff5
Development branch CMSSW_10_5_X_Patatrack at 96828db
Testing PRs:

bugfix - change treatment of GPU tracks not on CPU #284 at fe1bb42

`makeTrackValidationPlots.py` plots

/RelValTTbar_13/CMSSW_10_4_0_pre3-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.52
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.53

/RelValZMM_13/CMSSW_10_4_0_pre3-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

tracking validation plots and summary for workflow 10824.5
tracking validation plots and summary for workflow 10824.52
tracking validation plots and summary for workflow 10824.51
tracking validation plots and summary for workflow 10824.53

logs and `nvprof`/`nvvp` profiles

/RelValTTbar_13/CMSSW_10_4_0_pre3-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.52
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.51
- step3.py: log, visual profile and summary
development release, workflow 10824.53
- step3.py: log, visual profile and summary
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
testing release, workflow 10824.52
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.51
- step3.py: log, visual profile and summary
testing release, workflow 10824.53
- step3.py: log, visual profile and summary

/RelValZMM_13/CMSSW_10_4_0_pre3-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

reference release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.5
- step3.py: log, visual profile and summary
development release, workflow 10824.52
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
development release, workflow 10824.51
- step3.py: log, visual profile and summary
development release, workflow 10824.53
- step3.py: log, visual profile and summary
testing release, workflow 10824.5
- step3.py: log, visual profile and summary
testing release, workflow 10824.52
- step3.py: log, visual profile and summary
- profile.py: log, visual profile and summary
- ✔️ cuda-memcheck --tool initcheck --track-unused-memory no (report, log) did not find any errors
- ✔️ cuda-memcheck --tool memcheck --leak-check full --report-api-errors all (report, log) did not find any errors
- ✔️ cuda-memcheck --tool synccheck (report, log) did not find any errors
testing release, workflow 10824.51
- step3.py: log, visual profile and summary
testing release, workflow 10824.53
- step3.py: log, visual profile and summary

Logs

The full log is available at https://fpantale.web.cern.ch/fpantale/patatrack/pulls/61e0fe24b9ce170ae65b1b57c509f4fc23b89b02/log .

makortel · 2019-03-13T14:34:22Z

Does the inconsistency originate from here

cmssw/RecoPixelVertexing/PixelTrackFitting/plugins/PixelTrackProducerFromCUDA.cc

Lines 158 to 160 in 96828db

    
           for (uint32_t it=0; it<tuples_->nTuples; ++it) { 
        
             auto q = tuples_->quality[it]; 
        
             if (q != pixelTuplesHeterogeneousProduct::loose ) continue; // FIXME

?

(there is also another continue in

cmssw/RecoPixelVertexing/PixelTrackFitting/plugins/PixelTrackProducerFromCUDA.cc

Lines 193 to 197 in 96828db

    
           std::unique_ptr<reco::Track> track( 
        
                 builder.build(pt, phi, cotTheta, tip, zip, chi2, iCharge, hits, 
        
                   fieldESH.product(), bs)); 
        
           if (!track) continue; 
        
           // filter???

but inspecting PixelTrackBuilder::build() it does not return nullptr)

felicepantaleo · 2019-03-15T19:39:53Z

GPU tracks are copied to the CPU to fill legacy data formats. For every track we read the hits and the map associates a hit GPU to a converted GPU hit on CPU.
These hits CPU are created from a cluster GPU: the problem is that a cluster CPU can contain at most 1024 digis, while clusters on GPUs don't have this limit.
This means that when a cluster GPU is used to make a CPU cluster, some digis are lost. Matching of hits to cluster is done by matching the xmin,ymin corner (this issue will be solved by associating by id). This corner digi might be lost and if this happens there's no matching.
A GPU track however might have been created with a hit that does not have a ref to a matched cluster, and if this track is used in vertexing (which is the first consumer of pixel tracks) the assertion fails.
This PR mitigates the issue by skipping the track. The solution of the issue would consist in changing how the matching is done.

@VinInn

makortel · 2019-03-15T19:42:15Z

Thanks @felicepantaleo for the explanation.

These hits CPU are created from a cluster GPU: the problem is that a cluster GPU can contain at most
1024 digis, while clusters on GPUs don't have this limit.

Should the latter "GPUs" be "CPUs"?

felicepantaleo · 2019-03-15T19:46:10Z

no the error is on the previous GPU:
clusters on GPU don't have a limit, on CPU they have.

makortel · 2019-03-15T19:49:43Z

no the error is on the previous GPU:
clusters on GPU don't have a limit, on CPU they have.

Ok, thanks for the fix.

Update master-cmsswmaster to CMSSW_10_4_0

felicepantaleo added 2 commits March 13, 2019 10:35

a GPU track might be created from hits which are not clustered on the…

45be622

… CPU. If this is the case ignore the track in vertexing

replace cout with edm::LogWarning

fe1bb42

felicepantaleo added the bug-fix label Mar 13, 2019

felicepantaleo merged commit 0fe0b5e into cms-patatrack:CMSSW_10_5_X_Patatrack Mar 15, 2019

fwyzard added this to the CMSSW_10_5_X_Patatrack milestone Mar 26, 2019

fwyzard pushed a commit that referenced this pull request May 15, 2019

Merge pull request #284 from cms-nanoAOD/branch_10_4_0

3cfb786

Update master-cmsswmaster to CMSSW_10_4_0

fwyzard mentioned this pull request Jan 13, 2021

Patatrack integration - Pixel vertex reconstruction (11/N) cms-sw/cmssw#31723

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bugfix - change treatment of GPU tracks not on CPU #284

bugfix - change treatment of GPU tracks not on CPU #284

felicepantaleo commented Mar 13, 2019

fwyzard commented Mar 13, 2019

fwyzard commented Mar 13, 2019

felicepantaleo commented Mar 13, 2019

felicepantaleo commented Mar 13, 2019 •

edited

Loading

makortel commented Mar 13, 2019

felicepantaleo commented Mar 15, 2019 •

edited

Loading

makortel commented Mar 15, 2019

felicepantaleo commented Mar 15, 2019

makortel commented Mar 15, 2019

bugfix - change treatment of GPU tracks not on CPU #284

bugfix - change treatment of GPU tracks not on CPU #284

Conversation

felicepantaleo commented Mar 13, 2019

fwyzard commented Mar 13, 2019

fwyzard commented Mar 13, 2019

felicepantaleo commented Mar 13, 2019

felicepantaleo commented Mar 13, 2019 • edited Loading

Validation summary

makeTrackValidationPlots.py plots

/RelValTTbar_13/CMSSW_10_4_0_pre3-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre3-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

logs and nvprof/nvvp profiles

/RelValTTbar_13/CMSSW_10_4_0_pre3-PU25ns_103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

/RelValZMM_13/CMSSW_10_4_0_pre3-103X_upgrade2018_realistic_v8-v1/GEN-SIM-DIGI-RAW

Logs

makortel commented Mar 13, 2019

felicepantaleo commented Mar 15, 2019 • edited Loading

makortel commented Mar 15, 2019

felicepantaleo commented Mar 15, 2019

makortel commented Mar 15, 2019

felicepantaleo commented Mar 13, 2019 •

edited

Loading

`makeTrackValidationPlots.py` plots

logs and `nvprof`/`nvvp` profiles

felicepantaleo commented Mar 15, 2019 •

edited

Loading