Hcal Raw Decoding + Reco (method 0, mahi) on GPU #374

vkhristenko · 2019-07-19T16:03:39Z

PR description:

This is the replacemnt of #367 made on top of the #373

PR validation:

an exe is provided to validate against cpu version.
note, that validation branch should be used to match exactly.

method 0 changes:

containment correct removed
hbm legacy correction removed (to be removed in run 3 completely)

mahi changes:

do not use prefit. The logic is simple - prefit is there not to run mahi for a particular case and cut down runtime. if we are able to use mahi for all situations and improve runtime (speed-up) better run full mahi.
fixes for 2 bugs found in cpu version and reported

…ucts + cpu producer + validation

…merge-topic

mariadalfonso · 2019-10-09T11:32:46Z

general note:
you need to factorize the code

MahiGPU.cu should contain only MAHI not the conditions and neither nnls()

Can you put in conditions to validate the raw to the digi on CPU part and then only the MAHI on GPU ?

vkhristenko · 2019-10-09T11:45:14Z

so,

MahiGPU, at this point ecal and hcal are almost identical, exept for covariance matrix treatment. in there i separated fnnls and other computations. the goal is once merged, reuse the code and to completely remove various comptuation funs from the either ecal or hcal packages.
for validation for raw to digi, please carefully check what sits in EventFilter/HcalRawToDigi package and in particular standalone execs that were added.
both raw to digi and digi to rec hit are written and validated separately.

mariadalfonso · 2019-10-10T07:14:53Z

RecoLocalCalo/HcalRecAlgos/src/MahiGPU.cu

+            // move index to the right part of the vector
+            w_max_idx += npassive;
+
+            Eigen::numext::swap(pulseOffsets.coeffRef(npassive),


Q: how you avoided to swap all the vectors (and do only the fitted amplitude) ?
https://github.com/cms-sw/cmssw/blob/master/RecoLocalCalo/HcalRecAlgos/src/MahiFit.cc#L476-L484
If I do in the CPU version all the quantities are wrong.

check how pulseOffsets are used. you do not need to swap (on cpu it's prolly better, also you will not be able to use eigen in that case). but for gpu, it's better not to swap especially for small matrices. you just need to keep track of the active/passive set, which is what pulseOffsets does, and load things accordingly.

fwyzard · 2019-10-22T09:56:35Z

@vkhristenko could you remove the file RecoLocalCalo/HcalRecAlgos/src/.MahiGPU.cu.swk from the PR ?

vkhristenko · 2019-10-22T10:01:57Z

@vkhristenko could you remove the file RecoLocalCalo/HcalRecAlgos/src/.MahiGPU.cu.swk from the PR ?

done

fwyzard · 2019-10-22T22:28:38Z

Hi @vkhristenko,
I am trying to run your HCAL code, but I am not able to build it in a general working area.

If I checkout out only the minimal set of packages, with

cmsrel CMSSW_10_6_3_Patatrack
cd CMSSW_10_6_3_Patatrack/src
cmsenv
git cms-init -x cms-patatrack
git cms-fetch-pr cms-patatrack:374
git merge pull/374
git cms-addpkg CUDADataFormats/HcalCommon  CUDADataFormats/HcalDigi  CUDADataFormats/HcalRecHitSoA  EventFilter/HcalRawToDigi  HeterogeneousCore/CUDACore  RecoLocalCalo/HcalRecAlgos  RecoLocalCalo/HcalRecProducers

# edit HeterogeneousCore/CUDACore/interface/CUDAESProduct.h and add
#include <cassert> 

# edit RecoLocalCalo/HcalRecAlgos/src/HcalSiPMCharacteristicsGPU.cc and add
#include "FWCore/Utilities/interface/Exception.h"

scram b -j

it works.
Note that is seems a couple of files miss some of #includes.

However, if I add a couple more (unmodified) packages:

git cms-addpkg RecoLocalCalo/EcalRecAlgos RecoLocalCalo/EcalRecProducers
scram b -j

the build fails with a linker error:

>> Building shared library tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/libRecoLocalCaloHcalRecAlgos.so
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_79_tmpxft_0002b32b_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c':
link.stub:(.text+0x121): undefined reference to `__fatbinwrap_79_tmpxft_0002b32b_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_75_tmpxft_0002b336_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056':
link.stub:(.text+0x201): undefined reference to `__fatbinwrap_75_tmpxft_0002b336_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_68_tmpxft_0002b341_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9':
link.stub:(.text+0x2e1): undefined reference to `__fatbinwrap_68_tmpxft_0002b341_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_83_tmpxft_0002b34c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b':
link.stub:(.text+0x3c1): undefined reference to `__fatbinwrap_83_tmpxft_0002b34c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_59_tmpxft_0002b357_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c':
link.stub:(.text+0x4a1): undefined reference to `__fatbinwrap_59_tmpxft_0002b357_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_59_tmpxft_0002b362_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574':
link.stub:(.text+0x581): undefined reference to `__fatbinwrap_59_tmpxft_0002b362_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574'
collect2: error: ld returned 1 exit status

Have you seen this error before ?

vkhristenko · 2019-10-23T07:39:34Z

@fwyzard

first of, the errors are coming from ecal.... and not hcal
and what you check out is part of release already, therefore should've been compilable... just thinking out loud.
i've seen this kinda stuff before, but do not remember what i did to remove those. these undef refs are to cuda stuff... not user stuff...

sorry... would need to check out myself. i also never did minimalistic check outs, meaning i would always get head of the patatrack branch and do things on top... but here the problem is from what is already part of the release and what has been compiled before... may be try scram b clean; scram b -j 8

just to add

>> Building shared library tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/libRecoLocalCaloHcalRecAlgos.so

but next

undefined ref to __fatbinwrap_79_tmpxft_0002b32b_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c

this guy AmplitudeComputationCommonKernels this is ecal guy, not hcal

fwyzard · 2019-10-23T08:08:02Z

I think you can also reproduce it working directly in your branch.

Yes, the errors come from the ECAL packages, but are caused by the changes in the HCAL packages, because of the dependency among them.

I think we may have seen similar errors quite some time ago, and IIRC they were caused by multiple shared libraries (or plugins) trying to load the same CUDA kernels (for example, because they were linking a shared library that included those kernels).

fwyzard · 2019-10-25T12:17:22Z

@vkhristenko did you have a look at the link problems ?

vkhristenko · 2019-10-25T12:30:44Z

@fwyzard not yet, give me time until monday. i need that branch working on top of 10 6 3 patatrack for opendata in any case before then...

vkhristenko · 2019-10-28T08:59:38Z

@fwyzard

I"m not able to reproduce it. note that i've added the include of cms::Exception header directly into my branch. the cassert is missing in the Heterogeneous/CUDACore.

 1012  cmsrel CMSSW_10_6_3_Patatrack
 1013  cd CMSSW_10_6_3_Patatrack/src/
 1014  cmsenv
 1015  git cms-init -x cms-patatrack
 1016  git branch CMSSW_10_6_X_Patatrack --track cms-patatrack/CMSSW_10_6_X_Patatrack
 1017  git checkout CMSSW_10_6_X_Patatrack
 1018  git checkout -b test_merging
 1019  git cms-merge-topic vkhristenko:hcal_mahi_patatrack
 1020  ls
 1021  USER_CUDA_FLAGS="-Xptxas -v" scram b -v -j 20
 1022  git cms-merge-topic vkhristenko:hcal_mahi_patatrack
 1023  USER_CUDA_FLAGS="-Xptxas -v" scram b -v -j 20
 1024  ls
 1025  git cms-addpkg HeterogeneousCore/CUDACore
 1026  cd HeterogeneousCore/CUDACore/in
 1027  ls
 1028  cd HeterogeneousCore/CUDACore/interface/
 1029  ls
 1030  vim CUDAESProduct.h 
 1031  cd ../../..
 1032  ls
 1033  USER_CUDA_FLAGS="-Xptxas -v" scram b -v -j 20
 1034  git cms-addpkg RecoLocalCalo/EcalRecAlgos RecoLocalCalo/EcalRecProducers
 1035  ls
 1036  USER_CUDA_FLAGS="-Xptxas -v" scram b -v -j 20

compiles w/o any linking errors.

fwyzard · 2019-10-28T09:17:19Z

Hi @vkhristenko ,
it does fail for me if I checkout out the dependencies:

# create a working area for CMSSW_10_6_3_Patatrack
scram list
cmsrel CMSSW_10_6_3_Patatrack
cd CMSSW_10_6_3_Patatrack/src
cmsenv
git cms-init -x cms-patatrack
git branch CMSSW_10_6_X_Patatrack --track cms-patatrack/CMSSW_10_6_X_Patatrack
git checkout CMSSW_10_6_X_Patatrack

# merge vkhristenko:hcal_mahi_patatrack and checkout the modified packages
git checkout -b test_merging
git cms-merge-topic vkhristenko:hcal_mahi_patatrack
git diff --name-only $CMSSW_VERSION | cut -d/ -f-2 | uniq | xargs git cms-addpkg

# fix CUDAESProduct.h to add #include <cassert>
git cms-addpkg HeterogeneousCore/CUDACore
vim HeterogeneousCore/CUDACore/interface/CUDAESProduct.h 

# checkout the ECAL packages, and all dependencies 
git cms-addpkg RecoLocalCalo/EcalRecAlgos RecoLocalCalo/EcalRecProducers
git cms-checkdeps -a

# build
scram b -v -j

results in

/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_79_tmpxft_0002b32b_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c':
link.stub:(.text+0x121): undefined reference to `__fatbinwrap_79_tmpxft_0002b32b_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_75_tmpxft_0002b336_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056':
link.stub:(.text+0x201): undefined reference to `__fatbinwrap_75_tmpxft_0002b336_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_68_tmpxft_0002b341_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9':
link.stub:(.text+0x2e1): undefined reference to `__fatbinwrap_68_tmpxft_0002b341_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_83_tmpxft_0002b34c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b':
link.stub:(.text+0x3c1): undefined reference to `__fatbinwrap_83_tmpxft_0002b34c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_59_tmpxft_0002b357_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c':
link.stub:(.text+0x4a1): undefined reference to `__fatbinwrap_59_tmpxft_0002b357_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c'
/data/cmssw/slc7_amd64_gcc820/external/gcc/8.2.0-pafccj/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.3.1/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc820/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: in function `__cudaRegisterLinkedBinary_59_tmpxft_0002b362_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574':
link.stub:(.text+0x581): undefined reference to `__fatbinwrap_59_tmpxft_0002b362_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574'
collect2: error: ld returned 1 exit status

vkhristenko · 2019-10-28T09:23:23Z

@fwyzard

just added dependencies as well... after checking ecal stuff... still no errors

fwyzard · 2019-10-28T09:24:39Z

I am not sure how else to allow you to reproduce it...
where are you running ?
what architecture ?

vkhristenko · 2019-10-28T09:25:46Z

scram arch: slc7_amd64_gcc700 (note you have 820 gcc)
cmg-gpu1080:/data/patatrack/vkhriste/cmssw_releases/hcal/mahi/CMSSW_10_6_3_Patatrack/src

fwyzard · 2019-10-28T09:34:47Z

I've made a copy of your area, and I'll try to rebuild that...

fwyzard · 2019-10-28T09:39:34Z

...I did just

scram b clean
scram b -j

and I got the same problem

--- Registered EDM Plugin: RecoLocalCaloEcalRecProducersPlugins
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_79_tmpxft_0000e2c4_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c':
link.stub:(.text+0x121): undefined reference to `__fatbinwrap_79_tmpxft_0000e2c4_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_75_tmpxft_0000e2e3_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056':
link.stub:(.text+0x201): undefined reference to `__fatbinwrap_75_tmpxft_0000e2e3_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_68_tmpxft_0000e2fd_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9':
link.stub:(.text+0x2e1): undefined reference to `__fatbinwrap_68_tmpxft_0000e2fd_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_83_tmpxft_0000e31c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b':
link.stub:(.text+0x3c1): undefined reference to `__fatbinwrap_83_tmpxft_0000e31c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_59_tmpxft_0000e337_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c':
link.stub:(.text+0x4a1): undefined reference to `__fatbinwrap_59_tmpxft_0000e337_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_59_tmpxft_0000e365_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574':
link.stub:(.text+0x581): undefined reference to `__fatbinwrap_59_tmpxft_0000e365_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574'
>> Leaving Package RecoLocalCalo/EcalRecProducers
>> Package RecoLocalCalo/EcalRecProducers built
collect2: error: ld returned 1 exit status
gmake: *** [config/SCRAM/GMake/Makefile.rules:1733: tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/libRecoLocalCaloHcalRecAlgos.so] Error 1
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

Could you try just that ?

vkhristenko · 2019-10-28T09:45:45Z

yep, i did scram b clean; scram b and got the same

--- Registered EDM Plugin: RecoLocalCaloEcalRecProducersPlugins
>> Leaving Package RecoLocalCalo/EcalRecProducers
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_79_tmpxft_0000e2c4_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c':
link.stub:(.text+0x121)>> Package RecoLocalCalo/EcalRecProducers built
: undefined reference to `__fatbinwrap_79_tmpxft_0000e2c4_00000000_7_AmplitudeComputationCommonKernels_compute_70_cpp1_ii_9f04962c'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_75_tmpxft_0000e2e3_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056':
link.stub:(.text+0x201): undefined reference to `__fatbinwrap_75_tmpxft_0000e2e3_00000000_7_AmplitudeComputationKernelsV1_compute_70_cpp1_ii_443c6056'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_68_tmpxft_0000e2fd_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9':
link.stub:(.text+0x2e1): undefined reference to `__fatbinwrap_68_tmpxft_0000e2fd_00000000_7_TimeComputationKernels_compute_70_cpp1_ii_7bf553a9'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_83_tmpxft_0000e31c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b':
link.stub:(.text+0x3c1): undefined reference to `__fatbinwrap_83_tmpxft_0000e31c_00000000_7_EcalUncalibRecHitMultiFitAlgo_gpu_new_compute_70_cpp1_ii_27e3a93b'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_59_tmpxft_0000e337_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c':
link.stub:(.text+0x4a1): undefined reference to `__fatbinwrap_59_tmpxft_0000e337_00000000_7_inplace_fnnls_compute_70_cpp1_ii_a006763c'
tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o: In function `__cudaRegisterLinkedBinary_59_tmpxft_0000e365_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574':
link.stub:(.text+0x581): undefined reference to `__fatbinwrap_59_tmpxft_0000e365_00000000_7_KernelHelpers_compute_70_cpp1_ii_5fcd2574'
collect2: error: ld returned 1 exit status
gmake: *** [config/SCRAM/GMake/Makefile.rules:1733: tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/libRecoLocalCaloHcalRecAlgos.so] Error 1
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error

fwyzard · 2019-10-28T10:04:59Z

OK, I suspect you didn't get it before because of the order of the compilation and link commands :-(

Can you look into what is causing it ?

vkhristenko · 2019-10-28T10:05:42Z

yep yep

vkhristenko · 2019-10-28T18:05:10Z

Indeed this is a dependency in hcal code on ecal...

so, below is the command for final stage device side linking for RecoLocalCaloHcalRecAlgos_cudadlink.o

this command includes linking device side code from RecoLocalCalo/EcalRecAlgos -lRecoLocalCaloEcalRecAlgos_nv. I guess this goes back to cms-sw/cmssw-config#65

by removing this linkage (of -lRecoLocalCaloEcalRecAlgos_nv) and proceeding to the overall shared object linkage everything works fine.

@fwyzard , as far as i remember, you mentioned at some point although cms-sw/cmssw-config#65 got merged, still things did not work. I think what i see here is the result of this guy not working 100%...

Moreover, in here there is no device side dependency, only host side, but the build rules are still generated... this is actually something we should not add!

/data/patatrack/cmssw/slc7_amd64_gcc700/external/cuda/10.1.243/bin/nvcc -dlink -L/data/patatrack/vkhriste/cmssw_releases/hcal/mahi/CMSSW_10_6_3_Patatrack/static/slc7_amd64_gcc700 -L/data/patatrack/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_3_Patatrack/static/slc7_amd64_gcc700 -lRecoLocalCaloEcalRecAlgos_nv -L/data/patatrack/vkhriste/cmssw_releases/hcal/mahi/CMSSW_10_6_3_Patatrack/lib/slc7_amd64_gcc700 -L/data/patatrack/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_3_Patatrack/lib/slc7_amd64_gcc700 -L/data/patatrack/cmssw/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_3_Patatrack/external/slc7_amd64_gcc700/lib -L/data/patatrack/cmssw/slc7_amd64_gcc700/external/cuda/10.1.243/lib64/stubs -lcudadevrt -gencode arch=compute_60,code=sm_60 -gencode arch=compute_70,code=sm_70 -O3 -std=c++14 --expt-relaxed-constexpr --expt-extended-lambda --generate-line-info --source-in-ptx --cudart=shared --compiler-options '-O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -msse3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -Wno-error=unused-variable -Wno-error=unused-variable -DCUDA_ENABLE_DEPRECATED -DCUB_STDERR -DBOOST_DISABLE_ASSERTS -std=c++14  -fPIC   ' tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/MahiGPU.cu.o -o tmp/slc7_amd64_gcc700/src/RecoLocalCalo/HcalRecAlgos/src/RecoLocalCaloHcalRecAlgos/RecoLocalCaloHcalRecAlgos_cudadlink.o

…cmssw into hcal_mahi_patatrack

fwyzard · 2020-05-31T19:15:08Z

Replaced by #431 and #468.

vkhristenko added 15 commits July 16, 2019 15:31

preped a producer

bfb0fae

minimal kernel to read fed header

d4be6d1

decoding uhtr data

0c37f21

able to dump uhtr headers

96fab87

add eid -> did conversion for gpu

ac5e49c

add output ptrs alloc/dealloc

25c0668

running, number of channels is ~ same with cpu, need proper cuda prod…

f438e04

…ucts + cpu producer + validation

add cuda product for hcal digi

390a5bb

fixing, producing transient cuda products

e069518

quick speed up by using 12 threads per crate

f666cb2

validate cpu digis consumer

99f3b0c

add validation exe

74ff57a

clean up and add cerr for missing det id

353e50f

Merged hcal_rawdecode_patatrack from repository vkhristenko with cms-…

27feb5c

…merge-topic

rearranging

a26d268

vkhristenko mentioned this pull request Jul 19, 2019

[WIP] Hcal Reco on GPU #367

Closed

vkhristenko added 14 commits July 19, 2019 20:24

use cuda products as input collections

a3ae01b

rename

7a953d6

add qie types conditions

c5e35d9

add qie shapes generator

cd12bcb

charge computation

e21927d

adding sipm specific conditions. computing raw charge

e3138ff

need to get pedestals in fc units, similarly to cpu

3dc6b2c

refactor es producers

56e32ee

fixed usage of dependent records for converting pedestals

4988448

adding converted pedestals

44e56ce

fixed converted pedestals

dfb69b6

add converted pedestal widths

c25755f

working up to method 0 w/o containment and time slew

a03ffeb

fix conditions, method 0 quantities

b4ae00b

mariadalfonso reviewed Oct 10, 2019

View reviewed changes

remove vim leftovers

2841763

exception include header

c2d481b

vkhristenko added 2 commits November 4, 2019 06:44

fixing diffs w.r.t. cpu raw decoder

377994b

Merge branch 'hcal_mahi_patatrack' of https://github.com/vkhristenko/…

c4e21d4

…cmssw into hcal_mahi_patatrack

vkhristenko mentioned this pull request Dec 17, 2019

Hcal Reco w/ CUDA rebased #431

Closed

fwyzard added this to the CMSSW_11_1_0_Patatrack milestone Jan 22, 2020

fwyzard closed this May 31, 2020

vkhristenko mentioned this pull request Dec 7, 2020

Patatrack integration - HCAL local reconstruction (8/N) cms-sw/cmssw#31720

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hcal Raw Decoding + Reco (method 0, mahi) on GPU #374

Hcal Raw Decoding + Reco (method 0, mahi) on GPU #374

vkhristenko commented Jul 19, 2019 •

edited

Loading

mariadalfonso commented Oct 9, 2019

vkhristenko commented Oct 9, 2019 •

edited

Loading

mariadalfonso Oct 10, 2019

vkhristenko Oct 10, 2019

fwyzard commented Oct 22, 2019

vkhristenko commented Oct 22, 2019

fwyzard commented Oct 22, 2019

vkhristenko commented Oct 23, 2019 •

edited

Loading

fwyzard commented Oct 23, 2019

fwyzard commented Oct 25, 2019

vkhristenko commented Oct 25, 2019 •

edited

Loading

vkhristenko commented Oct 28, 2019 •

edited

Loading

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019 •

edited

Loading

fwyzard commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented May 31, 2020

Hcal Raw Decoding + Reco (method 0, mahi) on GPU #374

Hcal Raw Decoding + Reco (method 0, mahi) on GPU #374

Conversation

vkhristenko commented Jul 19, 2019 • edited Loading

PR description:

PR validation:

mariadalfonso commented Oct 9, 2019

vkhristenko commented Oct 9, 2019 • edited Loading

mariadalfonso Oct 10, 2019

Choose a reason for hiding this comment

vkhristenko Oct 10, 2019

Choose a reason for hiding this comment

fwyzard commented Oct 22, 2019

vkhristenko commented Oct 22, 2019

fwyzard commented Oct 22, 2019

vkhristenko commented Oct 23, 2019 • edited Loading

fwyzard commented Oct 23, 2019

fwyzard commented Oct 25, 2019

vkhristenko commented Oct 25, 2019 • edited Loading

vkhristenko commented Oct 28, 2019 • edited Loading

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019 • edited Loading

fwyzard commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

vkhristenko commented Oct 28, 2019

fwyzard commented May 31, 2020

vkhristenko commented Jul 19, 2019 •

edited

Loading

vkhristenko commented Oct 9, 2019 •

edited

Loading

vkhristenko commented Oct 23, 2019 •

edited

Loading

vkhristenko commented Oct 25, 2019 •

edited

Loading

vkhristenko commented Oct 28, 2019 •

edited

Loading

vkhristenko commented Oct 28, 2019 •

edited

Loading