index out of bounds in RecoHGCalTICLPlugins

[dan131riley]

While chasing another bug I found that UBSAN is getting index out of bounds in RecoHGCalTICLPlugins. Stack trace:

/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/array:57:36: runtime error: index 4372 out of bounds for type 'vector [4284]'
    #0 0x7fdc4341f399  (/mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/lib/el8_amd64_gcc11/pluginRecoHGCalTICLPlugins.so+0xb38399)
    #1 0x7fdc43500d4e  (/mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/lib/el8_amd64_gcc11/pluginRecoHGCalTICLPlugins.so+0xc19d4e)
    #2 0x7fdc437fcbb5  (/mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/lib/el8_amd64_gcc11/pluginRecoHGCalTICLPlugins.so+0xf15bb5)

I did a bit of debugging and got this backtrace by setting a gdb breakpoint on __ubsan::ScopedReport::~ScopedReport (it would have been smarter to set the breakpoint on __ubsan::__ubsan_handle_out_of_bounds as we get a lot of UBSAN hits from the Xilinx headers doing undefined shifts):

#0  __ubsan::ScopedReport::~ScopedReport (this=this@entry=0x7ffea3ec7230, __in_chrg=<optimized out>) at ../../../../libsanitizer/ubsan/ubsan_diag.cpp:388
#1  0x00007f43450842f4 in handleOutOfBoundsImpl (Data=Data@entry=0x7f416ba3caa0, Index=Index@entry=4372, Opts=...) at ../../../../libsanitizer/ubsan/ubsan_handlers.cpp:394
#2  0x00007f4345087fa5 in __ubsan::__ubsan_handle_out_of_bounds (Data=Data@entry=0x7f416ba3caa0, Index=Index@entry=4372) at ../../../../libsanitizer/ubsan/ubsan_handlers.cpp:399
#3  0x00007f416b3ac39a in std::__array_traits<std::vector<unsigned int, std::allocator<unsigned int> >, 4284ul>::_S_ref (__n=4372, __t=...) at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/array:57
#4  std::array<std::vector<unsigned int, std::allocator<unsigned int> >, 4284ul>::operator[] (__n=4372, this=0x7f40eea506a8) at /cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02785/el8_amd64_gcc11/external/gcc/11.2.1-f9b9dfdd886f71cd63f5538223d8f161/include/c++/11.2.1/array:198
#5  TICLLayerTileT<ticl::TileConstants>::operator[] (globalBinId=<optimized out>, this=0x7f40eea506a8) at /cvmfs/cms-ib.cern.ch/sw/x86_64/week1/el8_amd64_gcc11/cms/cmssw/CMSSW_13_2_UBSAN_X_2023-05-15-2300/src/DataFormats/HGCalReco/interface/TICLLayerTile.h:73
#6  ticl::PatternRecognitionbyCLUE3D<TICLGenericTile<std::array<TICLLayerTileT<ticl::TileConstants>, 104ul> > >::calculateLocalDensity (this=this@entry=0x7f414d40ce80, tiles=..., layerId=<optimized out>, layerId@entry=78, layerIdx2layerandSoa=std::vector of length 3816, capacity 3816 = {...}) at /mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/src/RecoHGCal/TICL/plugins/PatternRecognitionbyCLUE3D.cc:556
#7  0x00007f416b48dd4f in ticl::PatternRecognitionbyCLUE3D<TICLGenericTile<std::array<TICLLayerTileT<ticl::TileConstants>, 104ul> > >::makeTracksters (this=0x7f414d40ce80, input=..., result=std::vector of length 0, capacity 0, seedToTracksterAssociation=...) at /mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/src/RecoHGCal/TICL/plugins/PatternRecognitionbyCLUE3D.cc:276
#8  0x00007f416b789bb6 in TrackstersProducer::produce (this=0x7f4226720c00, evt=..., es=...) at /mnt/data1/dsr/tmp/CMSSW_13_2_UBSAN_X_2023-05-15-2300/src/RecoHGCal/TICL/plugins/TrackstersProducer.cc:181

which points back to

cmssw/RecoHGCal/TICL/plugins/PatternRecognitionbyCLUE3D.cc

Line 556 in 9d147a6

for (auto otherClusterIdx : tileOnLayer[offset + iphi]) {

with

(gdb) p offset
$3 = 4284
(gdb) p iphi
$4 = 88
(gdb) p ieta
$5 = 34
(gdb) p nPhiBin
$6 = 126

This was observed in wf 25234.911 with IB CMSSW_13_2_UBSAN_X_2023-05-15-2300.

[cmsbuild]

A new Issue was created by @dan131riley Dan Riley.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign reconstruction, upgrade

FYI @cms-sw/hgcal-dpg-l2

[cmsbuild]

New categories assigned: upgrade,reconstruction

@AdrianoDee,@mandrenguyen,@clacaputo,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

[clacaputo]

type hgcal

[felicepantaleo]

thanks, I have the fix ready, waiting for #41589 to be merged

[dan131riley]

I don't see any of these in the latest UBSAN builds, so it looks like this is fixed in #41589 and #41905, and the issue can be closed.