Recurring error: module 'std.codecvt' is incompatible with feature 'header_existence'

[rappoccio]

There is currently a recurring error (see here for instance) in some validation WFs that seems to be related to validate.C (see link here).

[cmsbuild]

A new Issue was created by @rappoccio .

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[rappoccio]

Affecting at least #39711

[rappoccio]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@mandrenguyen,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

The error message is

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj@.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.count
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.module.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.category.size()
plotting variable: HBHEDataFramesSorted_simHcalUnsuppressedDigis__RECO.obj.obj@.size()
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/etc//cling/std.modulemap:368:10: error: module 'std.codecvt' is incompatible with feature 'header_existence'
  module "codecvt" {
         ^
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/bin/../lib/gcc/x86_64-redhat-linux-gnu/10.3.0/../../../../include/c++/10.3.0/bits/fs_path.h:40:10: note: submodule of top-level module 'std' implicitly imported here
#include <codecvt>
         ^

 *** Break *** segmentation violation

#0  0x00002af647d053cb in waitpid () from /lib64/libc.so.6
#1  0x00002af647c66bff in do_system () from /lib64/libc.so.6
#2  0x00002af6474acd1b in TUnixSystem::StackTrace() () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCore.so
#3  0x00002af6474aa2b5 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCore.so
#4  <signal handler called>
#5  0x00002af64b565b6c in clang::CXXRecordDecl::getLambdaCallOperator() const () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCling.so
#6  0x00002af6490e3c38 in clang::RecursiveASTVisitor<cling::(anonymous namespace)::StaticVarCollector>::TraverseLambdaExpr(clang::LambdaExpr*, llvm::SmallVectorImpl<llvm::PointerIntPair<clang::Stmt*, 1u, bool, llvm::PointerLikeTypeTraits<clang::Stmt*>, llvm::PointerIntPairInfo<clang::Stmt*, 1u, llvm::PointerLikeTypeTraits<clang::Stmt*> > > >*) () from /cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/lib/libCling.so

Could the appearance be related to cms-sw/cms-bot#1856?

Is this pointing to some error in our ROOT setup? @vgvassilev @pcanal

[vgvassilev]

Which version of ROOT are you using. Do you have this commit: root-project/root@f8bcafc ?

[vgvassilev]

@makortel, @smuzaffar can you confirm that root-project/root#11571 fixes the issue for you?

[makortel]

The ROOT version is 6.24, more specifically https://github.com/cms-sw/root/commits/cms/v6-24-00-patches/b5aa8fd i.e. root-project/root@b5aa8fd + cms-sw/root@556667e. It does not contain root-project/root@f8bcafc.

[smuzaffar]

I have started the tests for root-project/root#11571 here cms-sw/root#176

[smuzaffar]

Could the appearance be related to cms-sw/cms-bot#1856?

@makortel , no this is not related to cms-sw/cms-bot#1856. log file has message like

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so

which means the tests were still using old cms-bot as it generated the validate_C.so. With latest cms-bot validateJR.sh change one should get something like this where we generate validate_C.so once

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10

[smuzaffar]

@vgvassilev , root-project/root#11571 worked for CMS. Comparison results here do not show any crashes

[smuzaffar]

@vgvassilev any idea why we suddenly hit this issue? Note that we have not changed root in CMSSW 12.6.X since May 2022. IBs and PR tests always run under singularity container which was re-built and is in use since 4th OCT 2022. Both CMSSW_12_5_X and CMSSW_12_6_X IBs use the same ROOT and same singularity container. Tests in CMSSW_12_5_X do not fail e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_5_X_2022-10-17-1100+296a35/53469/validateJR.html is all clean. Is it some thing we have integrated in CMSSW_12_6_X which exposed this bug?

[smuzaffar]

@makortel , @cms-sw/orp-l2 how should we proceed? should we cherry-pick just this in 12.6.X ?

[vgvassilev]

@vgvassilev any idea why we suddenly hit this issue? Note that we have not changed root in CMSSW 12.6.X since May 2022. IBs and PR tests always run under singularity container which was re-built and is in use since 4th OCT 2022. Both CMSSW_12_5_X and CMSSW_12_6_X IBs use the same ROOT and same singularity container. Tests in CMSSW_12_5_X do not fail e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_5_X_2022-10-17-1100+296a35/53469/validateJR.html is all clean. Is it some thing we have integrated in CMSSW_12_6_X which exposed this bug?

Did you switch recently to newer version of gcc, libstdc++ or bumped the language version? If you have then likely that is the cause.

[smuzaffar]

not really @vgvassilev . As I wrote tests do not fail for CMSSW_12_5_X which uses exact same root and same singularity image.

[smuzaffar]

ah could it be that rebuilt of root in new singularity container causes this. We do have rebuilt root for 12.6.X IBs ( due to dependency update). Let me rebuilt root for 12.5.X too and see if that fails too

[vgvassilev]

WARNING: AutoLibraryloader::enable() and AutoLibraryLoader.h are deprecated.
Use FWLiteEnabler::enable() and FWLiteEnabler.h instead
Info in <TUnixSystem::ACLiC>: creating shared library /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/all_OldVSNew_TTbar14TeV2026D88wf20834p0/./validate_C.so
Start making plots for Events with 10 events and refEvents with 10 events ==> check  10
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj@.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.count
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.module.size()
plotting variable: edmErrorSummaryEntrys_logErrorHarvester__RECO.obj.category.size()
plotting variable: HBHEDataFramesSorted_simHcalUnsuppressedDigis__RECO.obj.obj@.size()
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/lcg/root/6.24.07-743e141b5605864d08d853874a243e48/etc//cling/std.modulemap:368:10: error: module 'std.codecvt' is incompatible with feature 'header_existence'
  module "codecvt" {
         ^
/cvmfs/cms-ib.cern.ch/nweek-02754/el8_amd64_gcc10/external/gcc/10.3.0-84898dea653199466402e67d73657f10/bin/../lib/gcc/x86_64-redhat-linux-gnu/10.3.0/../../../../include/c++/10.3.0/bits/fs_path.h:40:10: note: submodule of top-level module 'std' implicitly imported here
#include <codecvt>
         ^

 *** Break *** segmentation violation

Our test started using bits/fs_path.h which seems to include codecvt and then we got the failure. So yes, I think you are right to say we integrated something that triggered this issue...

[smuzaffar]

CMSSW_12_5_X PR tests with root rebuilt in new container are all good . So the failure we see in 12.6.X is due to some new code in cmssw 12.6.X itself. PR tests based on CMSSW_12_6_X_2022-10-13-1100 IB were good and they start failing for CMSSW_12_6_X_2022-10-13-2300 and above. Looking at the PRs merged for CMSSW_12_6_X_2022-10-13-2300, it looks like #38341 introduced some change which exposed this issue.

[smuzaffar]

@makortel , @perrotta @rappoccio any objections on including cms-sw/root#176 ( suggested by @vgvassilev here )? This should fix these DQM comparison failures

[perrotta]

@makortel , @perrotta @rappoccio any objections on including cms-sw/root#176 ( suggested by @vgvassilev here )? This should fix these DQM comparison failures

@smuzaffar we have already started building pre4.
Are you suggesting to stop the build, merge that one, and restart the build afterwards?
Of course, after pre4 that fix is welcome in master

[smuzaffar]

no, do not stop pre4, we can include it after pre4.