Moved modules.py out of edmWriteConfig #44396

Dr15Jones · 2024-03-13T21:09:00Z

PR description:

is now a file in the template directory
modification of scram will use the new file and place in the correct directory.

PR validation:

Code compiles. Needs change to scram to fully test.

- is now a file in the template directory - modification of scram will use the new file and place in the correct directory.

Dr15Jones · 2024-03-13T21:09:16Z

@smuzaffar FYI

cmsbuild · 2024-03-13T21:09:24Z

cms-bot internal usage

Dr15Jones · 2024-03-13T21:11:27Z

please test

cmsbuild · 2024-03-13T21:16:13Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-44396/39464

This PR adds an extra 20KB to repository

cmsbuild · 2024-03-13T21:16:32Z

A new Pull Request was created by @Dr15Jones for master.

It involves the following packages:

FWCore/ParameterSet (core)

@Dr15Jones, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@makortel, @wddgit, @missirol this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2024-03-13T21:34:04Z

-1

Failed Tests: Build HeaderConsistency ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38113/summary.html
COMMIT: c374f9c
CMSSW: CMSSW_14_1_X_2024-03-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44396/38113/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38113/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38113/git-merge-result

Build

I found compilation error when building:

>> Leaving Package PhysicsTools/ONNXRuntime
>> Package PhysicsTools/ONNXRuntime built
Entering library rule at PhysicsTools/ONNXRuntime
>> Compiling  src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc
In file included from src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/onnxruntime_cxx_api.h: No such file or directory
   19 | #include "onnxruntime/onnxruntime_cxx_api.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/onnxruntime_cxx_api.h: No such file or directory

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package CalibTracker/SiStripLorentzAngle
>> Entering Package FWCore/ParameterSet
>> Entering Package PhysicsTools/NanoAOD
>> Entering Package PhysicsTools/ONNXRuntime
>> Compile sequence completed for CMSSW CMSSW_14_1_X_2024-03-13-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/tmp/el8_amd64_gcc12/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package Alignment/OfflineValidation
>> Compiling  src/Alignment/OfflineValidation/bin/Options.cc
>> Compiling  src/Alignment/OfflineValidation/bin/DMRmerge.cc

Dr15Jones · 2024-03-14T13:07:38Z

please test

cmsbuild · 2024-03-14T13:26:30Z

-1

Failed Tests: Build HeaderConsistency ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38133/summary.html
COMMIT: c374f9c
CMSSW: CMSSW_14_1_X_2024-03-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44396/38133/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38133/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38133/git-merge-result

Build

I found compilation error when building:

>> Leaving Package PhysicsTools/ONNXRuntime
>> Package PhysicsTools/ONNXRuntime built
Entering library rule at PhysicsTools/ONNXRuntime
>> Compiling  src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc
In file included from src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/onnxruntime_cxx_api.h: No such file or directory
   19 | #include "onnxruntime/onnxruntime_cxx_api.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from src/PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc:8:
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/src/PhysicsTools/ONNXRuntime/interface/ONNXRuntime.h:19:10: fatal error: onnxruntime/onnxruntime_cxx_api.h: No such file or directory

Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package CalibTracker/SiStripLorentzAngle
>> Entering Package FWCore/ParameterSet
>> Entering Package PhysicsTools/NanoAOD
>> Entering Package PhysicsTools/ONNXRuntime
>> Compile sequence completed for CMSSW CMSSW_14_1_X_2024-03-13-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
+ eval scram build outputlog '&&' '(python3' /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cms-bot/buildLogAnalyzer.py --logDir /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-03-13-1100/tmp/el8_amd64_gcc12/cache/log/src '||' 'true)'
++ scram build outputlog
>> Entering Package Alignment/OfflineValidation
>> Compiling  src/Alignment/OfflineValidation/bin/DMRmerge.cc
>> Compiling  src/Alignment/OfflineValidation/bin/Options.cc

Dr15Jones · 2024-03-14T13:59:13Z

@smuzaffar looks like other PR tests are failing in the same way.

valsdav · 2024-03-14T14:01:54Z

The change on the ONNX version was merged yesterday in cms-sw/cmsdist#9046
at 15:00. Here it is using IB CMSSW_14_1_X_2024-03-13-1100. Don't we have to refresh the tests?

smuzaffar · 2024-03-14T14:04:06Z

@Dr15Jones , we need to wait for newer IB. PR tests here are picking up #44354 for an IB which has old ONNXRuntime.

Last night's 23h IB is still in build phase for production arch. It will take few more hours as it has to build many externals

cmsbuild · 2024-03-15T10:26:11Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38152/summary.html
COMMIT: c374f9c
CMSSW: CMSSW_14_1_X_2024-03-14-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44396/38152/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

RelVals-INPUT

AddOn Tests

read312RV

----- Begin Fatal Exception 15-Mar-2024 11:16:41 CET-----------------------
An exception of category 'ConfigFileReadError' occurred while
   [0] Processing the python configuration file named /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-14-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py
Exception Message:
 unknown python problem occurred.
ModuleNotFoundError: No module named 'past'

At:
  /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44396/38152/CMSSW_14_1_X_2024-03-14-2300/src/FWCore/ParameterSet/python/Types.py(6): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/44396/38152/CMSSW_14_1_X_2024-03-14-2300/src/FWCore/ParameterSet/python/Config.py(15): <module>
  <frozen importlib._bootstrap>(228): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(850): exec_module
  <frozen importlib._bootstrap>(695): _load_unlocked
  <frozen importlib._bootstrap>(986): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1007): _find_and_load
  /cvmfs/cms-ib.cern.ch/sw/x86_64/week0/el8_amd64_gcc12/cms/cmssw/CMSSW_14_1_X_2024-03-14-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py(2): <module>

----- End Fatal Exception -------------------------------------------------

fastsim

[fastsim:1] cmsDriver.py TTbar_8TeV_TuneCUETP8M1_cfi  --conditions auto:run1_mc --fast  -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION  --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot Realistic8TeVCollision : FAILED - elapsed time: 0 sec (ended on Fri Mar 15 11:16:43 2024) - exit: 256

fastsim1

[fastsim1:1] cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi --conditions auto:run2_mc_l1stage1 --fast  -n 100 --eventcontent AODSIM,DQM --relval 100000,1000 -s GEN,SIM,RECOBEFMIX,DIGI:pdigi_valid,L1,DIGI2RAW,L1Reco,RECO,VALIDATION  --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --datatier GEN-SIM-DIGI-RECO,DQMIO --beamspot NominalCollision2015 --era Run2_25ns : FAILED - elapsed time: 0 sec (ended on Fri Mar 15 11:16:45 2024) - exit: 256

Expand to see more addon errors ...

cmsbuild · 2024-03-15T15:47:30Z

REMINDER @sextonkennedy, @rappoccio, @antoniovilela: This PR was tested with cms-sw/cmssw-config#104, please check if they should be merged together

Dr15Jones · 2024-03-20T13:18:44Z

please test

cmsbuild · 2024-03-20T16:26:59Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-40cb9f/38297/summary.html
COMMIT: c374f9c
CMSSW: CMSSW_14_1_X_2024-03-20-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/44396/38297/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 100 lines to the logs
Reco comparison results: 51 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3297369
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3297343
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2024-03-21T08:28:09Z

@smuzaffar Just to clarify, are the scram-side changes already in? I.e. this PR can safely be merged?

makortel · 2024-03-21T08:29:00Z

Comparisons show the usual

smuzaffar · 2024-03-21T09:01:39Z

@smuzaffar Just to clarify, are the scram-side changes already in? I.e. this PR can safely be merged?

yes, scram side changes are alreay in.

makortel · 2024-03-21T12:59:47Z

+core

cmsbuild · 2024-03-21T13:00:13Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2024-03-22T13:31:45Z

+1

Moved modules.py out of edmWriteConfig

c374f9c

- is now a file in the template directory - modification of scram will use the new file and place in the correct directory.

cmsbuild added this to the CMSSW_14_1_X milestone Mar 13, 2024

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Mar 13, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 13, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Mar 13, 2024

cmsbuild added tests-rejected and removed tests-started labels Mar 13, 2024

cmsbuild added tests-started and removed tests-rejected labels Mar 14, 2024

cmsbuild added tests-rejected and removed tests-started labels Mar 14, 2024

makortel mentioned this pull request Mar 14, 2024

Moved modules.py out of edmWriteConfig cms-sw/framework-team#849

Closed

smuzaffar mentioned this pull request Mar 14, 2024

Copy all CFIPYTHON_PACKAGE_FILES files to cfipython store cms-sw/cmssw-config#104

Merged

cmsbuild added the tests-started label Mar 15, 2024

cmsbuild added tests-rejected and removed tests-started labels Mar 15, 2024

cmsbuild added tests-started and removed requires-external tests-rejected labels Mar 20, 2024

cmsbuild added tests-approved and removed tests-started labels Mar 20, 2024

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Mar 21, 2024

cmsbuild added orp-approved and removed orp-pending labels Mar 22, 2024

cmsbuild merged commit 58d132a into cms-sw:master Mar 22, 2024
17 checks passed

Dr15Jones deleted the modulesTemplate branch March 26, 2024 20:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Moved modules.py out of edmWriteConfig #44396

Moved modules.py out of edmWriteConfig #44396

Dr15Jones commented Mar 13, 2024

Dr15Jones commented Mar 13, 2024

cmsbuild commented Mar 13, 2024 •

edited

Loading

Dr15Jones commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

Dr15Jones commented Mar 14, 2024

cmsbuild commented Mar 14, 2024

Dr15Jones commented Mar 14, 2024

valsdav commented Mar 14, 2024

smuzaffar commented Mar 14, 2024

cmsbuild commented Mar 15, 2024

cmsbuild commented Mar 15, 2024

Dr15Jones commented Mar 20, 2024

cmsbuild commented Mar 20, 2024

makortel commented Mar 21, 2024

makortel commented Mar 21, 2024

smuzaffar commented Mar 21, 2024

makortel commented Mar 21, 2024

cmsbuild commented Mar 21, 2024

antoniovilela commented Mar 22, 2024

Moved modules.py out of edmWriteConfig #44396

Moved modules.py out of edmWriteConfig #44396

Conversation

Dr15Jones commented Mar 13, 2024

PR description:

PR validation:

Dr15Jones commented Mar 13, 2024

cmsbuild commented Mar 13, 2024 • edited Loading

Dr15Jones commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

cmsbuild commented Mar 13, 2024

Build

Clang Build

Dr15Jones commented Mar 14, 2024

cmsbuild commented Mar 14, 2024

Build

Clang Build

Dr15Jones commented Mar 14, 2024

valsdav commented Mar 14, 2024

smuzaffar commented Mar 14, 2024

cmsbuild commented Mar 15, 2024

RelVals

RelVals-INPUT

AddOn Tests

cmsbuild commented Mar 15, 2024

Dr15Jones commented Mar 20, 2024

cmsbuild commented Mar 20, 2024

Comparison Summary

makortel commented Mar 21, 2024

makortel commented Mar 21, 2024

smuzaffar commented Mar 21, 2024

makortel commented Mar 21, 2024

cmsbuild commented Mar 21, 2024

antoniovilela commented Mar 22, 2024

cmsbuild commented Mar 13, 2024 •

edited

Loading