2024 Data RelVals and `InputInfo` Events Skimming (actually) Working #45055

AdrianoDee · 2024-05-27T13:38:38Z

PR description:

This PR proposes few things:

the introduction of a new script das-up-to-nevents.py that, given in input a dataset name, allows to get the list of files corresponding to N events. The dataset is also automatically skimmed to include only the run and the lumis from the latest and greatest Golden data certification json;
a fix to the fact that the InputInfo class, while accepting as input a variable events does not really use it to skim the result of the das query. It's only really used in a log. I find it ambiguous and a bit confusing. In practice it also means that for 2023 and 2022 RelVals we have been running on O(few millions) events per pre-release (see e.g. CMSSW_14_1_0_pre3__Data_Run3_2023_Data-RunEGamma2023D). While this is nice in terms of being able to do that (kudos on computing!) it usually slows down a lot the RelVal production. This leverages on the script introduced in (1).
the addition of 2024 data RelVals for Eras from B to F as standard wfs with 2024.* prefix. These are run with:
- Run3_2024 era;
- HLT:@relval2024 (currently pointing at GRun);
- running on 50k events using the script in (1) and the script in (2)
a new set of RelVals under relvals_data_highstats.py that should include all the higher statistics data RelVals workflows with 150k,250k,500k and 1M events. For the moment it includes only the 2024 wfs. This set is not turned on by default to avoid to have an over crowded matrix.

All the needed input RAW samples have been copied via Rucio at T2_CH_CERN.

PR validation:

Running (e.g.) runTheMatrix.py -w data,standard -l 12024.003015, 2024.00405, 2024.007005, 2024.0091, 2024.012005, 2024.013025, 2024.10305, 2024.108005, 2024.108015, 2024.210025, 2024.211015, 2024.2111, 2024.212025, 2024.213015, 2024.214025, 2024.21405, 2024.303015, 2024.310025, 2024.31305, 2024.3141, 2024.400005, 2024.402015

cmsbuild · 2024-05-27T13:39:06Z

cms-bot internal usage

AdrianoDee · 2024-05-27T13:41:47Z

test parameters:

workflows = 143.0010,143.0011,143.0020,143.0021,143.0030,143.0031,143.0040,143.0041,143.0050,143.0051,143.0060,143.0061,143.0070,143.0071,143.0080,143.0081,143.0090,143.0091,143.0100,143.0101,143.0110,143.0111,143.0120,143.0121,143.0130,143.0131,143.0140,143.0141,143.0150,143.0151,143.0161,143.0171,143.0181,143.0191,143.0201,143.0211,143.0221,143.0231,143.0241,143.0251,143.0261,143.0271,143.0281,143.0291,143.0301

cmsbuild · 2024-05-27T13:43:01Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45055/40365

This PR adds an extra 72KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): Add D110 Default Input Dataset #44965, Introducing SimOnGen and GenOnly wfs for 2024 and 2026D110 #45005

cmsbuild · 2024-05-27T13:43:24Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)

@sunilUIET, @AdrianoDee, @subirsarkar, @srimanob, @miquork, @cmsbuild can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @slomeo, @makortel, @fabiocos, @missirol this is something you requested to watch as well.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

AdrianoDee · 2024-05-27T13:51:02Z

please test

cmsbuild · 2024-05-27T19:28:35Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-016c90/39543/summary.html
COMMIT: d040c0f
CMSSW: CMSSW_14_1_X_2024-05-27-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45055/39543/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

The relvals timed out after 4 hours.

143.007

----- Begin Fatal Exception 27-May-2024 17:18:31 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024C/MuonEG/RAW/v1/000/380/049/00000/02ccb064-fb1b-4042-9449-0ebf9b2118ec.root?tried=+1213llrxrd-redir.in2p3.fr,&xrdcl.requuid=6501fbba-46b2-40bb-a3ad-e500e516a3b8
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

143.001

----- Begin Fatal Exception 27-May-2024 17:18:33 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024C/BTagMu/RAW/v1/000/380/049/00000/0023904f-d84c-4d4d-b230-9858791afb81.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=d8e00365-3d23-40da-b5f8-0816ee8d4f15
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

143.003

----- Begin Fatal Exception 27-May-2024 17:18:34 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024C/DisplacedJet/RAW/v1/000/380/049/00000/009b79a4-1719-4984-9e56-75c51e500694.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=003f9d3b-3f9d-4d08-a69a-654a395ec995
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

143.001143.001_RunBTagMu2024C/step2_RunBTagMu2024C.log
143.003143.003_RunDisplacedJet2024C/step2_RunDisplacedJet2024C.log
143.007143.007_RunMuonEG2024C/step2_RunMuonEG2024C.log

Expand to see more relval errors ...

AdrianoDee · 2024-05-28T08:35:35Z

test parameters:

workflows = 143.0011,143.0021,143.0031,143.0041,143.0051,143.0061,143.0071,143.0081,143.0091,143.0101,143.0111,143.0121,143.0131,143.0141,143.0151,143.0161,143.0171,143.0181,143.0191,143.0201,143.0211,143.0221,143.0231,143.0241,143.0251,143.0261,143.0271,143.0281,143.0291,143.0301

AdrianoDee · 2024-05-28T08:36:03Z

please test
(running only with the 100k wfs for which I have copied the samples at T2_CH_CERN)

cmsbuild · 2024-05-28T14:16:49Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-016c90/39563/summary.html
COMMIT: d040c0f
CMSSW: CMSSW_14_1_X_2024-05-27-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45055/39563/install.sh to create a dev area with all the needed externals and cmssw changes.

DAS Queries: The DAS query tests failed, see the summary page for details.

RelVals

The relvals timed out after 4 hours.

143.0111

----- Begin Fatal Exception 28-May-2024 12:10:37 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024C/ParkingLLP/RAW/v1/000/380/049/00000/08bbd238-ac10-41e4-b192-818b4a07e7a3.root?tried=+1213xrootd-redic.pi.infn.it,&xrdcl.requuid=5aea3c1c-c8fa-402b-be6b-329054739bfe
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

143.0191

----- Begin Fatal Exception 28-May-2024 12:10:38 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024D/EGamma0/RAW/v1/000/380/310/00000/112e3f5b-2543-4b35-aca4-fe2d37f3fd40.root?tried=+1213xrootd-redic.pi.infn.it,&xrdcl.requuid=95aabfe7-3b5b-4b6d-b3f1-850e8d7174c6
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

143.0101

----- Begin Fatal Exception 28-May-2024 12:10:38 CEST-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root'
   Additional Info:
      [a] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://cms-xrd-global.cern.ch//eos/cms/store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root
      [b] Calling RootInputFileSequence::initTheFile(): fail to open the file with name root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root
      [c] Input file root://xrootd-cms.infn.it//store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root could not be opened.
      [d] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [e] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/data/Run2024C/ParkingHH/RAW/v1/000/380/049/00000/001f71ed-c672-470d-a202-2c437d6deead.root?tried=+1213xrootd-cms-redir-int.cr.cnaf.infn.it,&xrdcl.requuid=1d1c026a-f637-47e3-96be-1c0393fabb3e
      [f] Problematic data server: cms-xrd-global.cern.ch:1094
      [g] Disabled source: cms-xrd-global.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

143.001143.001_RunBTagMu2024C/step2_RunBTagMu2024C.log
143.003143.003_RunDisplacedJet2024C/step2_RunDisplacedJet2024C.log
143.006143.006_RunMuon12024C/step2_RunMuon12024C.log

Expand to see more relval errors ...

143.007
143.009
143.0061
143.01
143.0031
143.011
143.0101
143.0111
143.012
143.013
143.014
143.0121
143.016
143.017
143.018
143.019
143.021
143.0191
143.023
143.022
143.0211
143.024
143.0231
143.025
143.026
143.0241
143.027
143.028
143.029
143.0281
143.0021
143.0071
143.0081
143.0091
143.0131
143.0141
143.0151
143.0161
143.0171
143.0181
143.0201
143.0221
143.0251
143.0261
143.0271
143.0291
143.0301

AdrianoDee · 2024-05-28T15:32:15Z

Ok, some files are missing a copy to T2_CH_CERN. Asked for the replicas. Will be done soon.

AdrianoDee · 2024-05-31T08:24:26Z

please test

cmsbuild · 2024-08-14T17:32:44Z

+1

Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-016c90/40921/summary.html
COMMIT: 92aef4d
CMSSW: CMSSW_14_1_X_2024-08-14-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/45055/40921/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 13 differences found in the comparisons
DQMHistoTests: Total files compared: 45
DQMHistoTests: Total histograms compared: 3422510
DQMHistoTests: Total failures: 15
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3422475
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 44 files compared)
Checked 196 log files, 165 edm output root files, 45 DQM output files
TriggerResults: no differences found

AdrianoDee · 2024-08-14T19:01:49Z

+pdmv

srimanob · 2024-08-19T12:16:25Z

+Upgrade

cmsbuild · 2024-08-19T12:16:47Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2024-08-20T10:59:56Z

+1

smuzaffar · 2024-08-21T14:49:38Z

@AdrianoDee , looks like after this update many relvals are failing for 14.1.X Ibs.

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/el8_amd64_gcc12/CMSSW_14_1_X_2024-08-20-2300/pyRelValMatrixLogs/run/2024.305005_RunMuon02024E_50k/step1_dasquery.log

2024/08/21 06:36:20 ERROR DAS QL ERROR, query=file , lumi , run dataset = /Muon0/Run2024E-v1/RAW, idx=4, msg=unable to parse DAS query
Traceback (most recent call last):
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 934, in _finalize_columns_and_data
    columns = _validate_or_indexify_columns(contents, columns)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 981, in _validate_or_indexify_columns
    raise AssertionError(
AssertionError: 3 columns passed, passed data had 21 columns

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_14_1_X_2024-08-20-2300/bin/el8_amd64_gcc12/das-up-to-nevents.py", line 136, in <module>
    df = das_lumi_data(dataset).merge(das_file_data(dataset),on="file",how="inner") # merge file informations with run and lumis
  File "/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_14_1_X_2024-08-20-2300/bin/el8_amd64_gcc12/das-up-to-nevents.py", line 58, in das_lumi_data
    df = pd.DataFrame(out,columns=["file","run","lumis"])
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/frame.py", line 782, in __init__
    arrays, columns, index = nested_data_to_arrays(
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 498, in nested_data_to_arrays
    arrays, columns = to_arrays(data, columns, dtype=dtype)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 840, in to_arrays
    content, columns = _finalize_columns_and_data(arr, columns, dtype)
  File "/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02851/el8_amd64_gcc12/external/py3-pandas/2.0.1-33b108e9d19def8c76b7c0615ce106ab/lib/python3.9/site-packages/pandas/core/internals/construction.py", line 937, in _finalize_columns_and_data
    raise ValueError(err) from err
ValueError: 3 columns passed, passed data had 21 columns

AdrianoDee · 2024-08-21T15:08:22Z

@smuzaffar let me have a look

cmsbuild added this to the CMSSW_14_1_X milestone May 27, 2024

cmsbuild added pending-signatures tests-pending orp-pending pdmv-pending upgrade-pending code-checks-pending labels May 27, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels May 27, 2024

cmsbuild added tests-started and removed tests-pending labels May 27, 2024

cmsbuild mentioned this pull request May 27, 2024

Introducing SimOnGen and GenOnly wfs for 2024 and 2026D110 #45005

Merged

cmsbuild added tests-rejected and removed tests-started labels May 27, 2024

cmsbuild added tests-started and removed tests-rejected labels May 28, 2024

cmsbuild added tests-rejected and removed tests-started labels May 28, 2024

cmsbuild removed the tests-rejected label May 31, 2024

cmsbuild added the tests-started label Aug 14, 2024

cmsbuild added tests-approved and removed tests-started labels Aug 14, 2024

cmsbuild added pdmv-approved and removed pdmv-pending labels Aug 14, 2024

cmsbuild mentioned this pull request Aug 16, 2024

Having Upgrade Matrix Failing for Duplicate Suffixes #45715

Merged

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Aug 19, 2024

cmsbuild added orp-approved and removed orp-pending labels Aug 20, 2024

cmsbuild merged commit 18f37d1 into cms-sw:master Aug 20, 2024
12 checks passed

This was referenced Aug 20, 2024

Issue7955 type first spec 2 cms-sw/root#208

Open

Added new ZDCHitReconstructor and ZDCRecAlgo to work with QIE10 Digis and Updated ZDCRecHit #45407

Merged

[TF] Update TF v2.16.1 cms-sw/cmsdist#9241

Merged

cmsbuild mentioned this pull request Aug 22, 2024

Apply suggested fix for Wdangling-reference #45765

Merged

AdrianoDee deleted the relval_2024_and_input_fix branch August 22, 2024 07:56

AdrianoDee restored the relval_2024_and_input_fix branch August 22, 2024 07:56

AdrianoDee mentioned this pull request Aug 22, 2024

Remove Duplicate RelVals from relval_highstats.py and Smaller Queries for das-up-to-nevents.py #45774

Merged

AdrianoDee mentioned this pull request Aug 29, 2024

Adding Golden Workflows for 2022 and 2023 Data #45836

Merged

AdrianoDee deleted the relval_2024_and_input_fix branch September 2, 2024 08:41

AdrianoDee mentioned this pull request Sep 8, 2024

Allow Event-based Input Skim with das-up-to-nevents.py cms-PdmV/RelVal#131

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2024 Data RelVals and `InputInfo` Events Skimming (actually) Working #45055

2024 Data RelVals and `InputInfo` Events Skimming (actually) Working #45055

AdrianoDee commented May 27, 2024 •

edited

Loading

cmsbuild commented May 27, 2024 •

edited

Loading

AdrianoDee commented May 27, 2024

cmsbuild commented May 27, 2024

cmsbuild commented May 27, 2024

AdrianoDee commented May 27, 2024

cmsbuild commented May 27, 2024

AdrianoDee commented May 28, 2024

AdrianoDee commented May 28, 2024

cmsbuild commented May 28, 2024

AdrianoDee commented May 28, 2024

AdrianoDee commented May 31, 2024

cmsbuild commented Aug 14, 2024

AdrianoDee commented Aug 14, 2024

srimanob commented Aug 19, 2024

cmsbuild commented Aug 19, 2024

mandrenguyen commented Aug 20, 2024

smuzaffar commented Aug 21, 2024

AdrianoDee commented Aug 21, 2024

2024 Data RelVals and InputInfo Events Skimming (actually) Working #45055

2024 Data RelVals and InputInfo Events Skimming (actually) Working #45055

Conversation

AdrianoDee commented May 27, 2024 • edited Loading

PR description:

PR validation:

cmsbuild commented May 27, 2024 • edited Loading

AdrianoDee commented May 27, 2024

cmsbuild commented May 27, 2024

cmsbuild commented May 27, 2024

AdrianoDee commented May 27, 2024

cmsbuild commented May 27, 2024

RelVals

RelVals-INPUT

AdrianoDee commented May 28, 2024

AdrianoDee commented May 28, 2024

cmsbuild commented May 28, 2024

RelVals

RelVals-INPUT

AdrianoDee commented May 28, 2024

AdrianoDee commented May 31, 2024

cmsbuild commented Aug 14, 2024

Comparison Summary

AdrianoDee commented Aug 14, 2024

srimanob commented Aug 19, 2024

cmsbuild commented Aug 19, 2024

mandrenguyen commented Aug 20, 2024

smuzaffar commented Aug 21, 2024

AdrianoDee commented Aug 21, 2024

2024 Data RelVals and `InputInfo` Events Skimming (actually) Working #45055

2024 Data RelVals and `InputInfo` Events Skimming (actually) Working #45055

AdrianoDee commented May 27, 2024 •

edited

Loading

cmsbuild commented May 27, 2024 •

edited

Loading