Allow an InputSource to declare a run/lumi is the last to be merged #43457

wddgit · 2023-11-30T23:06:47Z

PR description:

Allow an InputSource to declare that a run entry or a lumi entry is the last one that needs to be merged. The next run entry should have a different run number or ProcessHistoryID. Or the next lumi entry should be associated with a different run or have a different luminosity block number. Or the next run or lumi entry could be in a different file.

It is OK for InputSources not to make this declaration. The Framework can and already does figure this out by examining the next entry returned by getNextItemType. Existing sources should work OK.

This might be a useful optimization in an online source when the getNextItemType function takes a long time to return. If the last item is declared, then the global run transition or the global lumi transition can begin immediately. Otherwise, the Framework has to wait for getNextItemType to return for the next entry in order to know that the previous entry was the last. And that delays global begin run or global begin lumi.

This probably will not help much for offline input sources because getNextItemType returns quickly.

The Framework will detect the case where a source falsely declares an item is the last run or lumi entry. It will throw an exception indicating a bug in the InputSource.

This delay was noticed when investigating issue #42931, but this PR does not address either of the problems listed there.

There are some minor changes in the InputSource interface. The getNextItemType function now returns an object of type ItemTypeInfo instead of the enum ItemType. Also, the ItemType enum is changed to be an enum class (more of a style modernization, I could back that out if requested).

PR validation:

Three new unit tests are added for this feature. Existing core unit tests pass. Limited runTheMatrix passes.

Manually, you can also see the delay vanishes in the log file of one of the unit tests, but it is difficult to turn that into a pass/fail criteria as the timing of execution can vary and we don't want tests that occasionally fail due to timing variations.

cmsbuild · 2023-11-30T23:12:36Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43457/38008

This PR adds an extra 152KB to repository

cmsbuild · 2023-11-30T23:13:02Z

A new Pull Request was created by @wddgit (W. David Dagenhart) for master.

It involves the following packages:

DQMServices/FwkIO (dqm)
DQMServices/StreamerIO (dqm)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/Modules (core)
FWCore/Sources (core)
IOPool/Input (core)

@rvenditti, @cmsbuild, @antoniovagnerini, @makortel, @syuvivida, @tjavaid, @smuzaffar, @nothingface0, @Dr15Jones can you please review it and eventually sign? Thanks.
@missirol, @barvic, @makortel this is something you requested to watch as well.
@rappoccio, @sextonkennedy, @antoniovilela you are the release manager for this.

cms-bot commands are listed here

wddgit · 2023-11-30T23:13:33Z

please test

cmsbuild · 2023-12-01T01:59:13Z

-1

Failed Tests: AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-274562/36219/summary.html
COMMIT: 7b27eae
CMSSW: CMSSW_14_0_X_2023-11-30-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43457/36219/install.sh to create a dev area with all the needed externals and cmssw changes.

AddOn Tests

hlt_data_GRun

----- Begin Fatal Exception 01-Dec-2023 00:47:00 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
   Additional Info:
      [a] Another exception was caught while trying to clean up runs after the primary fatal exception.
----- End Fatal Exception -------------------------------------------------

hlt_data_HIon

----- Begin Fatal Exception 01-Dec-2023 00:47:03 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
   Additional Info:
      [a] Another exception was caught while trying to clean up runs after the primary fatal exception.
----- End Fatal Exception -------------------------------------------------

hlt_data_PIon

----- Begin Fatal Exception 01-Dec-2023 00:46:43 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Calling EventProcessor::runToCompletion (which does almost everything after beginJob and before endJob)
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)
   Additional Info:
      [a] Another exception was caught while trying to clean up runs after the primary fatal exception.
----- End Fatal Exception -------------------------------------------------

Comparison Summary

Summary:

You potentially removed 431 lines from the logs
Reco comparison results: 13 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3370032
DQMHistoTests: Total failures: 95
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3369915
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

wddgit · 2023-12-01T16:35:10Z

please test

Try again. See if the failures are reproducible. The only thing I noticed so far is that the 3 failures occurred within 20 seconds of each other in real time. Maybe some system glitch...

wddgit · 2023-12-01T18:33:15Z

abort test

cmsbuild · 2023-12-01T19:18:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43457/38041

This PR adds an extra 152KB to repository

wddgit · 2023-12-01T19:18:40Z

please test

Try again. I missed adding ServiceRegistry::Operate operate(serviceToken_); in a spot where it was needed. I squashed and force pushed a fix.

wddgit · 2023-12-11T22:15:15Z

I implemented all Chris's suggestions except for making the ItemTypeInfo constructor explicit (see comment above). Thanks, it is better. Also rebased. I'll rerun the tests when code checks passes.

cmsbuild · 2023-12-11T22:17:46Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43457/38140

This PR adds an extra 56KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/EventProcessor.h modified in PR(s): Allow a stream to skip a lumi if all events processed #43522
- File FWCore/Framework/src/EventProcessor.cc modified in PR(s): Allow a stream to skip a lumi if all events processed #43522

cmsbuild · 2023-12-11T22:18:15Z

Pull request #43457 was updated. @syuvivida, @nothingface0, @rvenditti, @antoniovagnerini, @tjavaid, @smuzaffar, @makortel, @Dr15Jones, @cmsbuild can you please check and sign again.

wddgit · 2023-12-11T22:22:03Z

please test

cmsbuild · 2023-12-12T03:42:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-274562/36423/summary.html
COMMIT: 1b3d395
CMSSW: CMSSW_14_0_X_2023-12-11-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43457/36423/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 377 lines to the logs
Reco comparison results: 16 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3430794
DQMHistoTests: Total failures: 1274
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3429498
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 214 log files, 167 edm output root files, 50 DQM output files
TriggerResults: no differences found

makortel · 2023-12-12T15:41:52Z

The usual set of

makortel · 2023-12-13T22:35:53Z

+core

makortel · 2023-12-14T15:01:07Z

@cms-sw/dqm-l2 Could you review and sign? The changes in your area should be straightforward.

tjavaid · 2023-12-14T16:45:34Z

+1

with spurious differences coming from WFs mentioned above by @makortel .

cmsbuild · 2023-12-14T16:46:01Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @antoniovilela, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

antoniovilela · 2023-12-19T15:34:59Z

+1

cmsbuild added this to the CMSSW_14_0_X milestone Nov 30, 2023

cmsbuild added dqm-pending core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 30, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 30, 2023

cmsbuild added tests-started and removed tests-pending labels Nov 30, 2023

cmsbuild added tests-rejected and removed tests-started labels Dec 1, 2023

cmsbuild added tests-started and removed tests-rejected labels Dec 1, 2023

makortel mentioned this pull request Dec 1, 2023

Add ability for Source to tell that a new LuminosityBlock is the only occurrence of that LuminosityBlock cms-sw/framework-team#716

Closed

cmsbuild added tests-pending and removed tests-started labels Dec 1, 2023

wddgit force-pushed the lastRunOrLumiToBeMerged branch from 7b27eae to a4df6b8 Compare December 1, 2023 19:10

cmsbuild added code-checks-pending and removed code-checks-approved labels Dec 1, 2023

cmsbuild removed the code-checks-pending label Dec 1, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 11, 2023

cmsbuild added tests-started and removed tests-pending labels Dec 11, 2023

cmsbuild added tests-approved and removed tests-started labels Dec 12, 2023

cmsbuild added core-approved and removed core-pending labels Dec 13, 2023

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Dec 14, 2023

cmsbuild added orp-approved and removed orp-pending labels Dec 19, 2023

cmsbuild merged commit 71e2c07 into cms-sw:master Dec 19, 2023
11 checks passed

This was referenced Dec 19, 2023

[14.0.X] Updated root to tip of branch v6-30-00-patches cms-sw/cmsdist#8896

Closed

Materialize terror symbols cms-sw/root#191

Closed

First PyTorch tests for TorchScript inference CPU/CUDA #43475

Merged

wddgit deleted the lastRunOrLumiToBeMerged branch March 11, 2024 14:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow an InputSource to declare a run/lumi is the last to be merged #43457

Allow an InputSource to declare a run/lumi is the last to be merged #43457

wddgit commented Nov 30, 2023

cmsbuild commented Nov 30, 2023

cmsbuild commented Nov 30, 2023 •

edited

Loading

wddgit commented Nov 30, 2023

cmsbuild commented Dec 1, 2023

wddgit commented Dec 1, 2023

wddgit commented Dec 1, 2023

cmsbuild commented Dec 1, 2023

wddgit commented Dec 1, 2023

wddgit commented Dec 11, 2023 •

edited

Loading

cmsbuild commented Dec 11, 2023

cmsbuild commented Dec 11, 2023

wddgit commented Dec 11, 2023

cmsbuild commented Dec 12, 2023

makortel commented Dec 12, 2023

makortel commented Dec 13, 2023

makortel commented Dec 14, 2023

tjavaid commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

antoniovilela commented Dec 19, 2023

Allow an InputSource to declare a run/lumi is the last to be merged #43457

Allow an InputSource to declare a run/lumi is the last to be merged #43457

Conversation

wddgit commented Nov 30, 2023

PR description:

PR validation:

cmsbuild commented Nov 30, 2023

cmsbuild commented Nov 30, 2023 • edited Loading

wddgit commented Nov 30, 2023

cmsbuild commented Dec 1, 2023

AddOn Tests

Comparison Summary

wddgit commented Dec 1, 2023

wddgit commented Dec 1, 2023

cmsbuild commented Dec 1, 2023

wddgit commented Dec 1, 2023

wddgit commented Dec 11, 2023 • edited Loading

cmsbuild commented Dec 11, 2023

cmsbuild commented Dec 11, 2023

wddgit commented Dec 11, 2023

cmsbuild commented Dec 12, 2023

Comparison Summary

makortel commented Dec 12, 2023

makortel commented Dec 13, 2023

makortel commented Dec 14, 2023

tjavaid commented Dec 14, 2023

cmsbuild commented Dec 14, 2023

antoniovilela commented Dec 19, 2023

cmsbuild commented Nov 30, 2023 •

edited

Loading

wddgit commented Dec 11, 2023 •

edited

Loading