Backport `ConditionalTask` to CMSSW_12_3_X #37977

fwyzard · 2022-05-17T11:27:51Z

PR description:

This PR backports from CMSSW 12.4.x/12.5.x the PRs that implement the ConditionalTask:

Added ConditionalTask #37305: Added ConditionalTask
Make SwitchProducer work with ConditionalTask #37563: Make SwitchProducer work with ConditionalTask
Fix dumpPython() for Task and ConditionalTask when they contain > 255 elements #37872: Fix dumpPython() for Task and ConditionalTask when they contain > 255 elements
Fix behavior of SwitchProducer in ConditionalTask with non-chosen EDAlias case(s) that are consumed explicitly with callWhenNewProductsRegistered() #38006: Fix behavior of SwitchProducer in ConditionalTask with non-chosen EDAlias case(s) that are consumed explicitly

PR validation:

Unit tests in FWCore/Integration pass.

if this PR is a backport please specify the original PR and why you need to backport that PR:

Backports #37305, #37563, #37872, #38006.

- added fillDescriptions - fixed a bug when filter not supposed to produce a data product - allow testing for missing data products

Use proper module label and avoid reusing a module.

- avoid unnecessary check on alias finding - break out of loop properly

The bit position of a module onto a path does not have to be the same as the modules order to be run on the path. This is needed to allow ConditionalTasks to be added to a path in a set run order but still be able to use the unaltered parameter value holding the list of module labels in order to later find the module labels. All ConditionalTask's modules are added to the end of the parameter and are bracketed by two non-module labels [# and @]. Modules added explicitly to the path are at the front of the parameter and in their run order. The TriggerNamesService knows how to translate the path bit position into the proper module label stored in the parameter.

This allows sharing of this code across two cases.

In preparation of using it for SwitchProducer in ConditionalTask whose chosen case is EDAlias.

…Task

… elements

cmsbuild · 2022-05-17T11:28:20Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_12_3_X.

It involves the following packages:

DataFormats/Provenance (core)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/ParameterSet (core)
FWCore/Utilities (core)
HLTrigger/HLTcore (hlt)

@Martin-Grunewald, @Dr15Jones, @smuzaffar, @makortel, @cmsbuild, @missirol can you please review it and eventually sign? Thanks.
@missirol, @makortel, @felicepantaleo, @rovere, @Martin-Grunewald, @silviodonato, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Backported from Added ConditionalTask #37305

fwyzard · 2022-05-17T11:28:31Z

This PR is probably not needed in the release; for the time being it is useful for local tests.

fwyzard · 2022-05-17T11:29:37Z

@cmsbuild, please test

fwyzard · 2022-05-17T11:30:05Z

backport #37305, #37563, #37872, #38006

missirol · 2022-05-17T11:37:30Z

Was #37562 also needed? Based on #37563 (comment).

fwyzard · 2022-05-17T11:39:31Z

I thought so, but it looks like #37562 is already included in #37933, which has already been merged into CMSSW_12_3_X .

cmsbuild · 2022-05-17T17:17:45Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-50df53/24781/summary.html
COMMIT: 0bf12a4
CMSSW: CMSSW_12_3_X_2022-05-17-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37977/24781/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test SiStripDAQ_O2O_test had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 11 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3696954
DQMHistoTests: Total failures: 18
DQMHistoTests: Total nulls: 14
DQMHistoTests: Total successes: 3696900
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2.0449999999999995 KiB( 48 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 0.031 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 1000.0,... ): 0.031 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 11634.0,... ): 0.027 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 11634.0,... ): 0.027 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 23234.0,... ): 0.023 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 23234.0,... ): 0.023 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.22,... ): 0.012 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 4.22,... ): 0.012 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

fwyzard · 2022-05-17T21:48:03Z

---> test SiStripDAQ_O2O_test had ERRORS

As usual ...

…fg.py

…lias case(s) that are consumed explicitly

cmsbuild · 2022-05-19T09:26:02Z

Pull request #37977 was updated. @Martin-Grunewald, @Dr15Jones, @smuzaffar, @makortel, @cmsbuild, @missirol can you please check and sign again.

missirol · 2022-05-19T09:30:38Z

backport of #37305
backport of #37563
backport of #37872
backport of #38006

backport label was missing :)

fwyzard · 2022-05-19T09:42:42Z

@cmsbuild, please test

cmsbuild · 2022-05-19T16:17:41Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-50df53/24852/summary.html
COMMIT: 762ae3c
CMSSW: CMSSW_12_3_X_2022-05-18-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37977/24852/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test SiStripDAQ_O2O_test had ERRORS

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 9 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3627938
DQMHistoTests: Total failures: 18
DQMHistoTests: Total nulls: 14
DQMHistoTests: Total successes: 3627884
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 2.0529999999999995 KiB( 48 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 0.031 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 1000.0,... ): 0.031 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 11634.0,... ): 0.027 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 11634.0,... ): 0.027 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 23234.0,... ): 0.023 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 23234.0,... ): 0.023 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.22,... ): 0.012 KiB MessageLogger/Errors
DQMHistoSizes: changed ( 4.22,... ): 0.012 KiB MessageLogger/Warnings
Checked 204 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

makortel · 2022-05-20T13:51:34Z

This PR is probably not needed in the release; for the time being it is useful for local tests.

Just to clarify, is this backport now considered as necessary for 12_3_X? (reading joint ops meeting notes)

fwyzard · 2022-05-20T13:53:00Z

We are discussing inside TSG if this would be needed or not.

Dr15Jones and others added 17 commits May 17, 2022 13:20

Improve testing modules

509272d

- added fillDescriptions - fixed a bug when filter not supposed to produce a data product - allow testing for missing data products

Added ConditionalTask object

6fe4313

Added code to handle ConditionalTasks

5fb1fdf

Improvements based on review comments

6ebe2ec

Added ConditionalTask test using consumesMany

f4212cc

Fix mayConsumes handling for ConditionalTasks

0206fb1

Use proper module label and avoid reusing a module.

Improved handling of regular consumes for ConditionalTasks

deab3ae

- avoid unnecessary check on alias finding - break out of loop properly

Update HLTConfigData to handle ConditionalTasks

5ca2959

Moved path configuration parsing to a standalone function

8ab785f

This allows sharing of this code across two cases.

Add tests for SwitchProducer in ConditionalTask

8099582

Propagate module labels to SwitchProducer case modules

1f8b0ed

Make SwitchProducer to work with ConditionalTask

b8cc3f6

Make processAliases() callable outside Schedule.cc

4dd72bb

In preparation of using it for SwitchProducer in ConditionalTask whose chosen case is EDAlias.

Make SwitchProducer with EDAlias chosen case to work with Conditional…

8f9de4d

…Task

Fix EDAlias treatment for ConditionalTask

242576c

Fix dumpPython() for Task and ConditionalTask when they contain > 255…

0bf12a4

… elements

cmsbuild added this to the CMSSW_12_3_X milestone May 17, 2022

cmsbuild added core-pending hlt-pending orp-pending pending-signatures tests-pending labels May 17, 2022

cmsbuild added tests-started and removed tests-pending labels May 17, 2022

cmsbuild added tests-rejected and removed tests-started labels May 17, 2022

makortel added 2 commits May 19, 2022 11:25

Add some clarifications to testSwitchProducerConditionalTaskEDAlias_c…

8555657

…fg.py

Fix behavior of SwitchProducer in ConditionalTask with non-chosen EDA…

762ae3c

…lias case(s) that are consumed explicitly

cmsbuild added tests-pending and removed tests-rejected labels May 19, 2022

cmsbuild added the backport-ok label May 19, 2022

cmsbuild added tests-started and removed tests-pending labels May 19, 2022

cmsbuild added tests-rejected and removed tests-started labels May 19, 2022

fwyzard closed this Jun 18, 2022

fwyzard deleted the backport_ConditionalTask_to_12.3.x branch July 31, 2022 13:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport `ConditionalTask` to CMSSW_12_3_X #37977

Backport `ConditionalTask` to CMSSW_12_3_X #37977

fwyzard commented May 17, 2022 •

edited

Loading

cmsbuild commented May 17, 2022 •

edited

Loading

fwyzard commented May 17, 2022

fwyzard commented May 17, 2022

fwyzard commented May 17, 2022 •

edited

Loading

missirol commented May 17, 2022

fwyzard commented May 17, 2022

cmsbuild commented May 17, 2022

fwyzard commented May 17, 2022

cmsbuild commented May 19, 2022

missirol commented May 19, 2022

fwyzard commented May 19, 2022

cmsbuild commented May 19, 2022

makortel commented May 20, 2022

fwyzard commented May 20, 2022 via email

Backport ConditionalTask to CMSSW_12_3_X #37977

Backport ConditionalTask to CMSSW_12_3_X #37977

Conversation

fwyzard commented May 17, 2022 • edited Loading

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented May 17, 2022 • edited Loading

fwyzard commented May 17, 2022

fwyzard commented May 17, 2022

fwyzard commented May 17, 2022 • edited Loading

missirol commented May 17, 2022

fwyzard commented May 17, 2022

cmsbuild commented May 17, 2022

Unit Tests

Comparison Summary

fwyzard commented May 17, 2022

cmsbuild commented May 19, 2022

missirol commented May 19, 2022

fwyzard commented May 19, 2022

cmsbuild commented May 19, 2022

Unit Tests

Comparison Summary

makortel commented May 20, 2022

fwyzard commented May 20, 2022 via email

Backport `ConditionalTask` to CMSSW_12_3_X #37977

Backport `ConditionalTask` to CMSSW_12_3_X #37977

fwyzard commented May 17, 2022 •

edited

Loading

cmsbuild commented May 17, 2022 •

edited

Loading

fwyzard commented May 17, 2022 •

edited

Loading