Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859

oljemark · 2023-06-02T19:03:39Z

PR description:

For the TOTEM special foreseen on 25.6.2023, the new T2 telescope will be used (and removed after the run) to measure inelastic rate and veto diffractive p+p collisions, when measuring the elastic p+p cross section.
Previously, PR #41472 introduced the 2-channel unpacking for T2 and PR #41777 included some T2 DQM plots and fixes. As part of the approval for PR #41777 , some fixes and syntax safety improvements were asked for, see #41777 (comment) . This meant that the tracked T2 parameter useOlderT2TestFile was added to several python config files in EventFilter/CTPPSRawToDigi, and also some other cosmetic changes to those files.
Last sunday @perrotta wrote a PR (#41784, #41785 ) to implement the asked-for changes, from which I include most changes into this PR.

Last week we also obtained new T2 test files with a new mapping (the final version), and a more diverse set of emulated T2 data frames. (Run 368080 had LE distributed around 5 and TE ~ 12 for all events, Run 368081 same LE value but TE ~9, and Run 368082 same as 368080 but with tracks in half the planes only). Testing the code with these new files exposed several more bugs in the T2 code in the common PPS unpacker.

On receiving more feedback from the detector experts, it turned out that one more member needed to be added to the T2Digi definition in DataFormats/TotemReco , to access header flags. @vavati also suggested that the DQM additions in PR #41777 needed to be expanded by adding the T2 as a DQMCalibrationSource, necessitating changes to RecoPPS/Configuration, also including the XML T2 geometry in a RecoPPS/Local T2 config file since the geometry was used in both the T2Rechit producer and T2 DQM module, but was not found in the database as far as we looked (thanks, @grzanka !) . The multichannel unpacker from PR #41472 was also turned on by default, needing era modifiers for 2016-2018 PPS data taken without the T2 detector, for which the dummy mapping file mapping_totem_nt2_2021.xml caused an exception since it was copied from 2017 diamond mapping without the changes in T2 xml schema introduced in PR #41472.

At present the T2 CRC field in the VFAT frame in the test files are still just a constant stub, but in the next version of the T2 firmware expected within a few days this will be calculated, and can validate the present calculation in the code. The code is using the same CRC formula as the RP, but due to the inverted order of reading the VFAT frame between T2 & Roman Pots, this needed new helper functions, as did the VFAT signature 'A','C','E' header word intializer checked by the footprint helper.

PR validation:

A subset of runTheMatrix tests relevant for the PPS detectors that might be affected by our code changes was run successfully after the era modifier fix noted above. Those were 136.793 & 1041 & 1042. The T2 unit test under RecoPPS/Local passed, while previously one unit test in in CalibPPS/TimingCalibration & 25 DQM/Integration were seen to fail when run on lxplus due to the files being on tape, but succeed when cmsbot/Jenkins ran on locally cached copies, see this comment: #41785 (comment)

The PR #41785 still passed the tests even so ( #41785 (comment) ), although more changes have been added than is included there.

The T2 unit test in RecoPPS/Local was successfully run with the older definition & xml mapping, producing T2Digis, and likewise the totemt2_dqm_test_nino_cfg.py script on last week's new files, produced the expected DQM plots. The totemt2_dqm_test_common_cfg.py script was run on PPS alignment run data with the other PPS detectors connected but not T2, showing that even though we include T2 in the common CTPPS RawToDigi and Reconstruction tasks, this does not negatively affect the other PPS detectors.
DQM_V0001_CTPPS_R000366192.zip

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

These three PR's, #41472 , #41777 and this one should be backported either to a special release on the 13_0_X cycle or the appropriate regular release on the 13_0_X . At a lower priority, backporting will also be done to 13_1_X.

cmsbuild · 2023-06-02T19:10:31Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41859/35761

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/CTPPS/test/totemt2_dqm_test_cfg.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/interface/RawToDigiConverter.h modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/ctppsRawToDigi_cff.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemT2Digis_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemTimingRawToDigi_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemVFATRawToDigi_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/src/RawToDigiConverter.cc modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File RecoPPS/Local/test/totemT2NewDigi_reco_cfg.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41859/35761/code-format.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41859/35761/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2023-06-02T19:28:57Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-41859/35763

This PR adds an extra 48KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQM/CTPPS/test/totemt2_dqm_test_cfg.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/interface/RawToDigiConverter.h modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/ctppsRawToDigi_cff.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemT2Digis_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemTimingRawToDigi_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/python/totemVFATRawToDigi_cfi.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File EventFilter/CTPPSRawToDigi/src/RawToDigiConverter.cc modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785
- File RecoPPS/Local/test/totemT2NewDigi_reco_cfg.py modified in PR(s): Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785

cmsbuild · 2023-06-02T19:29:27Z

A new Pull Request was created by @oljemark for master.

It involves the following packages:

CalibPPS/ESProducers (alca)
CondFormats/PPSObjects (alca)
DQM/CTPPS (dqm)
DataFormats/TotemReco (reconstruction)
EventFilter/CTPPSRawToDigi (reconstruction)
RecoPPS/Configuration (reconstruction)
RecoPPS/Local (reconstruction)

@micsucmed, @nothingface0, @emanueleusai, @clacaputo, @tvami, @cmsbuild, @saumyaphor4252, @pmandrik, @syuvivida, @tjavaid, @mandrenguyen, @francescobrivio, @rvenditti can you please review it and eventually sign? Thanks.
@fabferro, @rovere, @Martin-Grunewald, @missirol, @tocheng, @grzanka, @mmusich, @forthommel, @seemasharmafnal this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

CalibPPS/ESProducers/python/totemT2DAQMapping_cff.py

tvami · 2023-06-02T19:40:26Z

@cmsbuild , please test

grzanka · 2023-06-02T20:17:57Z

The DQM module for T2 is intended to run in the calibration mode,
to get the plots visible a change is needed here:

cmssw/DQM/Integration/python/clients/ctpps_dqm_sourceclient-live_cfg.py

Line 92 in b2aa6bb

# here: (un)comment to switch between normal and calibration mode

If I understood correctly this change will be done at a later stage by DQM DOC

cmsbuild · 2023-06-02T21:05:32Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-68e112/32959/summary.html
COMMIT: 2cbcb3f
CMSSW: CMSSW_13_2_X_2023-06-02-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/41859/32959/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-68e112/32959/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-68e112/32959/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOnlineClient-visualization had ERRORS
---> test TestDQMOnlineClient-visualization_secondInstance had ERRORS

RelVals

138.4

----- Begin Fatal Exception 02-Jun-2023 22:13:53 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Prefetching for module DQMMessageLogger/'DQMMessageLogger'
   [3] Prefetching for module LogErrorHarvester/'logErrorHarvester'
   [4] Prefetching for module TotemT2RecHitProducer/'totemT2RecHits'
   [5] Calling method for EventSetup module TotemGeometryESModule/'totemGeometryESModule'
   Additional Info:
      [a] Fatal Root Error: @SUB=TGeoManager::Init
Deleting previous geometry: cmsMagneticField:MAGF/Detector Geometry

----- End Fatal Exception -------------------------------------------------

138.5

----- Begin Fatal Exception 02-Jun-2023 22:14:04 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_8_step'
   [2] Prefetching for module LogMessageMonitor/'TrackFinderLogMessageMonMB'
   [3] Prefetching for module LogErrorHarvester/'logErrorHarvester'
   [4] Prefetching for module TotemT2RecHitProducer/'totemT2RecHits'
   [5] Calling method for EventSetup module TotemGeometryESModule/'totemGeometryESModule'
   Additional Info:
      [a] Fatal Root Error: @SUB=TGeoManager::Init
Deleting previous geometry: cmsMagneticField:MAGF/Detector Geometry

----- End Fatal Exception -------------------------------------------------

135.4

----- Begin Fatal Exception 02-Jun-2023 22:15:23 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Calling method for module TotemT2RecHitProducer/'totemT2RecHits'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edmNew::DetSetVector<TotemT2Digi>
Looking for module label: totemT2Digis
Looking for productInstanceName: TotemT2

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

138.4138.4_PromptCollisions2021/step2_PromptCollisions2021.log
138.5138.5_ExpressCollisions2021/step2_ExpressCollisions2021.log
140.6140.6_RunHI2022/step2_RunHI2022.log

Expand to see more relval errors ...

AddOn Tests

fastsim2

----- Begin Fatal Exception 02-Jun-2023 22:10:40 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 3 stream: 2
   [1] Running path 'AODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'AODSIMoutput'
   [3] Calling method for module TotemT2RecHitProducer/'totemT2RecHits'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edmNew::DetSetVector<TotemT2Digi>
Looking for module label: totemT2Digis
Looking for productInstanceName: TotemT2

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

hlt_mc_GRun

----- Begin Fatal Exception 02-Jun-2023 22:13:16 CEST-----------------------
An exception of category 'TotemDAQMappingESSourceXML::ParseTreeTotemT2' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 3
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module TotemVFATRawToDigi/'totemT2Digis'
   [4] Calling method for EventSetup module TotemDAQMappingESSourceXML/'totemDAQMappingESSourceXML_TotemT2'
Exception Message:
Payload position in fibre not given for element `nt2_tile'
----- End Fatal Exception -------------------------------------------------

hlt_mc_HIon

----- Begin Fatal Exception 02-Jun-2023 22:12:21 CEST-----------------------
An exception of category 'TotemDAQMappingESSourceXML::ParseTreeTotemT2' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 1
   [1] Running path 'FEVTDEBUGHLToutput_step'
   [2] Prefetching for module PoolOutputModule/'FEVTDEBUGHLToutput'
   [3] Prefetching for module TotemVFATRawToDigi/'totemT2Digis'
   [4] Calling method for EventSetup module TotemDAQMappingESSourceXML/'totemDAQMappingESSourceXML_TotemT2'
Exception Message:
Payload position in fibre not given for element `nt2_tile'
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

oljemark · 2023-06-09T17:02:06Z

Hi, @tvami . I did add the naming fix @perrotta asked for first.

emanueleusai · 2023-06-09T19:13:20Z

+1

cmsbuild · 2023-06-09T19:22:23Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-68e112/33072/summary.html
COMMIT: f932e9c
CMSSW: CMSSW_13_2_X_2023-06-09-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/41859/33072/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 8 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3219909
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3219884
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

perrotta · 2023-06-10T08:28:04Z

@mandrenguyen since your last signature this PR was just

renamed one method, as I asked in Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859 (review) (two lines changed with a renamed method)
all commits squashed

I would like this PR being tested in a few master IBs before getting backported in 13_0_X for the Totem run. Therefore, I will consider your previous @cms-sw/reconstruction-l2 signature as good, and merge this now in time for today's 1100 IB.

Please comment if you think there is anything else that need to be considered, instead.

perrotta · 2023-06-10T08:28:29Z

+1

perrotta · 2023-06-10T08:29:00Z

merge

mandrenguyen · 2023-06-10T08:31:14Z

+1

cmsbuild · 2023-06-10T08:31:40Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

…port of PR cms-sw#41859

Backport for Totem T2, to be used in the special run at 25.6, consisting of #41472 , #41777 and #41859

cmsbuild added this to the CMSSW_13_2_X milestone Jun 2, 2023

cmsbuild added reconstruction-pending dqm-pending alca-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jun 2, 2023

cmsbuild added code-checks-rejected code-checks-pending and removed code-checks-pending code-checks-rejected labels Jun 2, 2023

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 2, 2023

tvami reviewed Jun 2, 2023

View reviewed changes

CalibPPS/ESProducers/python/totemT2DAQMapping_cff.py Outdated Show resolved Hide resolved

cmsbuild added tests-started and removed tests-pending labels Jun 2, 2023

cmsbuild added tests-rejected and removed tests-started labels Jun 2, 2023

oljemark force-pushed the fo_nt2_mapfix2 branch from 2cbcb3f to 5753043 Compare June 4, 2023 18:53

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Jun 4, 2023

cmsbuild added alca-approved tests-started labels Jun 9, 2023

cmsbuild added dqm-approved and removed dqm-pending labels Jun 9, 2023

cmsbuild added tests-approved and removed tests-started labels Jun 9, 2023

cmsbuild added orp-approved and removed orp-pending labels Jun 10, 2023

cmsbuild merged commit 609dc7c into cms-sw:master Jun 10, 2023

perrotta mentioned this pull request Jun 10, 2023

Fix parameters in PPS RawToDigiConverter, and clean up a few related configs #41785

Closed

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Jun 10, 2023

oljemark added a commit to CTPPS/cmssw that referenced this pull request Jun 10, 2023

Naming fix from PR cms-sw#41859

6e281d4

oljemark added a commit to CTPPS/cmssw that referenced this pull request Jun 12, 2023

Update the Totem T2 mapping, data format, and DQM functionality, back…

67e31e8

…port of PR cms-sw#41859

oljemark added a commit to CTPPS/cmssw that referenced this pull request Jun 12, 2023

Naming fix from PR cms-sw#41859

ae1d0c8

cmsbuild added a commit that referenced this pull request Jun 13, 2023

Merge pull request #41916 from CTPPS/fo_nt2_PRs_backport130

d77e58b

Backport for Totem T2, to be used in the special run at 25.6, consisting of #41472 , #41777 and #41859

cmsbuild mentioned this pull request Jun 14, 2023

Fix Walloc-size-larger-than warnings in SimG4CMS/Forward/*NumberingScheme #41942

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859

Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859

oljemark commented Jun 2, 2023 •

edited

Loading

cmsbuild commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

tvami commented Jun 2, 2023

grzanka commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

oljemark commented Jun 9, 2023

emanueleusai commented Jun 9, 2023

cmsbuild commented Jun 9, 2023

perrotta commented Jun 10, 2023

perrotta commented Jun 10, 2023

perrotta commented Jun 10, 2023

mandrenguyen commented Jun 10, 2023

cmsbuild commented Jun 10, 2023

Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859

Update Totem T2 mapping, fix bug in VFAT footprint check, and implement suggestions from PR #41777 #41859

Conversation

oljemark commented Jun 2, 2023 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

tvami commented Jun 2, 2023

grzanka commented Jun 2, 2023

cmsbuild commented Jun 2, 2023

Unit Tests

RelVals

RelVals-INPUT

AddOn Tests

oljemark commented Jun 9, 2023

emanueleusai commented Jun 9, 2023

cmsbuild commented Jun 9, 2023

Comparison Summary

perrotta commented Jun 10, 2023

perrotta commented Jun 10, 2023

perrotta commented Jun 10, 2023

mandrenguyen commented Jun 10, 2023

cmsbuild commented Jun 10, 2023

oljemark commented Jun 2, 2023 •

edited

Loading