Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[12.4.X] New high granularity pixel alignment for the PCL #38497

Merged
merged 8 commits into from
Jun 27, 2022
Merged

[12.4.X] New high granularity pixel alignment for the PCL #38497

merged 8 commits into from
Jun 27, 2022

Conversation

mmusich
Copy link
Contributor

@mmusich mmusich commented Jun 24, 2022 •
edited
Loading

backport of #38449 and #38508

PR description:

This is the last PR towards using a higher granularity (HG) for the pixel alignment in the PCL, following #38371. It is planned to run the HG alignment in parallel to the LG alignment, with only the LG alignment updating the conditions. More details can also be found in [1].

In detail the following things are changed/added in this PR:

  • Adapt MillePedeFileReader and MillePedeDQMModule to read and store the high granularity alignment
  • Read and use the new thresholds in MillePedeDQMModule and MillePedeAlignmentAlgorithm
  • Store/read the pede results for the HG alignment in/from a separate folder, to avoid overwriting the LG result in the harvesting step.
  • Two new methods in the new HG threshold object, to get and store the float maps
  • Add workflows for the production and the harvesting required for the HG alignment in CommonAlignmentProducer
  • Add the new workflows to the PCL configuration
  • Add the LG and HG alignment to the 1001.2 workflows, which can be used with the runTheMatrix.py command
    * Add a relval customizer in CommonAlignmentProducer to override the 20 LS veto and required entries per structure in the HG aligment

PR validation:

The PR can be tested using the adapted relval workflow with runTheMatrix.py -l 1001.2.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

verbatim backport of #38449 and #38508. As explained in the description it's needed for deplyoment of the new workflow in the 2022 high intensity data-taking.

cc:
@dmeuser, @connorpa, @antoniovagnerini, @consuegs

[1] https://indico.cern.ch/event/1167313/contributions/4906905/attachments/2461217/4219804/AlCaDB_Meeting_13_06_22.pdf

@dmeuser @mmusich
Method to set floatMap and check floatMap in new Threshold object
ce54697
@dmeuser @mmusich
Use typedef map in new Threshold object
01bce85
@dmeuser @mmusich
Implementation of new HG PCL alignment in alignment package
e1e92ec
@dmeuser @mmusich
Add workflows for new HG PCL alignment
3028f33
@dmeuser @mmusich
Last changes and adaptions for HG PCL alignment, which include:
a904af3 
Add HG(LG) PCL alignment to 1001.2 workflow

add relval customization to override the 20 LS window for the Alignment PCL workflows

Fix entries in HG PCL alignment and change entries in relval customizer

Restructure and clean configs for HG PCL alignment

Use PixelTopologyMap for indexing alignables and booking histograms in HG PCL alignment

Rename TrackerAlignmenRcd and  update DropBoxMetadata and testPCLAlCaHarvesting for HG PCL alignment

Fix testCalibrationTkAlCaRecoProducers and use clone instead of deepcopy for HG PCL alignment

Don't use indexing helper for LG PCL alignment

Fix HG alignment threshold logic when using new threshold payload

Configurable tracker alignment record for HG PCL alignment
@cmsbuild cmsbuild added this to the CMSSW_12_4_X milestone Jun 24, 2022
@cmsbuild
Copy link
Contributor

cmsbuild commented Jun 24, 2022 •
edited
Loading

A new Pull Request was created by @mmusich (Marco Musich) for CMSSW_12_4_X.

It involves the following packages:

  • Alignment/CommonAlignmentProducer (alca)
  • Alignment/MillePedeAlignmentAlgorithm (alca)
  • Calibration/TkAlCaRecoProducers (alca)
  • CondFormats/Common (db, alca)
  • CondFormats/PCLConfig (db, alca)
  • Configuration/AlCa (alca)
  • Configuration/EventContent (operations)
  • Configuration/PyReleaseValidation (pdmv, upgrade)
  • Configuration/StandardSequences (operations)

@perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @ggovi, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@felicepantaleo, @kpedro88, @Martin-Grunewald, @tlampen, @trtomei, @threus, @mmusich, @slomeo, @pakhotin, @makortel, @ChrisMisan, @JanFSchulte, @dgulhan, @missirol, @seemasharmafnal, @beaucero, @GiacomoSguazzoni, @rovere, @VinInn, @tocheng, @ebrondol, @mtosi, @fabiocos, @adewit, @lecriste this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@tvami
Copy link
Contributor

tvami commented Jun 24, 2022

type new-feature,trk

@tvami
Copy link
Contributor

tvami commented Jun 24, 2022

test parameters:

  • workflows = 1001.2

@tvami
Copy link
Contributor

tvami commented Jun 24, 2022

urgent

  • needed for 12_4_X on Tuesday

@tvami
Copy link
Contributor

tvami commented Jun 24, 2022

@cmsbuild , please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-547554/25769/summary.html
COMMIT: a904af3
CMSSW: CMSSW_12_4_X_2022-06-24-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38497/25769/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 1001.2 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

@slava77 comparisons for the following workflows were not done due to missing matrix map:

  • /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-547554/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA+ALCAHARVD4+ALCAHARVDSIPIXELALIHG

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 51
  • DQMHistoTests: Total histograms compared: 3680359
  • DQMHistoTests: Total failures: 2
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3680335
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 208 log files, 45 edm output root files, 51 DQM output files
  • TriggerResults: no differences found

@tvami
Copy link
Contributor

tvami commented Jun 25, 2022

+1

@cmsbuild
Copy link
Contributor

Pull request #38497 was updated. @perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @ggovi, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please check and sign again.

@mmusich
Copy link
Contributor Author

mmusich commented Jun 25, 2022

please test

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-547554/25800/summary.html
COMMIT: a0df15d
CMSSW: CMSSW_12_4_X_2022-06-25-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38497/25800/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 50
  • DQMHistoTests: Total histograms compared: 3680359
  • DQMHistoTests: Total failures: 2
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3680335
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
  • Checked 208 log files, 45 edm output root files, 50 DQM output files
  • TriggerResults: no differences found

@tvami
Copy link
Contributor

tvami commented Jun 26, 2022 •
edited
Loading

+1

  • tests pass,
  • 1001.2 has the correct results
  • unit test is fixed

@kskovpen
Copy link
Contributor

+pdmv

@srimanob
Copy link
Contributor

+Upgrade

Backport

@tvami
Copy link
Contributor

tvami commented Jun 27, 2022

@cms-sw/orp-l2 we are essentially fully signed for this PR

@perrotta
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next CMSSW_12_4_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_12_5_X is complete. This pull request will be automatically merged.

@cmsbuild cmsbuild merged commit 3423e04 into cms-sw:CMSSW_12_4_X Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
alca-approved backport-ok db-approved fully-signed new-feature operations-approved orp-approved pdmv-approved tests-approved trk upgrade-approved urgent
Projects
None yet
Milestone
CMSSW_12_4_X
Development

Successfully merging this pull request may close these issues.
7 participants
@mmusich @cmsbuild @tvami @kskovpen @srimanob @perrotta @dmeuser