New proton transport without data dir #25355

mundim · 2018-11-27T17:57:23Z

As agreed, following up the PR 24490 (which as closed), a new PR with the same packages but removing the data directory from SimTransport module.

… adaptation

… changes in the code that depends on it

cmsbuild · 2018-11-27T17:57:47Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-11-27T18:00:45Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25355/7386

cmsbuild · 2018-11-27T18:01:08Z

A new Pull Request was created by @mundim for master.

It involves the following packages:

FastSimulation/CTPPSFastTrackingProducer
FastSimulation/CTPPSRecHitProducer
FastSimulation/CTPPSSimHitProducer
IOMC/ParticleGuns
SimPPS/PPSSimTrackProducer
SimTransport/HectorProducer
SimTransport/PPSProtonTransport
SimTransport/TotemRPProtonTransportParametrization
Utilities/PPS

The following packages do not have a category, yet:

Utilities/PPS
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@ssekmen, @lveldere, @perrozzi, @efeyazgan, @mdhildreth, @cmsbuild, @civanch, @alberto-sanchez, @qliphy can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @matt-komm, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

fabiocos · 2018-11-28T10:11:26Z

please test with cms-sw/cmsdist#4515

cmsbuild · 2018-11-28T10:11:47Z

The tests are being triggered in jenkins.
Using externals from cms-sw/cmsdist#4515
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31893/console

cmsbuild · 2018-12-20T20:14:39Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-12-20T20:19:47Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25355/7753

This PR adds an extra 1052KB to repository
Found files with invalid states:
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_0p4_185_reco_beam1.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR191.541_PreTS2.tfs: Added, Deleted
- PPSTools/Utilities/interface/PPSOpticsCalibrator.h: Added, Deleted
- Utilities/PPS/bin/BuildFile.xml: Added, Modified, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.30_6.5TeV_CR175_Nominal_2017.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR185_Nominal_2016.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR150_Nominal_2017.tfs: Added, Deleted
- Utilities/PPS/bin/PPSOpticsCalibration.cc: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_reco.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR179.394_PreTS2_TOTEM.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR191.541_PreTS2_TOTEM.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR179.394_PreTS2.tfs: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_transp.root: Added, Deleted
- PPSTools/Utilities/src/PPSUtilities.cc: Added, Deleted
- PPSTools/Utilities/bin/BuildFile.xml: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR185_Nominal_2016.tfs: Added, Deleted
- Utilities/PPS/interface/PPSOpticsCalibrator.h: Added, Modified, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR150_Nominal_2017.tfs: Added, Deleted
- PPSTools/Utilities/bin/PPSOpticsCalibration.cc: Added, Deleted
- PPSTools/Utilities/interface/PPSUtilities.h: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_transp_75.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.30_6.5TeV_CR175_Nominal_2017.tfs: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90p0_50urad_reco.root: Added, Deleted
- Utilities/PPS/bin/PPSOpticsCalibrator.cc: Added, Modified, Deleted
- PPSTools/Utilities/BuildFile.xml: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_0p4_185_reco_beam2.root: Added, Deleted
- PPSTools/Utilities/interface/PPSUnitConversion.h: Added, Modified, Deleted

cmsbuild · 2018-12-20T20:20:40Z

Pull request #25355 was updated. @efeyazgan, @lveldere, @perrozzi, @ssekmen, @mdhildreth, @cmsbuild, @civanch, @alberto-sanchez, @qliphy can you please check and sign again.

civanch · 2018-12-21T07:20:50Z

@smuzaffar , can you, please, comment on the recent update of this PR: how to avoid addition of extra 1052KB to repository?

smuzaffar · 2018-12-21T07:26:34Z

@civanch , as mentioned in the #25355 (comment) , there are many files which were added/removed. This also contains some data files. Please get the history of this Pr cleaned up and force push the changes. Or create a new PR with clean history.

civanch · 2018-12-21T07:41:23Z

@mundim, there is the problem with this PR as you see above. You need clean up the history or create a new PR for 10_5_X where manually copy all necessary files.

mundim · 2018-12-21T13:05:03Z

@civanch , as mentioned in the #25355 (comment) , there are many files which were added/removed. This also contains some data files. Please get the history of this Pr cleaned up and force push the changes. Or create a new PR with clean history.

I don't understand what is going on. I have not added any file, just removed the ones in the bin directory which was just causing problem and it is not important they go into the release and removed CLHEP vector and couts in the code. See below:

git status
On branch newProtonTransport_NoDataDir
Changes to be committed:
(use "git reset HEAD ..." to unstage)

modified:   FastSimulation/CTPPSFastTrackingProducer/plugins/CTPPSFastTrackingProducer.cc
modified:   FastSimulation/CTPPSRecHitProducer/plugins/CTPPSRecHitProducer.cc
modified:   IOMC/ParticleGuns/src/RandomtXiGunProducer.cc
modified:   SimTransport/HectorProducer/src/Hector.cc
modified:   SimTransport/HectorProducer/src/HectorProducer.cc
modified:   SimTransport/PPSProtonTransport/interface/HectorTransport.h
modified:   SimTransport/PPSProtonTransport/src/HectorTransport.cc
modified:   SimTransport/PPSProtonTransport/src/ProtonTransport.cc
modified:   SimTransport/PPSProtonTransport/src/TotemTransport.cc
modified:   SimTransport/TotemRPProtonTransportParametrization/interface/LHCOpticsApproximator.h
modified:   SimTransport/TotemRPProtonTransportParametrization/src/LHCOpticsApproximator.cc
modified:   SimTransport/TotemRPProtonTransportParametrization/src/TMultiDimFet.cc
deleted:    Utilities/PPS/bin/BuildFile.xml
deleted:    Utilities/PPS/bin/PPSOpticsCalibrator.cc
deleted:    Utilities/PPS/interface/PPSOpticsCalibrator.h
modified:   Utilities/PPS/src/PPSUtilities.cc

fabiocos · 2019-01-07T13:57:33Z

@smuzaffar this looks another example of the reports to be understood about added/removed file sizes. If I merge this PR as it is on top of CMSSW_10_5_X and look at the added/removed file sizes I see:

30 FastSimulation/CTPPSFastTrackingProducer/BuildFile.xml
-359 FastSimulation/CTPPSFastTrackingProducer/interface/CTPPSFastTrackingProducer.h
30 FastSimulation/CTPPSFastTrackingProducer/plugins/BuildFile.xml
-1205 FastSimulation/CTPPSFastTrackingProducer/plugins/CTPPSFastTrackingProducer.cc
22 FastSimulation/CTPPSFastTrackingProducer/python/CTPPSFastTrackingProducer_cfi.py
-18 FastSimulation/CTPPSRecHitProducer/plugins/CTPPSRecHitProducer.cc
160 FastSimulation/CTPPSRecHitProducer/python/CTPPSRecHitProducer_cfi.py
161 FastSimulation/CTPPSSimHitProducer/plugins/CTPPSSimHitProducer.cc
10 FastSimulation/CTPPSSimHitProducer/python/CTPPSSimHitProducer_cfi.py
30 IOMC/ParticleGuns/interface/BaseRandomtXiGunProducer.h
332 IOMC/ParticleGuns/src/RandomtXiGunProducer.cc
313 SimPPS/PPSSimTrackProducer/plugins/BuildFile.xml
7128 SimPPS/PPSSimTrackProducer/plugins/PPSSimTrackProducer.cc
2169 SimPPS/PPSSimTrackProducer/python/SimTrackProducerForFastSim_cff.py
1316 SimPPS/PPSSimTrackProducer/python/SimTrackProducerForFullSim_cff.py
-265867 SimTransport/HectorProducer/data/LHCB1IR5_5TeV.tfs
-265866 SimTransport/HectorProducer/data/LHCB1IR5_7TeV.tfs
-265866 SimTransport/HectorProducer/data/LHCB1IR5_v6.500.tfs
-265867 SimTransport/HectorProducer/data/LHCB2IR5_5TeV.tfs
-265866 SimTransport/HectorProducer/data/LHCB2IR5_7TeV.tfs
-265866 SimTransport/HectorProducer/data/LHCB2IR5_v6.500.tfs
-4143 SimTransport/HectorProducer/interface/CTPPSHector.h
-716 SimTransport/HectorProducer/interface/CTPPSHectorParameters.h
-1437 SimTransport/HectorProducer/interface/CTPPSHectorProducer.h
-2046 SimTransport/HectorProducer/python/FastSimWithCTPPS_cff.py
-1235 SimTransport/HectorProducer/python/FullSimWithCTPPS_cff.py
-1587 SimTransport/HectorProducer/python/HectorTransportCTPPS_cfi.py
-19770 SimTransport/HectorProducer/src/CTPPSHector.cc
-4269 SimTransport/HectorProducer/src/CTPPSHectorProducer.cc
-12 SimTransport/HectorProducer/src/Hector.cc
51 SimTransport/HectorProducer/src/HectorProducer.cc
419 SimTransport/PPSProtonTransport/BuildFile.xml
2561 SimTransport/PPSProtonTransport/interface/HectorTransport.h
1953 SimTransport/PPSProtonTransport/interface/ProtonTransport.h
2465 SimTransport/PPSProtonTransport/interface/TotemTransport.h
5147 SimTransport/PPSProtonTransport/python/HectorOpticsParameters_cfi.py
823 SimTransport/PPSProtonTransport/python/HectorTransport_cfi.py
2155 SimTransport/PPSProtonTransport/python/TotemBeamConditions_cff.py
460 SimTransport/PPSProtonTransport/python/TotemTransport_cfi.py
11117 SimTransport/PPSProtonTransport/src/HectorTransport.cc
4155 SimTransport/PPSProtonTransport/src/ProtonTransport.cc
9092 SimTransport/PPSProtonTransport/src/TotemTransport.cc
147 SimTransport/TotemRPProtonTransportParametrization/BuildFile.xml
8218 SimTransport/TotemRPProtonTransportParametrization/interface/LHCOpticsApproximator.h
11460 SimTransport/TotemRPProtonTransportParametrization/interface/TMultiDimFet.h
41050 SimTransport/TotemRPProtonTransportParametrization/src/LHCOpticsApproximator.cc
568 SimTransport/TotemRPProtonTransportParametrization/src/SimTransportTotemRPProtTranspParLinkDef.h
73263 SimTransport/TotemRPProtonTransportParametrization/src/TMultiDimFet.cc
187 Utilities/PPS/BuildFile.xml
807 Utilities/PPS/interface/PPSUnitConversion.h
964 Utilities/PPS/interface/PPSUtilities.h
2952 Utilities/PPS/src/PPSUtilities.cc
total -1440280

so apparently there is a reduction in the size of the master after this commit of about 1 MB. In my understanding what effectively matters from the point of view of the repository size is the fresh new additions, as the files removed are anyway present in the history. This should be 191715 bytes

fabiocos · 2019-01-07T14:13:18Z

@smuzaffar @mundim ok, the report I show is for the final added snapshot, but the intermediate history that can be looked at with a cms-checkout-topic shows that this PR is built on top of CMSSW_10_3_0_pre3, and that the addition of the unwanted files is brought inside in the initial commits:

15:03 cmsdev15 784> git log -30 --pretty=oneline
53c31a7 (HEAD -> pull/25355, cms-sw/refs/pull/25355/head) removing remaining CLHEP Vector
6f11bf0 removing cout/CLHEP/TFile that were somehow left behind
f2efc82 Removing output to cout and replacing CLHEP HepLorentzVector by the ROOT version
dba9d40 replacing std::cout by LogInfo in HectorTransport.cc
0f0af28 Removing data dir
2be631b Applying requested corrections
1b1d60b removing self assignment in HectorTransport.cc
7120ddb Removing CLHEP includes from header files
f3d483e Changing name and moving some CLHEP include to .cc
971c954 followig up PR comments
3e60d92 followig up PR comments
4c19208 Moving PPSTools/Utilities to Utilities/PPS as advised and the related changes in the code that depends on it
79bac8e removing typo
ff43c22 removing commented code that was not caught by PR testing
99bcc27 implementing PR suggestions
97e2ad7 implementing PR suggestions
43e133b Splitting into PPSTools + new transport with totem param. and fastsim adaptation
79bbbf1 (tag: CMSSW_10_3_0_pre3) Merge pull request #24468 from slava77/CMSSW_10_3_X_2018-09-04-1100/sign1038/dpHI2018

Between 79bbbf1 and 43e133b 8237049 bytes are added, including the root files, while between 2be631b and 0f0af28 9640757 bytes are removed (all the data files moved into the external package).

In order not to lose the history of the review i would suggest to open a new PR where the content of this one is squashed (so as the intermediate history is cleaned) and rebased on top of 10_5_X.

fabiocos · 2019-01-07T14:13:26Z

please test

cmsbuild · 2019-01-07T14:13:51Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/32435/console Started: 2019/01/07 15:15

cmsbuild · 2019-01-07T15:20:07Z

+1
Tested at: 53c31a7
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25355/32435/summary.html

smuzaffar · 2019-01-07T16:42:07Z

code-checks

cmsbuild · 2019-01-07T16:48:56Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-01-07T16:55:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25355/7845

This PR adds an extra 1804KB to repository
Found files with invalid states:
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_0p4_185_reco_beam1.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR191.541_PreTS2.tfs: Added, Deleted
- PPSTools/Utilities/interface/PPSOpticsCalibrator.h: Added, Deleted
- Utilities/PPS/bin/BuildFile.xml: Added, Modified, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.30_6.5TeV_CR175_Nominal_2017.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR185_Nominal_2016.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR150_Nominal_2017.tfs: Added, Deleted
- Utilities/PPS/bin/PPSOpticsCalibration.cc: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_reco.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR179.394_PreTS2_TOTEM.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR191.541_PreTS2_TOTEM.tfs: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR179.394_PreTS2.tfs: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_transp.root: Added, Deleted
- PPSTools/Utilities/src/PPSUtilities.cc: Added, Deleted
- PPSTools/Utilities/bin/BuildFile.xml: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.40_6.5TeV_CR185_Nominal_2016.tfs: Added, Deleted
- Utilities/PPS/interface/PPSOpticsCalibrator.h: Added, Modified, Deleted
- SimTransport/PPSProtonTransport/data/LHCB2_Beta0.40_6.5TeV_CR150_Nominal_2017.tfs: Added, Deleted
- PPSTools/Utilities/bin/PPSOpticsCalibration.cc: Added, Deleted
- PPSTools/Utilities/interface/PPSUtilities.h: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90_transp_75.root: Added, Deleted
- SimTransport/PPSProtonTransport/data/LHCB1_Beta0.30_6.5TeV_CR175_Nominal_2017.tfs: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_90p0_50urad_reco.root: Added, Deleted
- Utilities/PPS/bin/PPSOpticsCalibrator.cc: Added, Modified, Deleted
- PPSTools/Utilities/BuildFile.xml: Added, Deleted
- SimTransport/TotemRPProtonTransportParametrization/data/parametrization_6500GeV_0p4_185_reco_beam2.root: Added, Deleted
- PPSTools/Utilities/interface/PPSUnitConversion.h: Added, Modified, Deleted

smuzaffar · 2019-01-07T17:17:32Z

@mundim , in this PR 43e133b commit adds some files while 0f0af28 remove those files. As these commits are part of this PR history so this is going to pollute cmssw repository.

I agree with @fabiocos , please create a new PR with clean history.

smuzaffar · 2019-01-07T17:17:42Z

hold

cmsbuild · 2019-01-07T17:18:04Z

Pull request has been put on hold by @smuzaffar, @fabiocos
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

mundim · 2019-01-07T21:08:27Z

As adivised, closing this PR and issuing a new one from a clean area, rebased to CMSSW_10_5

mundim added 13 commits September 7, 2018 06:01

Splitting into PPSTools + new transport with totem param. and fastsim…

43e133b

… adaptation

implementing PR suggestions

97e2ad7

implementing PR suggestions

99bcc27

removing commented code that was not caught by PR testing

ff43c22

removing typo

79bac8e

Moving PPSTools/Utilities to Utilities/PPS as advised and the related…

4c19208

… changes in the code that depends on it

followig up PR comments

3e60d92

followig up PR comments

971c954

Changing name and moving some CLHEP include to .cc

f3d483e

Removing CLHEP includes from header files

7120ddb

removing self assignment in HectorTransport.cc

1b1d60b

Applying requested corrections

2be631b

Removing data dir

0f0af28

cmsbuild added this to the CMSSW_10_4_X milestone Nov 27, 2018

cmsbuild added code-checks-pending comparison-pending fastsim-pending generators-pending new-package-pending orp-pending pending-signatures simulation-pending tests-pending labels Nov 27, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 27, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Dec 20, 2018

cmsbuild added tests-started and removed tests-pending labels Jan 7, 2019

cmsbuild added tests-approved and removed tests-started labels Jan 7, 2019

cmsbuild added code-checks-pending and removed code-checks-approved labels Jan 7, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 7, 2019

mundim closed this Jan 7, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New proton transport without data dir #25355

New proton transport without data dir #25355

mundim commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

fabiocos commented Nov 28, 2018

cmsbuild commented Nov 28, 2018 •

edited

Loading

cmsbuild commented Dec 20, 2018

cmsbuild commented Dec 20, 2018

cmsbuild commented Dec 20, 2018

civanch commented Dec 21, 2018

smuzaffar commented Dec 21, 2018

civanch commented Dec 21, 2018

mundim commented Dec 21, 2018 •

edited

Loading

fabiocos commented Jan 7, 2019

fabiocos commented Jan 7, 2019

fabiocos commented Jan 7, 2019

cmsbuild commented Jan 7, 2019 •

edited

Loading

cmsbuild commented Jan 7, 2019

smuzaffar commented Jan 7, 2019 •

edited

Loading

cmsbuild commented Jan 7, 2019

cmsbuild commented Jan 7, 2019

smuzaffar commented Jan 7, 2019

smuzaffar commented Jan 7, 2019

cmsbuild commented Jan 7, 2019

mundim commented Jan 7, 2019

New proton transport without data dir #25355

New proton transport without data dir #25355

Conversation

mundim commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

cmsbuild commented Nov 27, 2018

fabiocos commented Nov 28, 2018

cmsbuild commented Nov 28, 2018 • edited Loading

cmsbuild commented Dec 20, 2018

cmsbuild commented Dec 20, 2018

cmsbuild commented Dec 20, 2018

civanch commented Dec 21, 2018

smuzaffar commented Dec 21, 2018

civanch commented Dec 21, 2018

mundim commented Dec 21, 2018 • edited Loading

fabiocos commented Jan 7, 2019

fabiocos commented Jan 7, 2019

fabiocos commented Jan 7, 2019

cmsbuild commented Jan 7, 2019 • edited Loading

cmsbuild commented Jan 7, 2019

smuzaffar commented Jan 7, 2019 • edited Loading

cmsbuild commented Jan 7, 2019

cmsbuild commented Jan 7, 2019

smuzaffar commented Jan 7, 2019

smuzaffar commented Jan 7, 2019

cmsbuild commented Jan 7, 2019

mundim commented Jan 7, 2019

cmsbuild commented Nov 28, 2018 •

edited

Loading

mundim commented Dec 21, 2018 •

edited

Loading

cmsbuild commented Jan 7, 2019 •

edited

Loading

smuzaffar commented Jan 7, 2019 •

edited

Loading