DeepDoubleCvL and DeepDoubleCvB tagger integration

[andrzejnovak]

DeepDoubleCvL and DeepDoubleCvB

This PR is to integrate the two DeepDoubleC taggers following up on the integration of the DeepDoubleB tagger #23033 and beind adjusted for the RecoBTag/TensorFlow changes in #23768

Documentation

The tagger is documented in CMS DP-2018/046 as well as regular BTV talks, for example https://indico.cern.ch/event/754267/contributions/3130748/attachments/1710994/2758428/DDX_80x_update_BTV.pdf

94X

Updated models are documented in https://indico.cern.ch/event/768759/contributions/3194255/attachments/1744684/2824149/DDX_update3110.pdf Notably slides 13 and 14 show all models added to RecoBTag/Combined/data
Original DDB model not replaced yet, but available.

Testing

The tagger predictions were tested on CMSSW_10_3_0_pre2, RelValTTbar_13 sample.
DeepDoubleCvL https://cernbox.cern.ch/index.php/s/Mi4CXyuw8lHNs0C

DeepDoubleCvB https://cernbox.cern.ch/index.php/s/U84clDNiKpwOEsn

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24918/6889

[cmsbuild]

A new Pull Request was created by @andrzejnovak (Andrzej Novak) for master.

It involves the following packages:

DataFormats/BTauReco
PhysicsTools/PatAlgos
RecoBTag/FeatureTools
RecoBTag/TensorFlow

@perrotta, @monttj, @cmsbuild, @slava77, @gpetruc, @arizzi can you please review it and eventually sign? Thanks.
@TaiSakuma, @gouskos, @rappoccio, @HeinerTholen, @seemasharmafnal, @mmarionncern, @imarches, @ahinzmann, @smoortga, @acaudron, @jdolen, @drkovalskyi, @hqucms, @ferencek, @rovere, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @clelange, @JyothsnaKomaragiri, @mverzett, @gpetruc, @mariadalfonso, @pvmulder this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[slava77]

@andrzejnovak
IIUC, this requires DeepDoubleC files to be made available in
https://github.com/cms-data/RecoBTag-Combined

Please clarify on the plans to update the external.

[andrzejnovak]

@slava77
Posted, cms-data/RecoBTag-Combined#17

[slava77]

@cmsbuild please test with cms-sw/cmsdist#4432

[cmsbuild]

The tests are being triggered in jenkins.
Using externals from cms-sw/cmsdist#4432
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31128/console

[slava77]

@smuzaffar
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31128/console
for this PR tests looks more like a full build (from the logs and from about 10 hours that it took).
I thought that an external file update doesn't lead to this much.
Is it just a bad luck overlapping with other cmsdist updates since the IB used for the tests (like the root.spec )?

[cmsbuild]

-1

Tested at: 40f8607

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31128/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

• Unit Tests:

I found errors in the following unit tests:

---> test testNumba had ERRORS
---> test testImports had ERRORS

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31128/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 2994843
• DQMHistoTests: Total failures: 84
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2994562
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24918/7391

[cmsbuild]

Pull request #24918 was updated. @perrotta, @monttj, @cmsbuild, @fgolf, @slava77, @peruzzim can you please check and sign again.

[perrotta]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/31894/console Started: 2018/11/28 12:33

[cmsbuild]

+1
Tested at: fc232c4
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31894/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
fc232c4
1658fcb
f74c4c1
230e8d3
5f8c3a4
92bb0d6
aef474e
f338373
183aa17
5d8e94f
e42d295
81b0d65
3c81d7b
7c16f68
0188aba
a90008a
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31894/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31894/git-merge-result

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24918/31894/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 990 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3131839
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3131634
• DQMHistoTests: Total skipped: 204
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[slava77]

+1

for #24918 fc232c4

• updates since the last signoff by reco https://github.com/cms-sw/cmssw/pull/24918/files/230e8d360021b246de07637923b151abf28daf9a..fc232c44c480cba52486bb36c97e86645d6767f5 are not supposed to change the behavior, they improve compliance with the code review
• jenkins tests pass and comparisons show differences only in slimmedJetsAK8 jet tag discriminants (as expected).

[fabiocos]

+1

[fabiocos]

merge

[Dr15Jones]

This pull request appears to have broken backwards compatibility. See https://hypernews.cern.ch/HyperNews/CMS/get/physTools/3647.html

There was a commit in this pull request which explicitly said it was to keep backwards compatibility. Unfortunately, a later commit changed DeepDoubleBFeatures to DeepDoubleXFeatures rather than keeping the original DeepDoubleBFeatures and making a new DeepDoubleXFeatures.

[slava77]

This pull request appears to have broken backwards compatibility. See https://hypernews.cern.ch/HyperNews/CMS/get/physTools/3647.html

There was a commit in this pull request which explicitly said it was to keep backwards compatibility. Unfortunately, a later commit changed DeepDoubleBFeatures to DeepDoubleXFeatures rather than keeping the original DeepDoubleBFeatures and making a new DeepDoubleXFeatures.

As mentioned in #24918 (comment), there were no files identified with this data persisted. As such, it was safe to cleanup.
It looks like that conclusion may have been wrong.

[andrzejnovak]

We can add back the .h file if there isn't a better solution.

[ferencek]

The problem has also been reported in https://hypernews.cern.ch/HyperNews/CMS/get/btag/1653.html From the example file provided in that post, I don't even see anything related to this tagger being run in the job and as Slava pointed out, this data is only used as transient so I am puzzled about why the error message appears in the first place.