Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228

iarspider · 2023-07-11T06:16:24Z

PR description:

Companion PR for cms-sw/cmsdist#8565

PR validation:

Tests in cms-sw/cmsdist#8565

cmsbuild · 2023-07-11T06:23:53Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42228/36238

ERROR: Build errors found during clang-tidy run.

    PROTOBUF_NODISCARD std::string* release_full_pathname();
                       ~~~~~^
DQMServices/Core/interface/ROOTFilePB.pb.h:201:35: error: expected ';' at end of declaration list [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_full_pathname();
                                  ^
                                  ;
DQMServices/Core/interface/ROOTFilePB.pb.h:222:5: error: unknown type name 'PROTOBUF_NODISCARD' [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
    ^
DQMServices/Core/interface/ROOTFilePB.pb.h:222:29: error: duplicate member 'string' [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                            ^
DQMServices/Core/interface/ROOTFilePB.pb.h:201:29: note: previous declaration is here
--
DQMServices/Core/interface/ROOTFilePB.pb.h:222:29: error: non-friend class member 'string' cannot have a qualified name [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                       ~~~~~^
DQMServices/Core/interface/ROOTFilePB.pb.h:222:35: error: expected ';' at end of declaration list [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                                  ^
                                  ;
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

…(CPP20)

cmsbuild · 2023-07-11T17:00:24Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42228/36248

ERROR: Build errors found during clang-tidy run.

    PROTOBUF_NODISCARD std::string* release_full_pathname();
                       ~~~~~^
DQMServices/Core/interface/ROOTFilePB.pb.h:201:35: error: expected ';' at end of declaration list [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_full_pathname();
                                  ^
                                  ;
DQMServices/Core/interface/ROOTFilePB.pb.h:222:5: error: unknown type name 'PROTOBUF_NODISCARD' [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
    ^
DQMServices/Core/interface/ROOTFilePB.pb.h:222:29: error: duplicate member 'string' [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                            ^
DQMServices/Core/interface/ROOTFilePB.pb.h:201:29: note: previous declaration is here
--
DQMServices/Core/interface/ROOTFilePB.pb.h:222:29: error: non-friend class member 'string' cannot have a qualified name [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                       ~~~~~^
DQMServices/Core/interface/ROOTFilePB.pb.h:222:35: error: expected ';' at end of declaration list [clang-diagnostic-error]
    PROTOBUF_NODISCARD std::string* release_streamed_histo();
                                  ^
                                  ;
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2023-07-12T07:15:08Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42228/36251

ERROR: Build errors found during clang-tidy run.

/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02793/el8_amd64_gcc11/external/tensorflow/2.6.4-94ea81ac8dcc00455e8150b7498bee77/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:51:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(8);
  ^~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02793/el8_amd64_gcc11/external/tensorflow/2.6.4-94ea81ac8dcc00455e8150b7498bee77/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:54:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(1);
  ^~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02793/el8_amd64_gcc11/external/tensorflow/2.6.4-94ea81ac8dcc00455e8150b7498bee77/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:55:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(max_n_pf_);
  ^~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02793/el8_amd64_gcc11/external/tensorflow/2.6.4-94ea81ac8dcc00455e8150b7498bee77/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:56:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(1);
  ^~~~~
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02793/el8_amd64_gcc11/external/tensorflow/2.6.4-94ea81ac8dcc00455e8150b7498bee77/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

iarspider · 2023-07-12T07:20:34Z

code-checks with cms.week1.PR_78d7858e/57.0-3187777e8b462de4843bc3736a08b743

cmsbuild · 2023-07-12T07:32:01Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-42228/36252

ERROR: Build errors found during clang-tidy run.

/cvmfs/cms-ci.cern.ch/week1/PR_78d7858e/el8_amd64_gcc11/external/tensorflow/2.12.0-174560a388c169544bfb91342f3aecd6/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:51:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(8);
  ^~~~~
/cvmfs/cms-ci.cern.ch/week1/PR_78d7858e/el8_amd64_gcc11/external/tensorflow/2.12.0-174560a388c169544bfb91342f3aecd6/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:54:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(1);
  ^~~~~
/cvmfs/cms-ci.cern.ch/week1/PR_78d7858e/el8_amd64_gcc11/external/tensorflow/2.12.0-174560a388c169544bfb91342f3aecd6/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:55:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(max_n_pf_);
  ^~~~~
/cvmfs/cms-ci.cern.ch/week1/PR_78d7858e/el8_amd64_gcc11/external/tensorflow/2.12.0-174560a388c169544bfb91342f3aecd6/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
RecoMET/METPUSubtraction/plugins/DeepMETProducer.cc:56:3: error: 'this' argument to member function 'AddDim' has type 'const tensorflow::TensorShape', but function is not marked const [clang-diagnostic-error]
  shape.AddDim(1);
  ^~~~~
/cvmfs/cms-ci.cern.ch/week1/PR_78d7858e/el8_amd64_gcc11/external/tensorflow/2.12.0-174560a388c169544bfb91342f3aecd6/include/tensorflow/core/framework/tensor_shape.h:206:8: note: 'AddDim' declared here
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

iarspider · 2023-07-25T08:10:19Z

@cms-sw/reconstruction-l2 gentle ping. This is also needed for gcc13 and cpp20 IBs.

clacaputo · 2023-07-31T10:12:45Z

+reconstruction

cmsbuild · 2023-07-31T10:13:11Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2023-07-31T10:20:33Z

@perrotta @rappoccio , this should go in along with cms-sw/cmsdist#8565

perrotta · 2023-07-31T16:42:07Z

please test with cms-sw/cmsdist#8565
(Previous tests were two weeks old, and they have expired by now)

cmsbuild · 2023-08-01T07:51:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-56091b/33990/summary.html
COMMIT: d487a13
CMSSW: CMSSW_13_3_X_2023-07-31-1100/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/42228/33990/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@iarspider [GCC13] Add include of cstdint in more places #42392
@missirol fix configuration of GPU Pixel unpacker in Run-3 HLT menus post-#41632 #42410
@cms-sw SCRAMV2: Do not create CMS_PAtH/etc/scramrc files which are taken care by SCRAmv1 cmsdist#8622

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-56091b/33990/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-56091b/33990/git-merge-result

Comparison Summary

Summary:

You potentially removed 11 lines from the logs
Reco comparison results: 28 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3150821
DQMHistoTests: Total failures: 1569
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3149230
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 207 log files, 159 edm output root files, 48 DQM output files
TriggerResults: no differences found

perrotta · 2023-08-01T10:36:49Z

@cms-sw/reconstruction-l2 are differences in reco/DQM outputs understood and/or accepted?

perrotta · 2023-08-01T19:51:31Z

+1

Fasthadd: move back to master repo after merging cms-sw/cmssw#42228

Update generated protobuf files

003102d

cmsbuild added this to the CMSSW_13_2_X milestone Jul 11, 2023

cmsbuild added dqm-pending pending-signatures tests-pending orp-pending code-checks-pending labels Jul 11, 2023

iarspider mentioned this pull request Jul 11, 2023

Update TF to 2.12 (backport from IB/CMSSW_13_2_X/g13 branch) cms-sw/cmsdist#8565

Merged

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jul 11, 2023

Fix "variable-sized array type ... is not a valid template argument" …

010a48d

…(CPP20)

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jul 11, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jul 11, 2023

Workaround

f30b94f

cmsbuild added reconstruction-pending code-checks-pending and removed code-checks-rejected labels Jul 12, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jul 12, 2023

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jul 12, 2023

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jul 12, 2023

smuzaffar modified the milestones: CMSSW_13_2_X, CMSSW_13_3_X Jul 18, 2023

iarspider mentioned this pull request Jul 18, 2023

Use C++20 with more packages cms-sw/cmsdist#8558

Merged

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels Jul 31, 2023

cmsbuild added tests-started and removed tests-approved labels Jul 31, 2023

cmsbuild added tests-approved and removed tests-started labels Aug 1, 2023

cmsbuild added orp-approved and removed orp-pending labels Aug 1, 2023

cmsbuild merged commit 972d35c into cms-sw:master Aug 1, 2023

cmsbuild mentioned this pull request Aug 2, 2023

[Do not merge] Test cmssw build image cms-sw/cmsdist#8626

Closed

iarspider added a commit to cms-sw/cmsdist that referenced this pull request Aug 2, 2023

Fasthadd: move back to master repo after merging cms-sw/cmssw#42228

edb5711

cmsbuild mentioned this pull request Aug 2, 2023

AVXVec: replace 2*insertf128 with 1*_mm256_set_m128d #42446

Merged

fwyzard mentioned this pull request Aug 3, 2023

[GPU] Unit test testEigenGPUNoFit_t failing when testing inverse 5x5 #42460

Closed

smuzaffar added a commit to cms-sw/cmsdist that referenced this pull request Aug 4, 2023

Merge pull request #8628 from cms-sw/fasthadd-official

ae82dfe

Fasthadd: move back to master repo after merging cms-sw/cmssw#42228

This was referenced Aug 29, 2023

Use consistent abseil version in TensorFlow cms-sw/cmsdist#8672

Closed

revert DeepMET TensorFlow workaround #42682

Merged

iarspider deleted the protobuf-3.21.9 branch February 27, 2024 10:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228

Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228

iarspider commented Jul 11, 2023

cmsbuild commented Jul 11, 2023

cmsbuild commented Jul 11, 2023

cmsbuild commented Jul 12, 2023

iarspider commented Jul 12, 2023

cmsbuild commented Jul 12, 2023

iarspider commented Jul 25, 2023

clacaputo commented Jul 31, 2023

cmsbuild commented Jul 31, 2023

smuzaffar commented Jul 31, 2023

perrotta commented Jul 31, 2023

cmsbuild commented Aug 1, 2023

perrotta commented Aug 1, 2023

perrotta commented Aug 1, 2023

Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228

Code changes for Protobuf 3.21.9 and TensorFlow 2.12 #42228

Conversation

iarspider commented Jul 11, 2023

PR description:

PR validation:

cmsbuild commented Jul 11, 2023

cmsbuild commented Jul 11, 2023

cmsbuild commented Jul 12, 2023

iarspider commented Jul 12, 2023

cmsbuild commented Jul 12, 2023

iarspider commented Jul 25, 2023

clacaputo commented Jul 31, 2023

cmsbuild commented Jul 31, 2023

smuzaffar commented Jul 31, 2023

perrotta commented Jul 31, 2023

cmsbuild commented Aug 1, 2023

Comparison Summary

perrotta commented Aug 1, 2023

perrotta commented Aug 1, 2023