[Tensorflow] Update to version 2.6.0 #7225

smuzaffar · 2021-08-21T14:27:13Z

Update TF to 2.6.0
Update Eigen to latest commit https://gitlab.com/libeigen/eigen/-/commit/82dd3710dac619448f50331c1d6a35da673f764a (version 3.4.90)
- I added the cms changes ( cms-externals/eigen-git-mirror@5c78a36 ) needed for powerpc and cuda/GPU. Few of our changes were rejected. Rejected changes are available here https://muzaffar.web.cern.ch/muzaffar/eigen-82dd3710dac619448f50331c1d6a35da673f764a.rej , @fwyzard @ericcano can you please check if these are needed and open a PR for cms-externals/eigen-git-mirror branch cms/master/82dd3710dac619448f50331c1d6a35da673f764a ?

cmsbuild · 2021-08-21T14:27:38Z

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_1_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2021-08-21T14:44:43Z

Shazad, for the GPU part, can you see if the branch at https://gitlab.com/fwyzard/eigen/-/tree/extend_CUDA_support merges cleanly ? I rebased on top of Eigen master a couple of days ago. .A

smuzaffar · 2021-08-21T15:09:17Z

Thanks @fwyzard , your branch is already using the latest eigen commit. So I only need to add

cms-externals/eigen-git-mirror@4ad2a6c
cms-externals/eigen-git-mirror@29be49c ( which is already included in upstream).

smuzaffar · 2021-08-21T15:11:35Z

please test

cmsbuild · 2021-08-21T15:11:47Z

Pull request #7225 was updated.

cmsbuild · 2021-08-21T15:39:22Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17944/summary.html
COMMIT: 15414a1
CMSSW: CMSSW_12_1_X_2021-08-21-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7225/17944/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

smuzaffar · 2021-08-21T17:42:24Z

please test

cmsbuild · 2021-08-21T17:42:38Z

Pull request #7225 was updated.

cmsbuild · 2021-08-21T22:48:18Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17946/summary.html
COMMIT: f361bfb
CMSSW: CMSSW_12_1_X_2021-08-21-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7225/17946/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ DEPS='py2-clang py2-keras '
+ '[' 'py2-clang py2-keras ' '!=' '' ']'
+ echo 'ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-clang py2-keras '
ERROR: Missing dependencies for py3-tensorflow (python3) found: py2-clang py2-keras 
+ exit 1
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.S0q0eY (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.S0q0eY (%build)

cmsbuild · 2021-08-22T07:55:06Z

Pull request #7225 was updated.

smuzaffar · 2021-08-22T07:55:14Z

please test

cmsbuild · 2021-08-23T12:33:32Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17958/summary.html
COMMIT: 51f7fe0
CMSSW: CMSSW_12_1_X_2021-08-22-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7225/17958/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test import-keras had ERRORS
---> test testTFGraphLoading had ERRORS
---> test testTheano had ERRORS
---> test testTFThreadPools had ERRORS
and more ...

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000352
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3000324
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

smuzaffar · 2021-08-23T12:46:24Z

please test

cmsbuild · 2021-08-23T12:46:29Z

Pull request #7225 was updated.

smuzaffar · 2021-08-23T13:27:22Z

FYI @riga
I see the following warnings while running cmsml

Running .2021-08-23 15:20:29.705176: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From py3-cmsml/0.1.1-llifpc2/lib/python3.9/site-packages/cmsml/tensorflow/tools.py:99: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.convert_variables_to_constants`
WARNING:tensorflow:From py3-tensorflow/2.6.0-3523f1ae2b020b77593e6809dd6db596/lib/python3.9/site-packages/tensorflow/python/framework/convert_to_constants.py:927: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`

these warning are also there for TF 2.5 ( https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_1_X_2021-08-22-2300/unitTestLogs/PhysicsTools/TensorFlow#/ )
I see that you are maintainer of https://pypi.org/project/cmsml/ , so may be you can update provide a new version of cmsml to avoid this warning?

smuzaffar · 2021-08-23T16:05:24Z

please test for slc7_aarch64_gcc9

smuzaffar · 2021-08-23T16:10:17Z

please test for slc7_ppc64le_gcc9

cmsbuild · 2021-08-23T18:50:11Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17968/summary.html
COMMIT: 48f2096
CMSSW: CMSSW_12_1_X_2021-08-22-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7225/17968/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 3000352
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3000330
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: no differences found

cmsbuild · 2021-08-23T21:03:09Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17975/summary.html
COMMIT: 48f2096
CMSSW: CMSSW_12_1_X_2021-08-22-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7225/17975/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test TestHeterogeneousCoreSonicTritonProducerGPU had ERRORS
---> test test_conddb had ERRORS
and more ...

cmsbuild · 2021-08-23T22:20:14Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-cfb39c/17978/summary.html
COMMIT: 48f2096
CMSSW: CMSSW_12_1_X_2021-08-22-2300/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7225/17978/install.sh to create a dev area with all the needed externals and cmssw changes.

smuzaffar · 2021-08-24T07:02:57Z

+externals

cmsbuild · 2021-08-24T07:03:23Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[Tensorflow] Update to version 2.6.0

81e2e27

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Aug 21, 2021

Update eigen.spec

15414a1

cmsbuild added tests-started and removed tests-pending labels Aug 21, 2021

cmsbuild added tests-rejected and removed tests-started labels Aug 21, 2021

Update tensorflow-sources.file

f361bfb

cmsbuild added tests-started and removed tests-rejected labels Aug 21, 2021

cmsbuild added tests-rejected and removed tests-started labels Aug 21, 2021

added py3-clang; updated py3-tensorflow deps

9ce2ad1

cmsbuild added tests-pending and removed tests-rejected labels Aug 22, 2021

cmsbuild removed the tests-pending label Aug 22, 2021

cmsbuild added tests-started and removed tests-rejected labels Aug 23, 2021

cmsbuild added tests-rejected and removed tests-started labels Aug 23, 2021

remove keras-nightl; rename Keras to keras

48f2096

cmsbuild added tests-started and removed tests-rejected labels Aug 23, 2021

cmsbuild added tests-approved and removed tests-started labels Aug 23, 2021

smuzaffar merged commit 23eed0a into IB/CMSSW_12_1_X/master Aug 24, 2021

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Aug 24, 2021

cmsbuild mentioned this pull request Aug 24, 2021

XrootD - backport 531 from devel to master #7233

Merged

smuzaffar mentioned this pull request Aug 25, 2021

Revert "[Tensorflow] Update to version 2.6.0" #7239

Merged

This was referenced Aug 25, 2021

[Do not merge] Testing private header usage warning for DEVEL IBS #7235

Closed

[Do not merge][Tensorflow] version 2.6.0 for CC8 branch #7244

Closed

smuzaffar deleted the tf260 branch August 28, 2021 10:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tensorflow] Update to version 2.6.0 #7225

[Tensorflow] Update to version 2.6.0 #7225

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

fwyzard commented Aug 21, 2021 via email

smuzaffar commented Aug 21, 2021

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

cmsbuild commented Aug 22, 2021

smuzaffar commented Aug 22, 2021

cmsbuild commented Aug 23, 2021

smuzaffar commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

smuzaffar commented Aug 23, 2021 •

edited

Loading

smuzaffar commented Aug 23, 2021

smuzaffar commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

smuzaffar commented Aug 24, 2021

cmsbuild commented Aug 24, 2021

[Tensorflow] Update to version 2.6.0 #7225

[Tensorflow] Update to version 2.6.0 #7225

Conversation

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

fwyzard commented Aug 21, 2021 via email

smuzaffar commented Aug 21, 2021

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

External Build

smuzaffar commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

cmsbuild commented Aug 21, 2021

External Build

cmsbuild commented Aug 22, 2021

smuzaffar commented Aug 22, 2021

cmsbuild commented Aug 23, 2021

Unit Tests

Comparison Summary

smuzaffar commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

smuzaffar commented Aug 23, 2021 • edited Loading

smuzaffar commented Aug 23, 2021

smuzaffar commented Aug 23, 2021

cmsbuild commented Aug 23, 2021

Comparison Summary

cmsbuild commented Aug 23, 2021

Unit Tests

cmsbuild commented Aug 23, 2021

smuzaffar commented Aug 24, 2021

cmsbuild commented Aug 24, 2021

smuzaffar commented Aug 23, 2021 •

edited

Loading