Updates for tensorflow on cc8

[mrodozov]

Changes taken from the slc7 PR for tensorflow.
The forward porting of cmsdist master to cc8 is not straightforward, so it had to be done by hand.
To be tested when the next cc8 IB is available

[cmsbuild]

A new Pull Request was created by @mrodozov (Mircho Rodozov) for branch IB/CMSSW_11_1_X/cc8.

@cmsbuild, @smuzaffar, @mrodozov, @tulamor can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

[mrodozov]

please test

[cmsbuild]

The tests are being triggered in jenkins.

[cmsbuild]

-1
Tested at: UNKNOWN
I was not able to find a release to test this PR. See the Jenkins logs for more details.

[mrodozov]

please test for cc8_amd64_gcc820

[cmsbuild]

The tests are being triggered in jenkins.
Test Parameters:

• ARCHITECTURE_FILTER = cc8_amd64_gcc820

[cmsbuild]

-1
Tested at: UNKNOWN
I was not able to find a release to test this PR. See the Jenkins logs for more details.

[mrodozov]

test parameters:

• architecture = cc8_amd64_gcc8

[mrodozov]

please test

[cmsbuild]

The tests are being triggered in jenkins.
Test Parameters:

• ARCHITECTURE_FILTER = cc8_amd64_gcc8
  https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4905/console Started: 2020/02/26 21:48

[cmsbuild]

-1

Tested at: b181ace

• Build:

I found compilation error when building:

�[32m[761 / 1,769]�[0m checking cached actions

�[1A�[K�[31m�[1mFAILED:�[0m Build did NOT complete successfully

�[1A�[K�[31m�[1mFAILED:�[0m Build did NOT complete successfully
�[0m
ERROR: Could not build Bazel
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wBV8gA (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+bazel+0.29.1-45459d
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wBV8gA (%build)


You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-45459d/4905/summary.html

[cmsbuild]

+1
Tested at: cd5aa6d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-45459d/4932/summary.html
CMSSW: CMSSW_11_1_X_2020-02-27-2300
SCRAM_ARCH: cc8_amd64_gcc8

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-45459d/4932/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 42529 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2679706
• DQMHistoTests: Total failures: 385725
• DQMHistoTests: Total nulls: 9
• DQMHistoTests: Total successes: 2293653
• DQMHistoTests: Total skipped: 319
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -6514.8 KiB( 33 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): -259.963 KiB DQM/TimerService
• DQMHistoSizes: changed ( 10224.0 ): -0.352 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 1306.0,... ): -204.107 KiB DQM/TimerService
• DQMHistoSizes: changed ( 1325.7,... ): -204.104 KiB DQM/TimerService
• DQMHistoSizes: changed ( 136.731,... ): -204.115 KiB DQM/TimerService
• DQMHistoSizes: changed ( 136.793 ): -0.004 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 136.874 ): -0.016 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 250202.181 ): 0.170 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 25202.0 ): -0.176 KiB SiStrip/MechanicalView
• DQMHistoSizes: changed ( 7.3 ): -0.173 KiB SiStrip/MechanicalView
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[mrodozov]

@slava77 could you have a look please there are comparison differences

[slava77]

@mrodozov
accodring to this https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_11_1_X_2020-02-27-2300/
the baseline is slc7_amd64_gcc820,
compared to this PR arch cc8_amd64_gcc8.
I do not expect bitwise agreement between the two, even if the CMSSW code is the same.

Is there a way to set the baseline to be from the same IB/arch that is used for the (special) tests like this?

[mrodozov]

I have to extend my understanding about published baselines, from what I see it's only the production architecture. which explains the differences.

Is there a way to set the baseline to be from the same IB/arch that is used for the (special) tests like this?

there is, I think what we did in the past was to produce the baseline by hand for the arch and publish it, but I've never done it myself. we did it at least once for this PR:
cms-externals/onnxruntime#4

[mrodozov]

+externals
fixes the cc8 IBs, differences in the comparison are to be expected

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_1_X/cc8 IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)