Update ROCm to version 6.1.0

[fwyzard]

Add amd-smi and ROCProfiler binaries and libraries.

[fwyzard]

please test

[cmsbuild]

A new Pull Request was created by @fwyzard for branch IB/CMSSW_14_1_X/master.

@iarspider, @smuzaffar, @aandvalenzuela can you please review it and eventually sign? Thanks.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38900/summary.html
COMMIT: f76bc04
CMSSW: CMSSW_14_1_X_2024-04-17-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9143/38900/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[fwyzard]

please test

[cmsbuild]

Pull request #9143 was updated.

[cmsbuild]

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38907/summary.html
COMMIT: f1b8c12
CMSSW: CMSSW_14_1_X_2024-04-17-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9143/38907/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @belforte Update crab-dev to latest tag #9138
• @cms-sw Rivet: use eigen flags; update rivet analysis path for multi-arch #9141

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38907/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38907/git-merge-result

Build

I found compilation error when building:

>> Leaving Package HeterogeneousCore/ROCmServices
>> Package HeterogeneousCore/ROCmServices built
Entering library rule at src/HeterogeneousCore/ROCmServices/plugins
>> Compiling edm plugin src/HeterogeneousCore/ROCmServices/plugins/ROCmMonitoringService.cc
>> Compiling edm plugin src/HeterogeneousCore/ROCmServices/plugins/ROCmService.cc
src/HeterogeneousCore/ROCmServices/plugins/ROCmService.cc:9:10: fatal error: rocm_version.h: No such file or directory
    9 | #include 
      |          ^~~~~~~~~~~~~~~~
compilation terminated.
gmake: *** [tmp/el8_amd64_gcc12/src/HeterogeneousCore/ROCmServices/plugins/HeterogeneousCoreROCmServicesPlugins/ROCmService.cc.o] Error 1
>> Building edm plugin tmp/el8_amd64_gcc12/src/HeterogeneousCore/ROCmServices/plugins/HeterogeneousCoreROCmServicesPlugins/libHeterogeneousCoreROCmServicesPlugins.so


[fwyzard]

please test with cms-sw/cmssw#44777

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38939/summary.html
COMMIT: f1b8c12
CMSSW: CMSSW_14_1_X_2024-04-18-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9143/38939/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @mmusich fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins cmssw#44764
• @cms-L1TK Fix bug in L1 track duplicate removal for displaced tracking cmssw#44574
• @makortel Ignore errors from alpaka::enqueue() in CachingAllocator::free() cmssw#44730
• @makortel Add tests for ProductID metadata consistency for ROOT and streamer formats cmssw#44753
• @iarspider [GCC13] Temporary convert array-bounds errors into warnings cmssw#44646
• @cms-sw Add cmssw/external/lib to RIVET_ANALYSIS_PATH #9146
• @fwyzard Add NVML python bindings #9134

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38939/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-200d6f/38939/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test runtestPhysicsToolsPatAlgos had ERRORS

Comparison Summary

Summary:

• You potentially removed 90 lines from the logs
• Reco comparison results: 46 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3319599
• DQMHistoTests: Total failures: 311
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3319268
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 202 log files, 165 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[fwyzard]

runtestPhysicsToolsPatAlgos has been failing for weeks.

@smuzaffar do you think it would be possible to teach the bot to ignore failures that are already present in the base release ?

[smuzaffar]

runtestPhysicsToolsPatAlgos has been failing for weeks.

@smuzaffar do you think it would be possible to teach the bot to ignore failures that are already present in the base release ?

we had discussed this in core sw meeting. One reason to not do this is that sometimes a PR is suppose to fix a failure but if it does not provide the right fix (and test still fails) then bot is going to ignore the error ( as we told it to ignore IB errors) and mark it as passed.

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_1_X/master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)
Notice This PR was tested with additional Pull Request(s), please also merge them if necessary: cms-sw/cmssw#44777

[fwyzard]

One reason to not do this is that sometimes a PR is suppose to fix a failure but if it does not provide the right fix (and test still fails) then bot is going to ignore the error ( as we told it to ignore IB errors) and mark it as passed.

What about ignoring the tests that are expected to fail (because they fail in the IB), but report if they actually pass ?
This way if a PR is supposed to fix a test, it should be easy to see that it does.

[smuzaffar]

but if a PR is suppose to fix a test but it still fails during PR tests then bot will ignore the failure and mark PR test passed ... right?

[fwyzard]

but if a PR is suppose to fix a test but it still fails during PR tests then bot will ignore the failure and mark PR test passed ... right?

In such case I would expect the person that submitted the PR specifically to fix a test, the relevant L2, or the release manager, to check if the bot reports the test being fixed.

[smuzaffar]

In such case I would expect the person that submitted the PR specifically to fix a test, the relevant L2, or the release manager, to check if the bot reports the test being fixed.

this is a big expectation :-) I will see if we can come up with reasonable solution

[cmsbuild]

REMINDER @rappoccio, @sextonkennedy, @antoniovilela: This PR was tested with cms-sw/cmssw#44777, please check if they should be merged together

[fwyzard]

@cms-sw/orp-l2 could you merge this and cms-sw/cmssw#44777 for the next IB ?

[rappoccio]

+1

[fwyzard]

thanks

[makortel]

Given cms-sw/cmssw#44821 maybe we should revert this update before building 14_1_0_pre3? @cms-sw/orp-l2

[fwyzard]

OK for me to revert.

The main reason for the update was to fix the linker warnings about executable stack, but we can live with them.

Does cms-sw/cmssw#44821 mean that we have to start building HIP/ROCm from sources ?

[makortel]

Does cms-sw/cmssw#44821 mean that we have to start building HIP/ROCm from sources ?

That is certainly one way forward, but I'm not sure if we understand the full picture well enough to judge if that is the best (or only?) way forward. (e.g. I still wonder if there is something (non-)LTO-specific in this failure mode, or if the set of so-far-impacted IBs is just "luck")

[smuzaffar]

@makortel , all the IBs with this failure are non-lto ( https://github.com/search?q=repo%3Acms-sw%2Fcms-bot%20BUILD_OPTS%3Dno-lto&type=code )