Update ROCm to version 5.6.1 #9050

fwyzard · 2024-03-06T11:37:34Z

Update ROCm to version 5.6.1:

see https://rocm.docs.amd.com/en/docs-5.6.1/release.html for the list of changes and fixes.

Add the ROCm packages to support OpenMP and Thrust:

the OpenMP packages are required to compile any source that uses #pragma omp ... with hipcc;
the Thrust packages are needed for compatibility with CUDA, and are used by the HIP backend of PyTorch.

fwyzard · 2024-03-06T11:37:44Z

please test

cmsbuild · 2024-03-06T11:38:01Z

A new Pull Request was created by @fwyzard for branch IB/CMSSW_14_1_X/master.

@iarspider, @smuzaffar, @aandvalenzuela can you please review it and eventually sign? Thanks.
@rappoccio, @antoniovilela, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-03-06T11:38:02Z

cms-bot internal usage

cmsbuild · 2024-03-06T12:04:15Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37928/summary.html
COMMIT: 64510f3
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37928/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Build successful rocm.
(, , ) done
Trying to install the rpm package external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e just built.
Checking local path dependency for rpm package external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e just build.
RPM installation stderr rocm:
error: Failed dependencies:
	libffi.so.6()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64
	libpython3.6m.so.1.0()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64

Failed to install RPM for rocm
Requested to quit.

fwyzard · 2024-03-06T12:47:12Z

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37928/summary.html COMMIT: 64510f3 CMSSW: CMSSW_14_1_X_2024-03-05-2300/el8_amd64_gcc12 User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37928/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Build successful rocm.
(, , ) done
Trying to install the rpm package external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e just built.
Checking local path dependency for rpm package external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e just build.
RPM installation stderr rocm:
error: Failed dependencies:
libffi.so.6()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64
libpython3.6m.so.1.0()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64

Failed to install RPM for rocm
Requested to quit.

@iarspider @aandvalenzuela do you have any idea why we see these errors ?

ROCm 5.6.1 should not be significantly different from ROCm 5.6.0, and that one used to build correctly.

See https://rocm.docs.amd.com/en/docs-5.6.1/release.html for the list of changes and fixes.

cmsbuild · 2024-03-06T13:13:12Z

Pull request #9050 was updated.

fwyzard · 2024-03-06T13:13:29Z

please test

fwyzard · 2024-03-06T13:22:06Z

please test for slc7_amd64_gcc12

fwyzard · 2024-03-06T13:22:33Z

please test for el9_amd64_gcc12

fwyzard · 2024-03-06T13:23:00Z

please test for el8_aarch64_gcc12

fwyzard · 2024-03-06T13:23:15Z

please test for el8_ppc64le_gcc12

cmsbuild · 2024-03-06T18:05:29Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37937/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el8_aarch64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37937/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-06T18:39:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37939/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el8_ppc64le_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37939/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-06T23:38:26Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37938/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el9_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37938/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-06T23:41:23Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37932/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37932/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-06T23:59:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37936/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37936/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-07T03:56:51Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37938/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el9_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37938/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-07T04:27:04Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37936/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/slc7_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37936/install.sh to create a dev area with all the needed externals and cmssw changes.

cmsbuild · 2024-03-07T05:23:46Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3f070/37938/summary.html
COMMIT: 9d9dab6
CMSSW: CMSSW_14_1_X_2024-03-05-2300/el9_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/9050/37938/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 343 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 53561 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3293028
DQMHistoTests: Total failures: 113485
DQMHistoTests: Total nulls: 242
DQMHistoTests: Total successes: 3179281
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -9.552000000000001 KiB( 47 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.596 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11634.0,... ): -0.439 KiB Physics/NanoAODDQM
DQMHistoSizes: changed ( 13034.0 ): 0.586 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.405 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.244 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): 0.481 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 8.0 ): -4.839 KiB SiStrip/MechanicalView
Checked 202 log files, 165 edm output root files, 48 DQM output files

fwyzard · 2024-03-07T13:34:23Z

type bugfix

fwyzard · 2024-03-07T13:35:56Z

@iarspider @aandvalenzuela @smuzaffar can any of you approve this for 14.1.x ?

iarspider · 2024-03-07T13:41:23Z

+externals

cmsbuild · 2024-03-07T13:41:51Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_14_1_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @rappoccio, @sextonkennedy, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

smuzaffar · 2024-03-11T14:58:39Z

libffi.so.6()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64
libpython3.6m.so.1.0()(64bit) is needed by external+rocm+5.6.1-6320ba1f36f64f70428881e859817e3e-1-1.x86_64

ROCm 5.6.1 should not be significantly different from ROCm 5.6.0, and that one used to build correctly.

@fwyzard , this is because the newly added openmp-extras-devel-16.56.0.50601-93.el%{rhel}.%{_arch}.rpm is build against system python 3.6. Following are the two libs which are linked against python 3.6

llvm/lib-debug/ompd/ompdModule.so
llvm/lib-debug/ompd/ompdModule.cpython-36m-x86_64-linux-gnu.so

fwyzard · 2024-03-11T17:10:00Z

I see... I guess we should drop those specific files ?
The only alternative is that we start building ROCm from sources, so we can link our version of python.

fwyzard added 2 commits March 6, 2024 12:19

Add OpenMP support for ROCm

d3c1ff6

Add Thrust libraries for ROCm

5fda1a8

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Mar 6, 2024

This was referenced Mar 6, 2024

Add OpenMP support for ROCm #8676

Closed

Update ROCm to version 5.6.1 [14.0.x] #9052

Merged

cmsbuild added tests-rejected and removed tests-started labels Mar 6, 2024

Update ROCm to version 5.6.1

9d9dab6

See https://rocm.docs.amd.com/en/docs-5.6.1/release.html for the list of changes and fixes.

fwyzard force-pushed the IB/CMSSW_14_1_X/master_rocm-5.6.1 branch from 64510f3 to 9d9dab6 Compare March 6, 2024 13:12

cmsbuild added tests-pending and removed tests-rejected labels Mar 6, 2024

cmsbuild added tests-started and removed tests-pending labels Mar 6, 2024

fwyzard mentioned this pull request Mar 6, 2024

Update OpenMPI, UCX and related libraries #8904

Merged

cmsbuild added tests-started and removed tests-approved labels Mar 6, 2024

cmsbuild added tests-approved tests-started and removed tests-started tests-approved labels Mar 6, 2024

cmsbuild added the bug-fix label Mar 7, 2024

iarspider merged commit f8fff94 into cms-sw:IB/CMSSW_14_1_X/master Mar 7, 2024
33 checks passed

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Mar 7, 2024

This was referenced Mar 8, 2024

[DO NOT MERGE] Test making Eigen alignment setting consistent #9043

Closed

Update gbl to V03-01-01 and millepede to V04-16-00 #9053

Merged

Refactor scatterer addition in aGblPoint to explicitly specify type for scatPrecision.inverse() cms-sw/cmssw#44340

Merged

fwyzard deleted the IB/CMSSW_14_1_X/master_rocm-5.6.1 branch April 7, 2024 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update ROCm to version 5.6.1 #9050

Update ROCm to version 5.6.1 #9050

fwyzard commented Mar 6, 2024 •

edited

Loading

fwyzard commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024 •

edited

Loading

cmsbuild commented Mar 6, 2024

fwyzard commented Mar 6, 2024

External Build

cmsbuild commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

fwyzard commented Mar 7, 2024

fwyzard commented Mar 7, 2024

iarspider commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

smuzaffar commented Mar 11, 2024

fwyzard commented Mar 11, 2024

Update ROCm to version 5.6.1 #9050

Update ROCm to version 5.6.1 #9050

Conversation

fwyzard commented Mar 6, 2024 • edited Loading

fwyzard commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024 • edited Loading

cmsbuild commented Mar 6, 2024

External Build

fwyzard commented Mar 6, 2024

External Build

cmsbuild commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

fwyzard commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 6, 2024

cmsbuild commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

Comparison Summary

fwyzard commented Mar 7, 2024

fwyzard commented Mar 7, 2024

iarspider commented Mar 7, 2024

cmsbuild commented Mar 7, 2024

smuzaffar commented Mar 11, 2024

fwyzard commented Mar 11, 2024

fwyzard commented Mar 6, 2024 •

edited

Loading

cmsbuild commented Mar 6, 2024 •

edited

Loading