Update UCX to version 1.12.1 #7809

fwyzard · 2022-04-20T14:04:42Z

Add the xpmem library from the HEAD of the master branch as of 2022.03.08, corresponding to the commit 61c39efdea943ac863037d7e35b236145904e64d.
Based on v2.6.3 with updates for Linux kernel up to 5.17.

Enable additional libraries in UCX:

enable the use of xpmem for intra-node communication;
enable the use of ROCm for AMD gpus;
remove the ROCm GDR module, which is not compatible with GDRCopy v2.x.

Update UCX to version 1.12.1:

change the default for UCX_MEM_CUDA_HOOK_MODE from "reloc" to "bistro";
various bug fixes for CUDA and ROCm;
see https://github.com/openucx/ucx/releases/tag/v1.12.1 for the full change log.

cmsbuild · 2022-04-20T14:05:05Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_12_4_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2022-04-20T14:05:28Z

@smuzaffar do I need to add xpmem to cmssw-tool-conf.spec ?

fwyzard · 2022-04-20T14:08:49Z

please test

cmsbuild · 2022-04-20T14:08:50Z

Pull request #7809 was updated.

fwyzard · 2022-04-20T14:09:36Z

This PR includes #7795 to ease testing. It can be rebased once that is merged.

cmsbuild · 2022-04-20T14:28:30Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24064/summary.html
COMMIT: bc787d8
CMSSW: CMSSW_12_4_X_2022-04-20-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24064/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

Requested to quit.
* The action "build-external+ucx+1.12.1-871f2c8f3832a729236a3a4b83fb7b49" was not completed successfully because Failed to build ucx. Log file in /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc10/external/ucx/1.12.1-871f2c8f3832a729236a3a4b83fb7b49/log. Final lines of the log file:
67 |         ret = gdr_copy_from_bar(buffer, (void *)remote_addr, length);
|               ^~~~~~~~~~~~~~~~~
|               gdr_copy_from_mapping
rocm_gdr_ep.c:67:15: error: nested extern declaration of 'gdr_copy_from_bar' [-Werror=nested-externs]
cc1: all warnings being treated as errors
make[4]: *** [libuct_rocm_gdr_la-rocm_gdr_ep.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
make[4]: Leaving directory `/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/slc7_amd64_gcc10/external/ucx/1.12.1-871f2c8f3832a729236a3a4b83fb7b49/ucx-1.12.1/src/uct/rocm/gdr'
make[3]: *** [all-recursive] Error 1

smuzaffar · 2022-04-20T15:46:22Z

@smuzaffar do I need to add xpmem to cmssw-tool-conf.spec ?

No, there is no need to explicitly add it in cmssw-tool-conf.

cmsbuild · 2022-04-20T21:03:57Z

Pull request #7809 was updated.

fwyzard · 2022-04-20T22:48:02Z

please test

cmsbuild · 2022-04-20T22:48:15Z

Pull request #7809 was updated.

cmsbuild · 2022-04-20T23:06:30Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24079/summary.html
COMMIT: ef20ef1
CMSSW: CMSSW_12_4_X_2022-04-20-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24079/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ sed '-e/SUBDIRS/s/ *\//' -i src/uct/rocm/Makefile.am
+ sed '-e/src\/uct\/rocm\/gdr\/configure\.m4/d' -i src/uct/rocm/configure.m4
+ rm -rf src/uct/rocm/gdr
+ ./autogen.sh
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Saz1OQ: line 48: ./autogen.sh: No such file or directory
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Saz1OQ (%prep)


RPM build errors:
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.Saz1OQ (%prep)

cmsbuild · 2022-04-21T17:57:36Z

Pull request #7809 was updated.

smuzaffar · 2022-04-21T20:54:48Z

ucx.spec

 Requires: rdma-core
+Requires: rocm


@fwyzard , this will package rocm for all archs. I think we should include and configure rocm only for x86_64 archs

you are right, of course.

Add the xpmem library from the HEAD of the master branch as of 2022.03.08, corresponding to the commit 61c39efdea943ac863037d7e35b236145904e64d. Based on xpmem v2.6.3 with updates for Linux kernel up to 5.17.

Enable additional libraries in UCX: - enable the use of xpmem for intra-node communication; - enable the use of ROCm for AMD gpus (only for x86_64); - remove the ROCm GDR module, which is not compatible with GDRCopy v2.x. Update UCX to version 1.12.1: - change the default for UCX_MEM_CUDA_HOOK_MODE from "reloc" to "bistro"; - various bug fixes for CUDA and ROCm; - see https://github.com/openucx/ucx/releases/tag/v1.12.1 for the full change log.

fwyzard · 2022-04-21T22:06:41Z

please test

cmsbuild · 2022-04-21T22:06:51Z

Pull request #7809 was updated.

fwyzard · 2022-04-21T22:07:26Z

@cmsbuild, please test for el8_amd64_gcc10

fwyzard · 2022-04-21T22:08:25Z

@cmsbuild, please test for el9_amd64_gcc11

fwyzard · 2022-04-21T22:08:50Z

@cmsbuild, please test for el8_ppc64le_gcc10

fwyzard · 2022-04-21T22:09:02Z

@cmsbuild, please test for el8_aarch64_gcc10

cmsbuild · 2022-04-22T05:20:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24116/summary.html
COMMIT: 431af70
CMSSW: CMSSW_12_4_X_2022-04-20-2300/el8_ppc64le_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24116/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24116/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24116/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreUtilities had ERRORS
---> test DRNTest had ERRORS

cmsbuild · 2022-04-22T08:21:47Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24118/summary.html
COMMIT: 431af70
CMSSW: CMSSW_12_4_X_2022-04-20-2300/el8_aarch64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24118/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24118/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24118/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test DRNTest had ERRORS

smuzaffar · 2022-04-22T10:15:46Z

please test with cms-sw/cms-bot#1751

cmsbuild · 2022-04-22T12:01:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24114/summary.html
COMMIT: 431af70
CMSSW: CMSSW_12_4_X_2022-04-20-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24114/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24114/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24114/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/pool/condor/dir_117278/jenkins/workspace/compare-root-files-short-matrix/data/PR-ee27b4/39434.75_TTbar_14TeV+2026D88_HLT75e33+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HLT75e33+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 62740 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695434
DQMHistoTests: Total failures: 594594
DQMHistoTests: Total nulls: 380
DQMHistoTests: Total successes: 3100438
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.31499999999999995 KiB( 48 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.063 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 11834.0 ): 2.372 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): 0.006 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 7.3 ): -2.009 KiB SiStrip/MechanicalView
Checked 205 log files, 45 edm output root files, 49 DQM output files
TriggerResults: found differences in 14 / 48 workflows

cmsbuild · 2022-04-22T19:23:17Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24128/summary.html
COMMIT: 431af70
CMSSW: CMSSW_12_4_X_2022-04-21-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7809/24128/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24128/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ee27b4/24128/git-merge-result

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ee27b4/39434.75_TTbar_14TeV+2026D88_HLT75e33+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HLT75e33+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695434
DQMHistoTests: Total failures: 19
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3695392
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 205 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

smuzaffar · 2022-04-28T06:04:03Z

+externals

smuzaffar · 2022-04-28T06:04:03Z

+externals

cmsbuild · 2022-04-28T06:04:26Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_4_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Apr 20, 2022

fwyzard mentioned this pull request Apr 20, 2022

Update OpenMPI to 4.1.0, and add extra support libraries #6523

Merged

fwyzard force-pushed the IB/CMSSW_12_4_X/master_UCX_updates branch from 34e7699 to bc787d8 Compare April 20, 2022 14:08

cmsbuild added tests-started and removed tests-pending labels Apr 20, 2022

cmsbuild added tests-rejected and removed tests-started labels Apr 20, 2022

fwyzard force-pushed the IB/CMSSW_12_4_X/master_UCX_updates branch from bc787d8 to fb64181 Compare April 20, 2022 21:03

cmsbuild added tests-pending and removed tests-rejected labels Apr 20, 2022

fwyzard force-pushed the IB/CMSSW_12_4_X/master_UCX_updates branch from fb64181 to ef20ef1 Compare April 20, 2022 22:47

cmsbuild added tests-started and removed tests-pending labels Apr 20, 2022

cmsbuild added tests-rejected and removed tests-started labels Apr 20, 2022

fwyzard force-pushed the IB/CMSSW_12_4_X/master_UCX_updates branch from ef20ef1 to 2c12a58 Compare April 20, 2022 23:10

smuzaffar reviewed Apr 21, 2022

View reviewed changes

fwyzard added 2 commits April 22, 2022 00:03

Add xpmem library from the master branch as of 2022.03.08 / 61c39ef

8f77585

Add the xpmem library from the HEAD of the master branch as of 2022.03.08, corresponding to the commit 61c39efdea943ac863037d7e35b236145904e64d. Based on xpmem v2.6.3 with updates for Linux kernel up to 5.17.

fwyzard force-pushed the IB/CMSSW_12_4_X/master_UCX_updates branch from 539a478 to 431af70 Compare April 21, 2022 22:06

cmsbuild added the requires-external label Apr 22, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 22, 2022

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Apr 28, 2022

smuzaffar merged commit 15be736 into cms-sw:IB/CMSSW_12_4_X/master Apr 28, 2022

This was referenced Apr 28, 2022

Dynamic Reduction Network for photon energy regression in miniAOD using the SonicTriton service cms-sw/cmssw#37134

Merged

DeepSC algorithm for ECAL SuperClusters cms-sw/cmssw#37115

Merged

fwyzard deleted the IB/CMSSW_12_4_X/master_UCX_updates branch July 6, 2022 22:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update UCX to version 1.12.1 #7809

Update UCX to version 1.12.1 #7809

fwyzard commented Apr 20, 2022 •

edited

Loading

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

smuzaffar commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

cmsbuild commented Apr 21, 2022

smuzaffar Apr 21, 2022

fwyzard Apr 21, 2022

fwyzard commented Apr 21, 2022

cmsbuild commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

cmsbuild commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

smuzaffar commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

smuzaffar commented Apr 28, 2022

smuzaffar commented Apr 28, 2022

cmsbuild commented Apr 28, 2022

Update UCX to version 1.12.1 #7809

Update UCX to version 1.12.1 #7809

Conversation

fwyzard commented Apr 20, 2022 • edited Loading

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

External Build

smuzaffar commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

fwyzard commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

cmsbuild commented Apr 20, 2022

External Build

cmsbuild commented Apr 21, 2022

smuzaffar Apr 21, 2022

Choose a reason for hiding this comment

fwyzard Apr 21, 2022

Choose a reason for hiding this comment

fwyzard commented Apr 21, 2022

cmsbuild commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

fwyzard commented Apr 21, 2022

cmsbuild commented Apr 22, 2022

Unit Tests

cmsbuild commented Apr 22, 2022

Unit Tests

smuzaffar commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

Comparison Summary

cmsbuild commented Apr 22, 2022

Comparison Summary

smuzaffar commented Apr 28, 2022

smuzaffar commented Apr 28, 2022

cmsbuild commented Apr 28, 2022

fwyzard commented Apr 20, 2022 •

edited

Loading