Update OpenMPI to 4.1.0, and add extra support libraries #6523

fwyzard · 2020-12-23T22:44:45Z

Update OpenMPI to version 4.1.0

See https://www.open-mpi.org/software/ompi/major-changes.php for major user-noticeable changes in the OpenMPI 4.1.x and 4.0.x series.
See https://raw.githubusercontent.com/open-mpi/ompi/v4.1.x/NEWS for a detailed list of changes in the OpenMPI 4.1.x and 4.0.x series.

Add the UCX libraries version 1.9.0

Unified Communication X (UCX) provides an optimized communication layer for Message Passing (MPI), PGAS/OpenSHMEM libraries and RPC/data-centric applications.
UCX utilizes high-speed networks for inter-node communication, and shared memory mechanisms for efficient intra-node communication.

For more information see:

Add the NVIDIA GDRCopy library v2.1

GDRCopy is a low-latency GPU memory copy library based on NVIDIA GPUDirect RDMA technology.

For more details, see

Add the Portable Hardware Locality (hwloc) package

The Portable Hardware Locality (hwloc) software package provides a portable abstraction (across OS, versions, architectures, ...) of the hierarchical topology of modern architectures, including NUMA memory nodes, sockets, shared caches, cores and simultaneous multithreading.
It also gathers various system attributes such as cache and memory information as well as the locality of I/O devices such as network interfaces, InfiniBand HCAs or GPUs.

See https://www.open-mpi.org/projects/hwloc/ for more details.

Add numactl utilities and libnuma library version 2.0.14

Simple NUMA policy support. It consists of a numactl program to run other programs with a specific NUMA policy and a libnuma shared library ("NUMA API") to set NUMA policy in applications.
See https://github.com/numactl/numactl/blob/v2.0.14/README.md for more information.

Add libpciaccess, a generic PCI access library

libpciaccess is a generic PCI access library originally developed for the Xorg project; it is now used by various other projects, for example the hwloc library developed by the OpenMPI project.

The upstream git repository is at https://gitlab.freedesktop.org/xorg/lib/libpciaccess

To avoid the dependency on the "xorg-macros" autotool package, the CMSDIST package for libpciaccess is based on the Debian package (where the autogen step has already been performed) instead of the upstream git sources.

cmsbuild · 2020-12-23T22:45:06Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_11_3_X/master.

@cmsbuild, @smuzaffar, @mrodozov can you please review it and eventually sign? Thanks.
cms-bot commands are listed here

fwyzard · 2020-12-23T22:46:03Z

Currently includes also #6522, because 12fdd3e and 3c037d5 are needed to build hwloc.

cmsbuild · 2020-12-23T23:52:11Z

Pull request #6523 was updated.

cmsbuild · 2020-12-23T23:58:49Z

Pull request #6523 was updated.

fwyzard · 2020-12-24T00:02:06Z

please test

cmsbuild · 2020-12-24T12:16:22Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5cf7e/11860/summary.html
CMSSW: CMSSW_11_3_X_2020-12-23-2300/slc7_amd64_gcc900

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-d5cf7e/1325.81_TTbar_13_106Xv1NanoAODINPUT+TTbar_13_106Xv1NanoAODINPUT+NANOEDMMC2017_106XMiniAODv1+HARVESTNANOAODMC2017_106XMiniAODv1
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-d5cf7e/136.8523_RunJetHT2018C_nanoULremini+RunJetHT2018C_nanoULremini+NANOEDM2018_106Xv2+HARVESTNANOAOD2018_106Xv2

Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716967
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2716938
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 156 log files, 37 edm output root files, 37 DQM output files

cmsbuild · 2020-12-30T00:52:06Z

Pull request #6523 was updated.

fwyzard · 2020-12-30T01:00:39Z

please test

cmsbuild · 2020-12-30T06:14:32Z

Pull request #6523 was updated.

fwyzard · 2020-12-30T06:14:55Z

please test

Unified Communication X (UCX) provides an optimized communication layer for Message Passing (MPI), PGAS/OpenSHMEM libraries and RPC/data-centric applications. UCX utilizes high-speed networks for inter-node communication, and shared memory mechanisms for efficient intra-node communication. For more information see - https://www.openucx.org/ - https://github.com/openucx/ucx/tree/v1.9.0

See https://www.open-mpi.org/software/ompi/major-changes.php for major user-noticeable changes in the OpenMPI 4.1.x and 4.0.x series. See https://raw.githubusercontent.com/open-mpi/ompi/v4.1.x/NEWS for a detailed list of changes in the OpenMPI 4.1.x and 4.0.x series.

fwyzard · 2021-01-08T15:22:50Z

Rebased to exclude the CUDA update from the changes.

fwyzard · 2021-01-08T15:22:54Z

please test

cmsbuild · 2021-01-08T15:22:56Z

Pull request #6523 was updated.

fwyzard · 2021-01-08T20:31:23Z

please test for slc7_aarch64_gcc9

fwyzard · 2021-01-08T20:31:52Z

please test for slc7_ppc64le_gcc9

cmsbuild · 2021-01-08T21:37:36Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5cf7e/12028/summary.html
COMMIT: 143c147
CMSSW: CMSSW_11_3_X_2021-01-08-1100/slc7_amd64_gcc900

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 1 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716967
DQMHistoTests: Total failures: 4
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2716941
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

cmsbuild · 2021-01-09T02:45:51Z

-1

Failed Tests: UnitTests AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5cf7e/12033/summary.html
COMMIT: 143c147
CMSSW: CMSSW_11_3_X_2021-01-07-2300/slc7_aarch64_gcc9

Unit Tests

I found errors in the following unit tests:

---> test testFWCoreConcurrency had ERRORS
---> test TestFWCoreServicesDriver had ERRORS
---> test TestRunnerFWCoreTFWLiteSelector had ERRORS
---> test TestConfigDP had ERRORS

fwyzard · 2021-01-09T21:12:57Z

The workflows failing on slc7_ppc64le_gcc9 are

11624.911

The unit tests failing on slc7_ppc64le_gcc9 are

DD4hepGeometryTestDriver
DetectorDescriptionDDCMSTestDriver
GeometryDTGeometryBuilderTestDriver
GeometryTrackerGeometryBuilderTestDriver
testAlignmentOfflineValidation
TestConfigDP
testDD4hepCompactView
testDD4hepFilteredView
testDD4hepFilteredViewFind
testDD4hepFilteredViewFirstChild
testDD4hepFilteredViewGet
testDD4hepFilteredViewGoTo
testDD4hepFilteredViewLevel
testFWCoreConcurrency
test_MpsWorkFlow
test_PrepareInputDb
testUnits

All of them were already failing in the CMSSW_11_3_X_2021-01-08-2300 IB (see https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/ib/CMSSW_11_3_X).

fwyzard · 2021-01-09T21:28:24Z

please test for slc7_aarch64_gcc9

cmsbuild · 2021-01-10T04:18:53Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5cf7e/12041/summary.html
COMMIT: 143c147
CMSSW: CMSSW_11_3_X_2021-01-08-2300/slc7_aarch64_gcc9

Unit Tests

I found errors in the following unit tests:

---> test TestFWCoreServicesDriver had ERRORS
---> test TestFWCoreServicesDriver had ERRORS
---> test PhiTest had ERRORS
---> test TestConfigDP had ERRORS

smuzaffar · 2021-01-10T21:34:41Z

please test for slc7_ppc64le_gcc9

cmsbuild · 2021-01-11T05:09:27Z

-1

Failed Tests: UnitTests RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d5cf7e/12046/summary.html
COMMIT: 143c147
CMSSW: CMSSW_11_3_X_2021-01-10-0000/slc7_ppc64le_gcc9

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test test_PrepareInputDb had ERRORS
---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test testUnits had ERRORS
---> test testDD4hepCompactView had ERRORS
---> test testDD4hepFilteredViewGoTo had ERRORS
---> test testDD4hepFilteredViewFind had ERRORS
---> test testDD4hepFilteredView had ERRORS
---> test testDD4hepFilteredViewFirstChild had ERRORS
---> test testDD4hepFilteredViewGet had ERRORS
---> test testDD4hepFilteredViewLevel had ERRORS
---> test DetectorDescriptionDDCMSTestDriver had ERRORS
and more ...

RelVals

11624.911 step1

11624.911_TTbar_13+2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA/step1_TTbar_13+2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA.log

smuzaffar · 2021-01-11T06:30:23Z

+externals
looks like bot is reporting same tests multiple times.

cmsbuild · 2021-01-11T06:30:46Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_11_3_X/master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

iarspider · 2022-04-12T08:51:43Z

Hi @fwyzard,

this recipe calls for --with-xpmem, but there is no recipe for xpmem in cmsdist. Also, xpmem requires kernel sources at build-time, which is something that should probably be avoided. The build did not fail because ucx will silently ignore --with-xpmem if it can't find xpmem.

Question: should we change the recipe to use --without-xpmem, or should we add xpmem to cmsdist?

Thanks,
Ivan

fwyzard · 2022-04-12T09:06:46Z

Hi @iarspider good timing, I'm in the process of reviewing the build options for UCX and MPI to improve support for RDMA transfers, I'll add this to the list of things to check.

fwyzard · 2022-04-20T07:09:30Z

Based on this comparison from a few years back, it looks like we should try to enable XPMEM:

I'll have a look in parallel to enabling ROCm.

fwyzard · 2022-04-20T14:06:14Z

Should be done in #7809 .

cmsbuild added the externals-pending label Dec 23, 2020

cmsbuild added orp-pending pending-signatures tests-pending labels Dec 23, 2020

fwyzard force-pushed the IB/CMSSW_11_3_X/master_OpenMPI_4.1.0 branch from 2a202db to ba6fa0d Compare December 23, 2020 23:58

fwyzard changed the title ~~Add libpciaccess 0.16 and hwloc 2.4.0~~ Update OpenMPI to 4.1.0, and add libpciaccess and hwloc libraries Dec 24, 2020

cmsbuild added tests-started and removed tests-pending labels Dec 24, 2020

cmsbuild added tests-approved and removed tests-started labels Dec 24, 2020

fwyzard force-pushed the IB/CMSSW_11_3_X/master_OpenMPI_4.1.0 branch from ba6fa0d to 39328c1 Compare December 30, 2020 00:51

cmsbuild added tests-pending and removed tests-approved labels Dec 30, 2020

cmsbuild added tests-started and removed tests-pending labels Dec 30, 2020

fwyzard changed the title ~~Update OpenMPI to 4.1.0, and add libpciaccess and hwloc libraries~~ Update OpenMPI to 4.1.0, and add extra support libraries Dec 30, 2020

fwyzard force-pushed the IB/CMSSW_11_3_X/master_OpenMPI_4.1.0 branch from 39328c1 to 96ceafe Compare December 30, 2020 06:14

cmsbuild added tests-pending and removed tests-started labels Dec 30, 2020

cmsbuild removed the tests-pending label Dec 30, 2020

fwyzard added 2 commits January 8, 2021 16:22

fwyzard force-pushed the IB/CMSSW_11_3_X/master_OpenMPI_4.1.0 branch from e4a3eea to 143c147 Compare January 8, 2021 15:22

cmsbuild added tests-approved and removed tests-started labels Jan 8, 2021

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jan 11, 2021

smuzaffar merged commit e9ba972 into cms-sw:IB/CMSSW_11_3_X/master Jan 11, 2021

cms-sw deleted a comment from cmsbuild Jan 11, 2021

fwyzard deleted the IB/CMSSW_11_3_X/master_OpenMPI_4.1.0 branch May 10, 2021 21:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update OpenMPI to 4.1.0, and add extra support libraries #6523

Update OpenMPI to 4.1.0, and add extra support libraries #6523

fwyzard commented Dec 23, 2020 •

edited

Loading

cmsbuild commented Dec 23, 2020

fwyzard commented Dec 23, 2020

cmsbuild commented Dec 23, 2020

cmsbuild commented Dec 23, 2020

fwyzard commented Dec 24, 2020

cmsbuild commented Dec 24, 2020

cmsbuild commented Dec 30, 2020

fwyzard commented Dec 30, 2020

cmsbuild commented Dec 30, 2020

fwyzard commented Dec 30, 2020

fwyzard commented Jan 8, 2021

fwyzard commented Jan 8, 2021

cmsbuild commented Jan 8, 2021

fwyzard commented Jan 8, 2021

fwyzard commented Jan 8, 2021

cmsbuild commented Jan 8, 2021

cmsbuild commented Jan 9, 2021 •

edited by smuzaffar

Loading

fwyzard commented Jan 9, 2021

fwyzard commented Jan 9, 2021

cmsbuild commented Jan 10, 2021 •

edited by smuzaffar

Loading

smuzaffar commented Jan 10, 2021

cmsbuild commented Jan 11, 2021 •

edited by smuzaffar

Loading

smuzaffar commented Jan 11, 2021

cmsbuild commented Jan 11, 2021

iarspider commented Apr 12, 2022

fwyzard commented Apr 12, 2022

fwyzard commented Apr 20, 2022

fwyzard commented Apr 20, 2022

Update OpenMPI to 4.1.0, and add extra support libraries #6523

Update OpenMPI to 4.1.0, and add extra support libraries #6523

Conversation

fwyzard commented Dec 23, 2020 • edited Loading

Update OpenMPI to version 4.1.0

Add the UCX libraries version 1.9.0

Add the NVIDIA GDRCopy library v2.1

Add the Portable Hardware Locality (hwloc) package

Add numactl utilities and libnuma library version 2.0.14

Add libpciaccess, a generic PCI access library

cmsbuild commented Dec 23, 2020

fwyzard commented Dec 23, 2020

cmsbuild commented Dec 23, 2020

cmsbuild commented Dec 23, 2020

fwyzard commented Dec 24, 2020

cmsbuild commented Dec 24, 2020

Comparison Summary

cmsbuild commented Dec 30, 2020

fwyzard commented Dec 30, 2020

cmsbuild commented Dec 30, 2020

fwyzard commented Dec 30, 2020

fwyzard commented Jan 8, 2021

fwyzard commented Jan 8, 2021

cmsbuild commented Jan 8, 2021

fwyzard commented Jan 8, 2021

fwyzard commented Jan 8, 2021

cmsbuild commented Jan 8, 2021

Comparison Summary

cmsbuild commented Jan 9, 2021 • edited by smuzaffar Loading

Unit Tests

fwyzard commented Jan 9, 2021

fwyzard commented Jan 9, 2021

cmsbuild commented Jan 10, 2021 • edited by smuzaffar Loading

Unit Tests

smuzaffar commented Jan 10, 2021

cmsbuild commented Jan 11, 2021 • edited by smuzaffar Loading

Unit Tests

RelVals

smuzaffar commented Jan 11, 2021

cmsbuild commented Jan 11, 2021

iarspider commented Apr 12, 2022

fwyzard commented Apr 12, 2022

fwyzard commented Apr 20, 2022

fwyzard commented Apr 20, 2022

fwyzard commented Dec 23, 2020 •

edited

Loading

cmsbuild commented Jan 9, 2021 •

edited by smuzaffar

Loading

cmsbuild commented Jan 10, 2021 •

edited by smuzaffar

Loading

cmsbuild commented Jan 11, 2021 •

edited by smuzaffar

Loading