Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ROCM] avoid creating symlinks #7813

Merged
merged 1 commit into from
Apr 22, 2022
Merged

[ROCM] avoid creating symlinks #7813

merged 1 commit into from
Apr 22, 2022

Conversation

smuzaffar
Copy link
Contributor

llvm in rocm does not match llvm from cms externals and this causes cmssw to fail. This change should allow to not create symlinks for rocm libs

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_4_X/master.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

@smuzaffar
Copy link
Contributor Author

FYI @fwyzard, I am merging as last night Ibs failed

error: Failed dependencies:
	libclang.so.13()(64bit) is needed by cms+cmssw+CMSSW_12_4_X_2022-04-21-2300-1-1.x86_64

@smuzaffar smuzaffar merged commit 3aa6cfa into IB/CMSSW_12_4_X/master Apr 22, 2022
@fwyzard
Copy link
Contributor

fwyzard commented Apr 22, 2022

@smuzaffar do you know why the problem did not come up in the PR tests ?

@smuzaffar
Copy link
Contributor Author

because we do not build cms+cmssw+CMSSW_12_4_X RPM during PR tests

@fwyzard
Copy link
Contributor

fwyzard commented Apr 22, 2022

Ah... so the problem is when building the RPM, not when building the actual release.
Can I check the logs somewhere ?

@smuzaffar
Copy link
Contributor Author

smuzaffar commented Apr 22, 2022 •
edited
Loading

logs of IBs are available https://cmssdt.cern.ch/SDT/jenkins-artifacts/build-any-ib/CMSSW_12_4_X_2022-04-21-2300/

@smuzaffar
Copy link
Contributor Author

cmssw-tool-conf was build properly but then cmssw failed with error

Done fetching git://github.com/cms-sw/cmssw.git?protocol=https&obj=master/CMSSW_12_4_X_2022-04-21-2300&module=CMSSW&export=src&output=/src.tar.gz
(<function fetchSources at 0x2ad12bffa950>, <Package name=cmssw>, <scheduler.Scheduler object at 0x2ad12bdb3b38>) done
Creating directory [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300) if not existing.
Building cmssw. Log can be found in [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/log.>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/log.)
rm -rf [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/pkgtools-pkg-src-move2git>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/pkgtools-pkg-src-move2git) [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp;>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp;) export _CMSBUILD_BUILD_ENV_=1; mkdir -p [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp;>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp;) source [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/el8_amd64_gcc10/rpm-env.sh>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/el8_amd64_gcc10/rpm-env.sh) ;TMPDIR=[<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/cmsdist-tmp)  rpmbuild  --buildroot [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/tmp/BUILDROOT/04a7d80d4a4918ff3f25a2fe641a0769>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/tmp/BUILDROOT/04a7d80d4a4918ff3f25a2fe641a0769) -ba --define '_topdir [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w'>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w') --define "compiling_processes 33" --define "ignore_compile_errors 1" --define "cmsdist_directory <https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/CMSDIST"> --define "compilerv 1030" --define "cmscompilerv 10" --define "cmsos el8_amd64" --define "almalinux_ver 8" --define "almalinux 8" --define "centos_ver 8" --define "centos 8" --define "rhel 8" --define "dist .el8" --define "el8 1" --define "package_vectorization %{nil}" [<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/SPECS/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/spec>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/SPECS/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/spec) >[<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/log>](https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/w/BUILD/el8_amd64_gcc10/cms/cmssw/CMSSW_12_4_X_2022-04-21-2300/log) 2>&1
Build logs cleanup cmssw
Build successful cmssw.
(<function buildPackage at 0x2ad12bffae18>, <Package name=cmssw>, <scheduler.Scheduler object at 0x2ad12bdb3b38>) done
Trying to install the rpm package cms+cmssw+CMSSW_12_4_X_2022-04-21-2300 just built.
Checking local path dependency for rpm package cms+cmssw+CMSSW_12_4_X_2022-04-21-2300 just build.
Failed to install RPM for cmssw
Requested to quit.
Requested to quit.
Requested to quit.
Requested to quit.
Requested to quit.
Requested to quit.
Requested to quit.
* The action "install-cms+cmssw+CMSSW_12_4_X_2022-04-21-2300" was not completed successfully because Traceback (most recent call last):
File "<https://cmssdt.cern.ch/jenkins/job/build-any-ib/ws/PKGTOOLS/scheduler.py",> line 267, in doSerial
result = commandSpec[0](*commandSpec[1:])
File "PKGTOOLS/cmsBuild", line 3651, in installPackage
File "PKGTOOLS/cmsBuild", line 3399, in installRpm
RpmInstallFailed: Failed to install package cmssw. Reason:
error: Failed dependencies:
	libclang.so.13()(64bit) is needed by cms+cmssw+CMSSW_12_4_X_2022-04-21-2300-1-1.x86_64

* The action "build-cms+cmssw-ib+CMSSW_12_4_X_2022-04-21-2300" was not completed successfully because The following dependencies could not complete:
install-cms+cmssw+CMSSW_12_4_X_2022-04-21-2300
* The action "install-cms+cmssw-ib+CMSSW_12_4_X_2022-04-21-2300" was not completed successfully because The following dependencies could not complete:
build-cms+cmssw-ib+CMSSW_12_4_X_2022-04-21-2300
* The action "final-job" was not completed successfully because The following dependencies could not complete:
install-cms+cmssw+CMSSW_12_4_X_2022-04-21-2300
build-cms+cmssw-ib+CMSSW_12_4_X_2022-04-21-2300
install-cms+cmssw-ib+CMSSW_12_4_X_2022-04-21-2300

@smuzaffar
Copy link
Contributor Author

I am afraid we might have to remove LIBDIR=$ROCM_BASE/llvm/lib from the rocm.xml otherwise it will still end up in LD_LIBRARY_PATH and can cause issues at runtime

@fwyzard
Copy link
Contributor

fwyzard commented Apr 22, 2022

I guess we cannot see the build area for the failes IBs ?

Looking at the installation from the previous tests of #7795 there is a problem with the ROCm external libraries: the symlinks point to paths that no longer exist...

Is it possible that by doing the links in %install instead of %post they point to some intermediate installation, which then gets cleaned up ?

@fwyzard
Copy link
Contributor

fwyzard commented Apr 22, 2022

I am afraid we might have to remove LIBDIR=$ROCM_BASE/llvm/lib from the rocm.xml otherwise it will still end up in LD_LIBRARY_PATH and can cause issues at runtime

Mhm, OK, looks like (outside CMSSW) the compilation still works fine without

    <environment name="LIBDIR"    default="$ROCM_LLVM/lib/linux"/>
    <environment name="LIBDIR"    default="$ROCM_BASE/llvm/lib"/>
    <environment name="INCLUDE"   default="$ROCM_LLVM"/>

@fwyzard
Copy link
Contributor

fwyzard commented Apr 22, 2022

See #7814.

@smuzaffar
Copy link
Contributor Author

I guess we cannot see the build area for the failes IBs ?

Looking at the installation from the previous tests of #7795 there is a problem with the ROCm external libraries: the symlinks point to paths that no longer exist...

Is it possible that by doing the links in %install instead of %post they point to some intermediate installation, which then gets cleaned up ?

If I run /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7795/24100 then it properly creates the symlink for rocm. It failed to create proper symlinks for cvmfs installation as the cvmfs publisher node does not mount /cvmfs/patatrack.cern.ch. I can ask CERN IT to mount /cvmfs/patatrack.cern.ch to our cvmfs publisher ( cms.cern.ch, cms-ib.cern.ch and cms-ci.cern.ch)

@smuzaffar
Copy link
Contributor Author

let me see if I can use https://github.com/cms-sw/cmsdist/blob/IB/CMSSW_12_4_X/master/scram-project-build.file#L293-L304 logic to relocate symlinks for PR tests

@cmsbuild cmsbuild mentioned this pull request Apr 22, 2022
Merged
@smuzaffar smuzaffar deleted the smuzaffar-patch-2 branch April 25, 2022 11:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
externals-pending orp-pending pending-signatures tests-pending
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@smuzaffar @cmsbuild @fwyzard